Skip to main content

Long Documents

Preview

Long document extraction is in Preview: ready to use, and it may still change based on feedback.

A model reads a document best a few pages at a time. Given a long table in one request, it often returns only part of its rows. With long document extraction, the Extract step reads such a document in parts of five pages, checks what it read, reads again where a page came back short, and puts the parts together. Every page and every table is read, up to the page limits the step sets, and what could not be read or settled is marked for a person to check. In our own readings almost every row comes back, and a few rows can still be missed without a mark: see What a reviewer sees.

It is on for every project. An administrator can switch it off for the whole deployment, for one tenant or for one project: see Switching it off.

Which documents are read this way​

  • A document whose class has a table: a repeating group, whose rows can run from page to page. It is read this way however short it is, because a table is what one request loses rows of.
  • Any document of more than 20 pages, counted within the step's page limits.

Every other document is read in one request, as before. So are:

  • a document whose class still has a field of the retired table type;
  • a document whose class has a table that holds a table or a group inside it.

A repeating group with Rows can span multiple pages off is not a table here: a class whose groups all have it off is read in one request up to 20 pages.

A step with Discover additional fields on (in a Playground run; Memory and Evaluation never discover) reads in parts every document of five pages or fewer, and a longer one when the step sends text and a page has some: see Discovering fields.

Up to 100 pages. A document of more than 100 pages is not supported yet. Nothing refuses it, and it is read the same way, but a transaction of several hundred pages is slow to open in the Transaction Viewer and may not finish processing. Until that limit is lifted, cut such a file into parts of 100 pages or fewer before it is imported, and import each part as a transaction of its own: the Split step makes shorter documents, but the transaction stays as long as the file.

How a document is read​

  1. The text is scanned. When the step sends text, the whole text is read first, a hundred pages at a time, to find on which pages each field and each table is.
  2. The pages are read in windows of five. Each window shares its last page with the next, so a row that starts at the foot of a page is not cut. A window is asked for the fields first, and then for each table on its own.
  3. The reading is checked. For each page the step counts the rows the page prints and compares them with the rows it read. A window that came back short is read again.
  4. The values are brought together. Where two windows read a field differently, the pages are read once more to decide. A table's rows are put in page order, each row with the page it is on.
  5. What could not be settled is marked. A value or a row the step is not sure of gets a confidence of 0 and an explanation, and goes to review.

The transaction's Activity Log shows each of these as it happens, with the windows read so far, and at the start how many model calls the document is expected to take and what they are expected to cost.

A long document that is being read when the deployment restarts is taken up where it stopped. The calls already made are not made again.

What a reviewer sees​

Where the step could not settle something, the Review step stops at it:

What the step foundWhat is markedWhat to do
A page prints more rows than were readThe first and the last row of that pageCheck the page, add the rows that are missing, confirm the rows
Some pages could not be readThe rows on either side of those pagesCheck those pages and add their rows
The rows of a page came back twiceThe first and the last row of that page. The first reading was keptCheck that no row of the page is there twice
The rows on a page may be the rows of the next pageThe first and the last row of that pageCheck them against the page, and delete them if they are not on it
Rows were read on a page that a second reading found none onThe rows were left out. The rows on either side of that page are markedCheck the page, and add its rows if it has any
Rows were read on a page where an earlier count found none, and no second reading reached the pageThe first and the last row of that pageCheck them against the page, and delete them if they are not on it
A field was read differently on different pagesThe fieldChoose the right value

The badge at the end of a marked row shows 0% and names the pages to check, with the confidence the row was read with. The rows in between are not marked: a table of a thousand rows read without a doubt is not a thousand stops.

The marks do not catch everything. In one of our readings of five documents, three rows of 15,557 were not read and nothing marked them; in a later reading of the same documents none was missed. Where every row matters, check a long table's row count against the document.

What it costs​

Reading every page takes more model calls than one request, and costs more. Measured on our own documents with the cloud service's default model, at list price, for the Extract step alone:

DocumentCost, compared with one requestPer 1,000 pagesModel calls
With a table, one pageAbout 1.7 timesAbout $3.00At least two: one for the fields and one for each table, and one more wherever a page is read again
With a table, 2 to 5 pagesAbout 1.5 times; up to twice for the densest tablesAbout $2.40The same
With a table, 6 to 20 pagesAbout 4.4 times. One request often loses most of the rows of such a tableAbout $3.65About 8 calls at 6 pages and 17 at 20: roughly one a page
More than 20 pagesOne request does not read such a document wholeAbout $2.40 on documents of every kind, up to about $4.30 on documents that are tables from end to endAbout two calls for every three pages

Where Classify also extracts, a document of five pages or fewer costs about 1.15 times one request, and one of 6 to 20 pages about 3 times.

Where you use model endpoints of your own, on a self-hosted deployment or with your own model on the cloud service, your provider bills these calls, and their number counts against your endpoints' rate limits. Cost and Budgets shows what each transaction cost.

How long it takes​

A document with a table takes about half a minute when it is one page, about a minute at 2 to 5 pages and about four and a half minutes at 6 to 20 pages; one request took 15, 40 and 53 seconds on the same documents. A document of 100 pages takes from a quarter of an hour to an hour, depending on how many rows its tables hold and on what else the deployment is reading at the time. The transaction stays In Progress meanwhile, and its Activity Log shows how far the reading is.

Settings that change how it reads​

  • What the model reads (the Model block of Extract). With text alone, the windows are read without their images, and a value that only a picture shows is missed, as it is today when no image is sent. With images alone, no text is scanned first, so nothing is known of where a table is before its pages are read.
  • The page optimizer of the Extract step does not apply to a document read this way: every page within the step's page limits is read.
  • Max image pages and Max text pages of the Extract step hold for a document read this way as they do for one request. The document is read over its first pages, up to the larger of the two limits; only the first Max image pages are sent as pictures and only the first Max text pages as text. Pages past both limits are not read, and nothing marks a table that goes on past them. Send Images of Pages Without Text still sends the picture of a page within the text limit that has no text.
  • Classify that also extracts. It extracts a document of five pages or fewer. One without a table is extracted there in one request, as before; one with a table is read on from Classify's request by Extract. A longer document is only classified.
  • Extraction policies are matched before the pages are read, on the values found in the document's text or on what its first pages read. The document is not read a second time for a policy. A policy whose condition the result meets, but which the reading did not apply, is named in the transaction's Activity Log.
  • Memory. Document examples go with every window of the document they are chosen for, so an example's pages are sent with each window: in our measurement about 11,500 input tokens more for a window's first request, the window's further requests being served from the model's prompt cache where it has one. An example made from a document read this way holds the pages its fields are on and the first page of each of its tables, five pages at most.

Discovering fields​

With Discover additional fields on, a document read in parts discovers as it reads: the fields and tables it finds are added to the draft while it is still being read, and a table it discovers is read in full, row by row. The documents of one class take turns to discover, so that two of them do not name the same data differently; the other documents of a class with no fields start once its first document's first pages are read. A class of many documents is therefore discovered more slowly than it is extracted once its fields are settled. The transaction's Activity Log says what each document added to the draft.

Two kinds of document are still read in one request while they discover: a document of more than five pages that has no text, and any document of a project with an active extraction policy.

In exports​

For a document read this way, the JSON export with field metadata carries with each cell of a row the page the row was read on (page_index), as it does for a simple field. See JSON Export.

Switching it off​

An administrator switches the feature off, or on again, for the deployment, a tenant or a project with the da admin features command: see Feature flags. On the DocAI Fabric cloud service we are that administrator: write to support@docaifabric.com to have it switched off for your organization or for one project. Off, Extract reads every document in one request, as before. A change reaches the whole deployment within 30 seconds and applies to the documents that start after it; a document that is being read in parts finishes that way, unless the feature is switched off for the whole deployment with --force.