Export Schema Version 2

Version 2 is a new arrangement of the same export. Nothing about how your documents are processed changes, and no data is lost: the file is reorganized so that the parts you actually want are easier to find and cheaper to carry.

It applies to both the JSON and XML formats, which produce the identical structured document in two serializations.

New export profiles are created at version 2. Existing profiles keep version 1 and are not changed. You can switch a profile at any time with the Schema Version option, and switch back.

Why version 2 exists

Version 1 grew one block at a time, and three problems came with it.

The page layer was repeated. Page text and OCR word layout were stored inside every document. A transaction whose pages were split across three documents carried each page's OCR three times, and asking for the pre-review snapshot doubled it again. In version 2 pages are written once for the whole transaction and documents point at them.

Business rule results were scattered. Transaction rules sat in one place, document rules inside each document, and field rules inside each field, so "show me every problem with this transaction" meant walking the whole file. Version 2 reports every issue in one list.

There was nowhere to put a second view of the transaction. The pre-review snapshot had to be bolted on as its own top-level block. Version 2 has a states array, so the transaction as it stands now, the transaction as automatic processing left it, and the transaction at any saved checkpoint are all the same shape.

The shape

{
  "schema_version": 2,

  "transaction": {
    "transaction_id": "…", "tenant_id": "…", "project_id": "…",
    "status": "completed",
    "created_at": "…", "exported_at": "…",
    "source": {
      "type": "web-upload",
      "files": [
        { "index": 0, "name": "invoice.pdf", "split_policy": "inherit",
          "content_type": "application/pdf", "size_bytes": 184320,
          "modified_at": "…", "page_numbers": [1, 2] }
      ]
    },
    "metadata": { "customer_ref": "ACME-42" },
    "processing": {
      "timing": { "total_seconds": 340, "review_seconds": 252, "…": "…" },
      "reviews": [
        { "step_name": "Human Review", "reviewer_email": "ada@example.com",
          "action_type": "accept", "comment": null, "completed_at": "…" }
      ]
    }
  },

  "pages": [
    { "page_number": 1, "page_id": "9c1e77a4-…",
      "source_filename": "invoice.pdf", "source_file_page_number": 1,
      "image_transform": { "…": "…" } }
  ],

  "states": [
    {
      "id": "final",
      "captured_at": "…",
      "summary": { "overall_status": "error",
                   "document_count": 2, "page_count": 3,
                   "requires_review": { "total_issues": 4, "…": "…" },
                   "rules": { "executed": 6, "…": "…" } },
      "issues": [ "…" ],
      "documents": [
        { "document_id": "…", "document_type": "Invoice",
          "pages": [ { "page_number": 1, "page_id": "9c1e77a4-…" } ],
          "fields": { "…": "…" } }
      ]
    }
  ]
}

Four top level keys, and each answers one question.

Key	Answers
`transaction`	What is this, where did it come from, what happened to it, and who touched it.
`pages`	What pages exist, and how do their coordinates map back to the original files.
`states`	What did the extracted data look like, at one or more points in time.
`schema_version`	Which layout this file uses, so your code can branch on it.

Pages are written once

In version 1 each document carried a full copy of its pages. In version 2 the page records live at the root and a document holds only references:

"pages": [ { "page_number": 3, "page_id": "1f0b2ad9-…" } ]

The array is the membership and its order is the reading order, so a page's position within the document is simply its position in this list.

Each reference carries two identifiers because they are useful in different situations. page_number is readable and joins to everything else in the file. page_id is stable: if splitting changed during review and a page moved to a different document, the same physical page keeps the same page_id, so it is the safer key when comparing one state to another.

const byId = Object.fromEntries(data.pages.map(p => [p.page_id, p]))
const doc = data.states.find(s => s.id === "final").documents[0]
const pages = doc.pages.map(ref => byId[ref.page_id])

States

states is an array, ordered oldest first, with the current state always last. Every entry has the same shape, so code written for one works for all of them.

State	What it is
`final`	The transaction as it stands now. Always available.
`pre_review`	The transaction as automatic processing left it, before any human touched it. Subtracting it from `final` is exactly what your reviewers changed.
`checkpoint:<id>`	Any saved checkpoint, for example the state right after OCR.

A state you asked for that cannot be reconstructed is simply absent from the array rather than present with an error in it. The most common reason is the ordinary one: a transaction with no review step has no pre-review state to report, which you can see for yourself from an empty transaction.processing.reviews list.

Choosing how much of each state to export

Each state can be exported at one of three levels, and they are cumulative:

Level	Contains
`summary`	Overall status, document and page counts, how many issues need attention, and how many rules ran.
`issues`	The above, plus the complete list of issues.
`documents`	The above, plus every document with its fields.

final defaults to documents and every other state defaults to summary, which is usually what you want: the full data for the current state, and cheap counts for the historical ones so you can measure how much review changed without doubling the file.

In the workflow editor, Include Pre-Review State and Include Full Pre-Review Document Tree set this for you. Over the API you can also set states and state_detail directly.

Every issue in one list

states[].issues is everything a reviewer has to act on in that state, from both of the things that can go wrong: a business rule objected, or the model was not confident.

"issues": [
  { "id": "iss_7f3a9c21", "reason": "validation_error", "scope": "transaction",
    "rule_name": "Total must match lines", "status": "error",
    "message": "Sum of line items (1200.00) does not match invoice total (1150.00)",
    "blamed_fields": [
      { "document_id": "d83dbd82-…", "field_path": "9a2f/total",
        "field_name": "Invoice Total" }
    ] },
  { "id": "iss_c405a719", "reason": "validation_error", "scope": "field",
    "document_id": "d83dbd82-…", "field_id": "1490f411",
    "field_name": "Invoice Number", "field_path": "1490f411",
    "rule_name": "Format check", "status": "error",
    "message": "Invoice number must match INV-#####" },
  { "id": "iss_9bdbb179", "reason": "low_confidence_fields", "scope": "field",
    "document_id": "d83dbd82-…", "field_id": "3f9a",
    "field_name": "Supplier Address", "field_path": "3f9a",
    "status": "review",
    "message": "Extraction confidence 0.98 is below the review threshold 0.99.",
    "confidence": 0.98, "threshold": 0.99 }
]

scope says what the issue is attached to: the whole transaction, one document, or one field. Counting errors is one filter rather than a walk over the file:

const state = data.states.find(s => s.id === "final")
const errors = state.issues.filter(i => i.status === "error")

The reason an issue is here

reason is one of five values, and it is the same vocabulary as summary.requires_review.by_reason:

`reason`	`status`	Meaning
`validation_error`	`error`	A business rule failed.
`validation_warning`	`warning`	A business rule raised a warning.
`low_confidence_fields`	`review`	An extracted field, group child or table cell was below the confidence threshold.
`low_confidence_classification`	`review`	The document type was chosen with low confidence.
`low_confidence_split`	`review`	The split decision was made with low confidence.

Because the list and the counts come from the same place, they always agree. issues.length equals summary.requires_review.total_issues, and grouping the list by reason reproduces by_reason exactly. You can use the cheap counts for dashboards and the list when you need the detail, without the two telling you different things.

Low confidence carries the number it was measured against

A confidence on its own is not enough to act on. A field flagged at 0.98 looks like a mistake until you know your project's threshold is 0.99, so the threshold travels with the issue:

{ "confidence": 0.98, "threshold": 0.99 }

Thresholds are set per type on the Review step (split, classification, extraction) and differ between projects, so a single figure elsewhere in the file would still leave you guessing which one applied.

If the threshold is changed after a transaction has been reviewed, the stored flag and the current setting no longer agree. In that case threshold is omitted rather than reported, because a threshold that does not explain the flag beside it is worse than none. confidence and the flag itself are always reported.

Only one issue per thing to fix

A field that is both low-confidence and failing a rule is one issue, not two, and the rule is the one reported. It is one stop for the reviewer either way, and the rule message is the more useful of the two.

A confirmed field never appears as a low-confidence issue: a human has already dealt with it. This is what makes the pre-review comparison meaningful. At the pre-review moment nothing is confirmed, so everything flagged is listed; on final the confirmed ones have dropped out.

`overall_status` is in the summary

"summary": { "overall_status": "error", "…": "…" }

It is the worst status anywhere in the state, ranked error > warning > review > success.

overall_status covers every issue in the state. In version 1 the top level status looked only at transaction level rules, so a file could report success while its documents carried errors. If you compare the two versions of the same transaction, this is the value most likely to differ, and version 2 is the accurate one.

A state whose only issues are low confidence reads review, not success. Low confidence never outranks a real warning: it is the model being unsure, not a rule objecting.

Issue ids are stable. id is derived from what the issue is, not from its position, so the same issue carries the same id in pre_review and in final. Which issues did review resolve? A set difference:

const idsIn = id => new Set(
  data.states.find(s => s.id === id).issues.map(i => i.id))
const resolved = [...idsIn("pre_review")].filter(x => !idsIn("final").has(x))

What the rules engine did

summary.rules reports rule activity, as opposed to what it found:

"rules": { "executed": 6, "skipped": 2, "passed": 3, "warnings": 1, "errors": 2 }

Turning off Include Validation removes this block and the rule-sourced issues. The low-confidence issues stay, because they are not business rule results.

Finding a field's problems

Documents and fields carry a simple validation_status flag (warning or error, absent when clean) so you can see at a glance that something is wrong. It is stamped at every level, including a single cell of a table. For the detail, filter issues by the same field_path:

state.issues.filter(
  i => i.document_id === documentId && i.field_path === "lineItems[2]/description")

Field paths use ids throughout: fieldId, fieldId/childId for a group child, and fieldId[row]/columnId for a table cell, with rows counted from zero. Using ids rather than labels means a path keeps working when someone renames a field.

Timing

transaction.processing.timing reports whole seconds, split into the time the system spent and the time people spent:

Counter	Meaning
`total_seconds`	The whole lifetime, from creation to completion.
`initial_queue_seconds`	Waiting before any processing started.
`automatic_seconds`	Automatic processing overall.
`automatic_queue_seconds`	Of that, waiting in queues or between steps.
`automatic_active_seconds`	Of that, actually executing.
`review_seconds`	Sitting at a review step, from arrival to the reviewer's decision.
`review_queue_seconds`	Of that, waiting for someone to pick it up.
`review_active_seconds`	Of that, a reviewer actively working on it.

The review split is measured from reviewer activity in the transaction viewer, and is only available for transactions reviewed after that measurement was introduced. Where it is not available, review_queue_seconds and review_active_seconds are omitted rather than reported as zero, because zero would claim a reviewer spent no time on it. review_seconds is always correct.

Knowing that something failed

transaction.status reports the transaction's state, and a failed transaction also carries the reason:

"processing": {
  "failure": {
    "step_id": "extract_1",
    "step_name": "Extraction",
    "message": "Azure OpenAI returned 429 after 6 attempts"
  }
}

Turn on Include Processing Steps to also get the full workflow timeline, with each step's status and duration:

"steps": [
  { "id": "ocr_1", "type": "ocr", "name": "OCR", "status": "completed",
    "queued_at": "…", "started_at": "…", "completed_at": "…",
    "queue_seconds": 4, "processing_seconds": 12 }
]

queue_seconds is how long the step waited to be picked up, and processing_seconds is how long it then took. Between them you can see whether a slow transaction was actually slow to process or simply spent its time waiting.

The two queue keys are omitted rather than reported as zero on transactions processed before this measurement was introduced, because zero would claim there was no wait. processing_seconds is always correct.

This is useful when diagnosing where time goes and unnecessary for a routine data export, which is why the whole block is off by default.

Moving from version 1

Switch the Schema Version option on the profile. Everything version 1 reported is still present, and this is where to find it:

Version 1	Version 2
`transaction_id`, `tenant_id`, `project_id` at the root	inside `transaction`
`documents` at the root	`states[id="final"].documents`
`summary` at the root	`states[id="final"].summary`
`validation.transaction_rule_results`	`states[].issues` where `reason` starts `validation_` and `scope` is `transaction`
`documents[].validation.field_rule_results`	`states[].issues` where `reason` starts `validation_` and `scope` is `field`
`documents[].validation.document_rule_results`	`states[].issues` where `reason` starts `validation_` and `scope` is `document`
`documents[].validation.overall_status`	`documents[].validation_status`, absent when clean
`validation.overall_status` at the root	`states[].summary.overall_status`
`summary.validation` (rule counts)	`states[].summary.rules`
`fields.<id>.review_required` (a flag you had to hunt for)	still there, and now also listed in `states[].issues`
`documents[].pages[]` (full records)	root `pages[]`, joined from `documents[].pages[]`
`pre_review` at the root	`states[id="pre_review"]`
`review.had_human_review`	`transaction.processing.reviews` is not empty
`review.pre_review_available`	a `pre_review` entry exists in `states`
`transaction.last_human_review`	the last entry of `transaction.processing.reviews`
`transaction.timing.total_duration_seconds`	`transaction.processing.timing.total_seconds`
`summary.requires_review.total`	`summary.requires_review.total_issues`
`summary.requires_review.documents_affected`	`summary.requires_review.documents_with_issues`
`summary.requires_review.by_reason.low_confidence`	`by_reason.low_confidence_fields`
`summary.requires_review.by_reason.classification`	`by_reason.low_confidence_classification`
`summary.requires_review.by_reason.split`	`by_reason.low_confidence_split`

Three things are reported differently rather than moved:

overall_status now covers every scope and both kinds of issue, as described above. The same transaction can read success in version 1 and error (or review) in version 2 without anything about it having changed.
error_fields became blamed_fields, and each entry now gives you a field path as well as the label, so it joins to the rest of the file the same way everything else does.
image_transform.effective_dpi was dropped. It was render_scale x 72 restated, and mapping a coordinate onto the original page uses render_scale alone.

Two counters were dropped as misleading and have no version 2 equivalent: documents_validated, which equalled the document count in every ordinary case, and fields_validated, which counted fields that had a rule attached rather than fields that were checked. summary.rules reports rule activity honestly.

Reducing file size

The same levers as version 1, and one of them is much more effective now:

Turn off Include Reasoning. The model's written explanations are usually the largest text in the file, and removing them costs you no data values at all.
Turn off Include Field Details to keep just the extracted values.
Field Filter to drop fields you do not need.
Turn off Include Page OCR Data and Include Page Full Text if you do not use them. In version 2 this matters less than it used to, because pages are no longer repeated per document.
Exclude Paths to strip specific branches. Paths address the finished file, so version 2 paths differ from version 1:
```
states.*.documents.*.fields.*.reasoning
states.*.issues
pages.*.ocr
transaction.processing.timing
```

Next steps

Export to JSON: the format itself, the coordinate contract, and field filtering.
Accessing OCR & Field Data: retrieving results over the API.

Why version 2 exists​

The shape​

Pages are written once​

States​

Choosing how much of each state to export​

Every issue in one list​

The reason an issue is here​

Low confidence carries the number it was measured against​

Only one issue per thing to fix​

overall_status is in the summary​

What the rules engine did​

Finding a field's problems​

Timing​

Knowing that something failed​

Moving from version 1​

Reducing file size​

Next steps​