Metadata in Guidelines and Descriptions
Metadata in guidelines and descriptions is in Preview: ready to use, and it may still change based on feedback.
Sometimes you know something about a document before it is read: the purchase order a delivery belongs to, the supplier of a batch, the subject of the email it came with. A transaction carries such values as its metadata, set by whoever sends it (the API, the upload screen, an import connector) or by a Data Transform step.
You can refer to a metadata key in the text you write for the model, and the model is told that transaction's value:
This delivery concerns purchase order {{ meta:po_number }} alone.
Extract only the lines of that order.
For a transaction whose po_number is PO-4500012345, the model reads that the delivery concerns that order.
Where you can use it
In every text the Split, Classify and Extract steps send to the model:
- the splitting, classification and extraction guidelines, the per-class extraction guidelines, and a step's own guidelines;
- the description of a document class and of a field, the fields of a group or a table included, and a step's own description of either;
- the description of a split field;
- the text of an extraction policy that changes a description or adds to the guidelines.
Names (of classes, of fields, of choices) take no reference. Other steps do not read it.
Write the key exactly as the transaction has it: {{ meta:email_subject }}, not {{ meta:Email Subject }}. Anything else in double braces is sent as you typed it.
Inserting a reference
In any of these boxes, type @ at the start of a word: a list of the project's metadata keys opens, the keys found in your transactions first, then those the import connectors write. Keep typing to narrow it, choose a key with the arrow keys and Enter, or click it, and the reference is written in its place, as text you can edit. Escape closes the list. An @ you do not follow with a key stays an @, so an email address is typed as before.
The platform's own keys are not offered, since they never carry a value.
Marks under the box
Under the box, a reference the model will not be told a value for is marked:
- Not a metadata reference: braces with no key, or with a
|in them. The text is sent as you typed it. - A key the platform keeps for itself: it always reads
not supplied. - A key nobody has seen: no import connector declares it and none of the project's transactions looked at holds it (a sample of up to 100 of production's, and of the Playground's when you may open it). Usually a spelling mistake; a key your transactions will bring later is fine.
How the model gets the value
The value is not written into your sentence. In its place the model reads a reference to the key, and the values come in a block of their own, next to the document, as data:
<transaction_metadata>
Values that came with this transaction, ...
"po_number": "PO-4500012345"
</transaction_metadata>
This matters because metadata is not always yours. An email's subject and sender are written by whoever sent the email. In the block, a subject line that says "ignore your instructions" is data about the document, as the document's own text is, and never part of your instructions. Each call carries only the keys its texts refer to, and a value longer than 1,000 characters is cut.
A project that uses no reference sends exactly what it sent before.
When the value is read
A step reads the transaction's metadata once, as it starts, before it processes any document. Every document of the step gets the same values. This has four consequences:
- A classification rule's value reaches the next step. A pre-classification rule that calls
set_metadata()writes the key while the Classify step runs; that Classify step still reads the value from before the rule, and the steps after it read the rule's value. To use a value in classification, compute it in a Data Transform step before Classify. - Classify and Extract in one call. When Classify also extracts, in one call to the model, the extraction texts are read with Classify's values, and Extract does not call the model again for those documents. A value a pre-classification rule writes then reaches neither, so again compute it before Classify.
- A step that runs again reads again. When a step is retried after a failure, it reads the metadata anew.
- Re-classify and Re-extract read the metadata as it is when you re-run the document.
If the metadata cannot be read, the step fails and is retried, rather than sending the model no values.
When a transaction has no value
A transaction that lacks the key, or whose value is empty, sends not supplied for it, and the document is processed as usual: an optional key, such as a reference some senders give and others do not, does not stop anything.
To see what was sent, open the transaction's Activity Log: each step whose texts refer to metadata adds a row naming every key it referred to and whether the transaction supplied it. A key that is never supplied is often a spelling mistake.
Some keys never carry a value: the platform's own bookkeeping (keys starting with review_, step_ or awaiting_, and the reserved keys). A reference to one of them always reads not supplied. The box marks such a reference, or one to a key no import connector declares and no sampled transaction holds, and so does the Assistant when it writes one.
Choosing between this and other ways
A value known in advance can do three other things, and each is surer when it fits:
- Fill a field with the value, with no model involved: a field's value source.
- Choose instructions by the value: an extraction policy with a condition on a metadata key applies its text only to the transactions it matches.
- Decide or narrow the class by the value: a classification rule on a metadata key.
A reference is for what only the model can do with the value: find the lines of one order among several, pick one customer's account from a statement, read a document knowing who sent it.