Transform

Category: Processing

The Transform activity (called Data Transform before, and still data_transform in the workflow API) runs a JavaScript script over the whole transaction. Use it to reshape extracted field values, read and write transaction metadata, restructure documents (move pages, split, merge, delete, reorder, reassign a class), and read or update catalog records, at any point in the workflow.

The script runs in a hardened sandbox with no network and no file access. It reads an in-memory copy of the transaction and returns the changes you make through a fixed set of helper functions.

When to use it

Reach for a Transform step when you need logic the standard activities do not cover, for example:

• Normalize or derive field values (uppercase a code, compute a total, map a country name to an ISO code).
• Set transaction metadata to drive later branching or export naming.
• Fix imperfect splitting: move a stray page into the right document, peel a page into its own document, or merge two documents.
• Reassign a document's class based on its content.
• Look up catalogs and maintain them: for example, register a vendor in the catalog when a document arrives from one you have not seen before.

If you only need to validate or normalize a single field, a business rule is usually simpler. If you need to call an external system, use a Notification webhook instead; a Transform step cannot do network calls.

Configuration

The Configure panel opens on Describe it to the assistant: say what the step should do in your own words and the assistant writes the script and sets the step up for you. Everything below it is there for when you would rather do it yourself.

Two folded sections follow. Both start closed, and each says on its cover what it holds, so a step you did not write can be read without opening anything.

Scope is what the script may reach outside the transaction. Most transforms need nothing here.

Setting	Description	Default
Catalog access	The catalogs the script may read with the catalog API and write with the catalog record helpers. Only the catalogs you select here are accessible.	(none)
Include source files	Load the original file bytes into the script context. Off by default because loading the bytes adds processing time; turn it on only when the script actually needs the raw bytes.	No

Transform script is the code. Folded, it shows one line saying what the script does, written by the assistant from the code itself and refreshed when you fold the section back up. Open it for the editor and everything described under Editing the script.

Timeout sits under both, in seconds (5 to 300, 30 by default): how long the script may run before the step is stopped and the transaction fails.

A newly added Transform node is a valid no-op until you write a script, so you can place it in the workflow first and fill in the code later. Until then the folded Script section says so.

Two further settings exist on the node but have no control in the panel, so they are set through the workflow API (or by editing the workflow definition): parameters, an object handed to the script as the parameters global, and max_source_bytes, the byte budget for Include source files (5 MB by default).

Runtime interface

Your script has access to these globals. Everything is read-only unless noted.

Global	What it holds
documents	Array of documents, in order. Each has document_id, page_indices (ordered page ids), document_type (the class name), document_class_id, and extracted_fields. Mutable through the assembly helpers.
pages	Array of pages, in reading order: each document's pages in document order, then any page in no document. Each has page_index, text (the page's text from the OCR step: the PDF text layer or OCR, "" when the page has none) and barcodes (each with value and format, [] when none was found). A page whose text or barcodes cannot be read fails the step instead of reaching the script as blank. page_index is the page's id, the number a document's page_indices holds; it is not the page's position, so find a document's pages with pages.filter(p => doc.page_indices.includes(p.page_index)).
transaction	The transaction record.
transaction_metadata	User-defined metadata. Mutable through set_metadata; new keys are allowed.
project_settings	The project configuration: classes, fields, guidelines, Memory. A step's own settings (models, inputs, thresholds, all of Split) are on the workflow's steps, not here.
validation_results	Business-rule results, or null until Validate has run.
parameters	The node's parameters object, empty unless it was set through the API.
workflow_context	Outputs of the activities that already ran in this workflow, keyed by activity (for example extract_output, classify_output). Empty when the transform is the first step.
source_files	One entry per uploaded file: filename, content_type, and split_policy. With Include source files on, each entry also carries content_base64, or content_truncated: true when the file did not fit the byte budget.
catalog	The catalog API (lookups and record writes). Available for the catalogs selected under Catalog access; see Working with catalogs.

Field and metadata helpers

Helper	Effect
get_field_value(doc, nameOrId)	Returns a field value, matched by field id or display name, or undefined if absent. Matches top-level fields only; a field inside a group or repeating group (table) is not found by its own name (see Groups and repeating groups).
set_field_value(doc, nameOrId, value)	Sets a top-level field value. The key must be a field the project declares for that document's class (its field id or its display name), otherwise the call fails the step: a transform can update fields, not create them (see Writing fields).
get_metadata(key?)	Returns one metadata key, or the whole object if no key is given.
set_metadata(key, value)	Writes transaction metadata (new keys allowed).
normalize(text, options?)	Cleans text. Options: trim, lowercase, uppercase, removeSpaces, removeSpecialChars, foldDiacritics.
fold_text(text)	Lower-cases and removes accents for comparing two values: Şişli and SISLI both become sisli.
strip_diacritics(text)	Removes accents but keeps the case, for a value that is stored back: Gökçe becomes Gokce.
resolve_country(input, format?)	Maps a country name to an ISO code (alpha2 by default), or null if unresolved.

Your script may return a value, which is surfaced in the step result. Returning nothing is fine.

How to refer to a field

get_field_value and set_field_value take a field's display name, exactly as it appears in the extraction schema, or its field id. There is no class prefix: a document sees the general (class-less) fields and its own class's fields as one flat set.

Field	How the script refers to it
A general field (belongs to no class)	get_field_value(d, 'Currency')
A field of the document's class	get_field_value(d, 'PO Number')
Any field, by id	get_field_value(d, 'a1b2c3d4')
A field inside a group	Not reachable by its own name: read the group, then the child (see Groups and repeating groups)
A cell of a repeating group (table)	Not reachable by its own name: read the table, then the row, then the cell (same section)

Two rules are worth knowing before you write the first line:

Field paths do not work here. A code rule accepts group and cell paths such as 'Vendor/Name' and 'Line Items[1]/Amount'. A Transform step does not: its helpers match top-level entries only, so a path matches nothing. get_field_value returns undefined, and set_field_value fails the step with a message naming the key. Traverse the group's value instead.

The field id is the only unambiguous key. A name is resolved against the fields declared for the document's class, general fields first, and the first declaration of a display name wins. So if a general field and a class field share a name, the name reaches the general one; pass the field id to reach the other. Ids also survive a rename, which names do not, so a transform that has to keep working after someone renames a field should use them.

A field's id is shown, with a copy button, in its Properties dialog (open the field's menu in the Transaction Viewer or the extraction schema and choose Properties).

Writing fields

A transform can change what a field holds, but it cannot bring a new field into existence. set_field_value resolves the key against the fields the project declares for that document's class (plus the class-less fields that apply to every document), and matches either the field id or the display name:

• the field already has a value, or is declared and simply was not extracted → the value is written under the declared field id;
• anything else - a typo, a field belonging to another class, a group child, a name you made up - throws, and the step fails with a message naming the key.

The reason is that everything downstream reads extracted_fields by field id: the viewer, exports, catalog lookups, validation and analytics. An entry stored under a display name is invisible to all of them, so the value would be there for nothing while the real field still reads as empty.

If you need somewhere to put free-form values, use set_metadata - transaction metadata accepts new keys by design. If you need a genuinely new field, add it to the project's extraction schema first, then set it from the transform.

Groups and repeating groups

The field helpers above match top-level fields only. When a field lives inside a group or repeating group, get_field_value(doc, 'Name') returns undefined even though the document clearly shows a Name: the child is stored inside the group's value, not as a field of its own. Fetch the group and traverse its value instead:

• A group's value is an object keyed by child field id, where each child is { name, value, ... }. For example a Vendor group holds { v001name: { name: 'Name', value: 'TAON Ltd' }, v003ctry: { name: 'Country', value: 'CY' }, ... }.
• A repeating group (also called a table) has the value { instances: [ ... ] }, with one such child object per row.
• Groups can nest: a child whose own value has one of these shapes is a nested group.

A small helper makes children addressable by display name:

// Find a child of a group (or of a repeating-group row) by display name.
function sub_field(group, name) {
  if (!group || typeof group !== 'object') return null;
  if (group[name]) return group[name]; // also works with a child field id
  for (const id in group) {
    if (group[id] && group[id].name === name) return group[id];
  }
  return null;
}

// Read a child of a group.
const vendor = get_field_value(d, 'Vendor');
const vendorName = (sub_field(vendor, 'Name') || {}).value;

// Write a child: mutate it in place. set_field_value would reject the child's
// name, because a group child is not a top-level field.
const country = sub_field(vendor, 'Country');
if (country && country.value) country.value = resolve_country(country.value);

// Repeating group: iterate the rows.
const items = get_field_value(d, 'Line Items');
for (const row of (items && items.instances) || []) {
  const qty = sub_field(row, 'Quantity');
  if (qty && qty.value === '') qty.value = '1';
}

Two details to watch:

• Extraction writes every child of a group, so a cell the document did not fill is still present, with value: ''. Check for emptiness (!child.value), not for existence.
• In validation_results, results for group children are keyed by path, <groupFieldId>/<childFieldId> (for example b2c3d4e5/p001ctry). Split the key on / to find the group and the child; passing the whole path to get_field_value matches nothing.

Activity Log messages and failing the transaction

The script can report messages into the transaction's Activity Log (the event timeline you open from the transactions table), and it can explicitly fail the transaction.

Helper	Effect
log_info(message)	Record an info message in the Activity Log.
log_warning(message)	Record a warning in the Activity Log.
log_error(message)	Record an error in the Activity Log. Reporting only: it does not fail the transaction.
fail(message)	Stop the script immediately and set the transaction to failed, with message as the failure reason.

Notes on how these behave:

• Messages accept any value; objects are JSON-stringified. Each run records up to 100 messages of up to 500 characters each.
• The three log_* helpers never change the flow of the workflow. Use log_error for problems worth flagging that should not stop processing (for example, "3 of 120 rows had no catalog match").
• fail() is deliberate and final: the step is not retried, and none of the script's changes are persisted (field values, metadata, document restructuring, and queued catalog writes are all discarded). Messages logged before the fail() call are still written to the Activity Log, so log the details first, then fail.
• A script error (an uncaught exception) also fails the step, but it is treated as a script bug and retried by the workflow engine. Use fail() when the failure is a business decision, so the transaction fails once, immediately, with your message as the reason.

// Fail the transaction when a required document is missing.
const invoice = documents.find(d => d.document_type === 'Invoice');
if (!invoice) {
  log_error('Transaction has ' + documents.length + ' document(s), classes: '
    + documents.map(d => d.document_type).join(', '));
  fail('No Invoice document found in this transaction');
}

// Report progress without affecting the workflow.
log_info('Matched ' + matched + ' of ' + total + ' line items against the price catalog');
if (matched < total) log_warning((total - matched) + ' line items had no catalog match');

In the Activity Log, each message appears as its own timeline entry with an info, warning, or error icon. A fail() shows as a failed step with your message, and the transaction status changes to failed.

Working with catalogs

Select catalogs under Scope > Catalog access in the node settings to unlock the catalog API. Scripts can then look records up and add, update, or delete records.

Reading

Helper	Effect
catalog.first(id, criteria)	First matching record, or null.
catalog.exact(id, criteria)	All matching records.
catalog.page(id, criteria, { offset?, limit? })	A page of matches: { records, total, offset, limit }.
catalog.count(id, criteria)	Number of matches.
catalog.get(id, recordId)	One record by its id, or null.

criteria is an object of { column: value } pairs, combined with AND. Values match with the catalog engine's exact normalization (case, whitespace, and punctuation insensitive), the same matching a catalog lookup rule uses. Each record comes back as { record_id, label, data }.

Writing

Helper	Effect
catalog.add_record(id, data, { on_conflict? })	Add a record. data is { column: value }; the record number (internal_id) is assigned automatically.
catalog.update_record(id, recordId, data)	Partial update: the columns in data are merged into the record, other columns keep their values.
catalog.delete_record(id, recordId)	Delete a record. Deleting a record that is already gone is not an error.

Writes are queued, not immediate: they are applied through the catalog service after the script finishes successfully. If the script throws, nothing is written. Each write still goes through the normal catalog checks (column validation and duplicate detection on unique columns), and the step result reports what was added, updated, deleted, or skipped.

on_conflict controls what happens when add_record collides with an existing record on a unique column:

• 'skip' (default): keep the existing record and move on.
• 'update': merge the new data into the existing record.
• 'error': fail the step.

Because reads see a snapshot taken when the script starts (and another transaction may write the same value at the same time), do not treat a catalog.first() miss as a guarantee that an add_record cannot collide. The default 'skip' makes the register-if-missing pattern safe under concurrency.

Example: register unknown vendors

A typical setup pairs a Validate step (a catalog lookup rule flags documents whose vendor is not in the catalog) with a Transform step that registers the missing vendor:

for (const d of documents) {
  const name = get_field_value(d, 'Vendor Name');
  if (!name) continue;
  if (!catalog.first('cat_vendors', { name: name })) {
    catalog.add_record('cat_vendors', {
      name: name,
      country: get_field_value(d, 'Vendor Country') || null,
    });
  }
}

This works when Vendor Name and Vendor Country are top-level fields. If they are children of a Vendor group, read them through the group instead: const vendor = get_field_value(d, 'Vendor'); then sub_field(vendor, 'Name'), as shown in Groups and repeating groups.

Restructuring documents (assembly API)

The assembly model is positional and live. You address documents by their position in documents and pages by their position within a document's page_indices, and every operation changes the model immediately, so later positions reflect earlier operations. If an operation empties a document, that document is removed automatically.

A document reference is either a document object from documents or its document_id string.

Operation	Signature	Effect
Move page	move_page(fromDoc, fromPos, toDoc, toPos?)	Move one page between documents or within a document. Returns the target document. Omit toPos to append.
Split (spin off)	move_page(fromDoc, fromPos, { new_document: { document_class? } })	Same call, but the target is a spec object. Creates a new document holding that page and returns it.
Move document	move_document(doc, position)	Move a document to position in the transaction.
Delete page	delete_page(doc, position)	Delete one page. If it was the document's last page, the document is deleted too.
Delete document	delete_document(doc)	Delete a document and all its pages.
Set class	set_document_class(doc, className)	Reassign the document's class.
Set split policy	set_split_policy(doc, policy)	Record a per-file split constraint: 'inherit', 'single_document', or 'self_contained'. Stored via the document's source file and read by a later Split step, so it only has an effect in a transform placed before Split, where each document is exactly one uploaded file. See Per-File Split Policies.

There is no empty create_document (a document must always have at least one page, so spin one off with new_document) and no one-call merge_documents (a merge is just moving all of one document's pages into another; the emptied document deletes itself).

Examples

// Normalize a field on every document.
for (const d of documents) {
  const v = get_field_value(d, 'Country');
  if (v != null) set_field_value(d, 'Country', resolve_country(v));
}

// Set metadata for later steps.
set_metadata('ready_for_export', true);

// Split: peel the first page of document 0 into a new Invoice document.
if (documents[0] && documents[0].page_indices.length > 0) {
  move_page(documents[0], 0, { new_document: { document_class: 'Invoice' } });
}

// Merge document 1 into document 0 (move all its pages, in order).
// Move position 0 each time because every move shifts the rest.
const src = documents[1], dst = documents[0];
if (src && dst) {
  while (src.page_indices.length) move_page(src, 0, dst, dst.page_indices.length);
}

// Reorder: make document 2 the first document.
if (documents[2]) move_document(documents[2], 0);

// Drop a blank page.
if (documents[0] && documents[0].page_indices.length > 3) delete_page(documents[0], 3);

// Reassign a class.
set_document_class(documents[0], 'Receipt');

// In a transform placed before Split: pin a cover sheet as one document
// so intelligent splitting never merges it with the files that follow.
for (const d of documents) {
  if (/^cover/i.test(d.source_filename || '')) set_split_policy(d, 'single_document');
}

What happens after a restructure (reprocessing)

The Transform activity does not reprocess documents itself. Instead, for each document it changed, it resets that document's completion flags, so any Classify or Extract step later in the workflow reprocesses exactly those documents. Documents you did not touch keep their flags and are skipped.

The rule per document:

• Pages changed (moved, added, removed, or reordered) and you did not set the class: both classification and extraction are reset, so a downstream Classify and Extract reprocess the document.
• set_document_class pins the new class and resets extraction only. A downstream Classify skips the document (keeping the class you assigned), and a downstream Extract re-reads it under the new class.
• Newly split-off documents start fresh, so a downstream step picks them up automatically.

The same applies to validation: a Transform step does not re-run business rules. If the transform changes field values or restructures documents and you want the rules checked again, add an explicit Validate step downstream as well.

Because of this, if you want changed documents re-classified, re-extracted, and re-validated, add explicit Classify, Extract, and Validate steps after the Transform node:

... → Split → Transform → Classify → Extract → Validate → Review

If nothing runs downstream, the changed documents are simply left pending; the transform does not reprocess them on its own.

Validation and self-correction

The script is checked before it can run, both in the editor and when the workflow is saved. Validation catches:

• Syntax errors.
• Unknown helpers: calling a function that does not exist (often a removed or misremembered helper) is rejected, with the list of the real helpers.
• Runtime errors on sample data: the script is smoke-run against sample documents, so a script that assumes data which may be absent is caught early.

Write defensively so a correct script is not rejected: guard against missing documents and fields (for example if (!documents.length) return; and check get_field_value(...) != null).

Editing the script

Open the node's Configure panel and expand Transform script to edit it in a code editor with autocompletion and inline validation for the helpers and globals above. Use the expand button in the editor's corner for a full-screen editor when the script grows.

Two things sit with the editor, the same two a code rule gets:

Reference. A folded panel under the editor with three tabs. What it can reach names what this step's script is allowed to touch: every document type in the project and every field of each one, spelled the way the script has to spell them, plus the catalogs you selected under Scope and whether Include source files is on. A transform script is not scoped by a picker the way a rule is; it runs once over the whole transaction and can name any field of any type, so the tab lists them rather than sending you to the Fields screen to check a name. What you can call lists every helper and global the sandbox provides, grouped, with a one-line description each; Naming a field shows worked calls built from your own project's field names, including the two that catch people out, a group child and a table cell. Clicking any row in the last two tabs inserts it at the cursor. The list is generated from the sandbox itself, so it cannot describe a helper that does not exist. It is there in the full-screen editor too.

Describe a change. Type what you want in plain language and the assistant writes the script, or rewrites the one you have. A change to existing code is shown as a diff against it, so you read what would change before accepting; nothing is written into the editor until you press accept, and nothing is saved until you save the step. Before you see it, the draft is checked for syntax and for calls to helpers that do not exist, and anything wrong with it is shown in amber above the diff rather than hidden. This box is not there on a deployment with no AI assistant configured.

The assistant is told what this step can actually reach: the catalogs you selected under Scope, whether Include source files is on, and the document classes and fields your project declares. If you ask for something that is outside all of that, such as a catalog you have not given the step, the content of a source file while that setting is off, or a field the project does not have, it still writes the code and marks the line with a comment starting // UNAVAILABLE: saying what has to change first. It also puts one line above the diff telling you the same thing. Nothing is hidden from you and nothing is quietly left out, so you can accept the draft, change the setting or add the field, and come back to working code.

Changes take effect after you publish the project.