Data Transform

Category: Transform

The Data Transform activity runs a JavaScript script over the whole transaction. Use it to reshape extracted field values, read and write transaction metadata, restructure documents (move pages, split, merge, delete, reorder, reassign a class), and read or update catalog records, at any point in the workflow.

The script runs in a hardened sandbox with no network and no file access. It reads an in-memory copy of the transaction and returns the changes you make through a fixed set of helper functions.

When to use it

Reach for a Data Transform when you need logic the standard activities do not cover, for example:

• Normalize or derive field values (uppercase a code, compute a total, map a country name to an ISO code).
• Set transaction metadata to drive later branching or export naming.
• Fix imperfect splitting: move a stray page into the right document, peel a page into its own document, or merge two documents.
• Reassign a document's class based on its content.
• Look up catalogs and maintain them: for example, register a vendor in the catalog when a document arrives from one you have not seen before.

If you only need to validate or normalize a single field, a business rule is usually simpler. If you need to call an external system, use a Notification webhook instead; Data Transform cannot do network calls.

Configuration

The Configure panel groups the settings into Script access (what the script can touch) followed by the Script itself:

Setting	Description	Default
Catalog access	The catalogs the script may read with the catalog API and write with the catalog record helpers. Only the catalogs you select here are accessible.	(none)
Include source files	Load the original file bytes into the script context. Off by default because loading the bytes adds processing time; turn it on only when the script actually needs the raw bytes.	No
Script	The JavaScript to run, edited in a full code editor with autocompletion.	(starter template)
Timeout (seconds)	Maximum script run time before it is stopped.	30

A newly added Data Transform node is a valid no-op until you write a script, so you can place it in the workflow first and fill in the code later.

Runtime interface

Your script has access to these globals. Everything is read-only unless noted.

Global	What it holds
documents	Array of documents, in order. Each has document_id, page_indices (ordered page ids), document_type (the class name), document_class_id, and extracted_fields. Mutable through the assembly helpers.
pages	Array of pages (page_index, OCR text, barcodes).
transaction	The transaction record.
transaction_metadata	User-defined metadata. Mutable through set_metadata; new keys are allowed.
project_settings	The project configuration.
validation_results	Business-rule results, or null until Validate has run.
parameters	Values passed in from the node config.
source_files	File metadata. The raw bytes are present only when Include source files is on.
catalog	The catalog API (lookups and record writes). Available for the catalogs selected under Catalog access; see Working with catalogs.

Field and metadata helpers

Helper	Effect
get_field_value(doc, nameOrId)	Returns a field value, matched by field id or display name, or undefined if absent.
set_field_value(doc, nameOrId, value)	Sets or creates a field value on a document.
get_metadata(key?)	Returns one metadata key, or the whole object if no key is given.
set_metadata(key, value)	Writes transaction metadata (new keys allowed).
normalize(text, options?)	Cleans text. Options: trim, lowercase, uppercase, removeSpaces, removeSpecialChars.
resolve_country(input, format?)	Maps a country name to an ISO code (alpha2 by default), or null if unresolved.

Your script may return a value, which is surfaced in the step result. Returning nothing is fine.

Working with catalogs

Select catalogs under Catalog access in the node settings to unlock the catalog API. Scripts can then look records up and add, update, or delete records.

Reading

Helper	Effect
catalog.first(id, criteria)	First matching record, or null.
catalog.exact(id, criteria)	All matching records.
catalog.page(id, criteria, { offset?, limit? })	A page of matches: { records, total, offset, limit }.
catalog.count(id, criteria)	Number of matches.
catalog.get(id, recordId)	One record by its id, or null.

criteria is an object of { column: value } pairs, combined with AND. Values match with the catalog engine's exact normalization (case, whitespace, and punctuation insensitive), the same matching a catalog lookup rule uses. Each record comes back as { record_id, label, data }.

Writing

Helper	Effect
catalog.add_record(id, data, { on_conflict? })	Add a record. data is { column: value }; the record number (internal_id) is assigned automatically.
catalog.update_record(id, recordId, data)	Partial update: the columns in data are merged into the record, other columns keep their values.
catalog.delete_record(id, recordId)	Delete a record. Deleting a record that is already gone is not an error.

Writes are queued, not immediate: they are applied through the catalog service after the script finishes successfully. If the script throws, nothing is written. Each write still goes through the normal catalog checks (column validation and duplicate detection on unique columns), and the step result reports what was added, updated, deleted, or skipped.

on_conflict controls what happens when add_record collides with an existing record on a unique column:

• 'skip' (default): keep the existing record and move on.
• 'update': merge the new data into the existing record.
• 'error': fail the step.

Because reads see a snapshot taken when the script starts (and another transaction may write the same value at the same time), do not treat a catalog.first() miss as a guarantee that an add_record cannot collide. The default 'skip' makes the register-if-missing pattern safe under concurrency.

Example: register unknown vendors

A typical setup pairs a Validate step (a catalog lookup rule flags documents whose vendor is not in the catalog) with a Data Transform that registers the missing vendor:

for (const d of documents) {
  const name = get_field_value(d, 'Vendor Name');
  if (!name) continue;
  if (!catalog.first('cat_vendors', { name: name })) {
    catalog.add_record('cat_vendors', {
      name: name,
      country: get_field_value(d, 'Vendor Country') || null,
    });
  }
}

Restructuring documents (assembly API)

The assembly model is positional and live. You address documents by their position in documents and pages by their position within a document's page_indices, and every operation changes the model immediately, so later positions reflect earlier operations. If an operation empties a document, that document is removed automatically.

A document reference is either a document object from documents or its document_id string.

Operation	Signature	Effect
Move page	move_page(fromDoc, fromPos, toDoc, toPos?)	Move one page between documents or within a document. Returns the target document. Omit toPos to append.
Split (spin off)	move_page(fromDoc, fromPos, { new_document: { document_class? } })	Same call, but the target is a spec object. Creates a new document holding that page and returns it.
Move document	move_document(doc, position)	Move a document to position in the transaction.
Delete page	delete_page(doc, position)	Delete one page. If it was the document's last page, the document is deleted too.
Delete document	delete_document(doc)	Delete a document and all its pages.
Set class	set_document_class(doc, className)	Reassign the document's class.

There is no empty create_document (a document must always have at least one page, so spin one off with new_document) and no one-call merge_documents (a merge is just moving all of one document's pages into another; the emptied document deletes itself).

Examples

// Normalize a field on every document.
for (const d of documents) {
  const v = get_field_value(d, 'Country');
  if (v != null) set_field_value(d, 'Country', resolve_country(v));
}

// Set metadata for later steps.
set_metadata('ready_for_export', true);

// Split: peel the first page of document 0 into a new Invoice document.
if (documents[0] && documents[0].page_indices.length > 0) {
  move_page(documents[0], 0, { new_document: { document_class: 'Invoice' } });
}

// Merge document 1 into document 0 (move all its pages, in order).
// Move position 0 each time because every move shifts the rest.
const src = documents[1], dst = documents[0];
if (src && dst) {
  while (src.page_indices.length) move_page(src, 0, dst, dst.page_indices.length);
}

// Reorder: make document 2 the first document.
if (documents[2]) move_document(documents[2], 0);

// Drop a blank page.
if (documents[0] && documents[0].page_indices.length > 3) delete_page(documents[0], 3);

// Reassign a class.
set_document_class(documents[0], 'Receipt');

What happens after a restructure (reprocessing)

The Data Transform activity does not reprocess documents itself. Instead, for each document it changed, it resets that document's completion flags, so any Classify or Extract step later in the workflow reprocesses exactly those documents. Documents you did not touch keep their flags and are skipped.

The rule per document:

• Pages changed (moved, added, removed, or reordered) and you did not set the class: both classification and extraction are reset, so a downstream Classify and Extract reprocess the document.
• set_document_class pins the new class and resets extraction only. A downstream Classify skips the document (keeping the class you assigned), and a downstream Extract re-reads it under the new class.
• Newly split-off documents start fresh, so a downstream step picks them up automatically.

The same applies to validation: Data Transform does not re-run business rules. If the transform changes field values or restructures documents and you want the rules checked again, add an explicit Validate step downstream as well.

Because of this, if you want changed documents re-classified, re-extracted, and re-validated, add explicit Classify, Extract, and Validate steps after the Data Transform node:

... → Split → Data Transform → Classify → Extract → Validate → Review

If nothing runs downstream, the changed documents are simply left pending; the transform does not reprocess them on its own.

Validation and self-correction

The script is checked before it can run, both in the editor and when the workflow is saved. Validation catches:

• Syntax errors.
• Unknown helpers: calling a function that does not exist (often a removed or misremembered helper) is rejected, with the list of the real helpers.
• Runtime errors on sample data: the script is smoke-run against sample documents, so a script that assumes data which may be absent is caught early.

Write defensively so a correct script is not rejected: guard against missing documents and fields (for example if (!documents.length) return; and check get_field_value(...) != null).

Editing the script

Open the node's Configure panel to edit the script in a code editor with autocompletion and inline validation for the helpers and globals above. The AI assistant can also write and adjust Data Transform scripts for you and validate them before saving.

Changes take effect after you publish the project.