Data Transform

Category: Transform

The Data Transform activity runs a JavaScript script over the whole transaction. Use it to reshape extracted field values, read and write transaction metadata, restructure documents (move pages, split, merge, delete, reorder, reassign a class), and read or update catalog records, at any point in the workflow.

The script runs in a hardened sandbox with no network and no file access. It reads an in-memory copy of the transaction and returns the changes you make through a fixed set of helper functions.

When to use it

Reach for a Data Transform when you need logic the standard activities do not cover, for example:

• Normalize or derive field values (uppercase a code, compute a total, map a country name to an ISO code).
• Set transaction metadata to drive later branching or export naming.
• Fix imperfect splitting: move a stray page into the right document, peel a page into its own document, or merge two documents.
• Reassign a document's class based on its content.
• Look up catalogs and maintain them: for example, register a vendor in the catalog when a document arrives from one you have not seen before.

If you only need to validate or normalize a single field, a business rule is usually simpler. If you need to call an external system, use a Notification webhook instead; Data Transform cannot do network calls.

Configuration

The Configure panel groups the settings into Script access (what the script can touch) followed by the Script itself:

Setting	Description	Default
Catalog access	The catalogs the script may read with the catalog API and write with the catalog record helpers. Only the catalogs you select here are accessible.	(none)
Include source files	Load the original file bytes into the script context. Off by default because loading the bytes adds processing time; turn it on only when the script actually needs the raw bytes.	No
Script	The JavaScript to run, edited in a full code editor with autocompletion.	(starter template)
Timeout (seconds)	Maximum script run time before it is stopped.	30

A newly added Data Transform node is a valid no-op until you write a script, so you can place it in the workflow first and fill in the code later.

Runtime interface

Your script has access to these globals. Everything is read-only unless noted.

Global	What it holds
documents	Array of documents, in order. Each has document_id, page_indices (ordered page ids), document_type (the class name), document_class_id, and extracted_fields. Mutable through the assembly helpers.
pages	Array of pages (page_index, OCR text, barcodes).
transaction	The transaction record.
transaction_metadata	User-defined metadata. Mutable through set_metadata; new keys are allowed.
project_settings	The project configuration.
validation_results	Business-rule results, or null until Validate has run.
parameters	Values passed in from the node config.
source_files	File metadata, including each file's split_policy. The raw bytes are present only when Include source files is on.
catalog	The catalog API (lookups and record writes). Available for the catalogs selected under Catalog access; see Working with catalogs.

Field and metadata helpers

Helper	Effect
get_field_value(doc, nameOrId)	Returns a field value, matched by field id or display name, or undefined if absent. Matches top-level fields only; a field inside a group or repeating group (table) is not found by its own name (see Groups and repeating groups).
set_field_value(doc, nameOrId, value)	Sets a top-level field value, or creates a new top-level field when the name matches nothing. Do not use it for a field inside a group: it would silently create a new top-level field instead (see Groups and repeating groups).
get_metadata(key?)	Returns one metadata key, or the whole object if no key is given.
set_metadata(key, value)	Writes transaction metadata (new keys allowed).
normalize(text, options?)	Cleans text. Options: trim, lowercase, uppercase, removeSpaces, removeSpecialChars.
resolve_country(input, format?)	Maps a country name to an ISO code (alpha2 by default), or null if unresolved.

Your script may return a value, which is surfaced in the step result. Returning nothing is fine.

Groups and repeating groups

The field helpers above match top-level fields only. When a field lives inside a group or repeating group, get_field_value(doc, 'Name') returns undefined even though the document clearly shows a Name: the child is stored inside the group's value, not as a field of its own. Fetch the group and traverse its value instead:

• A group's value is an object keyed by child field id, where each child is { name, value, ... }. For example a Vendor group holds { v001name: { name: 'Name', value: 'TAON Ltd' }, v003ctry: { name: 'Country', value: 'CY' }, ... }.
• A repeating group (also called a table) has the value { instances: [ ... ] }, with one such child object per row.
• Groups can nest: a child whose own value has one of these shapes is a nested group.

A small helper makes children addressable by display name:

// Find a child of a group (or of a repeating-group row) by display name.
function sub_field(group, name) {
  if (!group || typeof group !== 'object') return null;
  if (group[name]) return group[name]; // also works with a child field id
  for (const id in group) {
    if (group[id] && group[id].name === name) return group[id];
  }
  return null;
}

// Read a child of a group.
const vendor = get_field_value(d, 'Vendor');
const vendorName = (sub_field(vendor, 'Name') || {}).value;

// Write a child: mutate it in place. Never set_field_value a group child.
const country = sub_field(vendor, 'Country');
if (country && country.value) country.value = resolve_country(country.value);

// Repeating group: iterate the rows.
const items = get_field_value(d, 'Line Items');
for (const row of (items && items.instances) || []) {
  const qty = sub_field(row, 'Quantity');
  if (qty && qty.value === '') qty.value = '1';
}

Two details to watch:

• Extraction writes every child of a group, so a cell the document did not fill is still present, with value: ''. Check for emptiness (!child.value), not for existence.
• In validation_results, results for group children are keyed by path, <groupFieldId>/<childFieldId> (for example b2c3d4e5/p001ctry). Split the key on / to find the group and the child; passing the whole path to get_field_value matches nothing.

Activity Log messages and failing the transaction

The script can report messages into the transaction's Activity Log (the event timeline you open from the transactions table), and it can explicitly fail the transaction.

Helper	Effect
log_info(message)	Record an info message in the Activity Log.
log_warning(message)	Record a warning in the Activity Log.
log_error(message)	Record an error in the Activity Log. Reporting only: it does not fail the transaction.
fail(message)	Stop the script immediately and set the transaction to failed, with message as the failure reason.

Notes on how these behave:

• Messages accept any value; objects are JSON-stringified. Each run records up to 100 messages of up to 500 characters each.
• The three log_* helpers never change the flow of the workflow. Use log_error for problems worth flagging that should not stop processing (for example, "3 of 120 rows had no catalog match").
• fail() is deliberate and final: the step is not retried, and none of the script's changes are persisted (field values, metadata, document restructuring, and queued catalog writes are all discarded). Messages logged before the fail() call are still written to the Activity Log, so log the details first, then fail.
• A script error (an uncaught exception) also fails the step, but it is treated as a script bug and retried by the workflow engine. Use fail() when the failure is a business decision, so the transaction fails once, immediately, with your message as the reason.

// Fail the transaction when a required document is missing.
const invoice = documents.find(d => d.document_type === 'Invoice');
if (!invoice) {
  log_error('Transaction has ' + documents.length + ' document(s), classes: '
    + documents.map(d => d.document_type).join(', '));
  fail('No Invoice document found in this transaction');
}

// Report progress without affecting the workflow.
log_info('Matched ' + matched + ' of ' + total + ' line items against the price catalog');
if (matched < total) log_warning((total - matched) + ' line items had no catalog match');

In the Activity Log, each message appears as its own timeline entry with an info, warning, or error icon. A fail() shows as a failed step with your message, and the transaction status changes to failed.

Working with catalogs

Select catalogs under Catalog access in the node settings to unlock the catalog API. Scripts can then look records up and add, update, or delete records.

Reading

Helper	Effect
catalog.first(id, criteria)	First matching record, or null.
catalog.exact(id, criteria)	All matching records.
catalog.page(id, criteria, { offset?, limit? })	A page of matches: { records, total, offset, limit }.
catalog.count(id, criteria)	Number of matches.
catalog.get(id, recordId)	One record by its id, or null.

criteria is an object of { column: value } pairs, combined with AND. Values match with the catalog engine's exact normalization (case, whitespace, and punctuation insensitive), the same matching a catalog lookup rule uses. Each record comes back as { record_id, label, data }.

Writing

Helper	Effect
catalog.add_record(id, data, { on_conflict? })	Add a record. data is { column: value }; the record number (internal_id) is assigned automatically.
catalog.update_record(id, recordId, data)	Partial update: the columns in data are merged into the record, other columns keep their values.
catalog.delete_record(id, recordId)	Delete a record. Deleting a record that is already gone is not an error.

Writes are queued, not immediate: they are applied through the catalog service after the script finishes successfully. If the script throws, nothing is written. Each write still goes through the normal catalog checks (column validation and duplicate detection on unique columns), and the step result reports what was added, updated, deleted, or skipped.

on_conflict controls what happens when add_record collides with an existing record on a unique column:

• 'skip' (default): keep the existing record and move on.
• 'update': merge the new data into the existing record.
• 'error': fail the step.

Because reads see a snapshot taken when the script starts (and another transaction may write the same value at the same time), do not treat a catalog.first() miss as a guarantee that an add_record cannot collide. The default 'skip' makes the register-if-missing pattern safe under concurrency.

Example: register unknown vendors

A typical setup pairs a Validate step (a catalog lookup rule flags documents whose vendor is not in the catalog) with a Data Transform that registers the missing vendor:

for (const d of documents) {
  const name = get_field_value(d, 'Vendor Name');
  if (!name) continue;
  if (!catalog.first('cat_vendors', { name: name })) {
    catalog.add_record('cat_vendors', {
      name: name,
      country: get_field_value(d, 'Vendor Country') || null,
    });
  }
}

This works when Vendor Name and Vendor Country are top-level fields. If they are children of a Vendor group, read them through the group instead: const vendor = get_field_value(d, 'Vendor'); then sub_field(vendor, 'Name'), as shown in Groups and repeating groups.

Restructuring documents (assembly API)

The assembly model is positional and live. You address documents by their position in documents and pages by their position within a document's page_indices, and every operation changes the model immediately, so later positions reflect earlier operations. If an operation empties a document, that document is removed automatically.

A document reference is either a document object from documents or its document_id string.

Operation	Signature	Effect
Move page	move_page(fromDoc, fromPos, toDoc, toPos?)	Move one page between documents or within a document. Returns the target document. Omit toPos to append.
Split (spin off)	move_page(fromDoc, fromPos, { new_document: { document_class? } })	Same call, but the target is a spec object. Creates a new document holding that page and returns it.
Move document	move_document(doc, position)	Move a document to position in the transaction.
Delete page	delete_page(doc, position)	Delete one page. If it was the document's last page, the document is deleted too.
Delete document	delete_document(doc)	Delete a document and all its pages.
Set class	set_document_class(doc, className)	Reassign the document's class.
Set split policy	set_split_policy(doc, policy)	Record a per-file split constraint: 'inherit', 'single_document', or 'self_contained'. Stored via the document's source file and read by a later Split step, so it only has an effect in a transform placed before Split, where each document is exactly one uploaded file. See Per-File Split Policies.

There is no empty create_document (a document must always have at least one page, so spin one off with new_document) and no one-call merge_documents (a merge is just moving all of one document's pages into another; the emptied document deletes itself).

Examples

// Normalize a field on every document.
for (const d of documents) {
  const v = get_field_value(d, 'Country');
  if (v != null) set_field_value(d, 'Country', resolve_country(v));
}

// Set metadata for later steps.
set_metadata('ready_for_export', true);

// Split: peel the first page of document 0 into a new Invoice document.
if (documents[0] && documents[0].page_indices.length > 0) {
  move_page(documents[0], 0, { new_document: { document_class: 'Invoice' } });
}

// Merge document 1 into document 0 (move all its pages, in order).
// Move position 0 each time because every move shifts the rest.
const src = documents[1], dst = documents[0];
if (src && dst) {
  while (src.page_indices.length) move_page(src, 0, dst, dst.page_indices.length);
}

// Reorder: make document 2 the first document.
if (documents[2]) move_document(documents[2], 0);

// Drop a blank page.
if (documents[0] && documents[0].page_indices.length > 3) delete_page(documents[0], 3);

// Reassign a class.
set_document_class(documents[0], 'Receipt');

// In a transform placed before Split: pin a cover sheet as one document
// so intelligent splitting never merges it with the files that follow.
for (const d of documents) {
  if (/^cover/i.test(d.source_filename || '')) set_split_policy(d, 'single_document');
}

What happens after a restructure (reprocessing)

The Data Transform activity does not reprocess documents itself. Instead, for each document it changed, it resets that document's completion flags, so any Classify or Extract step later in the workflow reprocesses exactly those documents. Documents you did not touch keep their flags and are skipped.

The rule per document:

• Pages changed (moved, added, removed, or reordered) and you did not set the class: both classification and extraction are reset, so a downstream Classify and Extract reprocess the document.
• set_document_class pins the new class and resets extraction only. A downstream Classify skips the document (keeping the class you assigned), and a downstream Extract re-reads it under the new class.
• Newly split-off documents start fresh, so a downstream step picks them up automatically.

The same applies to validation: Data Transform does not re-run business rules. If the transform changes field values or restructures documents and you want the rules checked again, add an explicit Validate step downstream as well.

Because of this, if you want changed documents re-classified, re-extracted, and re-validated, add explicit Classify, Extract, and Validate steps after the Data Transform node:

... → Split → Data Transform → Classify → Extract → Validate → Review

If nothing runs downstream, the changed documents are simply left pending; the transform does not reprocess them on its own.

Validation and self-correction

The script is checked before it can run, both in the editor and when the workflow is saved. Validation catches:

• Syntax errors.
• Unknown helpers: calling a function that does not exist (often a removed or misremembered helper) is rejected, with the list of the real helpers.
• Runtime errors on sample data: the script is smoke-run against sample documents, so a script that assumes data which may be absent is caught early.

Write defensively so a correct script is not rejected: guard against missing documents and fields (for example if (!documents.length) return; and check get_field_value(...) != null).

Editing the script

Open the node's Configure panel to edit the script in a code editor with autocompletion and inline validation for the helpers and globals above. The AI assistant can also write and adjust Data Transform scripts for you and validate them before saving.

Changes take effect after you publish the project.