Test and Measure Models

The models supplied with the platform (the system models) serve every tenant that has not brought its own. Tenant administrators can only ping them; checking that they do their work, and measuring how fast they are and how much load they hold, is yours to do, with the CLI. The same commands can also probe a tenant's own models on their behalf.

This assumes the CLI is installed and pointed at your deployment (System Administration).

Two probes

Command	Question it answers	What it spends
da admin models test	Is each endpoint reachable, and can it perform the operations its tasks need?	a few thousand tokens per endpoint, about one document extraction's time
da admin models performance	How fast is it, and how much load does it hold?	on the order of 200,000 tokens per endpoint at the default concurrency, several minutes

Both send synthetic data only: a generated invoice labelled as test data. When the model is a tenant's, the tokens count against that tenant's budgets and show on its Usage tab; a hard budget that is reached refuses the run. System-scope runs have no tenant and no budget.

The concepts (operations, verdicts, single-stream rates versus capacity) are the same as on the tenant's Models page and are explained in Testing and Measuring Models. This page covers the commands.

See what you can probe

da admin models list                 # system models
da admin models list --tenant acme   # a tenant's own models

The model id in the first column is what the other commands take.

Test: can it do the work

da admin models test gpt-5.4
da admin models test gpt-5.4 --tenant acme
da admin models test gpt-5.4 --endpoint eu-1 --endpoint eu-2

One ping per endpoint, then on each endpoint that answered, the canonical operations the model's assigned tasks are made of: text completion, structured output, image input, tool calling, streaming. The output has three parts:

1. Reachability, one row per endpoint, with the round-trip time. That time includes the model writing its reply, so a few hundred milliseconds is normal.
2. Operations, one row per endpoint and operation, ok or fail with the detail (what came back).
3. A verdict per task: supported, limited (works, but an operation the default project configuration uses failed, typically image input) or unsupported (an operation the task cannot do without failed), each naming the operation.

The exit code is 0 when every endpoint answered and no task is unsupported, otherwise 1, so the command can sit in a check script. The structured-output operation is a full extraction; on a slow endpoint the command shows a spinner for a minute or so. That is a finding about the endpoint, not a fault of the command.

For a monitoring loop that runs every minute, the full sweep is too much. --ping-only keeps the old 36-token reachability check:

da admin models test gpt-5.4 --ping-only

--json prints the raw report for a script to parse.

Performance: how fast, and how much load

da admin models performance gpt-5.4
da admin models performance gpt-5.4 --tenant acme --concurrency 4
da admin models performance gpt-5.4 --concurrency 1     # single stream only, no ramp

Per endpoint, in this order:

1. Input and output token rates for a single stream (tokens per second).
2. What a page image costs: the seconds and tokens one page raster adds.
3. Capacity under concurrency: the same request at 1, 2, 4, 8 streams at once (doubling up to --concurrency), reported as tokens per minute, the level where more streams stop helping, and the level where the provider rate-limits (HTTP 429).

The run is a background job: the command starts it, follows it with the step it is on and how long it has been there, and prints the report when it is done. Closing the terminal does not stop the run; if the command gives up waiting (15 minutes), it stops the run first, so nothing keeps spending tokens unwatched.

caution

The ramp drives the endpoint at rising concurrency until throughput stops improving or the provider pushes back. A system model's endpoints carry every tenant's production traffic; the ramp can trip their rate limit and slow real work. Run it outside business hours, and use --concurrency 1 when you only need the single-stream rates.

Reading the report

• Per endpoint: the rates, the image cost and the capacity in tokens per minute. With several endpoints behind one model, compare them: two endpoints of the same model in different regions can be different builds, and the slow one hiding behind the fast one is the finding this exists to surface.
• Capacity across k endpoints is the model's capacity as deployed, and the number to divide by.
• Rate limited at about N tokens per minute means the ceiling is quota, not hardware. A higher quota or an endpoint in another region is the fix, not a faster model.

Sizing

The Performance Calculator turns a document volume into a required throughput in tokens per minute. Divide it by the measured capacity per endpoint and round up: that is how many independent endpoints the model needs. Never divide by a single-stream rate; it is several times lower than capacity and would oversize badly.

For an on-premises model, the calculator's per-GPU input and output token rates are in the same units this command reports; enter the measured rates there in place of the built-in ballparks.

OCR engines

OCR engines are measured separately, in seconds per page on a page with known text:

da admin models ocr-benchmark microsoft_read
da admin models ocr-benchmark tesseract --mode full

quick (the default) is three pages; full runs twice as many for a steadier figure. The engine keys are the ones the Models page and the workflow's OCR step list.

Where the results go

Every run is stored with the tenant it belongs to (system-scope runs under the system tenant), and finished performance runs are appended to a history, so drift over months is visible. A tenant's own runs, whether started from the page, by the assistant or by you with --tenant, all land in the same place and show on that tenant's Models page.