Testing and Measuring Models
A model you bring yourself is a promise: that it will do the work its tasks need, fast enough for your volume. The Models page (Integrate > Models) lets a tenant administrator check both before a project depends on it.
There are two probes, and they answer different questions:
| Probe | Question it answers | Where | What it spends |
|---|---|---|---|
| Test | Is each endpoint reachable, and can it do the tasks it is assigned? | the Test link next to each endpoint in the Endpoint health column | a few thousand tokens per endpoint, about one document's worth of time |
| Performance | How fast is it, and how much load does it hold? | the Performance action in the row menu | on the order of 200,000 tokens per endpoint at the default settings, several minutes |
Both need the tenant.manage_models permission, both count against your
budgets, and both show on the Usage tab like any
other traffic. Both send synthetic data only: a generated invoice that is
labelled as test data, never one of your documents.
System models (the ones supplied with the platform) get a reachability ping only. They are tested and measured by the platform administrator; see Measuring system models.
Test: can it do the work
Every task in the pipeline is made of a few basic operations that a model either performs or does not. Test runs those operations, one call each, on each enabled endpoint of the model, and reads a verdict per task off the results.
| Operation | What is sent | Needed by |
|---|---|---|
| Text completion | a short text and a question about it | every task |
| Structured output | a 5-page synthetic invoice with a 24-field schema, including a classification field with a fixed list of allowed labels | split, classify, extract, redact |
| Image input | a page raster carrying a code that exists only in the pixels | split, extract and redact when a project sends page images (the default for extract and redact) |
| Tool calling | a request that can only be answered by calling a declared function | assistant |
| Streaming | the same request with streaming on; the reply must arrive in pieces | assistant |
Only the operations your assigned tasks need are run, plus any the model declares (image, streaming) so that an optimistic setting is caught here rather than on a customer's document.
Reading the result
Under the endpoint, the Endpoint health column shows one line: Supports split, classify, extract, or Limited: extract, or Unavailable: redact. Click it for the working. Each task gets one of three verdicts:
- Supported: every operation the task needs passed.
- Limited: the task works, but an operation that most projects' default configuration uses failed. The usual case is image input: text-only extraction still works, but a project that sends page images will not get them read. The reason names the operation and the endpoint.
- Unavailable: an operation the task cannot do without failed. Structured output failing takes down split, classify, extract and redact; tool calling failing takes down the assistant.
The verdicts are never asserted per task. They are derived from the operations, so a red cross always names the operation that caused it, and that is the thing to fix: image input turned off at the provider, JSON mode unsupported, tool calling not enabled on the endpoint, streaming blocked by a gateway.
With several endpoints, the model's verdict per task is the worst any endpoint gave, because the pool will route to that endpoint sooner or later.
While it runs
A Test on a slow endpoint takes a while: the structured-output operation is a full extraction. The Endpoint health column shows which operation is in flight, how many are done, and for how long ("Testing structured output, 1 of 3 operations, 42 s"); click the line to open the dialog, which lists every operation and ticks them off as they land. Slow is a finding, not an error. The dialog has a Stop button; a stopped run keeps the operations that finished.
Reachability latency is a whole round trip including the model writing its reply, so a few hundred milliseconds is normal and does not mean the network is slow.
Performance: how fast, and how much load
Performance measures numbers, per endpoint. Whether the model can do its work is Test's question; an endpoint that fails an operation is measured anyway, because speed is a separate fact.
- Input and output token rates. How many input tokens per second the endpoint processes and how many output tokens per second it generates, for a single stream. Measured by streaming where the model supports it, otherwise by regression over three deliberately lopsided calls. These are the same units the Performance Calculator uses for on-premises models.
- What a page image costs. The seconds and tokens one page raster adds, measured by difference: the same document with and without images. This is the figure that decides whether an image-sending activity is affordable at your volume.
- Capacity under concurrency. The same request at 1, 2, 4, 8 streams at once (up to the cap you choose), reported as tokens per minute per endpoint, the point where more streams stop buying throughput, and the point where the provider rate-limits (HTTP 429). A single stream cannot tell you this: an endpoint often delivers several times more under concurrency than one stream shows.
Before you run it
The dialog lists the endpoints to measure (all enabled ones by default; each is measured separately, because throughput adds across endpoints while the time to process one document does not), the concurrency cap, and an estimate of the calls and tokens it will spend. There is deliberately no time estimate: how long it takes is what is being measured.
The ramp drives the endpoint at rising concurrency until throughput stops improving or the provider pushes back. On an endpoint shared with production traffic that can trip the rate limit and slow real work. Run it outside business hours, or choose concurrency 1 to measure a single stream and nothing more.
Reading the result
The report fills in as it is measured: the rates first, then the image cost, then a row per concurrency level in the capacity table. You can stop at any point and keep what has landed.
- Single-stream figures (the rates and the image cost) describe latency: how long one document takes on that endpoint.
- Capacity of this endpoint: N tokens per minute, reached at C streams is the number to size with. If a level says rate limited here, the ceiling is your quota, not the hardware; a higher quota or a second region is the fix, not a faster model.
- With several endpoints, Across k endpoints: N tokens per minute is the model's capacity as deployed.
Sizing endpoints
The Performance Calculator turns your document volume into a required throughput in tokens per minute. Divide that by the capacity measured here, per endpoint, and round up: that is how many independent endpoints you need.
For example, a required 120,000 tokens per minute against a measured 45,000 tokens per minute per endpoint needs three endpoints. Register them as endpoints of the same model and the pool spreads the load across them.
Never divide by a single-stream rate. It is several times lower than capacity and would tell you to buy far more than you need.
Where else you can run these
- Copilot. Ask the assistant to test a model or measure its performance; it runs the same probes and reports the same verdicts and numbers. It will ask before starting a performance run, because of what that spends.
- API and external agents. The same tools are available over the external agent interface for an agent holding a key with the models permission.
Runs started anywhere show on the Models page while they run and after.