Evaluation
Measure what
your workflow needs.
A correct field, a relevant result, and an answer with the right citation are different tests. Our evaluation framework measures task quality alongside latency, processing cost, and human review.
| Dimension | What to measure | What to disclose |
|---|---|---|
| Task quality | Field accuracy, routing errors, ranking relevance, or document structure | Dataset composition, sample size, held-out split, and baseline |
| Reliability | Schema validity, missing outputs, unsupported cases, and retry rate | Failure taxonomy and behavior on incomplete inputs |
| Latency | End-to-end p50 and p95 under defined concurrency | Hardware, input sizes, warm-up, batch size, and load |
| Cost | Inference and review cost per accepted result | Pricing unit, retry costs, and review assumptions |
| Evidence | Correct source spans, page references, and supported answers | Annotation process and human review method |
THE PROCESS
Define. Compare. Stress-test.
- Choose a narrow task and document the current workflow baseline.
- Build an authorized dataset covering typical inputs and difficult cases.
- Separate training, development, and held-out evaluation examples.
- Compare model approaches at the same quality target and operating conditions.
- Publish limitations and run a shadow pilot before production promotion.
Data boundaries
Evaluation starts with authorized data and an agreed scope. Data access, retention and processing requirements are defined before a collaboration begins.