Skip to main content
Avala measures your model on real-world scenes that your training data never touches. The same scenes score every model version, so you can compare versions directly.

​
Held-out evaluation scenes

An evaluation cohort is a fixed set of real-world scenes and tasks. Each cohort targets specific coverage cells and failure modes.
  • Never released for training. Sequences in a cohort carry evaluation_membership: "held_out_eval" in their outcome label. They never appear in a training release.
  • Separated by leakage group. No location, object set, mechanism unit, operator or environment family in a cohort appears in any training release. See leakage groups.
  • Checked for contamination. Before each release, Avala fingerprints the release and checks it against every active cohort. Contamination checks are versioned. Each report names the check version and the cohort version it ran against.
  • Versioned. A cohort changes only by a new version. Reports always name the cohort version, so numbers stay comparable.

​
Scoring

Every evaluation run is scored in two stages.
  1. VLM pre-score. A vision-language model scores each sequence and subtask against the task rubric. It proposes an outcome, subtasks with their boundaries, and a mistake_type and failure_stage from the failure taxonomy. A pre-score is stored with source: "model" and a confidence.
  2. Human audit. Trained reviewers check the pre-scores. They audit every disputed sequence and a random sample of the rest, and they correct the labels. A corrected label is a new version with source: "human". The earlier versions stay in the label history.
Scoring is a service Avala performs. Avala reports inter-rater agreement for each run. The report gives agreement between the VLM pre-score and the audit, and agreement between human reviewers on a double-scored sample. A metric is reported only when agreement meets the threshold set in the brief.

​
Metrics

Each metric is reported overall and per coverage cell. Cells with too few sequences for a stable interval are marked.

​
The capability report

Avala delivers a capability report after every evaluation, as part of the program. It answers one question: what can this model version do, and what changed? The report links every number back to its sequences. You can open any example in Mission Control.

​
What you provide

You provide the model version to evaluate. The brief sets where and how each evaluation run happens. Evaluation rollouts reach Avala the same way failures do. See Failure ingestion. If you already run evaluations with Inspect Robots, an open-source evaluation tool, you can import its logs. Each trial labels one sequence. See Import evaluation logs. The audited labels are stored as outcome labels on the evaluation sequences, so you can read them yourself. This lists the failed and aborted held-out sequences:
The API, the SDKs and the MCP server can also filter by model_version, so you can compare policy versions. See Read and write outcome labels.