> ## Documentation Index
> Fetch the complete documentation index at: https://avala.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation

> Held-out real-world evaluation scenes, scoring with audited agreement, capability metrics and the capability report

Avala measures your model on real-world scenes that your training data never touches. The same scenes score every model version, so you can compare versions directly.

## Held-out evaluation scenes

An evaluation cohort is a fixed set of real-world scenes and tasks. Each cohort targets specific [coverage cells](/docs/fleet-learning/coverage.md#coverage-cells) and failure modes.

* **Never released for training.** Sequences in a cohort carry `evaluation_membership: "held_out_eval"` in their [outcome label](/docs/fleet-learning/sequence-metadata.md#evaluation-membership). They never appear in a training release.
* **Separated by leakage group.** No location, object set, mechanism unit, operator or environment family in a cohort appears in any training release. See [leakage groups](/docs/fleet-learning/sequence-metadata.md#leakage-groups).
* **Checked for contamination.** Before each release, Avala fingerprints the release and checks it against every active cohort. Contamination checks are versioned. Each report names the check version and the cohort version it ran against.
* **Versioned.** A cohort changes only by a new version. Reports always name the cohort version, so numbers stay comparable.

## Scoring

Every evaluation run is scored in two stages.

1. **VLM pre-score.** A vision-language model scores each sequence and subtask against the task rubric. It proposes an `outcome`, `subtasks` with their boundaries, and a `mistake_type` and `failure_stage` from the failure taxonomy. A pre-score is stored with `source: "model"` and a `confidence`.
2. **Human audit.** Trained reviewers check the pre-scores. They audit every disputed sequence and a random sample of the rest, and they correct the labels. A corrected label is a new version with `source: "human"`. The earlier versions stay in the label history.

Scoring is a service Avala performs. Avala reports inter-rater agreement for each run. The report gives agreement between the VLM pre-score and the audit, and agreement between human reviewers on a double-scored sample. A metric is reported only when agreement meets the threshold set in the brief.

## Metrics

| Metric | Definition |
| - | - |
| Success probability | The share of sequences that reach the goal state, with a 95% confidence interval. |
| Partial progress | The mean share of subtasks completed per sequence. |
| Time to success | The median time to reach the goal state, over successful sequences. |
| Human-relative throughput | Tasks completed per hour, divided by the expert human reference for the same cohort. |
| Intervention rate | Human interventions per sequence and per hour of autonomous operation. |
| Failure-mode distribution | The share of failed sequences in each failure mode of the taxonomy. |
| Recovery rate | The share of sequences with an error where the robot recovers and completes the task without help. |

Each metric is reported overall and per coverage cell. Cells with too few sequences for a stable interval are marked.

## The capability report

Avala delivers a capability report after every evaluation, as part of the program. It answers one question: what can this model version do, and what changed?

| Section | Contents |
| - | - |
| Scope | Model versions compared, cohort IDs and versions, contamination-check version, run dates. |
| Headline | Each metric for each model version, with confidence intervals. |
| Per-cell results | Metrics per coverage cell, with thin cells marked. |
| Failure modes | The failure-mode distribution per model version. Example sequences for each mode. |
| Recovery | Recovery rate and the failure-and-recovery examples behind it. |
| Scoring quality | Audit sample size and inter-rater agreement. |
| Experiments | Each open [capability experiment](/docs/fleet-learning/capability-experiments.md), its hypothesis and its measured outcome. |
| Next allocation | The cells and failure modes the next brief targets, and why. |

The report links every number back to its sequences. You can open any example in Mission Control.

## What you provide

You provide the model version to evaluate. The brief sets where and how each evaluation run happens. Evaluation rollouts reach Avala the same way failures do. See [Failure ingestion](/docs/fleet-learning/failure-ingestion.md).

If you already run evaluations with Inspect Robots, an open-source evaluation tool, you can import its logs. Each trial labels one sequence. See [Import evaluation logs](/docs/fleet-learning/failure-ingestion.md#import-evaluation-logs).

The audited labels are stored as outcome labels on the evaluation sequences, so you can read them yourself. This lists the failed and aborted held-out sequences:

```bash theme={null}
avala sequence-outcomes list your-org kitchen-eval --evaluation-membership held_out_eval --outcome failure,aborted
```

The API, the SDKs and the MCP server can also filter by `model_version`, so you can compare policy versions. See [Read and write outcome labels](/docs/fleet-learning/sequence-metadata.md#read-and-write-outcome-labels).