Trials and statistics

An LLM is a stochastic system: send the same prompt twice and you can get two different answers. A single run samples that distribution exactly once, so a green suite might mean “the model reliably does the right thing” or “the model does the right thing 60% of the time and you got lucky.” One sample is not a measurement.

run.trials runs every case several times and folds the per-trial verdicts into one case verdict, so a suite can assert on how often a case passes rather than on a single roll of the dice.

Terminal recording: evalcore runs every case three times and folds the per-trial verdicts into one, tagging each PASS/FAIL line with its passing-trial count and summarizing the flaky count. — Run every case N times, with each verdict tagged by its passing-trial count.

Configure trials

Add trials to the run block. The integer shorthand runs each case that many times and requires every trial to pass:

run:
  trials: 3        # run each case 3 times; the case passes only if all 3 pass

The full form sets the fold policy explicitly:

run:
  trials:
    count: 5           # >= 1
    require: majority  # all | majority | any (default: all)

count must be at least 1 (trials: 0 is rejected: run.trials count must be at least 1). trials: 1, the default when trials is absent, behaves exactly like a run with no trials configured: identical results, identical cache keys, byte-identical reporter output. Trials are opt-in and cost nothing until you ask for more than one.

The `require` policy

A trial passes when every scorer passes for that trial (the same rule a non-trial case uses). The require policy folds the per-trial pass/fail results into the case verdict:

`require`	The case passes when…
`all` (default)	every trial passes
`majority`	strictly more than half the trials pass
`any`	at least one trial passes

majority is strictly more than half: 2 of 3 passes, 1 of 3 fails, and with an even count 2 of 4 fails (2 is not more than half of 4). Pick the policy that matches the claim you want to make: all for “this must never regress,” majority for “this should reliably work,” any for “the model can do this at least sometimes.”

How trials aggregate

A multi-trial case reports one aggregated CaseResult, computed from its trials:

Per-scorer score is the mean of that scorer’s value across the trials. Three trials scoring 1, 1, 0 for a scorer give a case score of 0.667. This mean is what suite gates and baselines see: a mean_score gate averages these per-case means, so trials feed statistical gating directly (see Gates and baselines). A trial whose target errored contributes no score to the mean.
Verdict (passed) follows the require policy above, not the mean. A case can have a 0.667 mean score yet fail under require: all.
Latency is the mean of the per-trial latencies. The individual per-trial latencies are kept in the case’s trial detail.
Cost is the sum across trials. Every trial’s tokens are billed, and every trial’s cost counts toward run.budget_usd. Running 3 trials of a costed target spends roughly 3× a single-trial run, and the budget stops dispatching new cases once that accumulated spend is exhausted.

The full per-trial breakdown (each trial’s pass/fail, scores, latency, and any target error) is preserved in the JSON and HTML reports; the terminal report summarizes it with a tag.

Reading the terminal report

When a case runs more than one trial, its PASS/FAIL line gains a [k/N trials] tag: k passing trials out of N. Single-trial cases are untagged, so their output stays byte-identical to a non-trial run.

Here is a real run of a two-case suite with trials: 3, where the first case passes every trial and the second fails every trial:

PASS greeting (6ms) [3/3 trials]
FAIL mismatch [0/3 trials]
     contains: trial 0: expected output to contain "hello", got: "goodbye"

1 passed, 1 failed, 2 total

FAILED

The reason line names the scorer and the first failing trial (in trial order), so it stays actionable; the reason for every trial lives in the JSON and HTML detail. The same suite with no trials configured drops the tag entirely:

PASS greeting (7ms)
FAIL mismatch
     contains: expected output to contain "hello", got: "goodbye"

1 passed, 1 failed, 2 total

FAILED

The example above uses a deterministic shell target, so every trial agrees and the tag reads [3/3] or [0/3]. Trials earn their keep against a stochastic target, such as a live model or a judge grading fuzzy output, where the trials genuinely diverge and [2/3 trials] tells you the case is flaky rather than solid.

Flakiness statistics

Once a run carries per-trial detail, the reporters distil it into flakiness figures. This is a reporter-layer computation over the trials, so nothing about the engine or the aggregation above changes.

A case is flaky when its trials split: it passed some but not all of them (0 < passed < count). A case that passes every trial or fails every trial is not flaky. It is solid, just solidly passing or solidly failing. The terminal summary line appends a · N flaky count whenever at least one case ran with trials detail:

PASS greeting (4ms) [3/3 trials]
FAIL mismatch [0/3 trials]
     contains: trial 0: expected output to contain "hello", got: "goodbye"

1 passed, 1 failed, 2 total · 0 flaky

FAILED

This is the same deterministic shell suite as above: every trial of a case agrees, so neither case splits and the count is 0 flaky. Against a stochastic target, where greeting might pass 2 of 3 trials, that case would be counted in the suffix. A single-trial run (or any run with no trials detail) omits the suffix entirely, so its summary line stays byte-identical to before.

The JSON report attaches a trial_stats object to each case that has trials detail. These are the flakiness numbers behind the terminal suffix, with sorted keys:

"trial_stats": {
  "pass_fraction": 0.0,
  "score_mean": {
    "contains": 0.0
  },
  "score_range": {
    "contains": [
      0.0,
      0.0
    ]
  }
}

pass_fraction is passing trials over count, the flakiness measure (there is no pass@k terminology; require: any already expresses “at least one of N”). score_mean and score_range give, per scorer, the mean and the [min, max] of that scorer’s value across the trials, so you can see both the central value the case reports and how far the trials spread. Cases with no trials detail gain no trial_stats key, so single-trial JSON is byte-identical to plain serialization.

The HTML report’s per-case Trials section carries the same figure in its header, k/N passed (pass fraction 0.67), above the per-trial table and a per-scorer mean/min/max summary.

Determinism and the cache

Trials are built on EvalCore’s record/replay cache without breaking the determinism guarantee that makes cassettes replay in CI:

Trial 0 keeps the pre-trials cache key, byte-for-byte. A cassette recorded before trials existed, or by a trials: 1 run, replays as trial 0 with no re-recording.
Trials 1..N re-key by adding the trial index to the cache key, so each trial is its own cache entry. This is what lets replayed trials differ instead of every trial returning the same recorded answer. Without it, trials over a cached target would measure nothing.
Judge and similarity calls re-key per trial the same way. An LLM-as-judge or embedding call made while scoring trial i is salted with i too, so a cached judge verdict varies across trials exactly as the target output does.

The practical consequence: record once with --cache auto (or --cache live), commit the cassette, and --cache replay reruns all N trials of every case offline, keyless, and deterministically. It is the same CI story as a single-trial suite, just with N recorded samples per case.