Core concepts

An EvalCore suite has four moving parts, declared in one evals.yaml: targets produce outputs, datasets supply inputs, scorers judge the outputs, and a run block ties it together with concurrency, budgets, and gates. Every feature starts as config surface, because the YAML file is the interface.

The run pipeline

Every run flows through the same stages, in order. Cases move through the middle in dataset order and results come back in that same order. Position is load-bearing.

Cases move through the engine in dataset order and results return in that same order. Position is load-bearing.

The target runs once per case; cacheable targets consult the cassette first (see Record / replay).
Every scorer runs on every case’s output.
Gates and the baseline diff are computed over the whole RunSummary after the last case finishes.
Reporters are pure functions of the summary; the exit code folds the per-case, baseline, and gate verdicts together.

Targets: what’s evaluated

A target is the thing under test. A run evaluates one target (selected with --target <name> when a suite defines several; implicit with one target), or several at once in a matrix run. There are four types:

Type	What it does
`openai-compatible`	POSTs to `{url}/chat/completions` in the OpenAI wire format. Supports `model`, `system`, pass-through `params`, `api_key_env`, retries, `timeout_seconds`, and `cost` rates.
`http`	Calls any HTTP/JSON endpoint (typically your own deployed app’s REST API) and caches it like an LLM call. `{{input}}` is substituted into the `url` and `body`; `response_path` pulls the answer out of the JSON.
`shell`	Runs a command with the case input piped to stdin; stdout is the output. Never cached, because it runs your local code.
`trace`	Ingests a recorded agent trace (native or OTel/OpenInference) named by each case, instead of invoking anything. Pair with the `trajectory` scorer.

targets:
  openai:
    type: openai-compatible
    url: https://api.openai.com/v1
    model: gpt-4.1-mini
    api_key_env: OPENAI_API_KEY

Secrets never live in the YAML: api_key_env names an environment variable, resolved at run time. Full field-by-field docs are in the configuration reference.

Datasets: the inputs

Datasets are JSONL files, one case per line, merged in listed order. Each case has an id and, depending on the target, an input (and optionally an expected, used by exact and passed to subprocess/judge scorers):

{"id": "refund-1", "input": "How do I get a refund for my order?"}

For trace targets, each case names a trace file instead of an input:

{"id": "refund-flow", "trace": "traces/run1.json"}

Results always come back in dataset order. Determinism is the product.

Scorers: how outputs are judged

Every scorer runs on every case. They form a ladder from cheap and deterministic to fully custom:

contains · exact · regexdeterministic, instant, no networkfree

subprocessyour script, any language, JSON over stdioyour code

judgeLLM-graded rubric, replay-deterministiccached

trajectoryagent path: tool calls, order, step budgettraces

The ladder: start deterministic, add judgment only where string checks stop being enough.

Deterministic checks: contains (substring, with case_sensitive), exact (equals value, or the case’s expected field), regex (matches a pattern). No network, instant, perfectly reproducible.
json-schema validates the output as JSON against a draft 2020-12 schema file. Failing reasons name violations by JSON pointer, deterministically ordered, and validation never touches the network. See the configuration reference.
subprocess is the any-language escape hatch. Your command receives {"input", "output", "expected"} as JSON on stdin and prints {"score": 0.0..=1.0, "passed"?: bool, "reason"?: string} on stdout. Write scorers in Python, Node, Go, or anything else that reads stdin and writes stdout. See Custom scorers.
similarity grades semantic closeness between the output and the case’s expected via any OpenAI-compatible embeddings endpoint, passing at a threshold (default 0.8). Embedding calls ride the record/replay cache like judge calls, so replayed scores are free and offline. See Semantic similarity.
judge is LLM-as-judge. It grades the output against a rubric using any OpenAI-compatible endpoint, with a configurable pass threshold. Judge calls go through the record/replay cache, so replayed verdicts are deterministic, which is what makes LLM-graded suites usable as CI gates. See LLM-as-judge.
trajectory asserts on an agent’s path (tool calls, ordering, step budget). It requires a trace target. See Agents and traces.

scorers:
  - type: contains
    value: "30 days"
  - type: subprocess
    cmd: python3 scorers/faithfulness.py
  - type: judge
    url: https://api.openai.com/v1
    model: gpt-4.1-mini
    rubric: "Is the answer grounded in the provided context?"
    api_key_env: OPENAI_API_KEY
    threshold: 0.7

Runs: concurrency, budgets, gates

The optional run block controls execution:

run:
  concurrency: 4          # max in-flight cases (default 4)
  budget_usd: 5.0         # stop dispatching new cases past this spend
  gates:
    - type: pass_rate
      min: 0.95
    - type: mean_score
      scorer: judge
      min: 0.8

concurrency bounds how many cases run at once (default 4). It never changes results, since order is preserved regardless.
budget_usd stops dispatching new cases once accumulated cost reaches the cap (requires the target to declare cost rates). Skipped cases are reported as failures with a reason, so the run completes rather than aborting. See Cost and budgets.
gates are absolute floors over the whole run: pass_rate (fraction of cases passing every scorer, in [0,1]) and mean_score (mean scorer value, optionally restricted to one scorer). See Gates and baselines.

The exit-code contract

evalcore run exits 0 when every case passes and 1 otherwise. Two mechanisms extend that contract additively:

With --baseline, the contract flips to “no regressions”. Failures already accepted into the baseline are tolerated, but a regression or a new failing case exits 1.
run.gates add absolute floors. Dropping below any floor also exits 1, even when it is not a per-case failure or a regression.

Users gate CI on this exit code and nothing else. The full truth table is in Gates and baselines.

Failures are data

A run never panics and one bad case never aborts the suite. Errors are captured and reported, not thrown:

A target error (a 500, a timeout, a budget skip, a replay cache miss) becomes a failed case with the error as its reason and no scores:
```
FAIL new
     target error: cache miss for case "new" in replay mode — record it first with --cache auto (or live)
```
A scorer error (a subprocess that crashes, an un-parseable judge verdict) becomes a failing score with a reason. The other scorers on that case still run.

The run always finishes, reports every case in dataset order, and lets the exit code carry the verdict. That is why an eval suite is safe to run unattended in CI: a flaky endpoint produces a red build with an explanation, never a hang or a stack trace.

Determinism is the product

Identical inputs produce identical outputs everywhere. That property is what makes an eval suite a test rather than a dashboard:

Results stay in dataset order, no matter the concurrency.
Reporters are pure functions of the run summary. The same summary renders byte-for-byte identical terminal, JSON, JUnit, and HTML output.
Nothing user-visible reads the clock except latency measurement.
The record/replay cache is built on this: cache keys hash the canonical request, so the same request always replays the same recorded response. Change the model, prompt, params, or input and the key changes, so a stale recording can never masquerade as a fresh one. See Record / replay and the cache and determinism reference.