Skip to content
EvalCore

Featured on Product Hunt

Know when your AI gets worse,
before your users do.

Change a prompt, swap a model, update a dependency. Did anything break? EvalCore records how your AI behaves and checks every change against it. One small binary, any language, no flaky tests.

Get startedInstall
  • Apache-2.0 open source
  • $0 in CI
  • offline replay
  • any language
evalcore · bash

One file. One live run. Replay forever.

An eval is a YAML file plus a JSONL dataset. Run it once against the real model to record a cassette, then CI replays the recording on every change: offline, keyless, deterministic.

  1. Describe the suite as data

    No SDK, no test harness. Point a target at your app, list the dataset, stack scorers on top. Anything that speaks HTTP or shell is a target; your app's language never matters.

    Configuration reference
    evals.yaml
    Target: your app
    targets:
  support-bot:
    type: openai-compatible
    model: gpt-4.1-mini
    Dataset: your cases
    datasets:
  - file: cases.jsonl
    Scorers: what "good" means
    scorers:
  - type: contains
    value: refund
  - type: judge
    rubric: Is the answer grounded?

  2. Run it once, live

    The first run calls the real model and records every request and response to a local SQLite cassette, keyed on a hash of the canonical request. Tokens and cost land on the summary.

    How recording works
    $ evalcore run evals.yaml
    PASS late-refund (843ms)
PASS fee-dispute (512ms)

2 passed, 0 failed, 2 total · 2202 tokens · $0.0038

✔ PASSED
    recorded to .evalcore/cache.db, commit it

  3. Replay in CI: $0, offline, deterministic

    Every PR replays the cassette. No network, no API keys, no flaky judge: identical inputs produce identical verdicts, and --baseline main fails only on regressions. CI gates on the exit code.

    Running in CI
    live run
    • 843ms /case
    • $0.0038
    • network + keys
    replay in CI
    • 0ms /case
    • $0
    • fully offline
    evalcore run evals.yaml --cache replay --baseline mainexit 0

Everything a suite needs, one binary

Pick a capability. Each one is a few lines of YAML or a flag, not a framework.

Cassettes make evals free and deterministic

Every model call is recorded to a local SQLite cassette, keyed on a hash of the canonical request. Commit it, and CI replays byte-for-byte: no network, no keys, no flaky judges.

--cache replayRead the guide

Works with OpenAI, vLLM, Ollama, any OpenAI-compatible gateway, your own REST APIs, shell commands, and OTel / OpenInference traces.

Ship your first eval in minutes

One dependency-free binary. No telemetry, no signup, no server.

cargo install evalcore
Get startedInstall

No Rust toolchain? Prebuilt binaries cover macOS, Linux, and CI runners.