Evals

What you’ll build

You changed the retrieval prompt yesterday. Did the assistant get better or worse? Today the same question returned a different answer than it did last week and you can’t tell whether that’s drift, improvement, or something you broke. This chapter is how you tell.

Chapter 7 gave the agent a feedback layer: a verifier that checks its output as it runs. An eval suite is the offline version of that. You run it before you ship a change, and it tells you whether the change helped or hurt.

At its smallest, an eval suite is a runner, a judge, and a handful of test cases, and that much already catches regressions. This assistant needs more: it has grounding tools, memory, and multi-step plans, and a team that wants to ship it, so its suite has to cover all of those surfaces.

By the end of this chapter the running assistant has:

• A six-test eval suite covering factual recall, refusal, and faithfulness.
• Five scorers per case: substring with paraphrase tolerance, regex, latency, faithfulness judge, refusal-discipline judge.
• Run persistence in ./runs/ with a Markdown report and a diff against the previous run.
• A CI gate that exits non-zero when any scorer drops below 70%.
• A calibrated judge rubric and the habit of adding a test every time you spot a real failure.

The code lives in code/chapter-10/. The runner extensions and the suite are already wired up.

The eval vocabulary

Five terms run through this chapter. They mean the same thing everywhere evals show up.

• Test case. A single (id, input, expected?) row. The runner feeds input to the task and hands the result to every scorer.
• Scorer. A small function that takes (input, output, expected?) and returns { pass: boolean, rationale?: string }. Two flavours: rule-based code (substring, regex, latency) and LLM judges.
• Judge. An LLM running as a scorer. You hand it a rubric and the output; it returns PASS or FAIL with a short rationale. Binary, structured JSON, low temperature.
• Rubric. The written instructions the judge reads: what counts as PASS, what counts as FAIL, ideally with a PASS example and a FAIL example built in. (Concept 3 covers how to write one that holds up.)
• Runner. A for loop over cases; runs the task once per case and every scorer against the output. Prints per-scorer pass rates.

Hold those five in your head and this chapter is mostly machinery on top.

Concept 1: Three layers, used in inverse proportion to cost

Evals come in three layers. Each is a different kind of check, with its own cost and its own kind of judgment. The first two do the everyday work; the third matters once real users are involved.

• Layer 1: deterministic checks. Plain code that returns PASS or FAIL: substring matches, regex, JSON-schema validation, latency limits. Free, fast, and the right tool for anything you can pin down in code. containsScorer is one. So is the latency scorer that fails any case slower than 45 seconds.
• Layer 2: an LLM judge. A second model call that scores the output against a rubric. This is the tool for judgment calls code can’t make, like faithfulness (is every claim backed by a cited source?) and refusal-discipline (did the agent correctly refuse a question it can’t answer?). llmJudge is the primitive; the suite here uses two judges.
• Layer 3: human review. A person who knows what good looks like reads the output and rates it. You can’t read everything, so you sample: a slice of production traffic, every case where layers 1 and 2 disagreed, every user complaint. Those ratings are how you keep the judge honest and grow the suite.

Lots of cheap deterministic checks, some judge calls on a sample, occasional human spot-checks. The pattern mirrors software testing: unit tests on every commit, integration tests on every merge, humans look at the product before release. Evals retrofit that pattern for systems that don’t return the same answer twice.

You don’t need a labelled dataset to start. Rule-based assertions need no labels at all, and reference-free judges score useful dimensions without ground-truth answers.

Reference-free judge. A judge that checks the answer against what the agent was given, not against a “right” answer you wrote in advance. It asks “is this claim supported by the context?” instead of “does this match the expected answer?” That means you can start scoring without a labelled dataset. RAGAS’s faithfulness metric is the canonical example (RAGAS docs).

Sidenote: catching hallucinations with NLI. Models hallucinate partly because benchmarks reward guessing over saying “I don’t know” (OpenAI’s Why Language Models Hallucinate). One way to catch a guess is natural language inference (NLI). An NLI model compares one claim from the answer against a trusted source and labels it: entailment (the source supports it), contradiction (the source disagrees), or neutral (the source is silent). Split the answer into single claims, check each against the retrieved context, and treat contradiction as a hallucination and neutral as an unsupported add-on. A small model like Xenova/nli-deberta-v3-base runs locally through Transformers.js, no API call needed.

Match the layer to the question. Pick the wrong layer and you’ll give up after one round. A regex for “is the tone friendly?” fails, because code can’t judge tone. An LLM judge for “does the JSON parse?” works, but it pays for a model call on every run when five lines of code would answer for free. Tone is a judge’s job; valid JSON is code’s job.

What the runner adds beyond the core

Those four primitives tell you what an eval is. To run evals day to day you need five more things, all in code/chapter-10/eval-runner.ts:

• More scorers. Latency, JSON-schema, and trajectory scorers join containsScorer and llmJudge, each under 30 lines. A trajectory is the sequence of tool calls the agent made to reach its answer; trajectoryScorer checks that the right tools fired in the right order (a retriever for factual questions, a calculator for math, web search for current events).
• Run persistence. Every run writes a JSON snapshot to ./runs/<timestamp>.json, and the runner can read prior runs back.
• Diff against previous. diffReports(prev, curr) lists each scorer’s pass-rate change, worst regression first. It catches one scorer quietly dropping while another improves, the kind of trade-off a single overall number hides.
• Markdown report. reportMarkdown(report, diff) renders the pass rates, the diff, and any failures with their model output. Save it as runs/latest.md or paste it into a PR.
• CI gating. gateOrExit(report, 0.7) exits non-zero when any scorer drops below 70%. A basic runner gates on one overall pass rate; this one gates each scorer on its own.

containsScorer skips cleanly when a case has no expected value, so one shared scorer list works across every case. Its expected field takes an array of acceptable answers, not just one string, so a case can list paraphrases (["60", "sixty", "k=60"]) and pass on any of them.

export const containsScorer = (name = "contains"): Scorer =>
  async ({ output, expected }) => {
    if (expected === undefined) {
      return { name, pass: true, skipped: true, rationale: "no expected value on this case" };
    }
    const alternatives = Array.isArray(expected) ? expected : [expected];
    const lower = output.toLowerCase();
    const hit = alternatives.find((a) => lower.includes(a.toLowerCase()));
    return {
      name,
      pass: hit !== undefined,
      rationale: hit ? `matched: "${hit}"` : `no match in ${alternatives.length} alternative(s)`,
    };
  };

When you outgrow even the extended runner

At team scale the local runner runs out of room in three places: a shared UI for annotating failures, a way to line eval scores up against live production traces, and dataset versioning once the cases outgrow a TypeScript file. A hosted platform like Braintrust fills those gaps, and the eval shape underneath stays the same.

The migration pattern when you adopt one: keep the local runner for fast CI (it costs nothing, runs offline), pipe high-signal runs into the hosted platform for team review. The two coexist as different tools. Move only when team workflows demand it.

---

Concept 2: Bootstrapping the eval set

The hardest part of evals is deciding what to test. It sounds like a big project. It’s really the same move as growing a unit-test suite: start with a few cases that matter, then add one case per failure.

Step 1. Seed with real failures. Open your support tickets, your error logs, your team’s “the agent gave a weird answer” Slack screenshots. Each is a real query the agent failed on. Add 10 to 15 to start. These are high-signal because they reflect real usage rather than imagined edge cases.

Step 2. Generate adversarial questions with an LLM. Feed the LLM a description of your agent’s domain and ask it for questions that probe edge cases: ones where the answer isn’t available publicly, ones with conflicting sources, ones needing multi-step reasoning. RAGAS (an open-source toolkit of reference-free metrics for RAG systems) has built-in synthetic generation; doing it manually with one prompt works fine.

import { GoogleGenAI } from "@google/genai";
import { z } from "zod";

const ai = new GoogleGenAI({});

const TestCases = z.object({
  cases: z.array(z.object({
    question: z.string(),
    why_hard: z.string(),
    expected_behaviour: z.string(),
  })),
});

const r = await ai.models.generateContent({
  model: "gemini-3-pro-preview",
  config: {
    responseMimeType: "application/json",
    responseJsonSchema: z.toJSONSchema(TestCases),
  },
  contents: `You are a QA engineer for an AI assistant that answers questions about ${domain}.
Generate 20 challenging test questions that probe its limits.
Include: edge cases, ambiguous questions, multi-step reasoning, questions requiring refusal.`,
});

const { cases } = TestCases.parse(JSON.parse(r.text ?? "{}"));

Step 3. Add a question every time the agent fails in production. Every “the agent gave a bad answer” report becomes a test. After six months you have a 100-question suite that covers your real workload, with no weekend lost to building it.

The eval set is a backlog. It’s never done. Aim coverage at real failures, not at every theoretical capability the agent might one day have.

What to put in the starter suite

The assistant has a few different surfaces, and each deserves its own slice of the suite. These are the categories the chapter’s suite uses.

Factual recall. Questions with a stable public answer (“What is BM25?”). The right answer doesn’t change week to week.

Refusal. Questions the agent shouldn’t try to answer: unreleased product internals, future events, anything outside public knowledge.

Faithfulness. Does every claim in the answer trace back to something in the cited sources?

Memory-driven personalisation. Does turn 2 correctly use what turn 1 established about the user?

Tool-call correctness. Did the agent call the right tools in the right order, regardless of what the final answer said?

How the Chapter 10 suite scores each:

• Factual recall through grounding (layer 1). Use containsScorer with paraphrase-tolerant expected arrays. Catches grounding regressions: search returning junk, fetch erroring, prompts that drift away from citing sources.
• Refusal on out-of-scope questions (layer 1 + 2). Layer 1 checks for a refusal phrase (“I don’t have information on that”); layer 2 confirms the refusal is the right call by judging whether the question is genuinely answerable from public web sources.
• Faithfulness to fetched sources (layer 2). The headline metric for grounded answers. The judge checks whether every claim is backed by something the agent actually cited. The worst failure is a confident answer pinned to sources that don’t support it, so watch for high faithfulness scores sitting next to hallucinated URLs.
• Memory-driven personalisation (layer 1 + 2). Two-turn tests. Turn one establishes a fact (“My name is Alice, I work in marketing”). Turn two asks something that should use it.
• Tool-call correctness (layer 1). Assert the right tool was called via trajectoryScorer. Vector retriever for factual questions, calculator for math, web search for current events.

code/chapter-10/assistant-evals.ts exercises six questions covering the first three slices (factual recall, refusal, faithfulness). Memory and tool-call tests are the natural next additions when you wire the runner into your real assistant.

---

Concept 3: Keeping the judge honest

Two terms power this section. Both are simple ideas with sharp consequences when you skip them.

Rubric. The written instructions the judge reads: what to look for (the criterion), how to decide (PASS or FAIL), and a PASS example plus a FAIL example baked into the prompt. A good rubric makes the verdict reproducible, the way two graders working from the same rubric land on similar grades. A vague one makes the judge cheap to run and impossible to trust.

Calibration. The habit of re-checking the judge’s verdicts against what a human would say. Without it, you can’t tell whether a rising pass rate means the agent improved or the judge got softer.

Your first rubric is almost always wrong: too strict, too lenient, or both at once. You find out by running the suite, reading the failures, and tightening the rubric against them, then running again. That is calibration at fast tempo, over an afternoon. The same drift happens slowly over months: the pass rate creeps toward 100% and you stop learning anything from it. When it slides from 78% to 92%, you need to know whether the agent improved or the judge went soft. The only way to tell is to re-rate a small batch of recent outputs by hand and compare.

Three habits.

Spot-check it. Once a month, pull 20 of the judge’s recent scores, have a human re-rate them blind, and compare. If they diverge, sharpen the rubric. You can also automate a cheaper check: run the judge on the same (input, output) pair N times at temperature 0 and see how often the verdict flips. The automated check catches an unstable rubric; the human spot-check catches drift the judge can’t see in itself.

Use a strong model for judging. Gemini Pro judging Flash works. Flash judging Flash is biased: Flash makes the same mistakes Flash is being asked to grade.

Anchor the rubric with examples. Don’t just say “score 0-5 on faithfulness.” Show what 5 looks like and what 0 looks like. The model anchors on examples better than on definitions.

const rubric = `Score the answer on faithfulness (0-5).

Score 5 example:
Sources: "Sales grew 14% in Q2."
Answer: "Sales grew 14% in Q2."
Reasoning: Every claim is exactly in the sources.

Score 2 example:
Sources: "Sales grew 14% in Q2."
Answer: "Sales grew dramatically and revenue is at an all-time high."
Reasoning: "Dramatically" is plausible but unsupported. "All-time high" is not in the sources.

Score 0 example:
Sources: "Sales grew 14% in Q2."
Answer: "The company is going bankrupt."
Reasoning: Directly contradicts the sources.

Now score this:
Sources: {{sources}}
Answer: {{answer}}`;

---

Other things worth knowing

Cost. Layer 1 is negligible. Layer 2 is the real spend: each judge call costs roughly an agent call, multiplied by how many rubrics per case. Layer 3 is engineering attention. The cost-control trick is to split the suite into a fast tier (deterministic only) gating every commit, and a thorough tier (deterministic + judge) on release branches.

Multi-step agents. Two things matter beyond the final answer. Trajectory: did the right tools fire in the right order? (trajectoryScorer checks this.) Cost and latency: a 10-step path on a 2-step question is a regression even when the answer is right. Same three layers; you’re just scoring the path instead of the answer.

Two failure modes that look like working evals.

Suite drift toward easy. You add new questions, old questions stay forever, the agent gets better at the old ones. Pass rate climbs to 95%, 98%, 100%. The bar is too low to detect regressions. Fix: when you add a question for a new failure, retire one that’s been passing for six months. The suite stays the same size and the difficulty floor stays meaningful.

Contamination between suite and training. You publish your eval questions in a public repo. Three months later, a model provider trains on that repo. The model passes your eval because it’s seen the answers. Fix: keep your private eval set private; the public version is a smaller subset.

Both look like working evals from the outside. Keep asking what the suite is actually catching.

Three rules to make it stick.

CI-gate it. “Did this PR regress evals” should be felt while the change is still hot. Nightly batch jobs are too slow.

Treat eval debt like tech debt. When pass rate slips, fix it before shipping new features. A team that ships through a failing eval is treating the suite as decoration.

Add a question on every bug. The suite reflects the agent’s actual failure surface, grown one real bug at a time.

---

What we shipped in Chapter 10

The assistant now has a measurable quality bar. You can tell whether a change made it better or worse by reading a number, which is the whole point.

What’s in the running build:

• A 6-test eval suite with five scorers per case (code/chapter-10/assistant-evals.ts).
• The extended runner: latency / json-schema / trajectory scorers, run persistence, diff, Markdown report, gateOrExit CI gate (code/chapter-10/eval-runner.ts).
• Runs persisted in ./runs/ with the full transcripts of the last green run.

What you don’t have yet:

• A safety layer. The agent will happily comply with prompt injection (Chapter 11).
• A sandbox. If you give the agent code execution it can do anything the host can do (Chapter 12).
• Production observability or cost caps. Local evals are not the same as runtime monitoring (Chapter 14).

Each gap is the topic of the next chapter. The eval suite stays with you across all of them, gating every change you make to the assistant from here on.

A note on evals from here on. The eval suite lives in code/chapter-10/, and the chapters that follow don’t reprint it as they grow the assistant. That’s a practical choice: every new behaviour (input filter, output filter, sandbox, planner steps) would need 5 to 15 new test cases plus prose to explain them, which would crowd out each chapter’s real topic. In your own work you’d add eval cases for every change. Skip that and the suite goes stale within a sprint. Treat this chapter’s suite as the seed and grow it.

---

Next up in Chapter 11: Safety, guardrails, prompt injection. Once your agent is good, the next concern is making it safe: defending against prompt injection, sandboxing risky tools, building guards that catch bad outputs before they ship.