Testing Your First Production AI Agent: Evaluation and Scaling

Which Part of an AI Agent Cannot Be Tested

An agent that calls tools, hits APIs and moves through states is mostly ordinary software. Input validation, prompt assembly, tool dispatch, retry logic, state transitions, response rendering — all of this is deterministic and all of it is testable with the tools you already own. If your agent crashes because a tool argument was serialized as a string instead of a number, that is a unit test you never wrote, not an AI problem.

The part that resists testing is one function call wide: the model output itself. There is no assertion that captures “the answer is correct and appropriately scoped.” You cannot write assert response == expected against free text, and you cannot write it against a tool-calling decision either, because two different tool sequences can both be right. This output needs an eval layer: scored, thresholded, statistical. It does not live inside the test pyramid, it sits beside it, and it needs its own owner. Teams that bolt evals onto the QA backlog as “AI tests” end up with neither tests nor evals.

What an Eval Is and Where It Differs From a Test

A test returns a binary. It passes or it fails, and a failure points at a line of code. An eval returns a score across a dataset, and a decision comes from comparing that score to a threshold. That single difference changes three things.

CI behavior. A failed test blocks the merge and the author fixes it. A dropped eval score needs interpretation: did quality regress, or did the sample move? You need a policy before you need a pipeline.

Failure reporting. A test failure is a stack trace. An eval failure is a list of examples with their scores, and someone has to read them.

Ownership. An eval has three components, and they rarely belong to the same person:

• The dataset — inputs plus expected behavior. Owned by whoever understands the product domain: PM, support lead, domain expert. QA curates, but does not invent.
• The scorer — the code or rubric that turns an output into a number. Owned by engineering, usually QA/SDET.
• The threshold — what score is acceptable, and what happens below it. Owned by whoever is accountable for shipping. This is a product decision wearing an engineering costume.

If no one owns the dataset, it rots in a month.

Splitting the Agent Architecture Into Testable Layers

Four layers. Three of them are normal software.

Layer 1 is pure functions: schema validation, truncation, context assembly, template rendering. Snapshot-test the assembled prompt — prompt changes are code changes and should show up in a diff.

Layer 2 is the model call. Mock it everywhere except in evals. Your integration tests should not be paying per token; see mocking and stubbing in test automation (in Turkish) for the boundaries.

Layer 3 is where real damage happens. A tool that deletes a record or charges a card must be covered by contract tests against the real API schema and integration tests with a fixture backend. Argument validation belongs here, not in the prompt — never rely on the model to stay inside an allowed range when a server-side check can enforce it. Related: why and how to automate API testing (in Turkish).

Layer 4 is parsing, citation rendering, streaming, error states. Deterministic, and the easiest place to ship bugs because people assume the model is at fault.

Everything except layer 2 can and must be covered by deterministic tests. Evals are expensive; do not spend them on bugs a unit test would have caught.

Building the Golden Dataset: Start With 30 Examples

The dataset comes from production logs, not from imagination. Hand-written examples encode what the team thinks users do, and the gap between that and reality is where your defects live. If the feature has not shipped yet, use support tickets, sales call notes, or an internal dogfooding period — anything with real phrasing in it.

Collect from three buckets:

1. Typical usage — the ten most common intents, in the user’s own words, including the typos.
2. Edge cases — empty input, very long input, ambiguous requests, requests outside scope, hostile prompts, multi-turn context.
3. Known production failures — every incident becomes an example. This is a regression suite, and it is the highest-value bucket.

Thirty well-chosen examples beat five hundred random ones, because for thirty examples a human can actually verify the expected behavior. A dataset whose expectations nobody checked measures nothing; it just produces a number that moves.

Two operational rules. Version the dataset in git next to the code, as JSONL, with a version field — when a score moves you need to know whether the model changed or the dataset did. And strip PII at ingestion, not later: replace names, emails, account numbers and free-text addresses with stable placeholders before the example ever lands in the repo. The same discipline that keeps test data isolated applies here; each test producing its own data (in Turkish) is the same argument in a different costume.

Choosing a Scorer: Deterministic, Model-Based and Human

Try deterministic checks first, and try harder than feels necessary. A surprising share of what people call “quality” is mechanical:

• JSON schema conformance of structured output
• required fields present and non-empty
• banned content: no invented URLs, no disclaimers you removed from the product, no PII echo
• tool-call correctness: was the right tool called, with arguments inside the allowed set
• grounding: does every cited document ID exist in the retrieved context
• exact or fuzzy match against a reference answer where one legitimately exists

These cost nothing per run, give the same answer every time, and fail in a way an engineer can debug.

Use an LLM-as-judge only where deterministic checks genuinely cannot reach: tone, helpfulness, faithfulness to a source document, whether the answer addressed the actual question. A judge is a measuring instrument, and an uncalibrated instrument is decoration.

Human evaluation stays in the loop by sampling, not by volume. Twenty to fifty outputs reviewed before a release, scored against the same rubric the judge uses, is enough to tell you whether your automated scores still mean anything.

Wiring the Deterministic Scorer Into CI

No special infrastructure is needed. A dataset file, a scorer module and pytest will do. The runner below reads JSONL, runs the agent, scores each example, writes a JSON artifact, and asserts against a threshold.

# evals/test_agent_eval.py
import json
import os
import pathlib
from dataclasses import dataclass, asdict

import pytest

from app.agent import run_agent  # returns {"text": str, "tool_calls": [{"name","args"}]}

DATA = pathlib.Path(__file__).parent / "golden.jsonl"
ARTIFACT = pathlib.Path(os.getenv("EVAL_ARTIFACT", "eval-results.json"))
THRESHOLD = float(os.getenv("EVAL_THRESHOLD", "0.85"))
REPEATS = int(os.getenv("EVAL_REPEATS", "3"))

BANNED = ("as an ai language model", "i cannot help with that")


@dataclass
class Result:
    id: str
    scores: list
    mean: float
    notes: list


def load_dataset(path):
    rows = []
    with path.open(encoding="utf-8") as fh:
        for line in fh:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def score_output(example, output):
    """Deterministic scorer. Returns (score in [0,1], list of failure notes)."""
    checks, notes = [], []

    expected_tool = example.get("expected_tool")
    called = [c["name"] for c in output.get("tool_calls", [])]
    if expected_tool is None:
        ok = len(called) == 0
        checks.append(ok)
        if not ok:
            notes.append(f"unexpected tool calls: {called}")
    else:
        ok = expected_tool in called
        checks.append(ok)
        if not ok:
            notes.append(f"missing tool {expected_tool}, got {called}")

    for key, value in example.get("expected_args", {}).items():
        actual = next(
            (c["args"].get(key) for c in output.get("tool_calls", []) if key in c["args"]),
            None,
        )
        ok = actual == value
        checks.append(ok)
        if not ok:
            notes.append(f"arg {key}: expected {value!r}, got {actual!r}")

    text = output.get("text", "").lower()
    for phrase in example.get("must_contain", []):
        ok = phrase.lower() in text
        checks.append(ok)
        if not ok:
            notes.append(f"missing phrase: {phrase!r}")

    ok = not any(b in text for b in BANNED)
    checks.append(ok)
    if not ok:
        notes.append("banned phrase present")

    score = sum(1 for c in checks if c) / len(checks) if checks else 0.0
    return score, notes


@pytest.fixture(scope="session")
def results():
    collected = []
    for example in load_dataset(DATA):
        scores, notes = [], []
        for _ in range(REPEATS):
            output = run_agent(example["input"], context=example.get("context"))
            score, run_notes = score_output(example, output)
            scores.append(score)
            notes.extend(run_notes)
        collected.append(
            Result(
                id=example["id"],
                scores=scores,
                mean=sum(scores) / len(scores),
                notes=sorted(set(notes)),
            )
        )
    payload = {
        "dataset": DATA.name,
        "repeats": REPEATS,
        "threshold": THRESHOLD,
        "overall": sum(r.mean for r in collected) / len(collected),
        "results": [asdict(r) for r in collected],
    }
    ARTIFACT.write_text(json.dumps(payload, indent=2), encoding="utf-8")
    return payload


def test_overall_score_above_threshold(results):
    assert results["overall"] >= THRESHOLD, (
        f"overall {results['overall']:.3f} < threshold {THRESHOLD}. "
        f"See {ARTIFACT}."
    )


def test_no_example_collapses(results):
    broken = [r["id"] for r in results["results"] if r["mean"] < 0.5]
    assert not broken, f"examples below 0.5: {broken}"

A golden.jsonl line looks like this:

{"id": "refund-001", "input": "I want my money back for order 4471", "expected_tool": "lookup_order", "expected_args": {"order_id": "4471"}, "must_contain": ["refund"], "version": 3}

Upload eval-results.json as a CI artifact. If your stack is JavaScript, the same structure maps onto the Playwright test runner without changes.

Calibrating a Model-Based Scorer

A judge is only useful if you have measured its agreement with humans. The procedure is fixed:

1. Write a rubric with explicit levels. Not “rate helpfulness 1–5” but “1 = does not answer the question; 3 = answers but omits a required constraint; 5 = answers and respects every constraint,” with one example per level.
2. Have humans label 50 outputs against that rubric. Two labelers on at least a subset, so you know how much humans disagree with each other — that number is your ceiling.
3. Run the judge on the same 50 and compute agreement. Exact-match rate for categorical rubrics; for ordinal scales, report how often the judge lands within one level.
4. Below roughly 85% agreement, fix the rubric — not the judge model. Low agreement almost always means the rubric is ambiguous, and an ambiguous rubric means the humans disagreed too.

Typical rubric repairs: splitting one criterion that was secretly two, replacing adjectives with observable conditions, forcing the judge to quote the span it is judging, and requiring a reason before the score rather than after.

The judge prompt is a versioned artifact. It lives in git, it has a version number, and changing it invalidates every baseline produced under the old version — exactly like changing the dataset.

Thresholds, Flaky Evals and the Regression Gate

Model output is stochastic, so a single low run is not a regression. Run each example n times — three is a workable floor — and track the distribution rather than the mean alone. An example that scores 1.0, 1.0, 0.0 is a different problem from one that scores 0.7 every time: the first is instability, the second is a capability gap. Report both the mean and the pass rate per example.

temperature=0 and a fixed seed reduce variance but do not eliminate it. Sampling is only one source: provider-side batching, floating-point non-determinism on GPU kernels, silent model point-releases behind a stable alias, and retrieval ranking that shifts when the index updates all move the output. Treat determinism as a reduction, not a guarantee. The diagnostic discipline is the same one you use for flaky UI tests (in Turkish): separate instability from genuine failure before you touch the code.

Gate on delta, not on an absolute number. Store a baseline per dataset version and model version, and block when the new score drops more than X points below it — X chosen from the observed run-to-run spread, not from a round number you liked. A good starting policy: warn below baseline, block below baseline minus two standard deviations of your measured variance, and always block when a previously-passing example in the known-failures bucket regresses.

Evaluation in Production: Offline Eval Is Not Enough

An offline eval sees exactly as much as its dataset covers. Users will find behavior you never sampled, usually within a week of launch.

Three production signals close the gap:

• Trace collection. Log the full chain for every request: input, assembled prompt, model version, tool calls and results, final output, latency, token counts. Without traces you cannot reconstruct a failure, and a failure you cannot reconstruct cannot become a dataset example.
• Online scoring by sampling. Run your deterministic scorer on 100% of traces — it is cheap — and the judge on a small random sample plus a targeted slice (new users, long conversations, tool-call failures).
• User feedback. Thumbs, regenerate clicks, abandonment after a response, support tickets referencing the feature. Weak signals individually, useful in aggregate as a filter for which traces to look at.

The loop is the point: a low-scoring production trace becomes a golden dataset example a week later, and that example blocks the regression from coming back.

Cost and Runtime: How Often to Run Evals

Running the full eval on every PR is slow and expensive, and the slowness is worse than the cost — engineers route around a gate that adds twenty minutes.

Layer it:

• On every PR: the deterministic fast set, 50–100 examples, no judge, minutes. Mostly tool-call correctness and schema checks.
• Nightly: the full dataset with repeats and the judge enabled. Baseline updates happen here.
• Before a release: human sampling against the rubric, plus a judge-vs-human agreement spot check.

Budget tokens explicitly. Cost is roughly dataset size × repeats × (agent tokens + judge tokens) × runs per month, and the judge is often a larger share than the agent because it reads both the input and the output. Compute that number before you build the pipeline, not after the first invoice.

Parallelism fights rate limits. Naive pytest -n 32 against a provider quota produces 429s that look like eval failures, and retries inflate cost silently. Use a bounded worker pool sized to your quota, exponential backoff with jitter, and — critically — count retried requests separately so a rate-limit storm does not get recorded as a quality drop.

What Breaks as You Scale

Going from one agent to a fleet surfaces four problems, each with a specific countermeasure.

Dataset maintenance debt. Each agent gets its own dataset; nobody prunes them. Expectations drift from current product behavior and the suite starts failing for reasons unrelated to quality. Countermeasure: a quarterly review where every example is re-confirmed or deleted, and a staleness field on each example.

Judge model version changes. A provider upgrades the model behind your judge alias and every baseline across every agent shifts at once. Countermeasure: pin judge models to explicit versions, never aliases; treat a judge upgrade as a planned migration with a re-calibration run and a re-baselining of all datasets.

Rate limit queueing. Ten nightly eval jobs sharing one quota turn into a six-hour serialized run. Countermeasure: a central scheduler with per-team quota allocation, staggered start times, and separate API keys for eval traffic so it cannot starve production.

Nobody reads the results. Eval output lands in an artifact no one opens. Countermeasure: name an owner per dataset, post the delta — not the absolute score — into the team channel, and escalate only on threshold breaches. Scores that nobody acts on are worse than no scores, because they create the appearance of coverage.

What to Do This Week

In order, and do not skip ahead:

1. Pull 30 examples from production logs — 10 typical, 10 edge cases, 10 known failures. Strip PII. Commit as golden.jsonl with a version field.
2. Write a deterministic scorer covering tool-call correctness, schema conformance and banned content. Nothing else yet.
3. Wire it into CI as a nightly job, not a blocking gate. Write the results to an artifact.
4. Record a baseline after three consecutive clean nights, and keep it in the repo alongside the model version.
5. Only then consider a judge, and only for the criteria your deterministic scorer provably cannot reach — with 50 human labels to calibrate it against before it influences any decision.

Do not start with LLM-as-judge. An uncalibrated judge gives you a number that feels like quality and tracks nothing, and it takes months to notice. Start with the checks that are exactly right and narrow, then widen.