Evaluation, testing and optimization

Everything in this domain reduces to one professional reflex: no claim about the system without an evaluation behind it. “It feels better” is not evidence; “it scores 91% on the 500-case golden set graded by two humans, up from 84%, at equal p95 latency and 60% of the cost per task” is. The exam’s scenarios are full of teams making decisions on feelings and asking you to name what would have settled the question — and the answer is almost always a metric, a dataset, or a controlled comparison that nobody built.

01Defining the metrics

The exam guide names five families, and the professional point is that they trade against each other: accuracy, latency, cost, safety, security. You cannot maximise all five, and every real design is a position among them — which is why “define the metric” comes before “build the system”: the metric is the specification. Accuracy for a generator is graded correctness against criteria (factuality to the source, completeness of extraction, policy compliance of a decision) — defined per task, never as a single global percentage. Latency is a distribution: p95 and p99 answer what users experience; the mean hides exactly the tail that generates complaints. Cost per task — not cost per token — is the business-meaningful unit, and it moves with model tier, caching, batch and retrieval depth. Safety metrics count the harms the product cannot afford (unsafe outputs per thousand, injection-resistance pass rate); security metrics cover what the system leaks or permits (sensitive-data disclosure rate, authorisation violations caught in test). Two families are guardrails: a design that improves accuracy while doubling unsafe-output rate has not improved.

02Datasets and mixed methodologies

An evaluation is only as honest as its dataset, and the exam’s recurring failure is the eval built from the prompts the team happened to write — easy, clean, unrepresentative. The professional dataset is drawn from real traffic (or a faithful simulation of it): the messy phrasings, the edge cases, the ambiguous inputs, the adversarial ones — with the distribution roughly matching production. Around it: a golden set with agreed correct answers or graded criteria, sized to the stakes (hundreds for direction, thousands for a pass/fail gate); held-out test data the team never tunes against; and adversarial cases planted deliberately — injection attempts, out-of-scope requests, conflicting instructions — because the traffic that hurts you is not average traffic.

“Mixed methodologies” is the objective’s key phrase: no single grading method scales and stays honest, so professionals combine three. Code-based checks — deterministic and free: schema validity, exact and fuzzy matches, unit-test assertions, length and format constraints, reference presence for citations. Human grading — the gold standard and the bottleneck: rubrics with criteria a second grader can apply, a double-graded sample to measure agreement, and human hours spent where disagreement lives. LLM-as-judge — scalable grading by a model against your rubric, and the exam wants its discipline: calibrate the judge against a human-graded subset before trusting it, check its consistency on re-runs, and keep it away from the cases where it is weakest (very long outputs, subtle domain judgment, its own failure modes). The division of labour that earns marks: code checks everything cheaply, humans grade the sample that defines truth, the judge scales that truth to the whole set — and the whole suite runs on every change, in CI, as a regression gate.

03A/B testing and iterative improvement

When the system is live, the eval question becomes causal: did the change help? The discipline the exam expects: one variable at a time (prompt change or model change or retrieval change — not all three, or you have learned nothing reusable); a control and treatment split on comparable traffic, randomised, run long enough that noise cannot explain the result (small effects on low volume need weeks, not a Tuesday afternoon); success measured on the metric the change targeted, plus guardrails — latency, cost, safety — that must not regress; and a shadow deployment option where the treatment runs silently alongside production with no user exposure, for changes too risky to A/B directly. Iteration is the loop that uses all of it: change, evaluate offline, A/B online, promote or roll back, and fold what you learned into the golden set so the next change is judged against a harder exam.

04Diagnosis: prompt failure, hallucination, model mismatch

The objective names three failure classes, and diagnosis-by-symptom is a directly examinable skill. The symptom table:

One professional habit binds the table together: keep a failure ledger. Every diagnosed incident becomes a golden-set case and a regression test — the system’s worst day becomes its permanent exam question, and the same failure cannot quietly return.

05Optimization: tokens, latency, cost-performance

The levers are the ones Domains 1–3 introduced; here they are managed as a portfolio against the metrics. Prompt caching on stable prefixes (reads at about a tenth of input cost) — the first lever, always. Batch for asynchronous work (half price, results within 24 hours) — anything that can wait overnight should. Model routing by difficulty (small tier on the easy majority, escalation on the hard tail). Retrieval depth tuned against the eval — every chunk beyond the useful k is latency and dilution. Output discipline: cap max_tokens where runaway output is a known mode, and stop sequences where the format has a natural end. Prompt bloat removed: the cheapest token is the one never sent. And thinking budgets set per lane — full reasoning where answers must follow from steps, none where a schema would do. The justification pattern the exam rewards: name the constrained metric, name the lever, state the measured trade (“cache hit rate 78% took cost per task from $0.09 to $0.04 with no accuracy movement on the golden set”).

06Monitoring: the eval that never stops

Production monitoring is the same domain applied to live traffic, and it leans on the observability telemetry from Domain 3. The standing measurements: quality on a live sample (the golden set was static; traffic is not — sample real outputs, grade them with the same mixed methodology, watch the trend); input distribution drift (new question shapes, new languages, traffic-mix shifts that quietly invalidate your dataset); output and cost drift (cost per task creeping as prompts accrete; latency p95 creeping as retrieval grows); and alert thresholds that page on user-felt symptoms. The loop closes when monitoring feeds evaluation: drift detected → cases added to the dataset → diagnosis run → fix shipped → A/B verified. A system whose evaluation stopped at launch is not being operated; it is being trusted — and the exam’s scenarios are what that trust costs.

07The traps

• —“The demo worked.” Demos sample the friendly cases; the eval samples the real ones. Every scenario where a launch decision rests on a demo is this trap.
• —“Ninety-two percent accurate.” Percentages without a dataset, a grader and a rubric are marketing; the exam will ask what the number is measured on.
• —“The LLM judge said so.” Judges are calibrated against human-graded subsets or they are noise — and they inherit blind spots from their own model family.
• —“Average latency is fine.” The mean is fine while the p95 is on fire; interactive systems live in the tail.
• —“Cheaper model, same results — ship it.” Same results on what set? Routing changes are eval-gated like any other change, with the hard tail specifically examined.
• —“We fixed it.” Fixed means: the case is in the golden set, the regression suite passes, and the fix is documented. Otherwise it is parked, not fixed.