Eval, testing, and debugging
One subdomain, 2.6% of the marks — a glance rather than a study session, per the official weighting. But the content is the debugging kit you will use on every other domain: what an API error is telling you, which failures retry, and how to tell a rate limit from a spend cap.
Every Claude API error arrives as JSON with the same anatomy: an error object carrying a machine-readable type, a human-readable message, and a request identifier you should log for support. That anatomy is also your inspection list, in order: the HTTP status, the error type, any subtype or error code inside the details (where enforced_spend_limit_reached and its kin live), the retry-after header when one is present — and, for the support ticket, the request identifier. The discipline the exam tests sits on top of it: branch on the type — or on the SDK’s typed exception classes, caught most-specific first — never on the message text. Messages are prose, and prose changes. Go is the one-language exception, with a single error type you switch on by status code.
01Two families of failure
Sort every status code into two buckets. Transient means the request was fine and conditions were bad — wait and retry. Request-change means the request itself is wrong — the identical resubmission fails identically, so fix the cause instead. The whole taxonomy in one screen:
| Status | Type | Family | Decision and fix |
|---|---|---|---|
| 429 | rate_limit_error | Transient — usually | Back off and honour retry-after. But confirm which of the three 429 conditions you are holding before trusting a retry (next section). |
| 500 | api_error | Transient | Retry with exponential backoff; open a support ticket with the request ID if it persists. |
| 504 | timeout_error | Transient family — but retry is not the fix | The call outran its window. The documented fix is switching to streaming or the Batches API, not retrying harder. |
| 529 | overloaded_error | Transient | Capacity on Anthropic’s side, not your request. Wait and retry. |
| 409 | conflict_error | Request-change with a retry-shaped recovery | Two writers collided. Resolve the conflict, then resubmit — the one 4xx you retry. |
| 400 | invalid_request_error | Request-change | Fix the malformed request — unless a spend limit you set yourself was hit, in which case raise or remove the limit. |
| 401–404 | authentication / billing / permission / not_found | Request-change | Credentials, billing, scope or wrong path. Fix; do not retry. |
| 413 | request_too_large | Request-change | Payload over the size cap — 32 MB for messages and token counting, 256 MB for batches, 500 MB for files. It must shrink; retrying cannot help. |
The row that blurs the buckets deserves its own paragraph, because the exam knows it blurs. Run the two-bucket test literally: would the identical resubmission fail identically? For a 409, yes — while the conflict stands, resending the same write fails again, which is request-change behaviour. What makes 409 unique among the 4xx is the recovery: once the collision is resolved, resubmitting the same logical operation is the documented path. Request-change in mechanism, retry-shaped in recovery — so "resolve, then resubmit" only counts as a retry in the loose sense, and that is the answer shape to reach for.
02What the SDK already does for you
The official SDKs retry transient failures themselves — connection errors, rate limits, 5xx — with exponential backoff, honouring any retry-after header, twice by default. Your error handler fires only after those attempts are spent. Two consequences follow. Your own retry loop layered on top doubles the waiting, unless you disable one side — max_retries on the client is the knob; turn it off when an outer job queue already retries. And "I saw the exception" means the SDK already retried, not that it failed once.
03Three things that look like a rate limit
The 429 family is where this domain earns its exam marks, because three different conditions wear the same clothes — and the wrong answer treats them as one. Side by side:
| True rate-limit 429 | Tier spend-cap 429 | Self-set limit — a 400 | |
|---|---|---|---|
| What tripped | One of three limits per model class: requests, input or output tokens per minute | The usage tier hit its spend ceiling | The org limit you set yourself |
| The giveaway | retry-after present; the error names the limit you tripped | No retry-after; enforced_spend_limit_reached in the details | Administrative — check the limits you set |
| Does retrying help | Yes — back off and honour retry-after | No — not until access resumes | No — raise or remove the limit |
| Accounting subtleties | Input limit counts uncached tokens only (most models); output limit counts tokens generated, not your max_tokens setting | — | — |
The third column is the trap the table exists to expose: the self-set organisation limit is not a rate limit at all — it returns a 400 invalid_request_error, and the fix is administrative. One carve-out runs the other way: Claude Code workspace limits are checked separately and do come back as a 429, with a retry-after header.
And the organisational fact that kills a whole class of wrong answers: rate limits attach to the organisation, not to the key. Rotating keys to "spread the load" does nothing — and is occasionally the exact wrong option the exam offers you.
04The eval habit
The domain’s name promises more than its single subdomain delivers — but evaluation does thread through this exam from other domains, so hold the pattern. A change to model, prompt or architecture is only judged by an eval built from your real prompts and data, with a pass threshold. The usage object on every response (input tokens, output tokens, cache reads and writes) is the ground truth for both billing and debugging. And when behaviour drifts mid-project, the first question is what changed in the context — not who to blame. Small domain, transferable kit.
05The traps in this domain
- —"Every 429 is transient" — the spend-cap 429 has no retry-after and no recovery by retry.
- —"413, just retry it" — the payload is over the cap; it must shrink.
- —"Rotating keys dodges rate limits" — limits are organisational.
- —"Parse the error message to classify it" — branch on the type or the exception class.
- —"Retry the 504 harder" — the call outran its window; change the request’s shape, not the retry schedule.
- —"409 means the request was malformed" — the request was fine; two writers collided. Resolve, then resubmit.