Skip to content
← Insights
Nov 15, 20264 min read

Keep the agent bug you can reproduce without calling the model again

Separate local regression cases, provider compatibility and model evaluations so agent changes remain reviewable.

A small test fixture card beside a modular connector, a sealed response envelope and a magnifying lens.

An agent bug is easier to fix when you can run the failing case again without paying for a new live attempt. That requires an explicit boundary between your application and the services the agent calls.

I would build that boundary before introducing another orchestration abstraction. A small interface with a recorded failing case can preserve more engineering time than a framework whose behavior nobody can reproduce.

A prompt is a versioned dependency

A prompt can look like a function: accept an input, return a structured result. The similarity is useful for organizing code. It does not make the model a pure function.

The result depends on the selected model and the surrounding instructions. External retrieval adds another changing input. A fixed temperature does not establish that a later request will produce identical output.

Keep a prompt's revision alongside its expected input and output shape. When behavior changes, that record lets you identify what changed deliberately. It also makes a rollback more specific than restoring an old text file from memory.

The application should decide what happens when the result is unusable. A schema can reject a missing field. A domain check can reject a validly shaped result that refers to an unavailable account. Give those failures distinct paths so a formatting retry does not become a way to bypass a business rule.

Capture an example that can fail locally

Suppose a hypothetical agent selects a shipping service. A provider returns a new status value, and the application mistakenly treats it as a completed booking. Save a sanitised version of that response with the expected application behavior.

A local test can then feed that response through the real decision code. It should fail on the faulty implementation and pass after the correction. The test's value comes from the defect it detects.

The pytest monkeypatch documentation illustrates replacing dependencies during tests, including blocking network access. The same principle applies in another language: substitute the external boundary while exercising the code you own.

Keep the fixture faithful to the failure. If you simplify away the unexpected status while cleaning up the example, the test will preserve the happy path and lose the bug.

Sensitive content needs separate handling. A regression fixture should retain the condition that triggered the defect without copying a customer's private document into the repository. Sometimes a synthetic input is enough; sometimes the failure requires a carefully controlled restricted fixture.

Test the adapter as well as the decision

A local fixture freezes an assumption about the provider. The provider can later change. That creates two questions: does your decision code handle the stored response, and does the current integration still speak the expected protocol?

Pact's documentation describes consumer-driven contracts that verify concrete interactions between services. That is useful when both sides can participate in the contract process. It does not magically make an arbitrary third-party API obey your fixture.

Use the testing method that fits the integration. A narrow live compatibility check may be needed for an external service you cannot verify locally. Keep it separate from the fast regression case, with appropriate test accounts and bounded effects.

Also distinguish message compatibility from correct state change. Pact's contract-testing comparison explicitly separates interaction checks from verifying side effects. Receiving the expected response shape does not prove a booking exists in the right account.

Evaluate model behavior with cases that can disagree

Some failures involve judgement: selecting an irrelevant source or answering a question that needed clarification. A single exact output snapshot can be too restrictive for those cases.

Define the behavior being assessed, then include examples where the model should take different paths. An ambiguous destination should test whether clarification occurs. An unavailable service should test whether the agent avoids claiming a booking.

Anthropic's agent evaluation guide discusses evaluating complete agent behavior and the limitations of different graders. Treat a model-based grader as another measurement component that needs calibration. A confident score is insufficient when the underlying evidence contradicts it.

Keep a held-out set when tuning the prompt. Repeatedly editing against the same handful of examples can produce a prompt that passes those examples while failing nearby cases.

The maintenance workflow then becomes concrete. A production failure yields a sanitised case. That case becomes an assertion about behavior. The correction is reviewed against the case and existing expectations. Live checks establish whether the external boundary still works.

You can change the orchestration library later. Keep the failing case that explains why the application behaves the way it does.

AI agentsTestingArchitecture