Skip to content
← Insights
Nov 15, 20264 min read

Trace the agent's tool boundary without logging the user's draft

Connect agent traces across tool calls, keep business counts separate from retries, and collect less sensitive data.

A magnifying glass over a connected timeline of tool-call blocks, beside a filter funnel and a small shield.

An agent request crosses a model provider, an MCP client and a tool server. The user sees a spinner. A single duration metric tells you how long the whole request took. It gives you very little help finding the part that consumed the time.

I would start tracing at those boundaries. Each owns a distinct operation, so each can report its duration without adding analytics code to every business function. The resulting trace should explain a slow request while keeping the user's draft out of the telemetry system.

Follow one operation across the boundary

Consider a hypothetical agent retrieving a document before drafting an answer. Give the user request a trace. Within it, record the retrieval call, including the tool name and a bounded result classification. The server can record the work needed to fulfil that call.

OpenTelemetry's context propagation documentation explains how trace context connects work across process boundaries. The receiving service needs to extract that context. Merely generating a new identifier at every hop produces several unrelated timelines.

Propagation also needs an explicit trust boundary. An external tool service may have no reason to receive your internal account identifier. Decide which context can leave your system, then test the actual outgoing request. A convenient interceptor can otherwise distribute more information than its author intended.

Keep the instrumentation wrapper narrow. It can observe the start of a call, record its completion and classify an error. Business rules stay in the implementation that owns them. If a telemetry backend changes, the invoice calculation should remain untouched.

A trace can explain a retry without counting another sale

Suppose the tool call times out. The agent retries and eventually receives a successful response. That history belongs in observability: two attempts, their durations and the relationship between them.

Product reporting needs a separate definition of success. A completed tool call might mean a preview was generated. It does not establish that a customer accepted it. Record the business outcome at the component that can establish that outcome, with a stable operation identifier where deduplication is required.

This prevents a common measurement error. Counting every successful invocation as a completed customer task rewards retries. Moving the same event to both the client and server introduces another source of duplicate counts. Write down who owns the event before building a chart around it.

Correlate the business event with its trace when permitted. That gives an investigator a route from an observed outcome to the technical request that produced it. The two records can still have different retention periods and access rules.

Sampling changes what a chart can claim

Retaining every trace may become expensive. OpenTelemetry describes sampling choices and their costs, including the engineering work needed to operate them. There is no sampling percentage that fits every agent workload.

A sample designed to retain failures is useful for incident investigation. Its failure share will differ from the failure share of all requests. A chart built directly from those retained traces needs to account for that selection.

Likewise, a count of exported spans is not automatically the count of completed business tasks. Use a measurement path designed for that count. If estimates from sampled data are acceptable, document the estimator and its assumptions. Otherwise keep the required outcome records through a separate path.

Measure the telemetry itself during a small rollout. Check export failures and dropped records. A beautifully named span that never reaches storage cannot explain yesterday's incident.

Hashing an identifier leaves a linkable identifier

Prompt bodies can contain confidential documents or personal information. Tool arguments can contain credentials. Copying either into a span turns a performance investigation into another data handling problem.

OpenTelemetry's sensitive data guidance discusses reducing or transforming collected data. Its warnings about predictable inputs are relevant to hashing: someone can hash a list of likely email addresses and compare the results. Hashing alone does not establish anonymity.

Prefer a minimal attribute allowlist. A tool name and an outcome category will often answer the first operational question. When content capture is needed for a specific investigation, give it a defined access policy and expiry.

Also inspect baggage propagation. Baggage can travel downstream; it is a poor place to put a secret simply because it is convenient to attach one there.

Finally, keep a versioned mapping between your internal operation names and the conventions used by your instrumentation. OpenTelemetry's GenAI documentation now points to its separate conventions repository. Check the maturity of the definitions you adopt. A local span called agent.step is your naming decision unless you have deliberately mapped it to a published convention.

Start the review with one failed request. Can you follow it through the tool boundary, identify the failed attempt and find the eventual business outcome? Then inspect the stored attributes. That second check tells you whether the explanation cost more user data than it needed.

AI agentsObservabilityPrivacy