Application telemetry can show that a request failed. Agent telemetry also needs to show which task was active, what operations occurred, which policy checks ran, and what changed in external systems.
It cannot prove the model's true reason for choosing an action. A model-generated explanation may help an operator, but it is not a guaranteed account of the factors that caused the decision.
Agent observability should reconstruct the sequence of operations and confirmed effects. It should do this without copying every prompt, document, or reasoning trace into a second data store.
Use a run as the unit of work
A run begins with a user request, schedule, webhook, or resumed checkpoint. It may include context retrieval, model calls, policy checks, tool calls, approvals, retries, and child work before it ends.
Give each run a stable opaque identifier. Record the agent, harness, model, tool, and environment versions. Associate the run with an authorized tenant or user reference only when needed. Do not place raw personal information in identifiers.
Within the run, spans can represent context assembly, model calls, policy checks, tool execution, persistence, and verification. Events can record changes such as created, running, waiting for approval, completed, cancelled, and failed.
Keep a durable run-state record separate from the conversation. This record can provide the application's status for the run. The external provider or transactional system remains the authority for an external effect.
Record proposals and effects as separate operations. A model tool request is a proposal. Policy approval is another event. Tool execution is another. Provider confirmation is another. A timeout may create an unknown outcome rather than a confirmed failure.
This separation helps operators identify whether a problem came from model selection, policy, tool execution, or missing confirmation. It also shows where the run spent time.
Connect work across services and processes
Distributed tracing uses trace and span context to connect operations across service boundaries. Pass this context through queues, tool gateways, sandboxes, and background workers where the trust policy permits it.
Asynchronous work does not always fit one parent-child span tree. OpenTelemetry span links can connect a child task or later process to the run that caused it. This is useful for queued work, retries, batch operations, and agents that create independent child runs.
Keep high-cardinality values such as run IDs, user references, and operation IDs in traces or logs. Do not use them as metric labels. Metrics need low-cardinality dimensions so storage and query costs remain controlled.
OpenTelemetry baggage propagates key-value context in request headers. It has no built-in integrity check and may travel to downstream or third-party services. Do not place credentials, personal data, or sensitive tenant information in baggage. Use opaque references and allow only fields required for propagation.
Instrumentation failure should not block the core agent operation. Buffer or drop telemetry according to policy when the collector is unavailable. Monitor the collector and exporter so operators can detect dropped spans rather than assuming the trace is complete.
Record telemetry health separately from agent health. Useful signals include exporter errors, queue depth, dropped span count, and collector availability. A run view should indicate when its trace may be incomplete.
Record metadata before raw content
Prompts and tool results can contain customer data, credentials, proprietary files, or content from untrusted sources. Logging complete payloads by default creates another copy with a new access and retention policy.
Start with metadata that supports diagnosis. Record component versions, source document IDs, tool names, argument shapes, and policy outcomes. Add token counts, latency, error categories, artifact hashes, and external operation IDs when needed.
References should point to the governed source system. Store a version or hash so an operator can identify the exact input used by the run. If that source can change and the investigation requires the historical content, retain a governed snapshot only when the data policy permits it.
The OpenTelemetry generative AI semantic conventions mark message content and system instructions as opt-in. The specification warns that these fields may contain sensitive information. Teams should also note that the agent semantic conventions remain in development and may change.
When raw content capture is necessary, define its purpose, access policy, retention period, and redaction rules. Enable detailed capture for a specific incident or controlled sample instead of every run. Record who enabled it and when it expires.
Reasoning traces and model explanations are not guaranteed to reveal all factors that influenced an answer. Use concise decision summaries as debugging aids, not as proof. The stronger evidence is the context supplied, request emitted, policy result, tool observation, and state change.
Logs, traces, and metrics answer different questions
Logs record events such as a policy denial, tool error, or state transition. Traces connect operations within a run. Metrics summarize behavior across many runs.
A trace can show that one run repeated a search. A metric can show that repeated searches increased after a release. A log can provide the error category. Use the three signals together.
Useful metrics include task completion, escalation, policy denial, tool errors, retries, duplicate prevention, latency, model turns, tool calls, tokens, and cost. Segment them by low-cardinality fields such as agent version, model family, task class, tool version, and account tier where policy permits.
Report distributions rather than averages alone. Track latency and cost percentiles, maximum turns, and repeated calls. A release can improve the median while creating a small group of long or expensive runs.
Connect system metrics to product outcomes with care. A lower escalation rate may mean better performance or less appropriate caution. A completed run may still require user correction. Review acceptance, correction, reversal, and downstream results when those measures fit the task.
Sample traces according to risk
Most systems cannot retain every detailed trace at full volume. Sampling decides which traces remain available.
Head sampling makes the decision when a trace begins. It is efficient, but it cannot know the final outcome. A low head-sampling rate may miss rare failures.
Tail sampling decides after enough of the trace is available. It can retain runs with policy denials, unknown external effects, severe errors, high latency, or repeated calls. It requires buffering and more collector capacity.
A practical policy can combine both. Keep a representative base sample and retain all traces that match high-risk outcomes. Check that sampling works across task classes and account tiers without collecting content beyond the applicable policy.
Sampling does not replace aggregate metrics. Metrics should show how often events occur even when individual traces are not retained. Record sampling configuration with releases so changes in trace volume do not appear to be changes in product behavior.
Design the operator view around confirmed facts
Start a run view with the goal, status, duration, cost, agent version, acting reference, and confirmed effects. Present a chronological sequence below it. Each consequential operation should show its input reference, policy result, execution status, and observed outcome.
Collapse repetitive low-level spans until an operator needs them. Link source identifiers to authorized views in their governing systems. Do not copy entire records into the trace interface for convenience.
Make version comparison available. Operators should be able to compare representative runs across model, prompt, tool, and harness changes. Release annotations help connect a metric change to a deployed component.
Record the incident resolution. Useful labels include corrected, retried, escalated, expected behavior, product defect, model error, tool error, and policy error. These labels support regression tests and show whether an alert produced useful action.
Alerts should identify an owner and link to representative evidence. Alert on service objectives and consequential failure modes, not every change in token count. Examples include unknown write outcomes, duplicate effects, cross-tenant policy denials, or a sustained drop in task success.
Keep operational telemetry separate from other data uses
Operational telemetry supports reliability, security, debugging, and evaluation. It does not automatically authorize model training or permanent evaluation storage.
Each use needs a defined policy. Operational traces may have short retention. Aggregate metrics may require no raw content. Evaluation sets can use reviewed and minimized examples. Model-improvement data must follow the product's consent, contract, retention, and account-tier rules.
Apply tenant scope through collection, queues, storage, dashboards, exports, and support tools. Redact sensitive fields before they enter the telemetry pipeline when possible. Test that disabled capture remains disabled. Check every schema change for new raw payload fields.
Verify deletion and retention jobs. Test that secrets do not enter traces. Keep raw identities out of metric labels and baggage. Operator links must enforce the same access rules as the source system.
Agent observability records the operations and confirmed effects of a model-directed run. Good instrumentation separates proposals from execution, uses governed references instead of raw content, and preserves serious failures through sampling. It helps operators diagnose the system without claiming access to the model's true reasoning.