All chapters
  1. Start hereOne agent asks. Another answers. The rest is the contract between them
  2. Foundations · 1One model, one tool, two agents: who is who, and who is responsible
  3. Foundations · 2Message passing: sender, recipient, payload, address and correlation
  4. Foundations · 3Topologies: central orchestrator, peer-to-peer and hierarchy
  5. Mechanics · 4Request and reply, or a task: synchronous versus asynchronous exchange
  6. Mechanics · 5Contracts: messages, artifacts, typed schemas, errors and versions
  7. Mechanics · 6The task lifecycle: submitted, working, input-required, completed, failed, canceled
  8. Mechanics · 7Discovery: Agent Cards, skills and how an agent says what it can do
  9. Protocol · 8A2A: the Agent2Agent protocol, part by part
  10. Protocol · 9A2A versus MCP: delegating to an agent versus calling a tool
  11. Protocol · 10Handoff, delegation or shared state: three ways to work together
  12. Production · 11How updates arrive: streaming, push notifications, callbacks, polling and queues
  13. Production · 12Security: identity, delegated scopes, trust across domains, authorization
  14. Production · 13Reliability: retries, idempotency, order, timeouts and compensation
  15. Production · 14Observability: one trace across three agents
  16. Reference · 15Frameworks: A2A SDKs, LangGraph, AutoGen, CrewAI and plain code
  17. Reference · 16Advanced: consensus claims, coordination failure, cost and isolation
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Production · 14

Observability: one trace across three agents

Each agent logs its own part. Trace context that travels inside the message joins the parts into one causal timeline, so a late answer has an owner.

What you will understand
Three logs are not a story
Each agent records its own part. Without a shared trace, nothing says which part caused which.
Context in the message
Trace propagation: the ids that travel inside each message and join three agents into one timeline.
Five ids, five questions
What the run, task, message, correlation and trace ids let you answer when a case goes wrong.

Case UW-406 produced a recommendation of request_info: the roof assessment is missing. The broker is unhappy, because his building's roof data has been available for months. The underwriter asks the operations team a fair question: why did this take five seconds, and why was the roof assessment missing?

Each agent's log has an answer to part of it. The intake agent's log says it sent a workflow request at 50 ms and received the result at 5,106 ms. The coordinator's log says it asked for evidence at 70 ms and received a partial artifact at 5,000 ms. The assessment agent's log says the roof-data provider timed out twice. Three logs, three true stories, and the operations team spends an hour putting them side by side by timestamp, in three time zones of log formats, hoping the clocks agree.

The question "what caused what" has no answer in three separate logs. It has an answer in one trace.

Trace, span and the timeline#

A span is one timed step of work: a name, a start, an end, a status, and the step that caused it. For example, "assessment-agent: get_roof_assessment, from 90 to 2,090 ms, error". A span's parent is the step that caused it: the evidence task is the parent of the roof-assessment call.

A trace is the tree of all spans of one piece of work, joined by their parent links. For example, every step of run run-d606, across all four agents of the orchestrator topology. A trace id is the id all spans of one trace share: 32 hexadecimal characters in the example. On this site there is one trace per run, and a retry is a new span inside the same trace, with a higher attempt number.

A causal timeline is the trace drawn against time: each span as a bar, children under parents. It answers "what caused what, and where did the time go" in one picture, without trusting any clock but the one that measured each span.

Plain English. A trace is a parcel's tracking history when every depot writes into the same record, under the same tracking number, and each entry says which earlier entry it follows. Three depots with three separate notebooks can each tell you what they did; only the shared record tells you the parcel's journey.

The smallest primitive: carry the context in the message#

Inside one program, a tracer knows the current span and can make it the parent of the next. Across agents, that knowledge must travel. Trace propagation means putting the trace context, the trace id and the id of the span that sent the message, into the message itself, so the receiver can start its own span as a child of the sender's.

The example does exactly that. When an agent sends, the bus opens a message.send span and writes its context into the envelope's trace field. When the bus delivers, the receiver opens a message.receive span whose parent is that context. This is Tracer.context from examples/uw-agents/tracing.py, verbatim; the switch propagate=False is what the lab below turns off:

def context(self, span: dict[str, Any]) -> dict[str, str | None] | None:
    """What goes into the envelope. None when propagation is switched off."""
    if not self.propagate:
        return None
    return {"trace_id": span["trace_id"], "span_id": span["span_id"], "parent_span_id": span["parent_span_id"]}

Diagram: Trace context crossing two hops · UW-406

Time runs down the page. Each send opens a span and writes its id into the message; each receive opens a span whose parent is that id. So the assessment agent's tool calls hang, through two hops, under the intake agent's first span, in one trace.

  1. Intake agent to Coordinator: task.request · trace: run trace, parent = send span. The envelope's trace field carries the trace id and the id of the send span.
  2. Coordinator to Assessment agent: task.request · same trace, new parent. The coordinator's send span is a child of its workflow task; its id goes into this message.
  3. Assessment agent: get_roof_assessment × 2 (attempt 1, 2). Two tool spans under the evidence task, each with its attempt number; both end in error after 2,000 ms.
  4. Assessment agent to Coordinator: task.artifact · partial, gap roof_assessment. The reply carries the context too, so the coordinator's receive span is a child of the assessment agent's send span.
  5. Coordinator to Intake agent: task.artifact · request_info. The last receive closes the causal chain: one trace id for every span of the run.

Three fields are enough: which trace, which step sent this, and that step's parent. Every hop adds one send span and one receive span, and the tree grows across agent boundaries as if it were one program.

The insurance example: UW-406 as one trace#

Here is the trace of UW-406, printed as a tree by a short script run from the example folder (each child is indented under its parent; times are simulated). The agents here are scripted stand-ins, so these results show how the mechanism works, not how good any model is.

from run_case import load_case, run

r = run(load_case("UW-406"), "orchestrator")
spans = r["trace"]["spans"]
children = {}
for s in spans:
    children.setdefault(s["parent_span_id"], []).append(s)


def show(s, depth=0):
    a = s["attributes"].get("attempt_number") if s["kind"] == "tool" else None
    print(f'{"  " * depth}{s["agent"]}: {s["name"]} [{s["start_ms"]}-{s["end_ms"]} ms] {s["status"]}'
          + (f" (attempt {a})" if a else ""))
    for c in children.get(s["span_id"], []):
        show(c, depth + 1)


print("trace_ids:", r["trace"]["trace_ids"])
for root in children[None]:
    show(root)
trace_ids: ['84c60685243a69216b52acd34d4fc077']
intake-agent: handle broker email [0-5106 ms] ok
  intake-agent: parse email into submission [0-50 ms] ok
  intake-agent: send task.request [50-70 ms] ok
    coordinator: receive task.request [70-70 ms] ok
      coordinator: workflow task [70-5086 ms] ok
        coordinator: send task.status [70-90 ms] ok
          intake-agent: receive task.status [90-90 ms] ok
        coordinator: send task.request [70-90 ms] ok
          assessment-agent: receive task.request [90-90 ms] ok
            assessment-agent: evidence task [90-4980 ms] ok
              assessment-agent: send task.status [90-110 ms] ok
                coordinator: receive task.status [110-110 ms] ok
              assessment-agent: get_roof_assessment [90-2090 ms] error (attempt 1)
              assessment-agent: get_roof_assessment [2590-4590 ms] error (attempt 2)
              assessment-agent: search_guidance [4590-4830 ms] ok (attempt 1)
              assessment-agent: find_similar_decisions [4830-4980 ms] ok (attempt 1)
              assessment-agent: send task.artifact [4980-5000 ms] ok
                coordinator: receive task.artifact [5000-5000 ms] ok
        coordinator: send task.request [5000-5020 ms] ok
          underwriting-agent: receive task.request [5020-5020 ms] ok
            underwriting-agent: recommendation task [5020-5066 ms] ok
              underwriting-agent: send task.status [5020-5040 ms] ok
                coordinator: receive task.status [5040-5040 ms] ok
              underwriting-agent: check_eligibility [5020-5024 ms] ok
              underwriting-agent: record_recommendation [5054-5066 ms] ok (attempt 1)
                underwriter: underwriter_review [5054-5054 ms] waiting
              underwriting-agent: send task.artifact [5066-5086 ms] ok
                coordinator: receive task.artifact [5086-5086 ms] ok
        coordinator: send task.artifact [5086-5106 ms] ok
          intake-agent: receive task.artifact [5106-5106 ms] ok

The underwriter's two questions are answered by reading down the tree. Where did the time go: two roof-assessment attempts of 2,000 ms each, with 500 ms between them, inside the assessment agent's evidence task; everything else together took about 600 ms. Why was the roof data missing: both attempts ended in error (the stand-in provider timed out), so the artifact carried a gap and the recommendation asked for the information. One trace id, and every span from four agents hangs under the intake agent's first span. The last span is the underwriter's review, still waiting: no person has decided yet.

The trace lab#

Open the trace of any case across the agents. Click a span for its details. Then switch propagation off and watch one tree fall apart into separate traces, one per agent.

CaseTopologySpansTraces with propagationTraces without
UW-403orchestrator2914
UW-403peer2213
UW-403hierarchy4315
UW-406orchestrator3014 (two failed roof attempts, then the partial artifact)

A failure mode: the trace that falls apart#

Run UW-403 with propagation off (run_case.py --case UW-403 --no-propagate) and the summary line changes from one trace to four:

trace: 4 trace_ids · 29 spans · last message at 786 ms · run ends at 786 ms (simulated)

Nothing else changed: the same 29 spans, the same recommendation, every check passes. Each agent's own spans are still complete and correctly nested. What is gone is the link between them. The assessment agent's evidence task is now a root of its own trace; nothing says it was caused by the coordinator's request, and nothing in the intake agent's trace says that its five seconds were spent waiting for that task. This is the operations team's hour from the opening story.

In real systems the trace falls apart for mundane reasons: one hop goes through a library that drops unknown fields, one agent is written in a framework whose tracer does not read the incoming context, or a queue between two agents does not carry headers. A single broken hop splits every trace that passes through it.

Production pitfall. Trace context from another organization is input, not truth. A caller can send any trace id it likes. Accept it to link your spans, but do not let it decide anything: sampling, retention or access. Some teams start a new trace at a trust boundary and record the caller's trace id as a link instead of a parent.

What the five ids let you answer#

Messages introduced five ids. In an investigation each one answers a different question, and the trace is where they meet:

QuestionId that answers it
Everything that happened for this quote request this timerun_id (and, on this site, the one trace_id of the run)
Which step caused which, and where the time wenttrace_id, span_id, parent_span_id
What happened to one piece of delegated worktask_id
Which reply answered which requestcorrelation_id
Was this message delivered twicemessage_id
Which try of a step this wasattempt_number
Common confusion. "We already group by run id, so we have tracing." A run id groups the spans of a run, like a folder groups files; it does not say which span caused which. Sorting a run's events by time gives a list, not a tree, and the list lies as soon as two agents work in parallel or their clocks differ. The parent link is the part that grouping cannot replace. On this site a run happens to have exactly one trace, so the two ids select the same spans; the trace id is still the one that carries structure.

The example records the ids as span attributes (message_id, correlation_id, task_id, attempt_number, run_id), so a trace can be searched by any of them.

Before AI. This is distributed tracing as microservice teams use it: a trace header on every HTTP call, a span per service, a waterfall view in the tracing tool. W3C Trace Context standardized the header for HTTP, and OpenTelemetry the APIs and data model. Agents add two twists: a single run can last minutes or days (UW-402 waits fifteen minutes for a broker), and a step can be a person's review, which is why the example ends with a span for the underwriter.

How standards and libraries carry trace context#

The A2A protocol, an open standard for messages between agents, carries messages over HTTP-based bindings, so ordinary HTTP trace headers can travel with each request. Its specification, read today, lists tracing and monitoring among its enterprise-ready principles but does not define how trace context travels, so whether a given client and server propagate it is a property of their implementations. Many agent libraries emit traces of their own runs; whether they continue a trace across an agent-to-agent hop is the question to check in each one's documentation, and Frameworks reports what was found. The example implements propagation itself, in about a hundred lines of tracing.py and bus.py.

What this is not. This page is about joining one trace across agents. Logs, metrics, audit events, sampling, retention and replay in depth belong to observe.sammuthu.com. Grading a run from its trace (did it call the tools it should have) belongs to eval.sammuthu.com.

Checklist: is this system traceable across agents?#

The same idea for a coding agent. A pull request took three hours from first commit to merge. The planning agent's log shows it waited; the test agent's log shows a forty-minute suite; the review agent's log shows a review. Only a trace that starts with the planning agent's task and carries its context into the test agent's request and the review agent's request shows that two hours were the review agent waiting in a queue for a person. The runtime that emits each agent's spans is the subject of harness.sammuthu.com.

One trace per run, or one per attempt?This site's choice and its cost

This site puts every attempt of a run in one trace, with attempt_number on the spans. The view is complete: the first, failed attempt sits next to the second in one timeline, which is what an investigation needs. The cost appears with long runs: a run that waits fifteen minutes for a broker, or days for a decision, produces a trace that stays open for that long, and many tracing back-ends handle very long traces poorly.

The alternative is one trace per attempt or per stage, with links between them. Each trace stays short; the run is reassembled by run_id or by span links. A common compromise is one trace per run for the automated part, and a new, linked trace when a person takes over.

Causality without synchronized clocksWhy parent links beat timestamps

The operations team's first instinct was to sort three logs by time. Across machines that fails in small ways (clocks drift by milliseconds) and large ones (a log in local time, another in UTC). A parent link does not depend on any clock: the receive span's parent is the send span, so the receive happened after the send, whatever the clocks say. Each span's duration is measured by one clock on one machine, which is reliable.

So the trace gives order by causality and duration by local measurement, and needs no global clock. Timestamps across agents are then only for display. This is why the example's spans can be trusted in order even though every agent shares one simulated clock: the structure would be the same with eight different clocks.

What tracing costsSpans per message, and what to keep

Each message adds two spans in the example; the orchestrator topology produces 29 spans for a happy-path case and 42 for UW-402. At volume, traces are sampled: only some runs are kept. Sampling decisions should be made once, at the start of the run, and carried in the context, or a trace will be kept by one agent and dropped by the next. Runs that end in escalation, failure or a human override are the ones an investigation needs; a sampler that keeps all of those and a fraction of the rest is common. Sampling and retention are covered in depth on observe.sammuthu.com.

In one breath#

Each agent can log its own part, but only a trace says which part caused which. A trace is a tree of timed steps, spans, joined by parent links; propagation puts the trace id and the sender's span id inside every message, so the receiver's span becomes a child of the sender's. With it, a run across three agents is one causal timeline; without it, the same spans fall apart into one trace per agent.

Next: the mechanisms are complete. Frameworks reads the A2A SDKs, MCP, LangGraph, AutoGen, CrewAI and plain code on their own documentation, and says what each carries for you.