All chapters
  1. Start hereOne agent asks. Another answers. The rest is the contract between them
  2. Foundations · 1One model, one tool, two agents: who is who, and who is responsible
  3. Foundations · 2Message passing: sender, recipient, payload, address and correlation
  4. Foundations · 3Topologies: central orchestrator, peer-to-peer and hierarchy
  5. Mechanics · 4Request and reply, or a task: synchronous versus asynchronous exchange
  6. Mechanics · 5Contracts: messages, artifacts, typed schemas, errors and versions
  7. Mechanics · 6The task lifecycle: submitted, working, input-required, completed, failed, canceled
  8. Mechanics · 7Discovery: Agent Cards, skills and how an agent says what it can do
  9. Protocol · 8A2A: the Agent2Agent protocol, part by part
  10. Protocol · 9A2A versus MCP: delegating to an agent versus calling a tool
  11. Protocol · 10Handoff, delegation or shared state: three ways to work together
  12. Production · 11How updates arrive: streaming, push notifications, callbacks, polling and queues
  13. Production · 12Security: identity, delegated scopes, trust across domains, authorization
  14. Production · 13Reliability: retries, idempotency, order, timeouts and compensation
  15. Production · 14Observability: one trace across three agents
  16. Reference · 15Frameworks: A2A SDKs, LangGraph, AutoGen, CrewAI and plain code
  17. Reference · 16Advanced: consensus claims, coordination failure, cost and isolation
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Production · 13

Reliability: retries, idempotency, order, timeouts and compensation

Messages arrive twice, late, out of order or not at all. Retries that are safe, keys that make a second delivery harmless, and what to undo when half the work is done.

What you will understand
At least once
A sender that retries until it hears back delivers every message, and some of them twice.
A key, not a promise
"Exactly once" is built by the receiver: an idempotency key makes a repeated write store one row.
When it never comes
A deadline ends the wait, a retry tries again, and compensation moves the case on safely.

At 734 milliseconds into case UW-410 (simulated time), the underwriting agent calls the write tool record_recommendation to store its recommendation for submission SUB-4010. The row is stored. The answer that says so never comes back. The agent waits 1,000 ms and has to decide.

If it does not call again, and the write had in fact failed, there is no recommendation and the underwriter never sees the case. If it calls again, and the write had in fact succeeded, there are two rows, and the underwriter cockpit shows the same recommendation twice, or worse, two different ones if anything changed in between. From where the agent sits, "not stored" and "stored, answer lost" look exactly the same.

The same run has a second problem earlier on. The message bus delivers the evidence request to the assessment agent twice. If the assessment agent treats the second copy as new work, it collects all the evidence again and sends a second artifact. Neither problem is rare. Any system that sends messages over something that can fail meets both, and this page is about the small set of mechanisms that make them harmless. (The UW-4xx case ids are this site's own; UW-410 to UW-412 are invented for this site, UW-406 reuses the facts of the series case UW-006.)

Four ways a message goes wrong#

A message can go wrong in four ways, and the receiver can see only some of them. It can be duplicated: the same message arrives twice. It can be lost: it never arrives. It can be late: it arrives after the sender stopped waiting. And it can arrive out of order: a later message overtakes an earlier one. A fifth case looks like loss from the outside: the receiver gets the message and never answers.

None of these is special to agents. They follow from two facts: networks drop and delay messages, and a sender cannot tell a lost request from a lost reply. Agents make the cost of getting it wrong higher, because each step can be slow, can call tools with effects, and can be retried by a caller that has its own deadline. Sync and async introduced the deadline as a concept; this page is where retries, keys, ordering and compensation are explained in full.

Before AI. A Kafka consumer that commits its offset after processing will reprocess a record when it crashes between the two steps, so the consumer is written to be idempotent: it remembers which records it has handled, or writes with a key that makes a second write a no-op. A REST API that takes a payment does the same with an idempotency key header on the POST. Agent-to-agent messaging needs exactly these two habits, for the same reasons.

At least once, and why "exactly once" is built from a key#

A retry is sending a request or calling a tool again because the first try failed or no answer came in time. For example, the underwriting agent calls record_recommendation a second time at 1,734 ms. A retry is a new attempt inside the same run, with a higher attempt_number, in the same trace; it is never a new run.

At-least-once delivery is what a sender gets when it retries until it hears back: every message arrives, and some arrive more than once. For example, the evidence request msg-0003 reaches the assessment agent twice in UW-410. The alternative, at-most-once, never repeats and sometimes loses. For a recommendation that a person must see, losing it is the worse failure, so agents retry, and the receiver must cope with repeats.

Idempotency is the property that doing an operation twice leaves the same state as doing it once. For example, reading the roof assessment twice changes nothing; storing a recommendation twice, without care, adds a row. Reads are idempotent by nature. Writes have to be made idempotent.

An idempotency key is a value the caller sends with a write so the receiver can recognize a repeat of the same intended write. For example, run-8a10:SUB-4010:rec: one key per run, per submission, per kind of write. Put together, at-least-once delivery plus an idempotent receiver gives an effect that happens once. That is the only honest meaning of "exactly once" between two programs that can fail independently.

The smallest primitive: two checks, two different keys#

The example uses two checks, and they protect different things. The first protects the receiver from the same message twice. The bus does it once for every agent, as each agent's inbox check (bus.py:Bus._deliver), keyed by message_id:

        elif self.dedupe and msg["message_id"] in seen:
            self.note("duplicate_dropped", msg, note="this message_id was delivered before")
        else:
            seen.add(msg["message_id"])

The second protects the stored row from the same intended write twice. The write tool remembers every key it has stored (tools_sim.py:World.record_recommendation) and answers a known key with the row it already has:

        if self.idempotency and idempotency_key in self._by_key:
            row = self._by_key[idempotency_key]
            self.audit.append({"event": "recommendation_replayed", "row_id": row["row_id"], "idempotency_key": idempotency_key,
                               "actor": token["principal"], "actor_type": "agent", "at_ms": at_ms})
            return row

(Both excerpts are verbatim and keep the indentation of their methods, so they are shown as excerpts, not as runnable code.) Why two keys? A retry of a request is a new message with a new message_id, so deduplication by message id cannot catch it; that is by design, or a receiver would drop every retry. The idempotency key is built from what the write is for, not from the message that carried it, so it stays the same across retries, duplicates and even a duplicated task. The message id recognizes a message; the idempotency key recognizes an intention.

The insurance example: a duplicate and a retried write#

Here is UW-410 in the orchestrator topology, run in a copy of examples/uw-agents. The agents here are scripted stand-ins, so these results show how the mechanism works, not how good any model is. A successful run shows one dropped duplicate, two calls to the write tool, and one row.

/opt/homebrew/bin/python3 run_case.py --case UW-410 --dry-run
UW-410 · topology orchestrator · run run-8a10 · agents: scripted stand-ins (no model called)
  time ms  from                 to                 type          state           ids
       50  intake-agent       → coordinator        task.request  -               msg-0001 corr msg-0001 task -        try 1
       70  coordinator        → intake-agent       task.status   working         msg-0002 corr msg-0001 task task-c1  try 1
       70  coordinator        → assessment-agent   task.request  -               msg-0003 corr msg-0003 task -        try 1
       70  · duplicated        coordinator → assessment-agent           msg-0003  fault: the bus will deliver this message twice
       90  assessment-agent   → coordinator        task.status   working         msg-0004 corr msg-0003 task task-a1  try 1
       95  · duplicate_dropped coordinator → assessment-agent           msg-0003  this message_id was delivered before
      660  assessment-agent   → coordinator        task.artifact completed       msg-0005 corr msg-0003 task task-a1  try 1
      680  coordinator        → underwriting-agent task.request  -               msg-0006 corr msg-0006 task -        try 1
      700  underwriting-agent → coordinator        task.status   working         msg-0007 corr msg-0006 task task-u1  try 1
     1746  underwriting-agent → coordinator        task.artifact completed       msg-0008 corr msg-0006 task task-u1  try 1
     1766  coordinator        → intake-agent       task.artifact completed       msg-0009 corr msg-0001 task task-c1  try 1
messages: 9 sent · 9 delivered · 1 duplicates dropped · 0 stale updates ignored · 0 deadlines fired
tasks (caller's view): workflow completed · evidence completed · recommendation completed
recommendation: approve · citations - · missing - · evidence complete · routed to the underwriter · stated_confidence 0.9
tool calls: get_roof_assessment 1 · search_guidance 1 · find_similar_decisions 1 · check_eligibility 1 · record_recommendation 2
rows stored: 1 · human_decision: none (no person has decided yet)
trace: 1 trace_id · 30 spans · last message at 1786 ms · run ends at 1786 ms (simulated)
expected vs actual: PASS (10 of 10 checks)

Read it against the opening. The second copy of msg-0003 arrives at 95 ms and is dropped: one task, task-a1, not two. The write tool is called twice (record_recommendation 2) and stores one row: the run's audit log records the first call at 734 ms as recommendation_recorded and the second at 1,734 ms as recommendation_replayed, both for row-1. The recommendation is approve; the model recommends; a person decides, and that decision is still open. The run with the key switched off differs in one line and fails its own check:

/opt/homebrew/bin/python3 run_case.py --case UW-410 --no-idempotency --dry-run
rows stored: 2 · human_decision: none (no person has decided yet)
trace: 1 trace_id · 31 spans · last message at 1786 ms · run ends at 1786 ms (simulated)
expected vs actual: FAIL (9 of 10 checks)
  ✗ rows: expected 1, actual 2

Diagram: UW-410 · a duplicate and a retried write, one row

Time runs down the page. The bus delivers the evidence request twice; the second copy is dropped by its message id. Later the write is stored but its answer is lost; the underwriting agent waits, calls again with the same idempotency key, and gets the row already stored. Two defenses, two different keys.

  1. Coordinator to Assessment agent: evidence request msg-0003. The first copy is delivered and becomes task-a1.
  2. Coordinator to Assessment agent: same msg-0003 again. The fault: the bus delivers the same message a second time. Same message_id, so it is dropped.
  3. Underwriting agent to Recommendations table: write, key run-8a10:SUB-4010:rec. The row is stored.
  4. Recommendations table to Underwriting agent: answer lost. The caller cannot tell 'not stored' from 'stored, answer lost'. It waits 1,000 ms.
  5. Underwriting agent to Recommendations table: write again, same key. Attempt 2 of the same intended write.
  6. Recommendations table to Underwriting agent: row-1, already stored. The key is known: the stored row is returned and nothing new is written. Audit: recommendation_replayed.

Switching the defenses off one at a time shows which one protects what. Each row below is a real run of UW-410 with the switches named.

Run of UW-410Duplicates droppedWrite callsRows stored
All defenses on121
--no-idempotency122
--no-dedupe021 (every read tool called twice; 2 stale updates ignored)
peer topology, --no-dedupe --no-idempotency033

The third row deserves a second look. Without deduplication the assessment agent accepts the second copy as new work, opens task-a2, and calls all three read tools again. The row count stays at 1 only because the idempotency key does not care how many tasks asked for the write. In the peer topology, where the assessment agent forwards its artifact straight to the underwriting agent, the duplicate task produces a second recommendation request, and with neither defense on, the lost reply adds a third write: three rows for one quote request.

The faults lab runs the same cases with the switches in your hands. Pick a fault, turn deduplication, the idempotency key, sequence numbers and deadlines on and off, and read the rows stored, the write calls and the caller's view of the evidence task.

Fault (case)With every defense onWhat breaks when one is off
Duplicate (UW-410)1 row2 rows without the idempotency key; 2 without dedupe and key; 3 with every switch off
Reorder (UW-411)evidence task completedworking without sequence numbers
Drop (UW-401, the first evidence request lost)deadline at 30,070 ms, attempt 2 succeeds, approve, 1 rowwithout deadlines: no recommendation, 0 rows, the evidence task stays submitted
Delay (UW-401, the artifact 35 s late)deadline, attempt 2 succeeds; the late artifact arrives at 35,680 ms and is ignored; 1 row, every read tool called twicethe evidence work is done twice even with every defense on
Silent (UW-412)deadline, attempt 2, deadline, compensation: escalate, evidence unavailable, 1 rowwithout deadlines: no recommendation, 0 rows, the evidence task stays submitted

(The drop and delay rows come from the lab's fault definitions in scripts/build_data.py, run here as two scratch case files copied from UW-401 with the fault added.)

Out of order: a sequence number per task#

Out-of-order delivery is a later message arriving before an earlier one. For example, in UW-411 the assessment agent's working update is held back by the bus and arrives after its completed artifact. A caller that applies updates in arrival order ends up believing a finished task is running again.

The example gives each task a sequence number, seq: the callee counts the messages it sends about one task (1, 2, 3), and the caller ignores any update whose number is not higher than the last one it applied. Together with the rule that a terminal state never moves, this keeps the caller's view right whatever the arrival order. The guard is in tasks.py:apply_update (an excerpt, verbatim):

    if use_seq and seq <= call.seq:
        return Refusal("STALE_UPDATE", f"seq {seq} is not higher than {call.seq}, the last update applied")
    if use_seq and call.state in TERMINAL:
        return Refusal("TASK_TERMINAL", f"the caller already holds {call.state}; a terminal state never moves")
/opt/homebrew/bin/python3 run_case.py --case UW-411 --dry-run
UW-411 · topology orchestrator · run run-4d11 · agents: scripted stand-ins (no model called)
  time ms  from                 to                 type          state           ids
       50  intake-agent       → coordinator        task.request  -               msg-0001 corr msg-0001 task -        try 1
       70  coordinator        → intake-agent       task.status   working         msg-0002 corr msg-0001 task task-c1  try 1
       70  coordinator        → assessment-agent   task.request  -               msg-0003 corr msg-0003 task -        try 1
       90  assessment-agent   → coordinator        task.status   working         msg-0004 corr msg-0003 task task-a1  try 1
       90  · held              assessment-agent → coordinator           msg-0004  fault: held back until the next message of this exchange has arrived
      660  assessment-agent   → coordinator        task.artifact completed       msg-0005 corr msg-0003 task task-a1  try 1
      680  coordinator        → underwriting-agent task.request  -               msg-0006 corr msg-0006 task -        try 1
      680  · stale_ignored     assessment-agent → coordinator           msg-0004  seq 1 is not higher than 2, the last update applied
      700  underwriting-agent → coordinator        task.status   working         msg-0007 corr msg-0006 task task-u1  try 1
      746  underwriting-agent → coordinator        task.artifact completed       msg-0008 corr msg-0006 task task-u1  try 1
      766  coordinator        → intake-agent       task.artifact completed       msg-0009 corr msg-0001 task task-c1  try 1
messages: 9 sent · 9 delivered · 0 duplicates dropped · 1 stale updates ignored · 0 deadlines fired
tasks (caller's view): workflow completed · evidence completed · recommendation completed
recommendation: approve · citations - · missing - · evidence complete · routed to the underwriter · stated_confidence 0.9
tool calls: get_roof_assessment 1 · search_guidance 1 · find_similar_decisions 1 · check_eligibility 1 · record_recommendation 1
rows stored: 1 · human_decision: none (no person has decided yet)
trace: 1 trace_id · 29 spans · last message at 786 ms · run ends at 786 ms (simulated)
expected vs actual: PASS (10 of 10 checks)

The held msg-0004 (seq 1) is released at 680 ms, after the artifact (seq 2), and is ignored as stale. With --no-seq the same run ends with evidence working in the caller's view and fails two checks. Nothing was lost; the order alone produced a wrong belief. Notice also that the recommendation went ahead correctly in both runs, because the coordinator acted on the artifact when it arrived. The damage of a wrong view shows up later, in a dashboard, a timeout handler or a cancel sent to a finished task.

Timeouts, retries and compensation#

A timeout is the moment a waiter stops waiting for one answer and treats the step as failed, without knowing whether it happened. For example, the roof-data provider stand-in gives up after 2,000 ms. A timeout ends the wait, not the work: after one, the caller can retry, compensate, or hand over to a person, but it cannot assume nothing happened.

Inside one agent: a tool timeout (UW-406). The assessment agent calls get_roof_assessment; the provider does not answer in 2,000 ms. The agent waits 500 ms and tries once more, with attempt_number 2, in the same run and the same trace (the trace has two get_roof_assessment spans, at 90 ms and 2,590 ms, both ending in error). The second try times out too. The agent does not fail the whole task: it finishes with a partial artifact that names the gap, {"need": "roof_assessment", "reason": "timeout", "attempts": 2}.

/opt/homebrew/bin/python3 run_case.py --case UW-406 --dry-run
recommendation: request_info · citations - · missing roof_assessment · evidence partial · routed to the underwriter · stated_confidence 0.6
tool calls: get_roof_assessment 2 · search_guidance 1 · find_similar_decisions 1 · check_eligibility 1 · record_recommendation 1
rows stored: 1 · human_decision: none (no person has decided yet)
trace: 1 trace_id · 30 spans · last message at 5106 ms · run ends at 5106 ms (simulated)
expected vs actual: PASS (10 of 10 checks)

The recommendation is request_info, with missing = roof_assessment: an unverified roof is not decided either way. The retry here is safe without any key, because the tool is a read. The same retry around a write would need the key from the section above.

Between agents: a silent callee (UW-412). Compensation is a forward step that makes up for one that failed, leaving the whole in a safe and honest state. For example, when evidence cannot be had, the case goes on to the underwriter marked "evidence unavailable" instead of stopping. Agents cannot roll back like a database transaction: messages were sent and tools were called. So compensation is written in advance, per step.

/opt/homebrew/bin/python3 run_case.py --case UW-412 --dry-run
UW-412 · topology orchestrator · run run-b712 · agents: scripted stand-ins (no model called)
  time ms  from                 to                 type          state           ids
       50  intake-agent       → coordinator        task.request  -               msg-0001 corr msg-0001 task -        try 1
       70  coordinator        → intake-agent       task.status   working         msg-0002 corr msg-0001 task task-c1  try 1
       70  coordinator        → assessment-agent   task.request  -               msg-0003 corr msg-0003 task -        try 1
       90  · ignored           coordinator → assessment-agent           msg-0003  assessment-agent is silent: it receives the message and never answers
    30070  · deadline          coordinator                              msg-0003  coordinator: no final answer to msg-0003 in time
    30070  coordinator        → assessment-agent   task.request  -               msg-0004 corr msg-0004 task -        try 2
    30070  · retry             coordinator → assessment-agent           msg-0004  attempt 2 replaces msg-0003
    30090  · ignored           coordinator → assessment-agent           msg-0004  assessment-agent is silent: it receives the message and never answers
    60070  · deadline          coordinator                              msg-0004  coordinator: no final answer to msg-0004 in time
    60070  · compensation      coordinator                                DEADLINE_EXCEEDED: the case goes to the underwriter marked evidence unavailable
    60070  coordinator        → underwriting-agent task.request  -               msg-0005 corr msg-0005 task -        try 1
    60090  underwriting-agent → coordinator        task.status   working         msg-0006 corr msg-0005 task task-u1  try 1
    60136  underwriting-agent → coordinator        task.artifact completed       msg-0007 corr msg-0005 task task-u1  try 1
    60156  coordinator        → intake-agent       task.artifact completed       msg-0008 corr msg-0001 task task-c1  try 1
messages: 8 sent · 6 delivered · 0 duplicates dropped · 0 stale updates ignored · 2 deadlines fired
tasks (caller's view): workflow completed · evidence failed · recommendation completed
recommendation: escalate · citations - · missing roof_assessment, guidance, similar_decisions · evidence unavailable · routed to the underwriter · stated_confidence 0.2
tool calls: check_eligibility 1 · record_recommendation 1
rows stored: 1 · human_decision: none (no person has decided yet)
trace: 1 trace_id · 21 spans · last message at 60176 ms · run ends at 60176 ms (simulated)
expected vs actual: PASS (11 of 11 checks)

The deadline of 30,000 ms fires at 30,070 ms. The coordinator marks its view of the evidence request failed and sends attempt 2 (msg-0004, try 2), a new message in the same run. That deadline fires at 60,070 ms, there are no attempts left, and the coordinator compensates: the underwriting agent receives the submission with no artifact and evidence_status unavailable, and recommends escalate with stated_confidence 0.2 and all three kinds of evidence listed as missing. Nothing is invented, and a person sees the case. The steps are in agents.py:Agent.on_deadline and Relay.on_call_failed.

Diagram: UW-412 · deadline, attempt 2, deadline, compensation

Time runs down the page. The assessment agent receives every request and never answers. The coordinator's deadline fires after 30 simulated seconds; it sends attempt 2 in the same run; that deadline fires too. Instead of waiting forever, the coordinator compensates: the case goes on, marked evidence unavailable, and a person sees it.

  1. Coordinator to Assessment agent: evidence request, attempt 1. deadline_ms 30000. No answer comes.
  2. Coordinator: deadline at 30,070 ms. The caller marks its view of the request failed with DEADLINE_EXCEEDED and retries once.
  3. Coordinator to Assessment agent: evidence request, attempt 2. A new message id, attempt_number 2, the same run and the same trace.
  4. Coordinator: deadline at 60,070 ms: compensate. No more attempts. The written-down compensation for this step: go on without evidence, say so.
  5. Coordinator to Underwriting agent: recommendation request, evidence unavailable. The case continues with artifact null and evidence_status unavailable.
  6. Underwriting agent to Underwriter: escalate, for review. Nothing is invented: missing names all three kinds of evidence. A person decides.

Dead letters: what the example does not do#

A dead letter is a message that failed every attempt and is set aside in a separate queue, with the reason, instead of being dropped or retried forever. For example, a request sent to an address where no agent is registered would land there marked "undeliverable", for a person to inspect.

The example does not implement a dead-letter queue. Its bus writes undeliverable and ignored events to the run's log, and that log is the only trace of such a message. In UW-412 that is acceptable because compensation already moved the case on. In a production system it is not: someone must be able to list the messages that never got through, see why, and decide to replay, repair or discard each one. A dead-letter queue without an owner is a slower way of losing messages.

A failure mode: retries without keys#

The opening's question has a common wrong answer: "retry, and it will be fine". Retries without idempotency turn every lost reply into a duplicate effect, and in a chain of agents they multiply. The peer run above stored three rows for one quote request, with each read tool called twice. Now add deadlines that are too short: the caller retries work that is still running, and the callee runs it twice. And add retries at several levels at once, a tool retry inside the assessment agent, a request retry in the coordinator, a workflow retry above it: two attempts at each of three levels can mean eight attempts at the bottom.

The opposite mistake is no deadline at all. With --no-deadline, UW-412 stops after three messages: the workflow stays working, the evidence task stays submitted in the caller's view, no recommendation is written, no row is stored, and nobody is told. A silent agent turns into a silent case.

Production pitfall. A retry policy is a load policy. When an agent slows down because it is overloaded, every caller's deadline fires at once and every caller retries at once, doubling the load on the agent that was already struggling. Retry budgets, backoff with jitter and circuit breakers are the tools; they belong to reliability as a discipline, below.
What this is not. This page covers what agent communication needs to survive faults: keys, sequence numbers, deadlines, compensation. Retry budgets, load shedding, fallbacks across providers and reliability as an engineering discipline belong to reliable.sammuthu.com.

How the A2A protocol handles it#

The A2A specification, version 1.0.0, read today, says several of these things in its own words. Send Message operations "MAY be idempotent", and agents may use the messageId to detect duplicate messages; get operations are idempotent by nature; cancel is idempotent, and a duplicate cancel may return TaskNotFoundError once the task has been purged. For push notifications, agents must attempt delivery at least once for each configured webhook, may retry with exponential backoff, should use a timeout for webhook requests (it recommends 10 to 30 seconds), and may stop after a number of consecutive failures; clients should process notifications idempotently, because duplicates may occur. For streaming, all implementations must deliver events in the order they were generated. How those delivery channels work is on Streaming.

What the sections read here do not define matters as much: an idempotency key for the side effects an agent performs, a dead-letter mechanism, or a compensation operation. The protocol makes a repeated message recognizable; making a repeated write harmless is the job of the agent and its tools, as in the example. How the A2A SDKs and the agent libraries handle retries and duplicates is on Frameworks.

Checklist: a reliability review of an agent-to-agent exchange#

The same idea for a coding agent. A planning agent asks a test agent to run the suite, and the test agent opens a comment on the pull request with the failures. The connection drops before the answer arrives, and the planning agent asks again. Without a key, the pull request gets two identical comments, and with a retry at each level, four. With a key such as the pull request number plus the commit hash plus "test-report", the second request updates the first comment instead. CI is the judge that is not an agent, and its own retries are configured where the change ships, which belongs to diff-to-deploy.sammuthu.com.

Designing the idempotency keyScope, lifetime, and what a replay must return

The key answers one question: are these two calls the same intended write? Too narrow, and real repeats slip through: a key that includes the message_id or the attempt_number changes on every retry and protects nothing. Too wide, and different writes collide: a key of SUB-4010 alone would make a second run for the same submission, after the broker sends corrected facts, return the first run's row. The example's run_id:submission_id:rec says "one recommendation per run per submission", which is the business rule. Three further decisions follow. Lifetime: the receiver must remember keys at least as long as any caller may retry, including retries after a restart, so the key store is durable, not in memory as in the example. Conflict: a repeat with the same key but a different payload is a bug in the caller, and the receiver should refuse it by name rather than silently return the old row. Replay: the answer to a repeat should be the original answer, so the caller cannot tell a replay from a first write, which is the point.

"Exactly once" in vendor languageWhat it can and cannot mean across two independent processes

When a product promises exactly-once delivery, read it as: at-least-once delivery inside the product, plus deduplication or transactions inside the product's own boundary. The moment the effect leaves that boundary (a tool writes to another database, an agent sends an email, a webhook fires), the guarantee ends and the receiver's idempotency takes over. Two processes that can fail independently cannot agree that one message was processed exactly once without a shared record of it; the idempotency key is that shared record, kept by the side that owns the effect. A design review should ask, for each side effect: where is the key stored, by whom, and does it survive a restart?

A defense that worked by accidentWhat the no-dedupe run of UW-410 really shows

In the orchestrator run with --no-dedupe, the duplicate request becomes a second task, task-a2, and its updates are ignored by the coordinator as stale. That looks like the sequence-number guard doing its job, but it is a coincidence. The coordinator matches replies by correlation_id, both tasks answer the same correlation, and both number their messages from 1, so the second task's seq 1 and seq 2 are "not higher" than the first task's. Had the second task been faster, its updates would have been applied and the first task's ignored, and the coordinator would have recorded the wrong task_id, so a later cancel would reach the wrong task. Sequence numbers order the messages of one task; they do not detect two tasks for one request. Deduplication at the door does, and in the example it is the only thing that does. When two defenses overlap by chance, test each one alone.

Compensation is a design artifactWrite it per step, before the incident

Compensation in the example is one line of policy: no evidence, so go on marked unavailable and escalate. Real workflows need a table: for each step, what effect it may already have had when it fails, what undoes or neutralizes that effect (a cancel, a correcting write, a message to a person), and who is told. Some steps have no compensation (an email sent to a broker cannot be unsent); those steps should come last, after everything that can fail. This ordering is the saga pattern from distributed transactions, and it applies unchanged. What agents add is that a step's effect is often not known to the caller at all: the callee is opaque. So the contract should say which effects a task can have, and the callee's artifact should report which ones it actually had, including on failure.

In one breath#

Messages between agents arrive twice, late, out of order or never, so every sender that cares retries, and every receiver must make repeats harmless. Deduplication by message_id stops the same message twice; an idempotency key built from the intended write stops the same write twice; a sequence number per task stops an old update from overwriting a new one. When an answer never comes, a deadline ends the wait, a bounded retry tries again in the same run and trace, and a written compensation moves the case on to a person, honestly marked.

Next: every mechanism on this page left marks in a log, but reading one agent's log at a time does not tell you what happened to a run. Observability joins the three agents' steps into one trace, so a duplicate, a retry or a stale update can be seen in one timeline.