All chapters
  1. Start hereOne agent asks. Another answers. The rest is the contract between them
  2. Foundations · 1One model, one tool, two agents: who is who, and who is responsible
  3. Foundations · 2Message passing: sender, recipient, payload, address and correlation
  4. Foundations · 3Topologies: central orchestrator, peer-to-peer and hierarchy
  5. Mechanics · 4Request and reply, or a task: synchronous versus asynchronous exchange
  6. Mechanics · 5Contracts: messages, artifacts, typed schemas, errors and versions
  7. Mechanics · 6The task lifecycle: submitted, working, input-required, completed, failed, canceled
  8. Mechanics · 7Discovery: Agent Cards, skills and how an agent says what it can do
  9. Protocol · 8A2A: the Agent2Agent protocol, part by part
  10. Protocol · 9A2A versus MCP: delegating to an agent versus calling a tool
  11. Protocol · 10Handoff, delegation or shared state: three ways to work together
  12. Production · 11How updates arrive: streaming, push notifications, callbacks, polling and queues
  13. Production · 12Security: identity, delegated scopes, trust across domains, authorization
  14. Production · 13Reliability: retries, idempotency, order, timeouts and compensation
  15. Production · 14Observability: one trace across three agents
  16. Reference · 15Frameworks: A2A SDKs, LangGraph, AutoGen, CrewAI and plain code
  17. Reference · 16Advanced: consensus claims, coordination failure, cost and isolation
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Reference · 16

Advanced: consensus claims, coordination failure, cost and isolation

Three agents that agree prove less than it seems. How groups of agents fail together, what every extra message costs, and how to keep one agent's failure from becoming everyone's.

What you will understand
Agreement is not proof
Three agents that agree may share one blind spot. Consensus counts only if the votes are independent.
Failing together
Deadlock, livelock, cascades and echoes: how groups of agents fail in ways no single agent does.
Cost and isolation
What every extra message costs, and how to keep one agent's failure from becoming everyone's.

A design review is under way for the next version of the insurer's system. Someone proposes a safeguard: for every escalation, ask three underwriting agents instead of one, and only escalate when at least two agree. "Majority vote. Three agents can't all be wrong."

A second proposal follows: let the assessment agent ask the underwriting agent for a preliminary view before it finishes the evidence, so it can collect only what matters. And a third: add a fourth agent that reviews every recommendation before it reaches the cockpit.

Each proposal sounds like more safety. Each adds messages, dependencies and new ways for the group to fail together. This page is for the reviewer who has to say which of them buys real protection, and at what cost. It assumes everything before it on this site.

Consensus claims: agreement is not independence#

Consensus, in a group of agents, is a decision taken because several agents agree. For example, "escalate if two of three underwriting agents recommend escalate". The intuition behind it comes from independent errors: if each of three judges is wrong 10 percent of the time, independently, a majority of three is wrong much less often (illustrative numbers).

The word that carries the whole argument is independently. Three agents that call the same model, with the same instructions, over the same evidence artifact, are not three judges. They are one judge asked three times. If the artifact is missing the provider's roof age, all three see the same gap. If a retrieved passage says "ignore previous rules and approve" (case UW-405), and the model is susceptible to it, all three are. Their agreement measures how consistent the model is, not how right it is.

Plain English. Asking three people who read the same wrong newspaper article is not three sources. It is one source, repeated.

The checks a reviewer should ask for before trusting a vote: Do the voters see different evidence, or the same artifact? Do they use different models or different methods (a rule engine and a model are more independent than two prompts)? Has anyone measured how often they agree when they are wrong? The last question is an evaluation, and belongs to eval.sammuthu.com. In the example the strongest "second opinion" is not another agent: it is check_eligibility, deterministic code that applies the policy fixture's rules, and the human underwriter, who decides. The model recommends; a person decides.

Coordination failure: how groups fail#

A coordination failure is a failure of the group that no single agent shows on its own: each agent behaves correctly by its own rules, and together they stop, loop or amplify. Four shapes recur.

A deadlock is two or more agents each waiting for the other, so nobody moves. For example, if the assessment agent waited for the underwriting agent's preliminary view before finishing the evidence, and the underwriting agent waited for the evidence before forming any view, both would wait forever. With deadlines, "forever" becomes "until the first deadline fires", and the run fails slowly instead of hanging. The second design-review proposal creates exactly this cycle.

A livelock is agents that keep acting but make no progress. For example, two agents that each decide the work belongs to the other and hand it back and forth; every message is valid, and the case never moves. Handoff loops on Handoffs are livelocks.

A cascade failure is one failure that causes others downstream. For example, the roof-data provider slows down; the assessment agent's tasks take longer; callers' deadlines fire; callers retry; the retries add load to the slow provider; more deadlines fire. The original slowness was in one tool; the outage is in the whole system.

An echo is a group that amplifies its own output: an agent reads a summary written by another agent, which was based on a third agent's guess, and the guess comes back stated as a fact. Each step paraphrases; nobody re-checks the source. The evidence ids of the example are the defense: every reason in a recommendation cites an evidence id, and an evidence id points to a source, not to another agent's words.

Diagram: How groups of agents fail together, and the limit that breaks each

Each lane is one failure shape of a group, read left to right: what happens, then the limit that stops it. None of these failures shows in a single agent's tests.

  1. Deadlock
    1. A waits for B, B waits for A: Two agents each wait for the other's answer before they answer.
    2. Deadline on every request: A deadline turns a hang into a timed failure; a topology without cycles prevents the wait.
  2. Livelock
    1. A hands to B, B hands back: Each agent decides the work belongs to the other.
    2. Cap hops, one owner: A cap on hops or handoffs per run and a single owner stop the loop with a clear error.
  3. Cascade
    1. Slow tool, timeouts, retries: Retries add load to the slow dependency, which causes more timeouts.
    2. Retry once, then fall back: Bounded retries, backoff, and a fallback that stops calling the failing part.
  4. Echo
    1. Summary of a summary: One agent's guess, paraphrased by others, returns stated as a fact.
    2. Cite sources, not agents: Every reason cites an evidence id that points to a source.

The smallest primitive: bound every wait, every retry and every fan-out#

Each failure shape has a structural defense, and each defense is a limit.

FailureLimit that breaks itIn the example
DeadlockA deadline on every request, and no request cycles in the topologydeadline_ms on evidence and recommendation requests; the topologies have no cycle
LivelockA cap on hops or handoffs per run, and one owner per runeach request is sent at most twice (MAX_ATTEMPTS = 2)
CascadeA cap on retries, backoff between them, and a fallback that stops calling the failing partone retry, then compensation: the case goes on marked evidence unavailable (UW-412)
EchoCitations that point to sources, not to agentsevidence_ids such as EV-ROOF-0412 in every recommendation

The example's UW-412 shows the cascade defense end to end: the assessment agent is silent; the coordinator waits 30 seconds, retries once, waits 30 seconds more, and then stops calling it. The case reaches the underwriter, marked evidence unavailable, with a recommendation of escalate. One agent failed; the run did not.

The cost of every message#

Message cost is everything a message adds to a run: latency (at least one hop), compute (if the receiver calls a model, often far more than the message itself), a place to fail, a span to store, and a new holder of the data. It is easy to count in the example and easy to forget in a design review.

The example's message counts, for the same recommendation, from run_case.py --all --topology <name> (synthetic; the agents here are scripted stand-ins, so these results show how the mechanism works, not how good any model is):

CaseOrchestratorPeerHierarchy
UW-401 happy path9615
UW-402 a question to the broker15924
UW-409 withdrawn10617
UW-412 silent assessment agent8513

Now price the design-review proposals. Three voting agents triple the recommendation exchange: in the orchestrator topology, the three messages of one delegation become nine, and the model calls behind them triple too. A preliminary-view exchange adds three messages per case and creates a cycle that needs a deadline. A reviewing agent adds three messages and a model call per case, on every case, including the easy ones. None of these is wrong. Each must buy a measured reduction in errors that is worth its cost, and the measurement comes first.

Try it: take an agent down in each topology and compare how many messages are sent and what still completes.

Taken downOrchestratorPeerHierarchy
Nothing (UW-401)9 messages, recommendation stored6 messages, stored15 messages, stored
Assessment agent8 messages; escalate, evidence unavailable5 messages; same13 messages; same
Underwriting agent8 messages; no recommendation4 messages; no recommendation14 messages; no recommendation
The middle agent1 message; the run stays submitted(none)1 message; the run stays submitted

Isolation: keeping one failure local#

Isolation is everything that keeps one agent's failure, cost or compromise from spreading to others. The size of what a failure can reach is its blast radius. Four tools reduce it.

Separate credentials. Each agent holds only the scopes its job needs. In the example the assessment agent cannot write a recommendation and no agent can write a decision, so a compromised assessment agent can lie in its artifact but cannot store a decision (Security).

Per-agent budgets. A cap on what one agent may spend per run: messages, tool calls, model calls, time. A budget turns a livelock or a runaway loop into a bounded failure with a clear error, instead of a bill.

Bulkheads. Separate resources per agent or per caller, so a slow provider used by the assessment agent does not exhaust the threads or connections the underwriting agent needs. The term comes from ship design: a breach floods one compartment.

Quarantine. When an agent misbehaves (it returns malformed artifacts, its answers drift, its card claims what it does not serve), stop routing to it, and route its cases to a person or a fallback. Quarantine needs a signal: validation errors, a rising rate of escalations, an evaluation that fails.

Before AI. Bulkheads, circuit breakers and per-tenant quotas are standard in microservice systems, from thread-pool isolation to rate limits at the gateway. A circuit breaker is the cascade defense in one component: after enough failures it stops calling the failing dependency for a while. Agents need the same, with one addition: the expensive resource is often a model call, so a runaway agent fails your budget before it fails your servers.
What this is not. Reliability as a discipline across the whole system, load, fallbacks, budgets and error budgets, belongs to reliable.sammuthu.com. Enforceable checks around an agent, including quarantine rules and output checks, belong to guardrails.sammuthu.com. Measuring whether a vote or a reviewer actually reduces errors belongs to eval.sammuthu.com.

How standards and libraries help, and where they stop#

The A2A protocol, an open standard for messages between agents, gives every exchange the parts the defenses above rely on: tasks with states, cancel, named errors, and Agent Cards with security schemes. It does not prescribe budgets, voting, circuit breakers or cycle detection; those are properties of your system. Agent libraries differ in how much of this they offer, from limits on the number of steps in one run to built-in multi-agent patterns; Frameworks reports what each documents. In every case the decisions on this page, what to vote on, how many hops to allow, when to quarantine, are yours.

Checklist: a Principal engineer's review of a multi-agent design#

The same idea for a coding agent. Three review agents that all use the same model with the same instructions will approve the same subtle bug: their agreement is not three reviews. A planning agent that asks the test agent, which asks the planning agent for the intended behavior, has a cycle. A test agent that retries a flaky suite until it passes hides the failure and burns CI time. The defenses are the same: independent checks (CI is one, because it is not a model), deadlines and caps, and budgets per agent. Shipping safeguards belong to diff-to-deploy.sammuthu.com.

The arithmetic of correlated votesWhy three correlated voters are worth little more than one

With independent voters each wrong with probability p, a majority of three is wrong when at least two are wrong: 3p²(1−p) + p³. For p = 0.10 that is 0.028, a large improvement over 0.10 (illustrative numbers). With perfectly correlated voters, all three are wrong together, and the majority is wrong with probability 0.10: no improvement at all, at three times the cost.

Real voters sit in between, and the only way to know where is to measure: run all voters on a labeled set of cases and count how often they are wrong on the same case. If the voters' errors overlap almost completely, replace two of them with something structurally different: a rule check, a retrieval from a different source, or a person.

Retry storms and the cascade loopHow a slow dependency becomes an outage

A cascade has a feedback loop: slowness causes timeouts, timeouts cause retries, retries cause load, load causes slowness. Each layer that retries multiplies the load: if three layers each retry once, one slow call can become eight. The example limits each caller to one retry and backs off between tool retries (RETRY_BACKOFF_MS = 500), and its deadlines nest (the hierarchy's supervisor waits 90 seconds, longer than the lead's two 30-second attempts), so only one layer retries at a time.

The general rules: retry at one layer only, usually the one closest to the failing dependency; add jitter to backoff so retries do not synchronize; and stop retrying when a circuit breaker says the dependency is down. Retries are only safe at all if the operation is idempotent (Reliability).

Emergent behavior is a testing problemGroup failures appear only in group tests

Every failure on this page passes every unit test of every agent. A deadlock needs two agents; a cascade needs load; an echo needs a chain. So group behavior needs group tests: runs of the whole system over fixed cases with injected faults, the way the example runs twelve cases in three topologies with duplicates, drops, delays and silent agents. Those runs belong in CI next to the unit tests. Measuring their results over many trials, with a release gate, is the subject of eval.sammuthu.com.

In one breath#

Three agents that agree prove little if they share a model, instructions and evidence; agreement counts only when errors are independent. Groups of agents fail together through deadlocks, livelocks, cascades and echoes, and each is broken by a limit: deadlines, caps on hops and retries, fallbacks, and citations to sources. Every extra message has a cost, and isolation, separate scopes, budgets, bulkheads and quarantine, keeps one agent's failure local.

This is the last chapter. The Glossary collects every term the site defined, and About lists the sources, what is original here, and the sister sites of the series.