Agent Reliability Starts With Idempotent Tools and Checkpoints
Agent failures rarely look dramatic in logs. More often, they look like an innocent retry, a duplicated side effect, or a workflow that forgets where it was after a timeout.
That is the important mental shift. Once a model can call tools, wait on external systems, and resume work later, you are no longer operating a clever chatbot. You are operating a small distributed system with a language model in the control plane.
Why tool-using agents fail differently
Reasoning quality still matters. But many production failures have less to do with intelligence than with execution semantics.
A tool-using agent can:
- submit the same write twice after a timeout
- retry a partially completed action without knowing which steps already landed
- lose intermediate state between approval gates or long waits
- continue planning from stale observations after the world has changed
This is precisely why benchmarks like τ-bench are useful. The hard part is not merely producing a plausible next token. It is interacting with users, APIs, and policy constraints consistently over multiple turns.
Idempotency is the missing primitive
If I could add one discipline to every agent stack tomorrow, it would be idempotency.
AWS has written for years that retries become much safer when an API accepts a unique request identifier and guarantees a semantically equivalent result for duplicate requests. That pattern maps almost perfectly onto agent tools. If the model calls create_invoice, send_message, or provision_vm, the runtime should be able to say: this action with this request key has either not happened, or it already happened and here is the canonical result.
What an idempotent tool contract should include
For write-capable tools, require at least:
- a caller-supplied operation ID or client token
- a defined deduplication window
- a stable success response for duplicate submissions
- clear distinction between safe retry, unsafe retry, and unknown state
Without that contract, the model is forced to guess whether it should retry. Models are quite fluent at guessing. That is not the same thing as being correct.
Checkpoints beat giant prompts
Many teams respond to reliability problems by expanding the system prompt. This is understandable. It is also frequently the wrong layer.
If an agent run can span minutes, approvals, or multiple external calls, the system needs durable checkpoints more than it needs another paragraph of instructions. Temporal’s durable execution model is a useful reference here: persist progress, replay deterministically, and resume from the last completed step instead of improvising after a crash.
Good checkpoints capture operational state, not just text
A useful checkpoint records:
- the current plan step
- the tool inputs that were attempted
- the tool results that were confirmed
- pending approvals or human decisions
- the observation timestamp or version the plan was based on
This matters because a transcript alone is ambiguous. “I already sent the email” is not a reliable state record. A durable event saying email.send(op_id=abc123) -> delivered is.
Budgets and stop conditions are not optional
Agents degrade when they are allowed to wander.
ReAct-style loops are powerful because they interleave reasoning and acting, but the same loop can spiral into waste or damage if it lacks hard limits. In production, every run should have explicit budgets and termination rules.
Minimum controls I would enforce
- max tool calls per run
- max retries per tool and per operation ID
- max wall-clock runtime
- explicit escalation path for unknown state after partial failure
- mandatory human approval for irreversible or external side effects
Anthropic’s guidance on building effective agents argues for simple, composable patterns over baroque autonomy. I agree. A small loop with strict boundaries is usually more reliable than a sprawling one with better rhetoric.
Multi-agent systems multiply the problem
In a multi-agent design, failure modes compound.
One planner may delegate a task, a worker may retry a tool, and a reviewer may validate against an observation that is already stale. If those agents do not share operation IDs, state checkpoints, and ownership boundaries, you get duplicate work at best and contradictory side effects at worst.
Rules that keep multi-agent systems sane
- assign one agent as the owner of each side effect
- propagate the same operation ID across delegations
- separate planning messages from committed state transitions
- treat handoffs as checkpoints, not casual chat
- make reviewers validate postconditions, not just prose quality
Bottom line
Reliable agent systems need more than better prompts. They need the same control surfaces that make distributed systems survivable: idempotent writes, durable checkpoints, bounded retries, and explicit stop conditions.
If your agent can act in the world, design its tools as though a timeout, duplicate request, or crash is inevitable. Because it is.