Every agent in our stack obeys one law before any other: running it twice must produce the same result as running it once. Not a similar result. The same result — one email, one invoice, one published post.
This property has a name, idempotency, and it is the least glamorous, most load-bearing idea in agent engineering. Agents crash mid-task, hit timeouts, get retried by schedulers, and get re-run by humans who are not sure the first run worked.
If your actions are not safe to repeat, every one of those ordinary events becomes an incident. Here is how we build for it at Avakata.
What idempotency means for an agent
An action is idempotent when performing it twice leaves the world in the same state as performing it once. For agents, that means a retried workflow does not send a second email, create a duplicate CRM record, or publish the same post twice. The first run changes the world. Every repeat recognizes the work is already done and does nothing.
The word comes from mathematics: an operation that can be applied repeatedly without changing the result beyond the first application. Pressing an elevator call button is idempotent. Sending an email is not, unless you build it to be.
Databases and payment processors solved this decades ago. Agent stacks keep relearning it, because language models make side effects feel casual — one tool call and an email is gone.
The model does not need to understand any of this. The tools it calls enforce it.
That division of labor is the theme of everything below: intelligence in the model, safety in the plumbing.
Why agents run twice more often than you think
Duplicate execution is the default condition of an agent stack, not a rare failure. Schedulers fire twice after a missed heartbeat. API calls time out after the server already did the work, so the client retries. Orchestrators resume a crashed workflow from the top. And humans re-run jobs manually because the dashboard looked stuck. We logged 41 duplicate executions in June alone.
Of June's 41, zero produced a duplicate side effect. That is not luck. Two years ago a retried invoicing job billed a client twice, and the phone call that followed is why this post exists.
The categories repeat everywhere: double-fires, retries after timeouts that actually succeeded, resumed workflows, and manual re-runs born of ambiguity.
You cannot prevent double execution. You can only make it harmless.
Give every action a deterministic key
The core mechanic is the idempotency key: a string that identifies the intent of an action, derived from stable facts rather than the moment of execution. 'weekly-digest-2026-07-24-client-17' names one specific send. Before executing, the tool checks a ledger for that key. Present means skip. Absent means execute, then record. Two runs, one email.
The key must derive from the work, not the run. Hash the stable inputs — recipient, template, billing period. A timestamp or a random run ID in the key defeats the entire mechanism.
Ours live in a single Postgres table: key, action type, result, created-at. About 40 lines of code guard the whole stack.
Name keys so a human can read them. Debugging 'weekly-digest-client-17' beats debugging a hash.
Check state before you change state
The second mechanic is read-before-write: the agent verifies the world's current state before modifying it. Before publishing, check whether the slug already exists. Before creating the task, search for it. Before updating the record, fetch it and compare. If the desired end state already holds, stop. This turns every 'create' operation into an 'ensure' operation, which is the idempotent frame.
Upserts are the database form of the same idea, and we use them anywhere an API offers one. Create-or-update-by-key cannot duplicate.
The trap is compound actions. 'Ensure the post exists' plus 'ensure the announcement exists' must each check independently, because a crash can land between them.
Read-before-write also catches a second failure class for free: acting on stale assumptions after a human changed something mid-run.
We timestamp every read as well. If the gap between check and write exceeds 30 seconds, the tool re-checks before committing. Cheap paranoia, twice justified this year.
Quarantine the actions you cannot undo
Some actions cannot be checked or reversed — sending an email, charging a card, posting to a social feed. These get the strictest pattern: an outbox. The workflow writes the intended action to a queue as data, commits it with its idempotency key, and a single dispatcher performs the send exactly once. Agents never touch the irreversible API directly.
The outbox buys a free review gate too. Anything in the queue can be inspected, edited, or deleted before dispatch. Ours holds outbound email for ten minutes as policy.
One dispatcher means one place to enforce exactly-once and one log of everything that ever left the building.
The delay has paid for itself. Refunds, apology emails, and correction posts all cost more than ten minutes of latency ever has.
If a tool call cannot be undone and cannot be checked, it does not belong in an agent's hands. It belongs in the outbox.
Make the whole workflow resumable
Idempotent steps are not enough if the workflow restarts from zero and repeats expensive work. We break every pipeline into named stages, each recording completion in the ledger. A resumed run replays the stage list, skips everything marked done, and picks up where the crash happened. Restart cost drops from the whole job to one stage.
This changed our economics more than our safety. A 40-minute research pipeline that dies at minute 38 now costs two minutes to finish instead of forty to redo.
It also makes the system braver. When resuming is cheap, workflows can fail fast instead of wrapping every step in defensive padding.
Stages should stay coarse. Five per pipeline is our norm — research, outline, draft, review, publish. Finer checkpoints add bookkeeping without adding much recovery value.
Stage names double as progress reporting. The dashboard is just the ledger, read aloud.
Test it by running everything twice
The test is brutally simple: run every job twice on purpose and count the damage. Once a month we re-fire the previous day's entire queue against a staging copy and diff the side effects. The pass condition is absolute — zero duplicate emails, zero repeated posts, zero double entries. Anything above zero is a bug with a name and a fix-by date.
Our first drill failed three of eleven workflows. The fixes took a week and have held since.
The staging copy matters. Running the drill against production tests your nerve, not your idempotency.
Put the drill on the calendar. Idempotency rots silently — a tool added in a hurry skips the ledger and works perfectly until the first real retry finds it.
Safe to run twice is really a trust property.
It is what lets you stop watching.