Designing a stack you can pause in under five minutes

Key takeaways

• Every agent in the stack should have a single pause signal it respects — a feature flag, an environment variable, or a kill switch endpoint — checked at the start of every task run.
• A pause is not a shutdown: paused agents stop accepting new tasks but complete their current task to a safe checkpoint before halting, protecting data integrity.
• We drill the pause quarterly: a planned "kill switch test" during a low-traffic window proves the system stops cleanly and restarts without data loss — an untested pause is a hypothesis.
• The longest we should need to pause all agents from a phone: five minutes. If it takes longer, the architecture has a single point of human-attention failure.

A bad prompt slips through review on a Friday afternoon. Your scheduler fires the same job twice. A client calls at 11pm because something is visibly wrong on their storefront — prices updating, emails going out, inventory moving — and you need it to stop right now. Not in the morning. Not after you find your laptop, connect to VPN, and SSH into a server. Now.

This is the scenario that exposes whether your agentic system has a real emergency stop or just the idea of one.

Here is how to build the real thing.

The global feature flag

Every agent in your system checks a single flag before it picks up a task. If the flag is false, the agent skips pickup, logs the skip, and sleeps. That is the entire mechanism. The power is in the universality: every agent, every time, no exceptions.

The flag itself can live anywhere — an environment variable, a feature flag service like LaunchDarkly or Unleash, or a single row in a database table. The implementation does not matter much. What matters is that the check is the first thing in the task loop, before any external call, before any state mutation.

Pseudocode — top of every agent task loop:

def run_task_loop():
    while True:
        if not get_flag("agents.enabled"):
            log.info("agents.enabled=false — skipping pickup")
            sleep(POLL_INTERVAL)
            continue
        task = queue.pickup_next()
        if task is None:
            sleep(POLL_INTERVAL)
            continue
        execute(task)

The log line matters. When you flip the flag at 11pm and then check your logging dashboard, you need to see every agent confirming it received the signal. Silence is not confirmation — it might mean the agent is mid-task, or it might mean the agent is not checking the flag at all.

A database row works well for small teams because it is easy to flip from anywhere, including a phone. A feature flag service adds audit trails and percentage rollouts you probably do not need for an emergency stop. An environment variable requires a process restart, which defeats the purpose. Use the database row.

Graceful checkpointing

The flag stops new task pickup immediately. But some agents are already mid-task when you flip it — halfway through a multi-step workflow, partway through writing a batch of records, three API calls into a five-call sequence.

Forcing an immediate halt on those agents creates a different problem: partial writes, inconsistent state, tasks that are neither complete nor cleanly rolled back. The solution is checkpointing.

An agent that is mid-task when the flag flips should complete to the nearest safe checkpoint, write its state, then halt. It does not abandon the task. It does not push through to completion. It finds the next natural boundary and stops there.

In practice, a checkpoint looks like this:

• A defined point in the task where all writes so far are internally consistent
• A state record written to durable storage: { task_id, step, payload, status: "paused" }
• A log entry confirming the checkpoint was written
• The agent releasing the task back to the queue with a paused status rather than marking it failed or complete

The key design question is: where are your safe checkpoints? For a product-sync agent, it might be after each product is fully written. For a campaign-build agent, it might be after each ad group is committed. You define these boundaries when you build the agent, not when the incident happens.

Agents that have no checkpoints — that treat the entire task as atomic — are the ones that cause the most damage in an emergency. If you cannot interrupt them safely, they will run to completion regardless of the flag. Audit your longest-running agents and add checkpoints now, before you need them.

Phone-accessible control surface

The flag is only useful if the person who needs to flip it can flip it from wherever they are. At 11pm, that person is probably on their phone.

A Slack bot command is sufficient. /pause-all flips agents.enabled to false. /resume-all flips it back. The bot posts a confirmation with a timestamp and the identity of who triggered it. That is your audit trail.

If you do not have a Slack bot, a simple authenticated webhook endpoint works. A bookmarked URL on your phone that hits POST /admin/agents/pause with a pre-shared token. Ugly, but functional. The point is that the action takes ten seconds, not ten minutes.

Do not gate this behind a VPN if you can avoid it. The one time you need it, you will not have easy VPN access. Put it behind strong authentication instead — a long token, a TOTP code, whatever your threat model requires — but make it reachable from the public internet.

What pause does not solve

Pause stops new damage. It does not undo damage already done.

If an agent wrote 200 records to an external CRM before you flipped the flag, those records are there. If it sent 50 emails, those emails are sent. Pause is the last resort, not the first line of defence.

The first line of defence is classifying your tool calls by reversibility before you build them. Every tool call an agent can make falls into one of three categories:

• Reversible: can be undone cleanly (delete a draft, roll back a DB write)
• Partially reversible: can be corrected but with side effects (update a record that was already read by another system)
• Irreversible: cannot be undone (send an email, charge a card, post to a public API)

Irreversible calls should require explicit confirmation steps, rate limits, or human-in-the-loop gates. They should never be the default path.

The outbox pattern extends this: instead of calling an external system directly, the agent writes its intended action to an outbox table. A separate process reads the outbox and executes the calls. Pausing the outbox processor is safe and immediate. The agent's work is preserved; the external calls have not happened yet. This is the architecture that makes pause genuinely effective rather than just damage-limiting.

The quarterly drill

A pause mechanism you have never tested is a pause mechanism you cannot trust.

Schedule a drill once per quarter during a low-traffic window — early Sunday morning works. The drill has four steps:

1. Flip agents.enabled to false
2. Wait one full poll cycle, then check logs — confirm every agent logged a skip
3. Flip agents.enabled back to true
4. Confirm agents resume cleanly — tasks picked up, no duplicate execution, no stuck paused records

Any agent that did not log a skip in step 2 is a P1 fix. It means that agent is not checking the flag, which means it will not stop in a real emergency. Fix it before the next drill.

Log the drill results: date, who ran it, which agents passed, which failed, what was fixed. This is the kind of operational record that matters when something goes wrong at scale.

The two-minute version

If you only do one thing: put a global feature flag check at the top of every agent's task loop. Check it before any external call. Log every skip. Make the flag flippable from your phone.

Checkpointing, the outbox pattern, the reversibility taxonomy — all of it is refinement on top of that single primitive. But without the primitive, none of the refinement matters. You will be the person at 11pm with no way to stop what is running.

Frequently asked questions

How do you build a kill switch for an AI agent stack?

Set a global feature flag — an environment variable or a flag service entry — that every agent task loop checks at the top of each cycle before picking up work. If the flag is off, the agent exits cleanly. Expose that flag through a phone-accessible control surface, such as a Slack bot command, so you can halt the entire stack in under 30 seconds from anywhere without touching a server.

What is the difference between pausing and shutting down an agent?

Pausing stops an agent from picking up new tasks but lets the current task run to a safe checkpoint before halting. Data integrity stays intact and the agent can resume without manual cleanup. Shutting down kills the process immediately, which risks partial writes, corrupted state, and orphaned queue items. Pause is the safe, reversible option for most situations; shutdown is the emergency break-glass when pause is not fast enough.

How often should you test your agent kill switch?

Quarterly is the minimum. Schedule a planned kill switch drill during a low-traffic window, verify every agent halts cleanly, confirm restart is clean with no data loss, and log any agent that did not respect the signal as a P1 fix before the next sprint. An untested kill switch is a hypothesis, not a control — and you do not want to discover it is broken during an actual incident.