The prompt works. In the demo, the agent reads the support ticket, drafts the refund, posts the reply, and closes the loop in about nine seconds. Everyone nods. Then someone asks the question that quietly stalls most agent pilots: what happens the first time it does that for a customer who should never have gotten a refund, and who catches it before the money leaves the account?
That question is not about the prompt. The prompt was fine. It is about everything the prompt never described - what the agent is allowed to touch, when it should stop and ask, what gets recorded, and who is accountable when the output is wrong. A prompt is an instruction for a single turn. An operating procedure is how something runs safely on the hundredth turn, the ten-thousandth, and the one that goes sideways at 2 a.m. Agents ship with the first and almost never with the second - the written procedure and the logged record it leaves behind.
Operations leaders already have a name for the second thing. Site reliability and IT teams have leaned on it for decades to run systems they cannot afford to break. It is called a runbook, and it is the missing layer between an agent that impresses in a pilot and an agent you would actually sign off on for production.
A prompt is not an operating procedure
A prompt shapes what an agent says or does in the moment. It sets the role, the tone, the task, maybe a few guardrails in plain language. That is genuinely useful, and it is also the entire extent of what most teams have written down.
Here is what a prompt does not cover, and what an operator needs before a system runs against real customers and real data:
- Permissions - which systems the agent may read, which it may write to, and which actions can never be undone.
- Escalation - the specific moment the agent must stop and hand off to a human.
- Logging - what every run records so a decision can be reconstructed later.
- Definition of done - the measurable standard for a good result versus a bad one.
- What to do when unsure - the safe default when confidence is low or a tool fails.
None of that fits naturally in a prompt, because a prompt is written for the model. These decisions are written for the operation - the people who own the process, the risk, and the sign-off. Confusing the two is the reason so many pilots feel finished and still are not safe to run.
What an agent runbook actually is
An agent runbook is the documented, repeatable operating procedure for a specific agent doing a specific job - the standing rules for how it behaves across every run, not just the next one.
The idea is borrowed on purpose. In site reliability practice, a runbook is the written procedure for a known operational task, so that any qualified person can execute it the same way under pressure (Google SRE). Vendors like BMC and SolarWinds define it the same way: a runbook turns an expert's judgment into a repeatable, auditable process. Port that discipline onto an autonomous agent and the payoff is identical. The agent stops being a clever one-off and becomes a system you can reason about, test, and hold to a standard.
It helps to separate three terms that get used interchangeably. A prompt is the instruction for one turn. A runbook is the operating procedure for one agent and one task. A playbook is the broader strategy for a class of situations, closer to policy than procedure. You need all three, but the runbook is the one almost nobody has written, and it is the one production depends on.
The 8 parts of an agent runbook
Every agent runbook worth the name answers eight questions. Cover each one explicitly and you have a spec your team can build against, test against, and audit against. Leave any of them implicit and you have a demo with good intentions.
| Component | What it answers | Example |
|---|---|---|
| Inputs | What data and context the agent is allowed and required to work from | A support agent reads the ticket, order history, and refund policy - nothing from the CRM notes field |
| Permissions | Which systems and actions are in scope, and the blast radius of each | Read all orders, issue refunds up to $200, never edit the ledger |
| Fallback | The safe default when the agent is uncertain or a tool fails | On low confidence or a timeout, take no action and route to the queue |
| Escalation | The explicit trigger and path to a human or another system | Any refund over $200, or a second contact on the same order, goes to a human |
| Logging | What every run records so a decision can be reconstructed | Inputs seen, action taken, confidence, and the policy clause cited |
| Success criteria | The measurable definition of a good result | Correct refund decision on 95% of sampled cases, zero policy violations |
| Failure handling | What counts as failure and what the agent does when it happens | A wrong decision is flagged, reversed if possible, and added to the review set |
| Human review mode | Whether a person approves before action or monitors after | In-the-loop for the first month, on-the-loop once the error rate holds |
Two of these are the ones teams skip, and they are the two that decide whether the agent is safe. Permissions are not one setting - they are a graded scale from harmless reads to irreversible writes, and the runbook should name the ceiling for each. Human review mode is a real design decision, not a checkbox: human in-the-loop means a person approves before the agent acts, while human on-the-loop means the agent acts and a person can step in. Choosing between them, and knowing when to move from one to the other, is exactly the kind of operating call a runbook exists to make explicit.
Why production-readiness fails without one
Demos are forgiving. The inputs are clean, the path is happy, and the person running it knows exactly what it is supposed to do. Production is none of those things. It has real permissions, real edge cases , and a real customer on the other end who did not read the script.
The cloud vendors building agent platforms describe the failure pattern in the same terms. AWS's guidance on operationalizing agentic AI (2025) frames the core challenge as non-deterministic behavior - agents that make different decisions on similar inputs, that are hard to reproduce and harder to debug, and whose costs can spiral without a ceiling. None of those are prompt problems. They are operating problems, and a prompt has no place to put an answer to them.
That is why a pilot can look complete and still fail its first production review. The reviewer is not asking whether the agent is smart. They are asking who it can act for, what it may touch, when it stops, and who catches the mistake . Those are runbook questions. A team that cannot answer them does not have a production-readiness gap in the model. It has a missing operating procedure, and no amount of prompt tuning fills it.
Runbooks are how you get quality gates for agents
You cannot gate what you have not defined. A quality gate - the automated check that decides whether an agent is good enough to ship or keep running - needs a standard to check against. That standard is the runbook's success criteria, its failure handling, and its logging. Without them, an evaluation has nothing to measure and a policy control has nothing to enforce.
This is the practical link between the runbook and the tooling that vendors now sell. Amazon Bedrock AgentCore, for example, added quality evaluations and policy controls for agents in preview in late 2025 (AWS, 2025). Those features are enforcement machinery. They still need a human-readable specification of what "good" means and what the agent may do - which is precisely what the runbook is. The runbook is the source of truth. The platform is how you enforce it at scale.
Read in that order, the sequence is clear. First you define the operating procedure. Then success criteria become evals, permission ceilings become policy controls, and logging becomes the audit trail. Skip the definition and you are buying enforcement for a standard nobody wrote down.
Two runbooks, side by side
The anatomy is easier to trust when you see it filled in. Here are two short, realistic examples for the readers most likely to own this decision.
A customer support refund agent (the Support and Ops reader):
- Inputs: the ticket, order history, and the current refund policy.
- Permissions: issue refunds up to $200. No access to the ledger or to customer payment methods.
- Escalation: any refund above $200, any second contact on the same order, or any fraud signal routes to a human agent.
- Human review mode: in-the-loop for the first month, moving to on-the-loop once the sampled error rate stays under 2% for a full month.
A back-office invoice-matching agent (the COO reader):
- Success criteria: matches invoices to purchase orders and receipts with a verified accuracy above the current manual baseline, zero silent mismatches.
- Failure handling: any unmatched or ambiguous invoice is set aside, never force-matched, and queued for a person.
- Logging: every match records the documents compared, the fields that agreed, and the confidence, so finance can audit any line later.
Notice that neither example is about the model. Both are about the operation around it - the ceiling on what it can do, the trigger for a human, and the record it leaves behind. That is the part you can hand to a team, and the part a reviewer will actually sign.
Common mistakes when teams skip the runbook
The failure modes are consistent enough to name in advance. Most teams that stall on the way to production hit some combination of these.
- Prompt-only governance. Rules for behavior live in the prompt and nowhere else, so nobody outside the model can see, test, or enforce them.
- No escalation path. The agent has no defined moment to stop, so it either acts when it should not or a human discovers the problem after the fact.
- Logging added after the incident. The first serious mistake is also the first time anyone realizes there is no record to reconstruct what happened.
- Rubber-stamp review. "Human review" exists on paper, but the human approves everything without real inspection, which is not oversight - it is latency.
- Write-once thinking. The runbook is drafted, filed, and never revisited, even as the process, the tools, and the policy change. Runbooks rot the same way SRE runbooks do (BMC), and a stale one is worse than none because it is trusted.
Each of these is cheap to prevent and expensive to discover. A runbook is where you prevent them, on purpose, before the agent runs against anything real.
How to start: your first agent runbook this quarter
This is not a research project, and it does not need a committee. Pick one agent that is already in a pilot - the one closest to touching real work - and write the eight sections for it. Be specific. Name the actual systems, the actual dollar or risk ceiling, the actual escalation contact, and the actual definition of a good result for that one task, and keep it in a shared, versioned document so anyone on the team can read, test, and challenge it.
That first filled-in runbook becomes your template. The second agent takes a fraction of the time, because you are adapting a working model instead of inventing one. Within a quarter you can move from "we have some prompts" to "we have a defined, auditable operating procedure for every agent in production" - which is the exact thing a production review is looking for.
If you want that operating layer built and pressure-tested with your own team rather than starting from a blank page, that is the work an AI Transformation Discovery Sprint is designed for - scoping the runbook, the quality gates, and the human-review model for a real agent in your environment. And when the hard part is capacity rather than clarity, a Fractional Agentic Team can own the human-in-the-loop and the ongoing maintenance so your runbooks stay current instead of rotting.
Key takeaways
- A prompt is an instruction for one turn. A runbook is the operating procedure for every run - and production depends on the second, not the first.
- An agent runbook answers eight questions: inputs, permissions, fallback, escalation, logging, success criteria, failure handling, and human review mode.
- There is no quality gate without a defined success and failure criterion - the runbook is the specification your evals and policy controls enforce.
- Human review is a designed mode, in-the-loop or on-the-loop, not an afterthought bolted on after the first mistake.
- Start with one agent already in a pilot, write its eight sections, and use that as the template for the rest.
