How to write an operations runbook for custom software
After custom software is in production, the operations runbook names who answers this shift, where the logs and dashboards are, the first check for each known failure, and when the person on duty calls the next person or follows the rollback plan. The handover names who holds the repository. The rollback plan says how a release is reversed. This page is what the person on duty does before either of those.
On this page
- Keep the runbook apart from handover and rollback
- Name who answers this shift, and who is next
- Write the company's own response window
- Put logs and dashboards where the alert can open them
- Give each known failure a first check
- Say when to call the next person and when to roll back
- Page only on alerts that have a first step
- Hand the runbook to the shift when names, paths, and routes are written
Keep the runbook apart from handover and rollback
The handover says who holds the system after go-live, where the repository and the permissions are, and which list still carries the open defects. The rollback plan says which visible symptom can start a reversal, who decides, and whether this reversal restores data or turns a feature flag off. The runbook answers a different question: once production is in use, who answers this shift, where the logs and dashboards open, what the first check is for a known failure, when to call the next person, and when to follow the rollback plan that is already written. The three documents can point at one another's versions. The handover roster stays a roster of who holds the system. The rollback triggers stay on the rollback plan. See How to prepare a custom software project handover and How to write a rollback plan for custom software go-live.
The order of work on go-live night stays in the cutover runbook. Whether the specification changes stays in the change record. See How to cut custom software over to production and How to control scope changes in custom software. Delivery still sits under Custom software, websites, and applications. The runbook cover carries its own version and the day it takes effect, and it points at the handover version and the rollback-plan version. When the three versions do not match, the person on duty reads the version named on the cover. The versions are not assembled during the call.
Name who answers this shift, and who is next
Read on 26 October 2026, Google SRE: Being On-Call is written by Andrea Spadaccini and edited by Kavita Guliani. The page shows no publication date. A note at the end says an earlier version appeared in ;login:, October 2015, volume 40, number 5. The page describes on-call as being available for calls during working and nonworking hours. Once a page is received and acknowledged, the on-call engineer triages the problem and works toward resolution, involving other people and escalating when needed. Many teams keep a primary and a secondary. Some use the secondary as the person who takes a page the primary misses. Some have the primary handle only pages, while the secondary handles other production work that is not urgent. Two related teams can also stand as secondary for each other.
The runbook therefore fills two names for a window with a start and an end: who answers first on this shift, and who is called when that person cannot be reached. One person may fill both lines. The lines stay separate. A line that says only "engineering" does not tell the next caller whom to reach. If the person is away that day, the deputy is already written on the window. At the start of the shift, the person on duty reads what the previous shift left. At the end of the shift, unfinished work is written for the next shift, with the date. Read on 26 October 2026, Google SRE: On-Call describes that handoff as one team's practice, reviewed before the team went on call: read the previous handoff at the start, reduce the effect on users first, and write the handoff for the next person at the end. The page shows no publication date. The authors are listed as Ollie Cook, Sara Smollett, Andrea Spadaccini, Cara Donnelly, Jian Ma, and Garrett Plasky (Evernote), with Stephen Thorne and Jessie Yang. That is one team's example in the chapter. The custom runbook keeps "read at the start, write for the next shift at the end". The company chooses the message format and dates it.
Write the company's own response window
The Being On-Call page says the person on duty works on production within the response time the team and the business owners have already agreed. The page then gives Google's own figures for response time, an availability example, the share of engineering time, how many people are on the rotation, and how many incidents a shift should carry. Those figures are the page's account of Google. The custom runbook writes the company's own window on the cover: which hours this shift covers, how soon after a page someone acknowledges it, and who agreed that window on which day. A window without a date is not filled with the page's example.
The same page says the most important on-call resources are a clear escalation path, a written incident procedure, and a review after the fact that looks at events rather than at people. The runbook keeps those three in fixed places: the escalation path later on this page, the first step of an incident in the failure entry, and the later review on its own list with a path. The page does not require a custom project to staff a separate operations rota, and it does not require a second site. The cover carries only the company's window and the day it was agreed. Google's figures stay on that page. A reader who wants them opens the page.
Put logs and dashboards where the alert can open them
The On-Call chapter says a page should link to the relevant monitoring console, the console should show where the system is outside its stated range, and alerts and graphs should be read together. For each kind of alert, the runbook records three things: the path or the query for the logs, the screen for the dashboard, and which issued account the person on this shift uses to open them. The path is specific enough that opening it shows this system's production environment. The product name stays with the set the company already runs. This page does not rank dashboards.
If the account cannot open the path, the line is not yet written. Permissions stay with the set the handover already gave the person on duty. Someone else's login is not shared in the middle of a failure. How the handover records the repository and the permissions is in How to prepare a custom software project handover. If the dashboard is meant to match a release, the version is the one already written on the rollback plan and the cutover runbook. If the screen opens and it is not that release, the entry names which environment it shows.
Give each known failure a first check
The On-Call chapter describes a playbook as high-level instructions for an automated alert: severity, impact, debugging suggestions, and actions that can reduce the impact and finish the response. The chapter says that when an alert is created, a matching entry is usually created with it, and that these guides reduce stress, the time to repair, and human error. That is the chapter's claim. The runbook writes each known failure as one entry: the name of the alert, the symptom a user can see, the first check as a log or a graph that opens, and what should change in that symptom after the check. The first check is a place another person can open.
Entries go stale at the same pace as production. The chapter says playbook detail goes out of date as fast as the production environment changes, and that a daily release can need an update on any day. When the steps do not match the system, the person on the shift edits that entry and writes the date and which sentence changed. Editing the entry does not edit a rollback trigger. Triggers stay on the rollback plan. The chapter also says that if an entry is a fixed list of commands run every time an alert fires, that list should become automation rather than a longer script. A fixed command is therefore written as the name of the automation, who may start it, and the first check if it fails. If there is no name yet, the cell says "not automated yet" and names who will make it so.
Say when to call the next person and when to roll back
Being On-Call says that when an issue is complex enough to involve more than one team, or when investigation still cannot put an upper bound on how long the incident will last, it is useful to switch to a written incident procedure. The runbook puts that sentence on names: after the first check, if the symptom is still there and another team has to continue, or if nobody can yet say roughly how long it will take, the person named as next on the window is told, and the time of that notice is written down. That name is already on the window. It is not chosen during the call.
Rollback is a separate route. When the symptom matches one the rollback plan already lists as able to start a reversal, the person on duty follows that page: who announces the reversal, whether this one restores data or turns a feature flag off, and who is told after the decision. The runbook only points at that version of the plan. See How to write a rollback plan for custom software go-live. If the symptom matches neither a rollback trigger nor the condition for calling the next person, it stays on the first check, and the handoff says it is unfinished. A spoken "it seems unstable" is not written down as an escalation or as a rollback.
Page only on alerts that have a first step
Being On-Call says paging alerts should line up with symptoms that threaten the service objective, and that every paging alert should be something the person who receives it can act on. A low-priority alert that sounds often and has no matching action trains the person on duty to give the same light attention to an alert that does need work. The same page says one abnormal condition should not fan out into repeated pages. Related alerts are grouped first, so the person on duty handles the one incident.
The runbook therefore keeps a page only for a symptom that has a first check. An alert with no entry does not go to this shift's pager. It is written as a record someone looks at during the day, and the runbook names who will add the entry. The service objectives and the cap on pages per shift on that page are Google's own wording, and they stay on that page. If the company writes a service objective for this system, the figures and the date go on a separate sheet, and the runbook only points at where that sheet is. Until the sheet exists, the runbook does not fill in an availability figure.
Hand the runbook to the shift when names, paths, and routes are written
If the person who answers first, the person who is next, the start and end of this shift, the log and dashboard locations, at least one first check for a known failure, the condition for calling the next person, or the condition for following rollback is blank, this copy is not yet the one the person on duty uses. The status is written as runbook still to be filled. The system owner on the handover is not treated as this shift's responder unless both lines carry the same name and the day it takes effect.
Before the copy is given to the person on duty, three things are on it: both names and the dates for this shift, a path for the logs and the dashboards that the alert can open, and a condition that already exists for escalation and a condition that already exists for rollback. When those three are there, the runbook is marked as the version this shift uses. After the version changes, the next shift reads the new one before it starts. A step the previous shift described aloud is followed only when the dated line on the runbook says the same thing.