19. Operating a Server Without Fear
“Every backup is a rumour until someone has restored it.”
The story: the night GitLab lost a database
Section titled “The story: the night GitLab lost a database”On 31 January 2017, GitLab.com, the code-hosting service that a great many teams trust with their work, had one of the worst days in its public history. We teased this story back in Chapter 6. Here it is in full. I am working from memory and from the company’s own postmortem, so treat the exact figures as roughly right. The lessons are exactly right.
It started with load. Someone was abusing the site with spam, the extra writes pushed a database replica behind its primary, and the lag kept growing. A replica that falls behind is a nuisance, not a catastrophe, until someone decides to fix it by hand.
Late in the evening, an engineer who had been at it for a long day tried to rebuild the lagging replica. One step meant clearing a data directory. The engineer believed they were on the secondary. They were on the primary. Before anyone could stop it, roughly 300 GB of production data was gone.
Then came the part that turns an outage into a lesson. The team reached for their backups and found the safety net had holes in it. The regular database dump had been failing for a while, quietly, because the dump tool was a different version from the database server. The failure notifications that should have woken someone were not arriving. Disk snapshots of the production database servers had not been enabled. The object-storage copy, which everyone hoped would save the day, was empty. The only recent copy that worked was a snapshot of a staging database, taken around six hours earlier.
So they restored from that snapshot, and about six hours of activity vanished: some issues, merge requests, comments and user records. The Git repositories were largely intact, because they lived on separate storage and were not what got deleted. Users lost conversation, not code. The team worked through the recovery in public, streamed much of it live, and then published a postmortem so candid that it listed their own failed checks one by one.
Notice what did not happen. Nobody wrote an exotic bug. Every failure was a check that existed on paper and had never been run. The dump job reported success while producing nothing useful. The alert existed and nobody received it. The restore had never been tried from start to finish on a clean machine. That is the whole chapter in one night: a backup is a claim, a restore is the proof, and monitoring is how you learn the claim is false before the 2 a.m. call does.
Your own stack is smaller and easier to get right, which is good news. Your dance-academy app keeps its members, bookings and payment records in Postgres, and Razorpay webhooks write into it. If that database vanished tonight, answer three questions in one breath. What would you lose? How long would you be down? How would you know the copy you restored was any good? If you can answer all three, you are ahead of most teams. If you hesitated on even one, keep reading.
A word on scope. This is the operator’s slice of the material in Book 1. The observability chapter of The Glue Book explains what monitoring is. This chapter is about what you do on a Monday morning so the night above does not happen to you.
Backups: a copy that counts only once it restores
Section titled “Backups: a copy that counts only once it restores”Everyone has a backup. Most people have three, and one of them is a folder they forgot existed. The useful question is never “do I have a backup?” It is “what happens on the worst Tuesday of my year, and do I know the answer from experience?”
The rule that has survived decades of bad days is 3-2-1. Keep three copies of anything you cannot afford to lose. Put them on at least two kinds of storage. Keep one copy somewhere else entirely. Modern practitioners bolt on two more rules, and they are the ones people skip. Make one copy immutable, so nothing holding your server’s passwords can delete or alter it. And aim for zero errors, which means you have proven the copy restores, not merely that the file exists.
Why does the rule exist? Because the failures we actually meet do not spread evenly. A dying disk takes out one copy. A mistaken delete takes out the live data, and if the backup sits right beside it, the backup goes too. A compromised server can remove anything it is allowed to write to, so an attacker with root can wipe your local backups as thoroughly as your production data. A ransomware gang does not check which copy is the original. The off-site, immutable copy is the one that survives the day your server is the problem.
Picture the world without the rule. You have one VPS, a cron job that dumps the database into a folder on the same disk, and a comfortable feeling. It works until the disk fills, until the provider terminates an instance by mistake, or until the one person who knew where the dumps went leaves the team. If the dance-academy app runs today on a single VPS with only that setup, you are standing where GitLab stood, with a smaller audience to be embarrassed in front of.
For a small production box, the arrangement I would choose is boring on purpose.
- Copy 1 is the live database, on the server itself.
- Copy 2 is a nightly dump or a volume snapshot, kept on different storage from the live disk. Many providers sell snapshots as a button, and Chapter 4 showed you where that button lives. Press it.
- Copy 3 is an encrypted, deduplicated backup pushed off the server to object storage such as Backblaze B2, Cloudflare R2 or Amazon S3, using a tool like restic or borg. Both encrypt before upload, so the storage provider never sees plaintext. Both deduplicate, so daily backups do not multiply your bill.
The property that matters most in copy 3 is who holds the key. The server should carry credentials that can add new backups but cannot delete old ones. Many providers let you scope a key that way. As of the mid-2020s, most object-storage services also offer versioning or object lock, which gives you the immutable copy if you switch it on. Check your provider’s current option, because the names and limits change.
Dumps and snapshots answer different questions
Section titled “Dumps and snapshots answer different questions”A dump is a logical copy. pg_dump writes the schema and the data out as a file that you can read back into a database, on the same provider or a different one, and usually into a newer major version of Postgres. It is consistent as of the moment it started, it is small enough to inspect, and it restores one database at a time. The cost is time. A large database can take hours to dump and hours more to load.
A snapshot is a physical copy. The provider captures the disk, usually in minutes, and a restore brings back the whole machine, including the files you forgot were important. It is fast and it covers everything. The catch is that a snapshot is crash-consistent unless the database is quiet when it is taken. A snapshot mid-write is a bit like pulling the power cord. Postgres usually recovers from that cleanly, but “usually” is a word to test rather than trust. Snapshots also tie you to one provider’s format, which matters on the day you move.
For a small production VPS I run both. Dumps give you a portable, inspectable copy that restores almost anywhere. Snapshots give you the whole box back quickly, which is what you want when a bad upgrade has wrecked the machine and you want yesterday’s state in ten minutes. In the GitLab story, the snapshot that saved the day had been taken for an unrelated reason, and the dumps that were supposed to be the safety net had been failing for a long time without anyone noticing.
Point-in-time recovery
Section titled “Point-in-time recovery”A nightly dump is a photograph taken at 02:00. If the mistake happened at 14:10, that photograph loses twelve hours. Point-in-time recovery (PITR) closes that gap. The database keeps a continuous log of every change, and you keep a base backup plus that log. You can then replay to any moment, for example one second before someone ran a DELETE without a WHERE.
Many managed Postgres services offer PITR as a setting you switch on. On a self-hosted box you set up WAL archiving yourself, and it is one of those jobs that feels like overkill until the morning you need it. For the lab, a nightly dump is the honest starting point. PITR is the next rung. You should know it exists even if you do not climb it yet.
Restore drills, or the backup is a hope
Section titled “Restore drills, or the backup is a hope”Here is the sentence I would put on the wall: a backup you have not restored is a hope. Once a quarter, and after any change to the backup job, run a restore drill. Build a fresh VPS, restore the newest off-server copy onto it, start the app, and time every step with a stopwatch and a text file.
The first drill almost always fails somewhere. An environment variable is missing. The restore needs a Postgres version you did not install. The backup host cannot decrypt the archive because someone rotated a key last month and forgot to update the server. That failure is the point. A drill that breaks on a Tuesday afternoon costs you an afternoon. The same break at 2 a.m. costs you the afternoon and your sleep.
The number on your stopwatch is your real recovery time, not the one in the runbook. Write it down next to the date, and watch it change over the months.
RPO and RTO: the two numbers to write down
Section titled “RPO and RTO: the two numbers to write down”Two terms from operations sound grander than they are. The recovery point objective (RPO) is how much data you can afford to lose, measured in time. The recovery time objective (RTO) is how long you can afford to be down. Neither number is a property of your tools. Both are decisions, and you make them before the incident, because at 2 a.m. nobody has the calm to choose well.
Read the figure from left to right. Your RPO is the gap between the last good copy and the failure. Your RTO is the gap between the failure and the moment users can use the app again. A nightly dump gives you an RPO of about a day. A restore that takes ninety minutes, plus the time you spend deciding what to do, gives you an RTO that is longer than the restore alone.
Apply that to your own apps. A dance academy that takes Razorpay payments and confirms bookings by email cannot comfortably lose a day of bookings, so its RPO pushes you toward more frequent snapshots and, eventually, WAL archiving. An internal team-status tool used by eight people can probably live with a day, and it does not need the same machinery. Write both numbers down, and let them choose the tooling, not the other way round.
Monitoring: know before your users do
Section titled “Monitoring: know before your users do”A server that is down with no one telling you is worse than a server that is down with a loud alarm. Monitoring is the set of signals that turn “something is wrong” into a message you read in time.
Start with the outside view. An uptime check runs from a machine that is not your server, hits https://app.pipeslab.in/ every minute or so, and alerts you when it gets an error or no answer. This catches the failure that internal monitoring cannot see: the one where your server is fine but the path to it is broken. Remember the DNS and firewall chapters. A check from inside the box tells you the app runs. Only a check from outside tells you that people can reach it.
Next, the inside view, but keep it small. Watch three things: disk space, memory pressure and load average. Disks are the classic killer. A log file that grows for six weeks will fill the root filesystem, and then Postgres cannot write, and then nobody can log in to fix it. Alert at 80 per cent full, not 99.
Logs matter when they tell you something. You do not need to read every line. You need a rule that notices the lines that matter, such as a spike in 5xx responses in the reverse proxy, or repeated failed SSH logins from one address. Those rules belong in your alerting, not in your memory.
Heartbeats for jobs that run quietly
Section titled “Heartbeats for jobs that run quietly”Uptime checks cannot see a backup job, because a backup job does not serve web pages. For that you want a heartbeat. The job pings a URL when it finishes successfully, and a monitoring service alerts you when the ping does not arrive on schedule. Silence is the signal. A heartbeat is the only way to catch the job that stopped running three weeks ago without any error at all, which is exactly how the GitLab dump failure stayed hidden.
Certificates and domains expire on schedule
Section titled “Certificates and domains expire on schedule”Two things expire no matter how healthy your server is. Certificates from Let’s Encrypt last ninety days and renew automatically, until the day the renewal breaks, as you saw in Chapter 9. Domains are paid yearly, and Chapter 11 showed how a lapsed domain takes your whole site with it. Monitor both. A certificate-expiry check and a domain-expiry check cost nothing, and they remove the most embarrassing category of outage, the one where the server was perfect and the paperwork was not.
Alert fatigue is a security problem
Section titled “Alert fatigue is a security problem”An alert that fires every night gets muted, and a muted alert is a dead alert. The rule I use is simple: page a human only for something that needs a human now. A site that is down, a backup that did not run, a certificate with two days left. Everything else goes into a daily digest you can read with a cup of chai. Doorknob-rattlers, the bots that probe every server on the internet, are weather. They should not wake you up. If your alert channel is full of them, tune the rule until the channel carries only things you would act on.
Patching: the boring habit that keeps you alive
Section titled “Patching: the boring habit that keeps you alive”It is a Tuesday morning. A security advisory lands for the web server in front of your app, and the post is climbing the feeds. You know you should patch. You also remember that the last update you applied broke something for an hour, and you cannot quite recall what. That is the patching dilemma in one sentence. Leave the server alone and you carry holes that are now public. Touch it carelessly and you break a house people are living in.
Patching means applying updates to everything between the hardware and your code: the kernel, the operating system libraries, the Node runtime, the database, the proxy and Docker itself. Ubuntu can install security fixes on its own, and for a small production VPS you should let it. The rest, you choose.
Why does this matter more than it used to? Servers once ran for years untouched, and the people attacking them were slower. Today, scanners find newly published holes within hours. A widely reported breach in 2017 came through a web framework flaw for which a fix had been out for roughly two months. The fix existed. The process to apply it did not reach every server.
A cadence you can keep
Section titled “A cadence you can keep”Here is the rhythm I recommend for a small production box run by you or a small team.
- Security updates install automatically, every day. This is the one place where automation beats memory.
- Full package upgrades happen once a month, in a window you pick and announce, and only after a snapshot.
- Reboots follow kernel updates. A patch that is installed but not yet booted is a patch you do not have. Ubuntu leaves a marker file when a reboot is needed, and a monitor can alert on it.
- Containers and services come back after a reboot through their restart policies, and you check that they did.
The update that breaks things
Section titled “The update that breaks things”Most updates are fine. The ones that hurt share a shape. A major version changes behaviour on purpose: a new Node major, a new Postgres major, a new TLS default. A library you never knew you depended on gets bumped as a side effect. Or a tag you trusted, like latest, moves under you while you were not looking.
A composite story, not a real incident: a dance academy’s server took an automatic image update overnight. The payment webhook handler still ran, but a changed default in a dependency made signature checks fail, and bookings stopped being confirmed. Nobody was hacked, and nobody was careless on purpose. The fix was a rollback that took minutes, because there was a snapshot and the image tag was pinned. Without the snapshot, the same bug would have been an afternoon of archaeology.
The habits that make this boring are three. Pin versions, so updates happen when you say so. Rehearse the upgrade on a staging copy first. Take a snapshot before you touch the real box, because the snapshot is your undo button. Figure 19.3 puts the order in one line.
Postgres is the special case. A minor bump, such as one point release to the next within the same major version, is usually a restart. A major bump changes the on-disk format, so you either dump and restore or run a documented upgrade tool. The exact steps depend on the version pair, so read the release notes before you start, and rehearse on a copy of the database, not on the database itself.
Secrets, least privilege, and leaving the door the way you found it
Section titled “Secrets, least privilege, and leaving the door the way you found it”Picture a Tuesday. A former contractor, who helped with the dance-academy app for six weeks in 2023, pings the team chat to say hello. Nobody remembers that their laptop key is still in authorized_keys on the box that runs the webhook handler. Nothing bad happens that week. The story is still the one you should worry about, because the key was never a secret anyone tracked. It was just a line in a file.
Access on a server is a pile of small grants that nobody added on purpose. Most of this chapter is about making them deliberate.
Least privilege is a habit, not a setting
Section titled “Least privilege is a habit, not a setting”Give each thing the smallest set of permissions that lets it do its job. The app runs as app, a user that owns the app directory and nothing else. It does not have sudo. ops has sudo, because ops is the human who has to fix things at 2 a.m. The database user the app connects with can read and write its own tables. It cannot create roles, drop databases, or read the other apps’ data.
You will be tempted to hand the app root because the deploy script failed once with a permission error. Resist. The permission error is the system telling you the grant you thought you gave was not the one you needed. Fix the grant. Do not widen the blast radius.
Secrets live outside the code
Section titled “Secrets live outside the code”Your database password, your Razorpay keys, your Resend API key, and any webhook signing secret should sit in an environment file owned by the app user, with permissions that only that user can read. Keep that file out of git. Keep it out of the Docker image, because image layers are copied around and nobody remembers which registry they went to. Keep it out of logs, and check that your error handler does not print the whole request body, which sometimes includes a token.
Test keys and live keys should never share a file. A test key in production is a small mistake. A live key in a laptop repo is a large one.
Rotation needs a map
Section titled “Rotation needs a map”Rotating a secret is easy when you know what uses it. It is painful when you find out at midnight that the same database password is also in a cron script, a backup job and a staging .env nobody opened for a year.
Keep a plain list: the secret, the place it lives, the things that read it, and the date it was last changed. It can be a text file in your password manager. It does not need to be fancy. When you rotate, change the value in the provider, update every reader, restart what needs restarting, and then revoke the old value. Where the provider allows two valid values at once, such as some webhook signing setups, use that overlap window so the old value keeps working until the new one is live.
Shared secrets rotate together. If two apps share a database password, rotate it once and update both, or you will have an outage that looks like a network fault. Anything that looks like a network fault on a Tuesday, check the secrets first.
Offboarding a person from a server
Section titled “Offboarding a person from a server”People leave. Some leave on good terms, some do not, and most leave with a list of accounts that is longer than they remember. Offboarding is the same work either way. Treat it as a checklist, not a feeling.
authorized_keyson every host they ever touched. Grep for their key comment on each box. This is the Chapter 6 lesson: a key is a door, and the file is the list of doors.- Their device in the tailnet, and their user if you have one. Chapter 16 is where that lives.
- Their identity in any Cloudflare Access policy and the matching tunnel rules. Chapter 17 covers the doorman.
- The registrar and DNS panel. A former teammate who can change a record can take your domain for an afternoon. Chapter 11 is the reminder.
- Your cloud and VPS provider accounts, the team members list, and any API tokens they created.
- GitHub or GitLab org membership, deploy keys, and any personal access token they generated for CI.
- Payment and email dashboards: Razorpay, Resend, and the team password manager vault.
- Rotate every shared secret they could have read. Do not ask whether they copied it. Assume they could have.
None of this is glamorous. It is the work that makes the next incident smaller, and it is the same work whether the person was your best engineer or a contractor you barely met. Write the list once, keep it next to your hardening checklist, and run it every time the roster changes. You will be surprised how often the first version of the list is wrong, and how much better the second one is.
Cost: the bill is a monitor too
Section titled “Cost: the bill is a monitor too”Most small servers do not get expensive in one dramatic afternoon. They get expensive the way a dripping tap wastes water: slowly, and nobody reads the meter until the month feels wrong.
Four places leak most often:
- Outbound bandwidth. Many providers bill for data leaving the machine, the egress line, and let data coming in go free or nearly so. A download-heavy site, or a backup pushed to the wrong region, can turn a quiet month loud. Read the egress line on the invoice itself, not the headline price on the pricing page.
- Snapshots and backup storage. Chapter 4 introduced snapshots and idle reclamation. Snapshots are cheap insurance right up until you take one every six hours and keep every one forever. Old snapshots are the most common orphaned cost on an account. Write the retention rule down, and delete by the rule.
- Machines nobody remembers. The load-test box from March. The staging copy from a hackathon. A reserved IP attached to nothing. Idle machines are the cheapest thing to forget and the easiest thing to reclaim, once you have a list of what exists.
- Free tiers with a cliff. A free tier is a trial with no end date anyone told you about. It ends on the day traffic grows, or the day a service adds a second region. Read the limit, not the marketing page.
Here is the rule I would give a small team. Put two billing alerts on the account: one at half the monthly budget, one at the budget itself. Then look at the list of running machines once a quarter. A cost alert is a resource alert with a wallet attached, and it is the cheapest monitoring you will ever set up.
The lab: a backup you have to earn
Section titled “The lab: a backup you have to earn”Goal. On the lab box, prove that your backups work by restoring them onto a machine that did not exist a few minutes earlier, and by proving that you hear about it when the job breaks. Nothing in this lab touches production. The lab VPS is throwaway, and that is the point.
Conceptual steps.
- Put a small Postgres database with a handful of rows behind the hello-pipes app. Write down the rows, because you will look for them later.
- Schedule a nightly dump. Decide where the file lands and how it is named, so that the name alone tells you when it was taken.
- Ship the dump off the server with a tool that encrypts, deduplicates, and keeps a retention rule. Restic or borg both fit. Keep the repository key somewhere that is not the lab box, because a key stored beside the data is not a key at all.
- Build a second lab machine from nothing. Install only what the app needs. Start a stopwatch the moment you begin.
- Restore the latest copy, start the app, and check that your rows are there. Stop the stopwatch. That number is your RTO for this setup. Next to it, write the RPO you actually have: a nightly job means up to a day of data can vanish.
- Point an outside uptime check at the app’s public hostname. Add a heartbeat for the backup job. Add alerts for the certificate and the domain expiry dates.
- Break the backup job on purpose. Change its destination, or revoke its key, and wait past the heartbeat window. Then watch which alert arrives, and how long it took.
How you’ll know it worked. The restore on the fresh machine returned the app with the rows you wrote down. You have a timed RTO on paper, and a sentence explaining the RPO. The outside uptime check shows green from a network that is not your own. When you broke the job, the heartbeat went silent and an alert reached you inside the window you chose. If nothing arrived, the alert is the bug, and you have found it on a quiet afternoon instead of at 2 a.m.
What you now know
Section titled “What you now know”You started this chapter thinking of backups as a file you make, and operations as the thing you do when something breaks. Look at what you have now. A backup is a loop: dump, encrypt, ship off the server, restore, and prove it. Monitoring is a set of questions you ask before your users do, and a good alert is one that needs a human right now. Patching is a rhythm, with a snapshot in front of every risky change and staging ahead of production. Secrets have owners, rotation dates and a list of places they live, and offboarding is a checklist rather than a memory. Cost is one more monitor, because the bill tells you about leaks that your dashboards miss.
That is a lot of habit to build, and most of it is boring on purpose. Boring is the goal. The GitLab night was not caused by missing knowledge. Everyone involved knew what a backup was. What was missing was the habit of checking, and you now know what that check looks like.
Go and do the lab on a throwaway box before you trust any of it. Then take the whiteboard checklist to your next fresh VPS, and tick every line by hand.
Self-quiz
Section titled “Self-quiz”1. What does the 3-2-1 rule ask for, and what do the “+1” and “0” add in modern practice?
Three copies of the data, on two different kinds of media, with one copy off the server. The “+1” is at least one copy that is immutable or write-once, so a stolen key or a bad script cannot quietly delete it. The “0” is zero errors on verification, which you only know by restoring and checking.
2. Why is a disk snapshot of a running database not, on its own, a trustworthy backup?
A snapshot captures the disk at one instant, and a running database may have writes in flight that are only partly on disk. You can get a copy that looks complete and restores into a corrupt state. A logical dump, or a snapshot taken with the database quiesced or in a consistent mode, avoids that. Snapshots are still valuable for fast rollback of the whole machine, so use both for different jobs.
3. Explain RPO and RTO in one sentence each, and say which one a nightly dump mainly limits.
RPO, the recovery point objective, is how much data you can afford to lose, measured as time since the last good copy. RTO, the recovery time objective, is how long you can afford to be down while you restore. A nightly dump gives you an RPO of up to about a day, so it sets the RPO, while the restore time you measured sets the RTO.
4. Why is an uptime check run from your laptop, or from the same server, a weak signal?
If the server is down, or your home network is the thing that failed, the check cannot tell a real outage from its own blindness. An outside check runs from a network that is not yours, so when it says the site is down, the site is probably down for real users too.
5. What does a cron heartbeat catch that a resource monitor does not?
A resource monitor sees the machine is alive, but it cannot see that a job stopped running. A heartbeat is a ping sent at the end of a good job. If the ping does not arrive inside the expected window, something is wrong with the job, even when the disk and CPU look fine.
6. What is alert fatigue, and what rule keeps your pager honest?
Alert fatigue is what happens when so many alerts fire for things that do not need action that people stop reading them, and then miss the one that matters. The rule is simple: page a human only for something that needs a human now. Everything else goes to a log, a daily digest, or a ticket.
7. Why do you patch staging before production, and what do you take first on the production box?
Staging shows you the update’s breakage on a box that does not carry real users, so a broken update costs you a test instead of a night. On the production box, you take a snapshot first, so that rollback is a single step even if the update rewrites something you depended on.
8. Why is offboarding a checklist and not a memory?
People forget the places they have access to. A key sits in an authorized_keys file, a device sits on the tailnet, an account sits at the registrar, and a shared secret sits in a password manager nobody else has opened in months. A checklist covers each place, and the rotation of shared secrets closes the gap the person left behind.
Exercises
Section titled “Exercises”Warm-up 1. Name the three copies. Your app’s database is backed up to a disk on the same VPS and to an object store in another region. Which rule of the 3-2-1 is still missing, and what would you add?
Answer
The copy on the same VPS does not count as off-site, and the object store copy is one copy of two media at most. Add a third copy that is off the server and immutable, such as a restic or borg repository with retention locks on the bucket. Then verify by restoring it.
Warm-up 2. Pick the right tool. You want to roll the whole VPS back after a bad package upgrade, and separately you want to recover one table from last Tuesday. Which is a snapshot for, and which is a dump for?
Answer
The VPS rollback is a snapshot job, since it restores the whole disk quickly. The single table from last Tuesday is a dump job, since a logical dump lets you restore one table without rewinding everything else.
Warm-up 3. Read the alert. A heartbeat for your nightly backup went silent at 03:40. Your disk is at 40 percent and the CPU is idle. What does that tell you, and what do you check first?
Answer
The machine is healthy, so the problem is the job. Check whether the cron entry ran, whether the job exited with an error, and whether the object store credentials changed. The resource numbers were never going to tell you that.
Lab 4. Time a restore from scratch. Use your lab VPS to restore last night’s dump onto a fresh box, and measure how long it takes from the first command to the app answering.
Hint: Start a timer, create a new empty database, pull the newest encrypted repository snapshot with restic or borg, decrypt it, load it with psql, then start hello-pipes and request it through curl.
Model solution: A timed log with four numbers: fetch, decrypt, load, and first healthy response. Your RTO is the total, and it should be an honest figure, not the one you hoped for.
Lab 5. Make the uptime check tell the truth. Add an outside uptime check for app.pipeslab.in that hits a route which touches the database, and not just the static front page.
Hint: Use a hosted monitor or a second tiny VPS in another network. Point it at a health route that runs a trivial query, and set a check interval of one to five minutes.
Model solution: A check that goes red when the app is up but the database is not. You will see the difference in a test where you stop Postgres for a minute.
Lab 6. Break the backup and confirm the alert. Stop the backup job, or make it fail on purpose, then confirm the heartbeat alert reaches you inside the window you chose.
Hint: Remove the heartbeat ping line from the job or point the job at a wrong bucket name, run it once by hand, and wait past the grace period. Check the alert arrived on the channel you chose.
Model solution: An alert that fired, with the time it fired recorded next to the time the job should have finished. If nothing arrived, the alert is the bug, and that counts as a finding.
Lab 7. Set a cost tripwire. Create a billing alert at half your monthly budget and another at the full budget on the cloud account that hosts the lab VPS.
Hint: Use the provider’s budget or billing alert screen, set the two thresholds, and send the alerts to an address you read. Then list every snapshot and bucket you own and mark which ones have an expiry.
Model solution: Two alerts, one list of storage with expiry dates, and a note on which resource you would reclaim first if the half-budget alert fired.
Staff 8. Design the backup for a team of eight. Your team’s status tool holds a Postgres database and a few uploaded files. Design the backup and restore plan in five sentences: copies, media, off-server destination, RPO and RTO targets, and the restore drill cadence.
Model answer
Dump the database nightly, and take a continuous WAL archive if the RPO must be under an hour. Keep uploads in an object store with versioning and a retention lock. Ship encrypted, deduplicated copies to a second provider or region. Target an RPO of one day with a WAL archive for under an hour, and an RTO you have measured on a fresh box, not one you guessed. Restore the whole thing from scratch once a quarter, and record the time.
Staff 9. Write the offboarding plan. A contractor who had shell access, a tailnet device, a Cloudflare Access grant and a registrar login is leaving today. List what you revoke, what you rotate, and what you check afterwards.
Model answer
Remove their key from authorized_keys on every server and remove their tailnet device. Remove their Cloudflare Access policy entry and their registrar and cloud account roles. Rotate any secret they could read, including shared database passwords, API keys and the deploy token. Afterwards, check the login logs for their last activity and confirm the old key no longer works from a test machine.
Book 2 of a series. Book 1: The Glue.