Skip to content

20. Capstone — Two Journeys, Full Resolution, and the 2 a.m. Drill

“Nothing is ever down. It is only somewhere between your fingertip and the app.”

The phone buzzed at 2:07 in the morning, because production has never once waited for a civilised hour. On the other end was the owner of a dance academy in Pune. Her site was down, and parents could not pay the term fees.

This is a composite, built from the kinds of incident that turn up on small Next.js servers. It did not happen to a real academy. The details are still the useful part, so here they are.

The fee page would not load. The Razorpay button spun for a while and then gave up. It was the first week of term, the deadline was midnight, and somebody had already posted a screenshot to the parents’ group. The screenshot was not flattering.

You know how this goes. You open the laptop with one eye, type the address, and the browser says what browsers say when they have stopped trying. Then you do what most of us did in our first year on a VPS. You restart something. You redeploy. You restart Docker and feel briefly like a wizard. Sometimes that works, and you learn nothing, which is the worst kind of working.

This chapter is for the other kind of fix. You now have a name for every box in the 2 a.m. story, and the chain you have been carrying through the whole book is this:

Name → Address → Route → Door → Guard → Receptionist → App

Hold on to one rule for the rest of the chapter. Do not guess the layer. Ask the one question that layer can answer, and move down the chain only when the answer is no.

We will do three things. We will replay the two journeys from the Prologue at full resolution. We will build a diagnostic tree that turns a symptom into a list of questions. And we will work five incidents, each solved one rung at a time, followed by ten drawings you should be able to make on a whiteboard without looking at this page.

There is a test I use on every explanation, and you should use it on this chapter. Could you say the fix to the academy owner over a cup of chai, in one breath? If you can, you understand it. If you cannot, the explanation is not finished yet.

The Prologue gave you two journeys. In the first, a visitor types an address and gets a page. In the second, you, at your laptop, type a command and get a shell. Both looked like magic then. Now every box has a name, and you can point at each one and say what it does and what it refuses to do.

Follow a parent in Pune opening the academy’s fee page on her phone. Her request crosses seven stops, and each stop answers exactly one question.

  1. Name. Her phone wants app.pipeslab.in. Her resolver walks the DNS hierarchy and comes back with an address, not a website. The walk is in chapter 10.
  2. Address. The name points at a Cloudflare edge address, not at your VPS. The subdomain record that makes this possible is in chapter 12.
  3. Route. Her packets cross the wild internet to the nearest edge. The packets themselves, with their source and destination fields, are in chapter 2.
  4. Door. The edge opens a TLS session on 443 and then opens its own connection to your origin, also on 443. Ports are the numbered doors from chapter 3.
  5. Guard. Two guards check that connection. The cloud firewall (the security list) comes first, then the host firewall (ufw) on the VPS. Both are covered in chapter 14.
  6. Receptionist. Caddy listens on 443. It reads the server name the browser asked for, picks the matching certificate, reads the Host header, and forwards the request to the right app. That is chapter 8 and chapter 9.
  7. App. Next.js listens on 127.0.0.1:3000. Nothing on the street can reach that address directly. The app asks Postgres for the fee rows, and Postgres listens only on localhost too.
The visitor’s trip through seven boxesTwo rows of boxes. Top row, left to right: DNS, Cloudflare edge, cloud firewall, host firewall. Bottom row, right to left: Caddy, Next.js on localhost, Postgres on localhost, and the visitor’s phone at the far left. Arrows show the request moving right along the top, down, then left along the bottom.1 · DNSname to address2 · EdgeCloudflare proxy3 · Cloud FWsecurity list4 · Host FWufw allows 443Visitorphone on a street7 · Postgreslocalhost only6 · Next.js127.0.0.1:30005 · CaddySNI, cert, Host1 askport 443The answer travels back the same road, in reverse.
Figure 20.1. The visitor's trip: one box per question, and the answer goes back the way it came.

Notice what is missing from that picture. Only one door faces the street, and it is 443 on the origin, guarded by two firewalls and staffed by a receptionist. The app has no street-facing door. The database has none either. If you ever find a port that faces the street and should not, you have found a mistake, not a feature.

Here is the part that trips people up. The visitor never talks to your VPS by name. She talks to the edge, and the edge talks to your VPS. Your server’s address is hidden behind the directory, which is why a DNS change can move the whole site in minutes, and why a stale DNS record can keep a dead server on the map for hours.

Now the other journey. You are on the couch in Pune, and you want a shell on the lab box.

  1. Laptop. Your laptop sits at 192.168.1.7 on your home network. Its address is private, so the street cannot route to it.
  2. NAT. Your home router rewrites your outgoing traffic to its public address. The gate guard writes down who went out, so replies can find their way back in. Chapter 14 has the gate guard in full. If your ISP puts you behind CGNAT, the gate is inside another gate, and direct connections become harder.
  3. Tailnet. Tailscale tries a direct private tunnel between your laptop and the VPS. If the gates refuse a direct path, traffic goes through a DERP relay. The relay cannot read it, but it does carry it, so it is slower. Chapter 16 covers the mesh.
  4. sshd. The VPS’s SSH daemon checks the host key you already trust, then checks your public key against the one on file for ops. No password is offered at all.
  5. Shell. You get a shell as ops. Your session, and everything you type in it, runs inside the encrypted tunnel from step three.
The operator’s trip from laptop to shellFive boxes in a row, left to right: laptop, home NAT, tailnet, sshd on the VPS, and a shell as ops. A dashed path below the tailnet box leads to a DERP relay box, then climbs back up to sshd when the direct path is blocked.Laptop192.168.1.7Home NATgate logs exitsTailnetdirect or DERPsshdyour key onlyShellas ops, no rootDERP relaywhen direct failsif no direct pathDirect if the gates cooperate, relay if they do not. Either way the tunnel is encrypted.
Figure 20.2. The operator's trip: four gates between your couch and a shell, and a relay for the day the direct path fails.

The operator’s path has a quality the visitor’s path lacks. Nothing on it asks the street for permission. Your laptop reaches out, and the replies come back through a gate that remembers you asked. Compare that with the visitor’s path, where the street knocks on a door you have chosen to leave open. Both paths are encrypted. Only one of them requires you to open a port, and the best setup opens none.

It is 2:07 a.m. and the message says the academy site is down, and parents cannot pay their fees. Do not restart anything yet. Restarting is how you destroy the evidence. Instead, walk the mantra from left to right and ask one question at each rung. The first rung that answers “no” is where you dig. Everything to its right is innocent until proven otherwise.

Diagnostic tree across the seven mantra rungsSeven rows from top to bottom, one per rung of the mantra: Name, Address, Route, Door, Guard, Receptionist, App. Each row shows the one question to ask and the symptom that usually points to that rung.RUNGTHE ONE QUESTIONSYMPTOM THAT POINTS HERE1 NameDoes the name resolve correctly?Server not found, or an old IP2 AddressIs the address the one I deployed?Resolves to the wrong address3 RouteDo packets reach the edge at all?Timeouts, packets go nowhere4 DoorIs the port open and listening?Refused, or a timeout on 4435 GuardDo the security list and ufw allow it?Timeout from outside only6 ReceptionistDoes the proxy match Host and the cert?Cert error, 502, redirect loop7 AppIs the app up, and the DB answering?500s, crash loop, DB errors
Figure 20.3. The diagnostic tree: seven rungs, one question each, and a symptom that usually points to each one.

The table gives you the first tool for each rung. A good answer is what you want to see before you move right.

Rung First tool A good answer looks like
Name dig an address you recognise, from a public resolver
Address dig against the authoritative server, and the TTL the record matches what you deployed
Route ping and mtr from outside replies, or a hop list that reaches the edge
Door nc or curl -v to 443 connected, or a clear refusal
Guard the cloud console’s security list, and ufw status 443 allowed at both guards
Receptionist nginx -t or caddy validate, then the error log config valid, Host matches, certificate present
App ss -tlnp, journalctl, docker logs app listening on 3000, database answering

Notice that the tree never asks “is it the server?” as a single question. That question is too big to answer. Each rung is small enough to test in a minute, and each one rules out a whole layer when it passes.

Every incident you will meet arrives as a scene before it is a diagnosis. Here are five, kept short. Each one is solved with the mantra, Name → Address → Route → Door → Guard → Receptionist → App, walking down the rungs in order. Stop at the first rung that disagrees with what you expected. That rung is usually where the fault lives.

Drill 1: works on Wi-Fi, dead on mobile data

Section titled “Drill 1: works on Wi-Fi, dead on mobile data”

Priya messages you from a train platform. The dance-academy site loads on her home Wi-Fi and hangs on her phone’s mobile data, same browser, same minute. Nothing shipped today. You tether your laptop to your own phone’s hotspot, which is a different network, and ask the first question.

Name. Does the name resolve the same way on both networks? It does not. On the hotspot, the name resolves to an IPv6 address, and that AAAA record still points at a VPS you retired in the spring. Her home Wi-Fi uses IPv4 and never asks for the old one. Her carrier leans toward IPv6, so her phone tries the dead address.

The detail that turns this into a long wait instead of an error is that a refused connection fails fast, and browsers then fall back to IPv4 in a short race. A silent address, one that never answers at all, makes the browser sit until its timeout. So the page does not say “refused”. It says nothing for thirty seconds and then gives up.

The fix is to delete the stale AAAA record. Then wait for the old answer to leave every cache. That wait is the part people forget. DNS didn’t fail to propagate; the old record is cached for its TTL. Lower TTL a day before.

Drill 2: 502 Bad Gateway, right after the deploy

Section titled “Drill 2: 502 Bad Gateway, right after the deploy”

It is 6:40 on a Friday. You shipped the new build at six. The homepage now shows a plain 502 from nginx. You go down the rungs anyway. The name resolves, the address is right, port 443 answers, the cloud security list and the host firewall both allow it, and the certificate is valid. The only one speaking is the receptionist, and it is saying the back office did not answer.

Receptionist. The nginx error log says connection refused to 127.0.0.1 on port 3000. A 502 means the proxy was fine and the upstream was not. Read it as “I knocked on the back door and nobody opened it.”

App. Nothing is listening on 3000. The process exited within two seconds of starting, and its startup log names a missing environment variable for the database URL. The new build reads a variable that the deploy script never wrote into the env file.

The fix is to add the variable, restart the service, confirm that something is listening on the port, and then reload the page. If the new build had moved to port 3001 instead, the symptom would be identical, and the fix would sit in the nginx upstream line, not in the app. The question at this rung is “Is anything listening on the port the receptionist is calling?” Ask it before you touch anything else.

Drill 3: the certificate expired on one subdomain only

Section titled “Drill 3: the certificate expired on one subdomain only”

status.pipeslab.in shows a warning: certificate expired four days ago. app and api are fine, on the same server, behind the same Caddy.

Door and Guard pass. Port 443 answers, and the browser reaches the right box. The fault is not in the building or at the gate. It sits at the receptionist, who is presenting an out-of-date certificate for that one name.

Receptionist. Caddy’s renewal log shows the HTTP-01 challenge failing for status only. Two weeks ago you put status behind a Cloudflare Access policy so that only your team could see it. The policy covers every path, including the one Let’s Encrypt fetches anonymously during validation. The validator arrives with no login, gets the login page instead, and the challenge fails.

The fix is to exempt the challenge path from Access, or move that name to the DNS-01 challenge so it no longer depends on the web path at all. HTTP-01 also needs port 80 reachable from the internet, and closing 80 “because we only need HTTPS” is the other common cause of this exact symptom. The question is “Which name is this certificate for, and what does its renewal log say?”

Drill 4: the redirect loop behind Cloudflare

Section titled “Drill 4: the redirect loop behind Cloudflare”

Visitors see “too many redirects” on app.pipeslab.in. Your laptop, asking the origin directly with the right Host header, gets a clean page. So the app is fine. Something between the visitor and the origin is arguing with itself.

Route. Traffic reaches Cloudflare first, and the SSL mode is set to Flexible. Flexible means the visitor talks HTTPS to Cloudflare, and Cloudflare talks plain HTTP on port 80 to your origin.

Receptionist. Caddy sees plain HTTP and does the polite thing: it redirects to the HTTPS version of the same address. Cloudflare follows, the request comes back to the edge, the edge sends plain HTTP to the origin again, and the origin redirects again. Nobody in the loop is wrong about their own part, and the loop never ends.

The fix is to set the SSL mode to Full (strict), so that Cloudflare speaks HTTPS to the origin with a valid certificate there. The origin’s redirect was never the bug. It was doing its job for a request that arrived in plain speech. The question is “What protocol does the origin see, and what does the edge think it is talking to?” When those two answers disagree, you have found the loop.

You added Postgres to the dance-academy compose file with the port mapping 5432:5432, so a GUI client on your laptop could reach it. UFW says deny incoming, allow 22, 80 and 443. Two weeks later a scan you run on purpose, from your home connection, shows 5432 open. The logs show failed logins from addresses you have never heard of. The doorknob-rattlers found it first. They are weather, but this door was open.

Guard. You check UFW and it says no. The port answered anyway, so the guard you checked was not the guard that acted.

Why. Docker writes its own firewall rules for published ports, and those rules forward the packet before UFW’s usual input chain gets a look. Docker published the DB on 0.0.0.0 and bypassed UFW. Bind to localhost or drop the mapping.

The fix is to publish the port on 127.0.0.1 only, or to remove the mapping and let the app reach Postgres over the compose network. For the GUI, forward the port over SSH or Tailscale instead. Then prove the fix from outside, with the same scan that found the problem. Assume the password may have been tried, and rotate it if the logs show a success. The question is “Which door is open, and which guard let it in?”

Five scenes, one habit. Name the rung where the evidence disagrees, then ask the one question that rung owns. None of these was solved by restarting things first.

Whiteboard templates: ten diagrams to draw from memory

Section titled “Whiteboard templates: ten diagrams to draw from memory”

Ten drawings. You should be able to reproduce each one from memory, in about thirty seconds, while saying one sentence out loud. If you can, you own that layer. If you can’t, the gap tells you which chapter to reread. Figure 20.4 shows the first six. The last four are short enough to draw from the list below.

Six whiteboard templates in a gridNAT gate, TCP handshake, three firewall layers, Host routing to three apps, DNS resolver asking a zone, and the ACME HTTP-01 challenge on port 80.1. NAT: gated societyhome networklaptopgate logserverreplies ride the open entry2. TCP: three-way startyouserverSYNSYN ACKACKthen the data flows3. Firewall: three layerscloud security listhost firewall (ufw)port 443 to Caddyouter first, each can drop4. Host routingCaddyapp :3000api :3001admin :3002Host: app.pipeslab.in5. DNS: ask the directoryresolverzone fileaskanswer 203.0.113.10the answer carries a TTLthe cache is the slow part6. ACME HTTP-01your serverport 80CA1. CA fetches a token on :802. you serve it, CA checks3. port 80 closed, no cert
Figure 20.4. Whiteboard templates one to six (seven to nine are Figure 20.5; ten is Figure 20.6). Draw them in this order: each one reuses a box from the one before.

The ten, with the sentence to say while your marker moves:

Whiteboard templates seven to nineThree panels. Panel 7, SSH key auth: a laptop holding a private key, a server holding the matching public key in authorized_keys; the server sends a challenge, the laptop returns a signature, and the private key never crosses the line. Panel 8, Tailscale mesh with DERP: two nodes behind NAT gates joined by a direct purple WireGuard line, with a DERP relay box below joined by dashed lines as the fallback; a coordination server above only exchanges keys. Panel 9, Cloudflare Tunnel: a visitor reaches the Cloudflare edge; the lab VPS has no street door and its cloudflared dials outward to the edge, and requests come back down that corridor.7 · SSH key authlaptopprivate keysshdauthorized_keyschallengesignatureThe private key neverleaves the laptop; theserver only checks thesignature.8 · Tailscale + DERPcoordinationlaptoplabNATNATdirectDERPRelay sees only ciphertext.9 · Cloudflare TunnelvisitorCF edgelab VPScloudflareddials outNo street door: 0 inbound ports.
Figure 20.5. Whiteboard templates seven to nine: the key that never travels, the mesh with a relay underneath, and the corridor dug outward.

1. NAT. Draw a gated society with one gate and a log, then an arrow out and the reply coming back through the same gate. Say: “The gate remembers who went out, so replies get back in and strangers don’t.”

2. TCP handshake. Two lifelines and three arrows: SYN, SYN ACK, ACK. Say: “Three messages agree on starting numbers, and only then does data flow.”

3. Firewall layers. Three nested guards, outermost first. Say: “The cloud list drops packets before the box sees them, the host firewall gets the next look, and the app comes last.”

4. Host routing. One public door, one receptionist, three offices. Say: “The Host header is the name on the envelope, and Caddy reads it to pick the office.”

5. DNS resolution. A resolver asks the zone file and gets an answer with a TTL on it. Say: “The answer is cached for its TTL, and that cache is usually the slow part.”

6. ACME HTTP-01. The CA hands you a token and fetches it over port 80. Say: “The CA makes you prove the name by answering on port 80, so close that port and your renewals die quietly.”

7. SSH key auth. A private key on the laptop, a public key in authorized_keys, and a signed challenge in the middle. Say: “The private key never leaves the laptop, and the server checks a signature, so no password crosses the wire.”

8. Tailscale mesh with DERP. Two nodes joined by a direct WireGuard line, with a relay box underneath for when the NATs refuse. Say: “Direct when the NATs allow it, relayed when they don’t, and encrypted either way.”

9. Cloudflare Tunnel. A visitor enters at the Cloudflare edge, and the lab dials out to that edge, with no door on the street. Say: “Nothing listens on the street. The visitor’s request comes back down the call the lab already made.”

10. The full request path. Draw the mantra as a row of boxes, Name → Address → Route → Door → Guard → Receptionist → App, then the answer walking home the same way. Say: “Name, address, route, door, guard, receptionist, app, and the reply walks back the same path.”

Draw each one twice: once badly, to find the gaps, and once cleanly, so it stays in your head. The second drawing is the one that lasts.

The gauntlet: thirty questions for the interview and the design review

Section titled “The gauntlet: thirty questions for the interview and the design review”

Answer each one out loud before you open it. If you can say it in thirty seconds, it’s yours.

1. What separates a name from an address, and which comes first?

A name is a label, like app.pipeslab.in. An address is where packets go, like 203.0.113.10. The name comes first: DNS turns it into an address before any packet leaves.

2. Why can’t a CNAME sit on the zone apex?

A CNAME says “this name is an alias”, which clashes with the records the apex must hold, such as SOA and NS. Many DNS hosts offer a flattening record for that job. Check yours.

3. “That subdomain has a CNAME to a deleted resource. Anyone can claim it, so remove it today.” Agree?

Agree. The name still points at a resource you no longer own, so anyone can create it and serve content under your domain. Remove the record, then audit the zone for others.

4. A teammate says the migration “failed to propagate”. What do you say?

“DNS didn’t fail to propagate; the old record is cached for its TTL.” Lower the TTL a day before the next move, move the record, then raise it once things are stable.

5. The site loads on Wi-Fi but fails on mobile data for some people. Where do you look first?

At the AAAA record. A stale IPv6 address can sit beside a fine A record, and carriers that prefer IPv6 reach a dead address. Compare both record types from a public resolver.

6. What does a route table decide, and what happens with no matching route?

It picks the next hop for each destination, longest match first, and a default route catches the rest. With no match and no default, the packet has nowhere to go.

7. How do a refusal and a timeout differ?

A refusal is an answer: a reset or an ICMP reject means a host got the packet and said no. A timeout is silence, so something dropped the packets before any host replied.

8. “It’s a timeout, not a refusal, so packets are dropped before the box. Check the cloud security list before nginx.” Defend that.

It’s the right instinct. A drop happens before the VM, so nginx logs stay empty. Check the cloud security list, then the host firewall, then the service.

9. An app listens on 127.0.0.1:3000. What does that mean for a client on another machine?

Only processes on the same host can connect. Outside clients get refused, even when the firewall allows port 3000. That’s fine behind a proxy and a bug if you expected direct access.

10. Name the firewall layers on a typical VPS, in the order packets meet them.

The cloud security list, outside the VM. Then the host firewall, such as ufw or nftables, then container rules, then the app. An allow at a later layer does nothing if an earlier one drops the packet.

11. “Docker published the DB on 0.0.0.0 and bypassed UFW.” Why, and what is the fix?

Docker writes its own firewall rules for published ports, and they run ahead of UFW’s chains. Bind the mapping to 127.0.0.1 or don’t publish it, then test from outside.

12. “We don’t open 22 at all: SSH over Tailscale, app out through a Cloudflare Tunnel.” What does it buy, and what does it cost?

No public SSH listener, so no password guessing from the internet, and no inbound app port. The cost: you need the tailnet to administer the box, so keep a provider console path for recovery.

13. Two sites share one IP. What lets the proxy pick the right one?

Two fields. SNI names the site during the TLS handshake, so the proxy picks the right certificate. The Host header names the site inside the request, so it picks the right app. One address can serve many names.

14. What is the difference between a 502 and a 504 from the proxy?

A 502 means the proxy got no usable answer from the upstream: refused, crashed, or the wrong port. A 504 means the upstream accepted the request and took too long. Fix the first by finding the process, and the second by finding the slow query.

15. Why does certificate renewal need port 80, and what breaks it?

HTTP-01 proves you control the name by having the certificate authority fetch a token over port 80. Close that port, or put the path behind an access gate, and renewal fails quietly until the certificate expires.

16. Why does Cloudflare’s Flexible SSL mode loop back on itself?

Flexible means Cloudflare talks plain HTTP to your origin. If the origin redirects HTTP to HTTPS, Cloudflare asks again over HTTP, and the redirect repeats. Use Full (strict) with a valid origin certificate, and stop the origin forcing HTTPS on its own.

17. Why keep Postgres off the public interface when it has a strong password?

A strong password stops guessing, not bugs. Every open port is a surface for unpatched software and a place for scanners to knock. Keep the database on localhost or a private network, and let the app reach it there.

18. What does ss -tlnp show that ufw status does not?

It shows what is actually listening: the port, the address it is bound to, and the process. ufw shows the rules you wrote. A service bound to 0.0.0.0 listens on every interface, so only the firewall stands between it and the internet.

19. The app answers from the server on localhost, but not from outside. Which rung, and which question?

Door. Ask: is the port listening on an address the outside can reach? If it is bound to 127.0.0.1, that is your answer, and the proxy should be the only door. If it is public, move to the guard: cloud list first, then the host firewall.

20. A deploy crash-loops the container. What is your first tool, and what do you look for?

docker logs, or journalctl if the app runs as a service. Look for the last line before the exit: a missing environment variable, a port already in use, or a failed database connection. A rising restart count tells you it is a loop, not a one-off.

21. What does the DERP fallback cost you, and what does it not expose?

It costs latency, because traffic takes a relay hop instead of a direct path. The relay forwards traffic that stays encrypted end to end. It can see that two machines talk and roughly how much, but not what they say.

22. Why start the diagnostic tree at Name, not at the app?

A packet that never arrives cannot reach the app, so your app logs stay silent. The name tells you where the client is trying to go. If that is wrong or stale, nothing below it matters. Start where the packet starts.

23. “It’s a timeout, not a refusal.” What does that sentence tell you, and where do you look first?

A refusal means a packet arrived and something answered no, with a reset or an unreachable message. A timeout means nothing came back. The packets were dropped before the box, so nginx never saw them and its logs stay silent. Check the cloud security list first, then the host firewall. Only then nginx.

24. Docker published the database on 0.0.0.0 and bypassed UFW. How do you explain that, and how do you fix it?

Docker writes its own firewall rules, and a published port with no address answers on every interface. In the usual setup that traffic is forwarded before UFW’s rules apply, so ufw status can say deny while the port is open. Bind the mapping to 127.0.0.1, or drop the mapping and let the app reach Postgres over the Docker network. Then scan from outside to prove it. The status page is not the proof.

25. We don’t open 22 at all: SSH over Tailscale, app out through a Cloudflare Tunnel. Why is that stronger than an open port 22 with a strong key?

A strong key stops guessing, but an open port still gets knocked on all day, and every daemon bug is reachable by strangers. With SSH on the tailnet, the public interface never answers on 22, so there is nothing to knock. The tunnel runs the other way: the server dials out to Cloudflare, so web traffic needs no inbound door. Keep one break-glass path, such as the provider’s console, for the morning Tailscale has a bad day.

26. “DNS didn’t propagate.” What is really happening, and what do you do before the next change?

Usually nothing failed to propagate. Resolvers around the world hold the old answer, and they are allowed to, for as long as its TTL says. Lower the TTL a day before the change, make the change, and raise the TTL again once it has settled.

27. A subdomain has a CNAME to a deleted resource. Why is that a security problem and not a tidy-up?

Whoever claims the deleted name on the other provider now serves content under your subdomain, and visitors trust it because the name is yours. That is subdomain takeover. Remove the record today, then audit the zone for CNAMEs that point at services you no longer run.

28. Why does TCP need a three-way handshake before the first byte of your request?

The client sends SYN with a starting sequence number. The server replies SYN-ACK with its own number and an acknowledgement of yours. The client sends ACK. Both sides now know the other can send and receive. That costs a full round trip before your request leaves, which is why a distant server feels slow even on a fast box. TLS stacks another round trip on top.

29. Why can your laptop reach out from a home network while nothing can reach in?

NAT keeps a table. Each outgoing connection gets a line that maps your private address and port to a public one, and replies match a line. A stranger’s first packet has no line, so the gate guard drops it. CGNAT adds a second gate above yours, so even a router port forward cannot reach the street. That is why the tailnet and the tunnel exist.

30. How does an ACME HTTP-01 challenge work, and why does it fail behind a closed port 80?

The certificate authority asks your server, from the public internet, to serve a token file on port 80 at the name you want. If it cannot reach port 80 at that moment, the check fails. Behind a proxy, the edge has to forward that path too, and an access policy can block it. A DNS-01 challenge avoids the door entirely by asking for a TXT record instead, which is why wildcard certificates use it.

Here is the game. You break the throwaway lab VPS on purpose, ten times, and diagnose each break with the mantra before you look at the answer. The lab is never production. Never run this on the dance-academy box.

The goal is the order of your questions, not the fix. For each card:

  1. Read the symptom only, the way a visitor would see it.
  2. Walk the mantra one rung at a time, and ask the one question for each rung.
  3. Name the rung that broke, and check it from outside the box.
  4. Only then turn the card over and compare.

The board below shows the whole request path. The ten red marks are where the cards bite.

The full request path with ten sabotage marksEight boxes in a row, from the visitor’s browser through DNS, Cloudflare, the cloud firewall, the host firewall, Caddy and Next.js to Postgres. Numbered red marks sit on the boxes where each lab sabotage card breaks the path.VisitorbrowserDNSnameCloudflareedgeCloud FWcloud listHost FWufwCaddyport 443Next.jsport 3000Postgresport 543212345761089The reply takes the same road back, in reverse.Red marks: the ten lab sabotage cards, numbered to match.
Figure 20.6. The full request path, with the ten places the lab breaks it.

Run each card on the lab VPS, one at a time, and restore the lab before the next. Read the symptom, walk the mantra from the top, and open the solution only after you have a guess.

Card 1. Point the app name at the wrong address. Change the record for app.pipeslab.in to an old or unused address.

Solution

The name is the first rung, so check what the world sees for it. If the answer is the old address, the record changed but resolvers still hold the old answer for its TTL. That is waiting, not breakage, and the lesson is to lower the TTL a day before any planned change. If the answer is nothing at all, the record is missing from the zone.

Card 2. Switch the Cloudflare SSL mode to Flexible. Leave the origin serving HTTP and redirecting HTTP to HTTPS.

Solution

Flexible means the visitor reaches Cloudflare over HTTPS, but Cloudflare talks to your origin over plain HTTP. The origin sees HTTP, redirects to HTTPS, and the request comes back through Cloudflare as HTTP again. The browser shows a redirect loop. The one question at the receptionist rung: what scheme does the origin think the visitor used?

Card 3. Close 443 in the cloud security list. Remove the ingress rule for port 443 and leave everything else alone.

Solution

This is a timeout, not a refusal. The cloud firewall drops packets before they reach the box, so the browser waits and gives up. Check the cloud security list before you touch nginx or Caddy. Nothing on the box has failed, and the logs there will be quiet, which is itself the clue.

Card 4. Close 443 in the host firewall. Leave the cloud list open and remove the host rule for 443.

Solution

A silent host drop also looks like a timeout, so the symptom alone cannot tell cards 3 and 4 apart. Check the guards in order: the cloud list first, then the host firewall’s rule list. The fix is to restore the rule you removed, and the lesson is that every guard you pass adds a place a silent drop can hide.

Card 5. Stop Caddy. Leave the firewalls open and stop the proxy service.

Solution

Now the packets arrive, but nothing is listening on 443, so the kernel answers with a reset. That is a fast refusal, the opposite of card 3. The one question at the door rung: is anything listening on 443? A refusal points you to the box, a timeout points you to the guards. This contrast is the most useful sentence in the chapter.

Card 6. Bind the app to the wrong port or interface. Change Next.js so it listens on a different port, or only on a private address.

Solution

Caddy is alive, so the proxy logs show the upstream refusing the connection, and visitors see a 502. The door rung is the question: where is the app actually listening? Compare that with the address the proxy is sending to. The app is innocent here, and the fix is to make the two agree.

Card 7. Remove the certificate, or break the ACME path. Delete the issued certificate and block port 80, so renewal cannot finish.

Solution

Visitors get a certificate warning, and the proxy may still serve HTTP. The receptionist rung asks which name the certificate covers and whether the file is present. Renewal uses HTTP-01, which needs port 80 open to the public. Two faults hide here: a missing file, and a closed door that blocks the challenge. Fix the door first, or the renewal fails again on the next attempt.

Card 8. Fill the disk on the Postgres host. Write a large file until the root filesystem is full.

Solution

The app starts returning errors on writes, and the logs stop growing. Sometimes new SSH sessions also struggle, because the shell cannot write its own files. Walk the app rung, but treat the root cause as a resource. The one question: which resource is exhausted? Disk, memory, and open file limits all look like app errors from the outside.

Card 9. Publish Postgres on 0.0.0.0. Map the database port so it listens on every address, and leave the host rule in place.

Solution

Docker published the database on 0.0.0.0 and bypassed UFW, so the rule you thought protected it never sees the traffic. Run a scan from outside your own network, as a doorknob-rattler would. The fix is to bind the mapping to localhost, or drop it, since the app talks to Postgres over the internal network anyway. Verify from the outside, not from the box.

Card 10. Remove an environment variable. Delete the database connection string from the app’s environment and restart it.

Solution

Next.js starts, fails on the first query, and restarts in a loop. The proxy shows 502s, and the first instinct is to blame nginx or Caddy. Read the app’s logs first. The app rung is where this lives, and it is the same shape as the 502 after a deploy in drill two.

Write one of these for every real incident, and for every lab break you care about. Keep it to five lines.

  • What the user saw: the symptom in their words, with the time it started.
  • Timeline: the three or four moments that changed what you believed.
  • Failed rung: the mantra layer where the fault was, and the one question that found it.
  • Why it was not caught earlier: the missing alert, the missing check, or the untested change.
  • One change so it cannot recur: a single concrete fix, with an owner and a date.

Stop for a second and look at what changed. At the start of this book, “the site is down” was a feeling. Now it is a list. You can say where a name turns into an address, and which resolver and which zone file gave the answer. You can draw the trip a packet takes from a phone on a Pune tower to a Node process on a lab VPS, and you can point at the door, the guard and the receptionist on that path. You know why a port is only a numbered door, why a dropped packet looks different from a refused one, and why “works on localhost” was never the same as “works on the internet.”

You also know how to be wrong slowly, which sounds like a joke and is actually the most useful skill in this chapter. The diagnostic tree is a habit, not a trick. Ask one question per rung, and do not climb until the rung below has answered. That habit is what turns a 2 a.m. call into a four-minute fix instead of a four-hour one.

The mantra is the whole book in one line: Name → Address → Route → Door → Guard → Receptionist → App. Say it when you are tired. Say it when someone is shouting. Then ask the one question for the rung you are on.

Well done. Really. You started this book by deploying Next.js from tutorials, and you finish it able to explain every hop between a visitor’s thumb and your code, and to fix the hop that broke. That is a real skill, and the parents paying dance fees on Sunday night are the people it protects.

If you want the older plumbing underneath all of this, the story of DNS, TCP, TLS, HTTP and BGP in more depth, read the network chapter of The Glue. If you want the two journeys again with fresh eyes, go back to the prologue.

1. A visitor’s browser says it cannot resolve app.pipeslab.in, but dig @1.1.1.1 app.pipeslab.in returns an address. Which rung is at fault, and what do you ask next?

The Name rung, on the visitor’s side or their resolver. The authoritative answer exists, so the zone is fine. Ask whether the failing resolver is serving a stale or negative cached answer, or whether it is pointed at a broken DNS server. Compare its answer with a public resolver’s answer before you touch the server.

2. Your site times out from a laptop, but curl to the origin works from inside the VPS. What does a timeout tell you that a refusal would not, and where do you look first?

A timeout means the packets were dropped silently before they reached the box. A refusal means something answered with a reset or an error, so the packets got through. Check the cloud security list and any upstream filter before you open the nginx or Caddy logs, because the proxy never saw the traffic.

3. Why is a firewall that drops packets harder to debug than one that rejects them?

Silence looks the same whether the port is closed, the host is down, or the network is slow. A rejection tells you who said no, and that is a fact you can act on. With a drop you have to rule out each layer by testing it, which is why the tree asks you to test the lower rungs first.

4. Caddy serves a valid certificate for api.pipeslab.in, but staging.pipeslab.in presents an expired one. What do you check?

Renewal for that one name. The usual causes are that port 80 is closed for the HTTP-01 challenge, that an access rule or a redirect in front of the origin is blocking the challenge path, or that the name no longer resolves to this server. Read the proxy’s logs for that hostname, not for the whole server.

5. One public IP serves three subdomains over HTTPS. What decides which app answers, and at which layer?

The proxy decides, and it uses two names. For TLS, the SNI value in the ClientHello picks the certificate, before any HTTP is spoken. For HTTP, the Host header inside the request picks the upstream. One door, one receptionist, several offices.

6. Two tailnet machines connect through DERP for a while, then switch to a direct path. Why both, and what changes?

WireGuard tries to punch a direct hole through each side’s NAT first. When both sides sit behind strict NAT or carrier-grade NAT, the punch fails, and the traffic goes through a DERP relay. The relay sees only encrypted packets, but adds latency and a hop. Once the direct path works, traffic moves off the relay, and you should see the path change with the status command, not guess.

7. What is the single property that makes a Cloudflare Tunnel different from a port forward?

The connection starts from inside. The origin dials out to Cloudflare and keeps that connection open, so you need no inbound port open on the origin at all. A port forward asks the internet to knock on your door. A tunnel means you call the doorman yourself.

Do these with the lab VPS and the laptop open side by side. Warm-ups are for thinking, lab items are for typing, and staff items are for arguing with a colleague. Open each answer only after you have a guess.

Warm-up 1. The stale record. You changed the A record for app.pipeslab.in an hour ago. Some visitors still reach the old server and some reach the new one. Which rung is it, and what do you tell the team?

Answer

The Name rung. Resolvers cache answers for the record’s TTL, so the old answer lives on until its timer runs out. Nothing is broken, and you cannot force every cache to forget. The lesson is to lower the TTL a day before a planned move, then change the record.

Warm-up 2. Refused or dropped. From outside, port 443 accepts a connection, port 22 is refused straight away, and port 80 hangs and then times out. Which ports did the packets reach, and which layer dropped them?

Answer

Port 443 is open, so the packets reached the box and a program accepted them. Port 22 refuses, so the host answered and nothing is listening or a rule rejected it. Port 80 times out, so something upstream dropped the packets silently, probably the cloud security list or a guard in front of the box. Check that layer before the proxy logs.

Warm-up 3. Two offices, one door. Two subdomains share one IP. Visitors to the admin site land in the student app. Which field decides which app answers, and where do you look?

Answer

The Host header inside the HTTP request, and for TLS the SNI value that picked the certificate. Look at the server name blocks in the proxy config, check which block is the default, and confirm that each name has its own upstream.

Lab 4. The 502 after a deploy. Name and Address are fine. The proxy returns 502 after you deploy a new build. Walk the mantra and state the one question for the rung you suspect.

Answer

Hint: Work at the Receptionist and App rungs. Ask whether anything is listening on the port the proxy points at, then read the app’s own log. Listing listening ports and the service journal are the first two tools.

Model solution: The app crashed at startup because a required environment variable was missing from the new deploy. The proxy could not reach 127.0.0.1 on port 3000, so it reported a bad gateway. Restore the variable, restart the service, and check the listening port again.

Lab 5. The database that appeared. You add a Compose file with the database mapped as 5432 to 5432. An outside scan now shows 5432 open, even though the firewall denies it.

Answer

Hint: List the running containers and read the port column. Then decide what address the mapping should bind to.

Model solution: Docker writes its own firewall rules, and they run ahead of the host firewall rules, so the mapping published the port on every address. Bind the mapping to the loopback address only, recreate the container, and scan again from outside. The database must still be reachable from the app on the same host.

Lab 6. The redirect loop. Cloudflare shows a too many redirects error. The origin redirects plain HTTP to HTTPS, and the Cloudflare SSL mode is set to the one that talks to the origin over HTTP.

Answer

Hint: Find the SSL mode in the Cloudflare dashboard, then ask what protocol Cloudflare uses when it talks to your origin.

Model solution: Cloudflare connects to the origin over plain HTTP. The origin sends a redirect to HTTPS, Cloudflare follows it back to HTTP, and the loop never ends. Change the mode to full strict so Cloudflare uses HTTPS to the origin and checks the certificate.

Lab 7. The stale subdomain. A name in your DNS still points to a hosting resource you deleted last month. Nothing breaks yet, but a stranger could claim that resource.

Answer

Hint: Find the record by name, then confirm that the target no longer answers.

Model solution: This is a dangling record. Remove the record today, not next sprint, and check the rest of the zone for targets that no longer exist. Then add a regular check that compares every record target with a live resource.

Staff 8. Admin access without a public port. Staff need the admin panel from home and from the clinic. The owner wants no public port open for it. Propose a design, and name one option you would reject.

Model answer

Put the admin host on the tailnet, with access rules that allow only the staff group, and reach it over SSH on the tailnet. Publish the admin web app through a Cloudflare Tunnel, behind an access policy that requires a named login. Reject the shared VPN password. It is one secret for everyone, it cannot be revoked per person, and it still leaves a port open.

Staff 9. The postmortem. The team-status tool was down for forty minutes. The cause was a subdomain pointing at a deleted hosting resource. Write the fix and one prevention step.

Model answer

Fix: remove the dangling record and confirm the name no longer resolves to the old target. Prevention: keep an inventory of records and their targets, and alert when a target stops answering. The five-line template in the lab section is the place to write it up.

Book 2 of a series. Book 1: The Glue.