7. Running Apps in Production
“The dev server is wonderful right up until it has to work without you.”
The story: the tab that vanished
Section titled “The story: the tab that vanished”A composite, stitched together from weekends that several of us have lived through.
It was a Saturday evening, and the dance academy’s enrolment window opened on Monday. You had deployed the app to the VPS the week before. The deployment, if we are being honest, was an SSH session, a git pull, and npm run dev running in a terminal window that you had wisely parked inside tmux. The app worked. Parents enrolled. You went to bed feeling like a senior engineer.
At 2:07 a.m. the cloud provider rebooted the VPS for a kernel update it had scheduled, and mentioned in an email nobody read. Nothing started again. The tmux session was gone, the terminal was gone, and the dev server, which had never been asked to survive anything, did not survive the reboot.
At 9:15 a.m. the phone rang. It was Priya, on her ₹12,000 phone, on a Pune mobile tower that cannot decide between 4G and a rumour. The enrolment page was blank. Razorpay was retrying its payment webhooks, and the retries were piling up against a server that was not listening. You logged in, and the first thing you checked was DNS. It was not DNS. For once.
Nothing in your code had broken. Everything had broken in the way the code was being run. Hold on to that, because it is the lesson of this chapter: a program that runs is not the same as a program that is operated. The gap between those two is where most of the 2 a.m. calls in our trade come from.
Here is a test I use, which I call the chai-stall test. Imagine the terminal window is a chai stall, and the app is the person running it. If the stall shuts the moment the owner walks off for the evening, the business is not running. It is parked. A production service keeps serving tea when nobody is standing behind the counter, restarts itself when the gas cylinder runs dry, and leaves a ledger you can read in the morning.
We are going to build that ledger, that restart, and that safe back room. We will meet two tools that do the operator’s heavy lifting, systemd and Docker, and one gotcha that has caught experienced engineers who believed they had read the manual. By the end you will know how to run an app on a small VPS so that it survives reboots, recovers from crashes, deploys without drama, and keeps its database out of reach. You will also know why “works on localhost” is the most cursed sentence in our profession, and why it is cursed for reasons that have nothing to do with your code.
One promise before we start. The prose keeps commands to a minimum. The concepts come first, and the commands wait in the lab, where you are allowed to break things on purpose.
Why npm run dev in a terminal is not production
Section titled “Why npm run dev in a terminal is not production”The dev server is a convenience. It watches your files, rebuilds on save, prints friendly errors in colour, and quietly assumes that a person is sitting in front of the terminal, ready to press Ctrl-C when things go strange. On your laptop that assumption holds. On a VPS it does not, and the program has no way of knowing.
Look at the process tree. Your npm process is a child of a shell. The shell is a child of an SSH session. The SSH session hangs off a network connection that a mobile tower in Pune can drop at any moment. Close the window, lose the connection, or reboot the machine, and the whole branch goes with it. The dev server did not fail. It was never given an owner.
A production process needs an owner that outlives your session. That owner is a process manager, and its job is four things. It starts the program at boot. It restarts the program when the program dies. It collects the program’s output into a log you can read later. And it gives you one small set of verbs, start, stop and status, so you stop relying on a half-remembered command from last month.
A widely told story, which I cannot verify and will not swear to in court, says that early Unix operators kept services alive with hand-written boot scripts and a good memory. Every generation since has tried to automate the memory part. What you have on a modern Ubuntu VPS is systemd, which is also the program that starts everything else on the machine. It runs as process ID 1, the first process to start, so it is already on your server whether you asked for it or not.
The lesson is not that one tool is holy. The lesson is that an owner is not optional. The only real choice is which owner you pick.
Process managers: systemd and pm2
Section titled “Process managers: systemd and pm2”pm2 is a process manager written for Node, and many people meet it first because a tutorial told them to. It runs your app as a background daemon, keeps a list of what is running, shows logs, and restarts on crash. On a laptop it is pleasant. On a VPS it is one more program supervising your program, and it ties the supervisor to your Node installation, so an upgrade can leave the supervisor pointing at a path that no longer exists. Its boot persistence works by generating a startup script that you then run once with elevated rights. That is a small ceremony, and it is the step most likely to be forgotten after a server migration.
systemd runs services from unit files: small text files that say what to run, as which user, from which directory, and what to do when it fails. Here is the shape. The shape is the lesson.
[Unit]Description=hello-pipesAfter=network-online.target
[Service]User=appWorkingDirectory=/srv/hello-pipesExecStart=/usr/bin/node server.jsRestart=on-failure
[Install]WantedBy=multi-user.targetRead it as four questions. Who runs it? User=app means the process runs as an ordinary account, so a bug in your code cannot quietly become root. What runs, and from where? ExecStart needs an absolute path to the Node binary, because a service does not inherit your shell’s PATH, which is the thing that makes node work when you type it. What happens on failure? Restart=on-failure brings the process back after a crash, after a non-zero exit, or after a signal that was not a clean stop. A clean exit with status zero counts as a decision, so the service stays down. If your app exits cleanly at 2 a.m. because of a bug in its shutdown path, nothing will restart it. That is a trap worth remembering. Finally, WantedBy=multi-user.target is the part that makes boot persistence work. Enabling the unit links it into that target, so systemd brings it up with the normal set of services every time the machine starts.
Two more habits come with the package. Output goes to journald, the system’s log keeper, which holds it across restarts and which you read with a tool called journalctl. You get the history of every crash without writing a log file yourself. And systemd limits restart storms. If a unit fails too many times in a short window, it stops trying and leaves the failure visible, which is good manners toward your CPU and your logs.
My recommendation for a small production VPS is systemd. It is already there, it is boring, and boring is the highest compliment you can pay a supervisor. Use pm2 if you already live in it and understand its startup step. Do not use it only because a tutorial made it look like the only option.
Docker for operators: the parts that touch production
Section titled “Docker for operators: the parts that touch production”Most of you have met Docker through a tutorial. You ran a Postgres container to follow a course, or you copied a docker-compose.yml out of a README and it started. That is enough to make containers run. It is not enough to run them on a server you answer for at 2 a.m.
Operators need four nouns and one file, and they fit together like this.
An image is a packed recipe: read-only filesystem layers plus the default command to run. You usually build one from a Dockerfile. Pin it to a specific tag such as node:22-slim and you have made a decision. Leave it on latest and you have made a bet on whatever shipped last Tuesday.
A container is one running instance of an image. It gets a thin writable layer on top of the read-only ones. Anything the app writes there vanishes when the container is removed, which is exactly why the next noun exists.
A volume is storage that lives outside the container’s life. Your Postgres data belongs in a named volume. Recreate the container, upgrade the image, and the rows stay where they were. Forgetting this is one of the most expensive mistakes a new operator makes, and it stays quiet until the day it is not.
A network is a private virtual network that Docker manages. Containers on the same user-defined network find each other by service name, so your app connects to db and never needs an IP address. Containers on different networks cannot see each other unless you wire them together on purpose.
Compose is the file that describes several of these at once: the app, the database, the proxy, and the networks and volumes that join them. docker compose up -d brings the set into existence. docker compose down takes it away and, unless you ask otherwise, leaves named volumes alone. On one VPS, Compose is the right tool. Kubernetes is for fleets, and you do not have a fleet.
Why it exists is easy to state. Before containers, each server slowly became a museum of installed libraries, half-remembered runtime versions and one cron job nobody claimed. A container bundles the app with what it needs, so the thing you ran on localhost is the thing you ship. “Works on localhost” stops being cursed and becomes a checksum you can trust. Docker made the packaging pleasant enough to spread widely around 2013, and that is about as precise as I will be on history.
Here is the world without it. Two apps share a VPS. One needs Node 18, the other needs Node 22, and the database wants a Postgres version your distribution does not ship yet. You end up with version managers, PATH surgery and a runbook nobody reads twice. With containers, each app carries its own userland and the host stays boring, which is the best thing a host can be.
My opinion for a small production VPS: keep the Compose file in the repository, pin image tags, put anything stateful in named volumes, and give each project its own Compose network. Run the app as a non-root user inside the container. Never mount the Docker socket into anything you do not fully trust, because that socket is root on the host wearing a nicer hat.
Ports are where the trouble starts. A container listening on port 3000 is reachable only from inside its network until you publish it. -p 3000:3000 tells Docker to bind host port 3000 on every interface and forward it to the container. “Every interface” is the important phrase, because on a public VPS that means the whole internet. -p 127.0.0.1:3000:3000 binds the same door to the loopback interface only, reachable from the host and nowhere else. Same container, different door. Keep that difference in your head, because the next section is about what happens when you think you chose the second one and got the first.
The famous gotcha: Docker can walk past your firewall
Section titled “The famous gotcha: Docker can walk past your firewall”Picture the scene. Last weekend you hardened your VPS the way the tutorials said: deny incoming by default, allow OpenSSH, allow 80 and 443, switch the firewall on. You felt responsible. On Tuesday you started Postgres for the dance-academy app, published it with -p 5432:5432 so your database client could connect, and went to bed.
On Wednesday, something you have never heard of has found your database. Nothing looks wrong. The firewall status lists 22, 80 and 443, and nothing lists 5432. The rules say no, and the port says yes.
That mismatch is the most useful surprise in this book, so let’s take it apart. The details shift between Docker and UFW releases, so treat the names below as a map and check the live tables on your own box.
When a packet arrives at your public address for port 5432, its first stop is the nat table, in a chain called PREROUTING. Docker has put a rule there that says, roughly, “anything arriving for host port 5432 now goes to the container’s private address on port 5432.” That rewrite is called DNAT, destination NAT. By the time the kernel decides what to do, the packet is no longer addressed to your host. It is addressed to a container on a Docker bridge.
So the packet is forwarded, through the FORWARD chain, straight to the container. It never reaches INPUT, the chain where UFW’s rules live. Your “allow 22, deny the rest” policy is a guard at the front door of the building. Docker has quietly built a side passage from the street to the back office, and the guard was never told.
Docker also adds its own accept rules to FORWARD, in a chain called DOCKER. A chain called DOCKER-USER is evaluated before those, and Docker leaves it for you to use. That is your hook for fixes.
Fixes, best first.
- Do not publish what should stay private. The database belongs on the Compose network, where the app reaches it by service name and nothing else can. No published port means the packet has no route in.
- If something on the host must reach it, bind to loopback. A mapping written with
127.0.0.1in front answers only on the loopback interface, so the outside world gets silence. - Write your own rules in DOCKER-USER when a published port must be public but only for certain sources. It works, but it is the fiddly option and the one most likely to rot. Keep it documented next to the Compose file.
- Turn off Docker’s iptables management. Possible, but then you own every container’s network by hand. On a small VPS, do not.
The standard small-production shape
Section titled “The standard small-production shape”Here is the shape I would put in front of anyone running a small app on one VPS. It is the shape the dance-academy app uses, and it fits on a single page of a notebook. Read it from the street inward.
Caddy is the only container that listens on ports 80 and 443. It holds the certificates, speaks HTTPS to the world, and forwards requests to the app. In the city story, this is the receptionist. It is the one door with a sign on it.
The app runs in a container with no published port. Nothing on the host can reach it directly, and nothing on the internet can either. It answers only to Caddy, and only because both sit on the same Compose network.
Postgres runs on that same private network, also with no published port. Your laptop cannot speak to it. The scanners rattling doorknobs cannot either. Neither can you, at two in the morning, unless you first get inside the building.
The data lives in a named volume, pgdata. Rebuilding the database container does not wipe the tables, because the tables live in the volume and not in the container. The volume sits on the host disk, but it is not a door. It is a cupboard.
Figure 7.4 draws the whole thing. Look at what is missing. There is no arrow from the street to the app, and no arrow from the street to the database. That absence is the design.
A word on what “private” means here. Docker creates the network as a bridge, and only containers attached to it can use that bridge. The app reaches the database by its service name, and that name resolves only inside the network. You do not need a firewall rule for this path. You need the absence of a published port, and that is a much simpler thing to check.
Deploying new versions: build, swap, check, and roll back
Section titled “Deploying new versions: build, swap, check, and roll back”A deploy is a short sequence, and the order matters more than the tool you use to run it.
First, build. You build the new image while the old version keeps serving traffic. A failed build costs you nothing, because no user ever saw it. The pipeline does this part, and the Glue Book’s CD chapter at https://the-glue-book.pages.dev/part-1/ch04-cd/ covers how the build gets triggered and where the image goes. I will not re-teach pipelines here. This chapter starts where the pipeline hands you an image.
Second, start the new version beside the old one. For a small app, the calm habit is to bring the new container up under its own name or tag, and leave the old one running until the new one has earned traffic.
Third, check. A health check is a small route the app exposes, one that answers only when the app can reach its database. The deploy waits for that answer. If it does not come within a time limit, the deploy stops and the old version keeps serving. This step turns “the container started” into “the app actually works,” and it is the step most tutorials skip.
Fourth, swap. Once the new version is healthy, Caddy points at it. Depending on how you wired it, that is a reload or a restart of one service, and a request in flight can get a small hiccup. So the honest phrase is near-zero downtime, not zero. On one VPS with one copy of the app, true zero needs more machinery than a small app should carry. I would rather you accept a two-second blip than run a cluster to avoid it.
Fifth, roll back. If the new version fails its check, you never swapped, so there is nothing to undo. If it passes and then misbehaves, you roll back by pointing the stack at the previous image tag. That is why you keep the old image around for a while, and why tags should be immutable. A tag called latest tells you nothing about what was running last Tuesday.
Database migrations deserve their own warning. A migration that drops a column breaks the old version the moment it runs, and then rollback becomes a mess. The safer pattern is expand then contract. Add the new column first. Deploy code that writes to both. Remove the old column in a later deploy, once nothing depends on it. Rollbacks stay possible because the old code still fits the schema.
Config and secrets: one VPS, two environments
Section titled “Config and secrets: one VPS, two environments”Configuration is everything that changes between environments and is not code: hostnames, database addresses, the Razorpay keys, the Resend key. The rule to live by is build once, configure at runtime. The image that passed staging must be the exact image that reaches production. Rebuild for production and you have tested a different artefact.
Secrets get one more rule. They never go into the repository, the Dockerfile, or build arguments, because build arguments end up in the image history. They live in an env file on the server, readable only by root and the deploy user, and Compose reads it when a container starts. It is not a vault, but it is a sensible arrangement for one VPS.
Staging and production on one box are fine as long as they share nothing by accident. Give each stack its own Compose project name, its own named volume, and its own Caddy hostname. Keep payments apart with particular care. A staging app holding a live Razorpay key can create real charges from a test run, and a test webhook pointed at production can write fake payments into real records.
The figure shows the shape. One Caddy routes by hostname, each stack gets the same image tag, and each has its own env file and volume. If you ever find yourself reaching into the other stack’s volume just to check something, stop and take a dump instead.
The lab: a unit that comes back, and a port that should stay shut
Section titled “The lab: a unit that comes back, and a port that should stay shut”You will prove two things on the throwaway lab VPS. First, a process under systemd returns after a crash and after a reboot, while one started in a terminal does not. Second, a database port can look protected and still be open, and the real fix lives in how you publish it.
Part one. Hand hello-pipes to systemd: a unit that runs the app as the app user from its own directory, restarts it on failure, and starts it at boot. Kill the process the way a crash would, and watch systemd bring it back with a new process ID. Read the journal to see the restart. Then reboot, touch nothing, and check that the app answers.
Part two. Run a Compose stack with Postgres and publish its port to all interfaces, as many tutorials do. Keep UFW denying 5432. From your laptop, make a plain TCP connection attempt to the lab’s public address on 5432. If it connects, the rule did not stop it. Work out where the packet went: it was rewritten and forwarded before the host’s input rules ever saw it.
Then check the cloud layer. On Oracle Cloud, the security list is a separate gate in front of the host. If it has no rule for 5432, the attempt fails before UFW is consulted. That result is weather, not policy, and the next edit to the list can undo it without warning.
Finally, fix it twice. Bind the published port to 127.0.0.1, and confirm that the outside attempt now fails while the app on the same host still connects. Then remove the mapping and let the app reach Postgres by service name over the Compose network. The second version is your default.
How you will know it worked: hello-pipes returns with a new process ID after the kill and again after the reboot, and the journal explains why. The outside attempt connects before the fix and fails after it. The app still reads and writes Postgres at the end, and the host shows no public listener for the database.
What you now know
Section titled “What you now know”A terminal is for experiments. Production needs a supervisor, a log, and a boot order. systemd gives you all three on a box that already runs everything else, so I would have you choose it over pm2 unless you already live inside Node tooling and want its restart behaviour.
Docker is not magic. An image is a recipe, a container is one running copy, a volume outlives containers, and a network lets containers find each other by name. Most trouble lives at the boundaries, and the most surprising boundary is the published port. Docker writes its own rules to make -p work, and those rules run ahead of the firewall you thought was guarding the door. The exact chain names have shifted between releases, so hold on to the mechanism rather than the listing. Bind to 127.0.0.1, or do not publish at all.
The small-production shape has one public door, Caddy on 80 and 443, with the app and Postgres behind it on a private network that publishes nothing. Deploys build, start beside the old version, check, swap, and keep the previous tag. Migrations expand before they contract. Config comes from the environment, and secrets stay out of the repository.
You have now watched an app survive a reboot and seen a port open that you believed closed. Most people learn the second one from an incident, so that is real progress. Chapter 8 brings in the receptionist, and Chapter 14 looks at firewalls properly.
Self-quiz
Section titled “Self-quiz”1. Why is a process started by npm run dev in a terminal not a production service?
It belongs to your terminal session. Close the window, log out, or reboot, and it is gone. Nothing restarts it when it crashes, nothing starts it at boot, and its output scrolls away unless someone went looking for it. A supervisor such as systemd owns the process on the box’s terms rather than yours.
2. What does -p 127.0.0.1:3000:3000 change compared with -p 3000:3000?
The first binds the published port to the loopback address, so only processes on the host itself can reach it. Without the address, Docker listens on every interface, the public one included. The image and the app inside the container are identical in both cases. Only the door changes.
3. UFW denies port 5432, and Postgres is still reachable from outside. What is the likely explanation?
A published port is redirected in the nat table and then forwarded to the container. That traffic travels through the FORWARD path and never reaches the INPUT chain, which is where UFW keeps its rules. The rule you wrote is not consulted for it. Check how the port is bound, check the Docker rules, and check your cloud provider’s firewall, which sits outside the box entirely.
4. In the standard small-production shape, which piece faces the internet, and what do the app and the database have in common?
The reverse proxy faces the internet, here Caddy on ports 80 and 443. The app and Postgres sit on a private Compose network and publish no ports, so nothing outside that network can reach them directly. The proxy reaches the app by its service name.
5. Why keep the previous image tag around after a deploy?
So that a rollback is a pointer change rather than a rebuild made under pressure. A rebuild can pull different dependencies or fail on the day you need it most. A rollback is only clean if the database schema still fits the old code, which is why migrations expand before they contract.
6. Why should the image not contain production secrets?
The same image should run in staging and in production, and an image is easy to copy, push, or inspect. Secrets arrive at run time through an env file or the service manager, with tight file permissions. They stay out of the repository and out of the image history.
7. What does a health check tell you that “the container is running” does not?
Running only says the process has not exited yet. A health check asks the app to answer a real request, so the swap waits for something that is up and serving. A process stuck in a crash loop can look running for a while, and a health check catches it.
Exercises
Section titled “Exercises”Warm-up 1. Stopped is not failed. A unit has Restart=on-failure, and someone runs systemctl stop on it. Does systemd bring it back? Why does that matter for a deploy script that stops the app first?
Answer
No. A deliberate stop is not a failure, so the restart policy stays quiet. That is what you want during a deploy, when the script stops the old process on purpose. If systemd restarted it, every deploy would fight the supervisor.
Warm-up 2. The port that says everyone. Your Compose file publishes "5432:5432". Which addresses accept connections for it? What would you change to limit it to the host?
Answer
Every address the host holds, the public one included. Change the mapping to "127.0.0.1:5432:5432", or remove it entirely and let the app reach Postgres over the Compose network by its service name. The second option is the better default for a database.
Warm-up 3. What outlives what. Name what a named volume survives, and say why a Postgres data directory belongs on one.
Answer
A named volume survives the removal of its containers and the replacement of an image. The data files persist while the code around them changes. Put the database’s data on a volume so that replacing the container does not replace the data.
Lab 4. Make hello-pipes a unit that comes back. Run hello-pipes under systemd as user app, with Restart=on-failure, enabled at boot. Then kill its main process the hard way and watch it return. Reboot once and confirm it is still up without anyone logging in to start it.
Hint
Write a unit file in /etc/systemd/system/, then load and start it:
sudo nano /etc/systemd/system/hello-pipes.servicesudo systemctl daemon-reloadsudo systemctl enable --now hello-pipessystemctl status hello-pipessudo kill -9 $(systemctl show -p MainPID --value hello-pipes)The unit needs User=app, a WorkingDirectory, an ExecStart that runs the Node entry point, and Restart=on-failure in the [Service] section.
Model solution
The unit names the user, the directory, the command, and the restart policy. After the kill -9, systemctl status shows a new main PID and a recent restart in the log. After the reboot, the service is active because enable linked it into the boot targets. Use journalctl -u hello-pipes to read its output, since systemd now owns the logs.
Lab 5. Publish the database, then close it again. Start Postgres in Compose with the mapping 5432:5432. From your laptop, run nc -vz 203.0.113.10 5432, with UFW set to deny 5432. Record what happens. Then change the setup so the same probe from outside fails, while the app on the box still reaches the database.
Hint
Check what the host listens on before and after. The socket view is more honest than the firewall config:
sudo ss -tlnp | grep 5432nc -vz 203.0.113.10 5432The fix is in the Compose ports list: bind to 127.0.0.1:5432:5432, or delete the mapping and keep the app and database on the same Compose network.
Model solution
On a box without a cloud firewall in front of it, the probe succeeds even though UFW says deny. The traffic was redirected and forwarded to the container, so the INPUT rules never saw it. On an Oracle Cloud instance, the VCN security list may block the packet before it reaches the host at all. That can make the probe fail for the wrong reason, so confirm with ss on the host and with a probe from another network. The fix is to stop publishing the port to all addresses, and then ss shows the listener only on 127.0.0.1 or not at all.
Lab 6. Two stacks, one box, no collisions. Run a staging and a production copy of hello-pipes on the lab VPS at the same time. Give each its own Compose project name and its own env file. Then take the staging stack down and confirm that production never noticed.
Hint
Keep the two stacks apart by project name, port, and env file:
docker compose -p hello-staging --env-file .env.staging up -ddocker compose -p hello-prod --env-file .env.prod up -ddocker compose -p hello-staging downdocker compose -p hello-prod psBind staging to 127.0.0.1:3001 and production to 127.0.0.1:3000, and let Caddy route the staging. and app. hostnames to them. Never add -v to a down you do not mean. That flag deletes the named volumes.
Model solution
Compose prefixes every container, network, and volume with the project name, so the two stacks share nothing by accident. The env files carry different secrets and different database URLs, and the staging stack points at its own database. Taking staging down removes only its containers and its network, and the production volume keeps its data. The thing to watch is the port. Two stacks on the same host port would fail on the second up, and that failure is the cheapest warning you will get all year.
Staff 7. Renaming a column during a rolling swap. Your team wants to rename full_name to display_name in a live Postgres table. Build-then-swap means the old and new app versions both run for a short time, and a rollback brings the old one back. Lay out the order of migrations and deploys that keeps all of this safe.
Model answer
Expand first. Add display_name as a nullable column, and ship code that writes to both columns and reads from the new one, falling back to the old one when it is empty. Backfill the new column in small batches, so that no single statement holds locks on a busy table. Then switch reads fully to the new column and stop writing the old one. Contract last, in a later deploy, when no running or rollback-able image still reads full_name, and only after the rollback window has passed. The reason for the order is that the schema must work for both versions during the swap, and a rollback must find the column it expects. Dropping the old column in the same deploy as the switch removes the safety net the moment you need it.
Staff 8. Answering “UFW is on, so we are fine.” A colleague reviewing your Compose file writes: “UFW is enabled, only 80 and 443 are allowed, so the Postgres container cannot be reached from outside.” Write a reply that is kind, exact, and gives them something to check.
Model answer
Something like this: “UFW is doing its job for the host’s own services, and I would keep it on. The part I do not trust is the assumption that its rules see every packet. A published container port is rewritten and forwarded before it reaches the INPUT chain where UFW’s rules sit, so the deny for 5432 may never be consulted. Rather than argue from the rules, let’s test from outside. Run ss on the host to see where 5432 listens, then probe the public address from a network that is not ours. If the probe connects, the fix is to stop publishing the port, so the app reaches Postgres on the Compose network by name. If we need it published for a reason, bind it to 127.0.0.1. I would add that probe to the deploy checklist, because the rules can change under us and the probe will not lie.”
The good version of this reply does not mock the colleague. It names the mechanism, offers the test, and makes the fix small.
Staff 9. Guarding a published port you cannot remove yet. A legacy stack publishes a port on all addresses, and the app behind it cannot change for a quarter. Design a host-level guard that allows only the Caddy container to reach that port. Say where the rules go, what must survive a reboot, and what you would test.
Model answer
Put the rules in the DOCKER-USER chain. Docker evaluates its own forwarding rules and then jumps to DOCKER-USER before them, and it leaves that chain alone when it rebuilds its own chains, so it is the one place you can add a guard that Docker does not overwrite. In that chain, accept traffic that belongs to an existing connection, accept new traffic that comes from the proxy’s network to the published port, and drop everything else headed for that port. The exact behaviour depends on your Docker version and firewall backend, so treat the details as something to confirm on the box and not as a recipe from this chapter.
Persistence is the part people forget. The rules live in kernel memory, so they vanish on reboot unless something restores them. Put them in a small systemd unit that runs after Docker starts, and keep it in version control next to the Compose file. Then test from outside the network, before and after a reboot, and after a Docker upgrade. The honest answer is still to stop publishing the port. The guard is the bridge you build while the app catches up.
Book 2 of a series. Book 1: The Glue.