Skip to content

5. Linux Survival for Server Owners

“Every server is a building. Somebody has to know which doors are locked.”

Your phone lights up at 2:10 in the morning. The message is from the dance academy’s front desk, and it is short. The enrolment page shows a grey error. Two students say their payment went through, and no confirmation email arrived.

You built this app and deployed it, and it worked on your laptop, which is the cruellest sentence in software. You open a terminal, get into the server, and nothing has crashed the way you expected. The app process is still running. It just cannot write anything. The database refuses inserts, the log file stopped growing hours ago, and the disk report says one hundred percent.

Your code did not change. The building around it did.

This night is a composite, stitched from several that will feel familiar to anyone who runs a small production box. You do not need to live this exact one to need this chapter. You will meet a version of it eventually. The question is whether you meet it with a checklist or with prayer.

Your laptop is a flat with one tenant, which is you. A server is an apartment block with dozens of tenants, most of them programs. Those programs have names, owners, file permissions, logs and limits, and when something breaks, those are the things you read. This chapter is a tour of that building. It is not a Linux manual. It is the set of ideas that stops you guessing.

We will cover who is allowed to do what, how the files are laid out, how programs run in the background, how updates arrive, where secrets belong, and what to check first when the disk, memory or CPU gives up. The lab moves a running app off the root user, which is the most valuable habit in the chapter.

A Linux server is a building with staff. Everything that touches it has an identity, and the system keeps a record of which doors that identity may open. Once you can see the staff, most permission errors stop looking like magic.

Start with the residents. A user is an identity: a name, a number, a home directory and a login shell. Some users are people. ops is the account you use to get into the lab box, and it is you wearing a work badge. Most users are programs. Your web server, your database and your app each run as some user, and that user’s keys decide what the program can open. Linux tracks the user by a number, and the name is a label on top of it.

Wings are groups. A group is a set of users who share access to something. If two people both need to read the same folder of invoices, you do not cut two keys. You put both of them in one group and let the folder answer to the group. Every user has a primary group, and may belong to several others.

Some residents are not people at all. Linux distributions create system users for background services, usually with no login shell, so nobody can sit down as them. On Ubuntu, a web server often runs as a system user called www-data. These accounts exist so that a service has its own identity and its own limited keys, rather than borrowing someone else’s.

The manager with the master keys is root. Root is the user with ID 0, and the kernel lets root open almost any door: read any file, kill any process, and change the rules themselves. Root is not a fancy word for administrator. It is the absence of a lock. That is why it is so useful and so dangerous. It is the one identity that no permission bit can stop, so a program running as root that gets tricked into doing something bad has no internal wall to slow it down.

Between the residents and the manager sits sudo. Sudo lets an allowed user borrow root’s authority for a single command, after signing the register. The register is a rule file that names who may do what, and it usually asks for your own password first. You are not logged in as root. You are a resident who has been told which doors you may briefly open with the manager’s keys, and every such use is written down. On a fresh Ubuntu 24.04 install, the first account you create is placed in the sudo group, and that group is listed in the register as allowed to borrow root. The lab’s ops account was set up that way in Chapter 4.

Building staff: users, groups, root and sudoA building with an app wing holding the app user, a sudo wing holding the ops user, and a manager office holding root. A sudo register sits below the ops wing, and a dashed arrow shows it borrowing root for one command.Wing: app groupuser appWing: sudo groupuser opsManager officeroot (UID 0)sudo register
Figure 5.1. Residents are users, wings are groups, and the manager's office holds root. Sudo borrows the master key for one command at a time, and the borrowing is written down.

Notice what sudo does not do. It does not make your app safer. It makes the admin safer, by turning root into a deliberate act rather than a default state. Sudo answers the question of who may borrow the master key. It does not answer the question of what your program should run as, and that is the question the rest of this section is about.

Every file on the box has an owner user and an owner group. When a process asks to read or change a file, the kernel asks three questions in order: is this process’s user the owner, is it in the group, or is it neither? It then applies the matching permission bits and stops there. That is the whole rule, and it is the subject of the next section.

Every file and folder carries three things: an owner user, an owner group, and nine permission bits. The bits come in three triples, one each for the owner, the group, and everyone else. Each triple has three letters. r means read, w means write, and x means execute. A dash means the letter is missing.

Read -rwxr-x--- as three short sentences. The owner may read, write and execute. The group may read and execute, but not write. Everyone else may do nothing at all. The leading dash marks an ordinary file. A folder shows a d in that first position instead.

Reading a permission stringThe string dash r w x r dash x dash dash dash is split into three boxes: owner with rwx, group with r-x, and others with no access. Arrows show the kernel checking owner first, then group, then others.- r w x r - x - - -Ownerrwx: full accessGroupr-x: read and enterEveryone else—: no accessKernel check order: owner first, then group, then everyone else
Figure 5.2. A permission string is three triples read in order: owner, group, everyone else. The kernel picks the first triple that matches the process, and stops there.

Two things trip people up, and both are worth a sentence before they cost you a night.

The first is that x means different things for files and folders. For a program, x means you may run it. For a folder, x means you may walk into it and reach files inside by their exact names, and r means you may list what is inside. A folder with r but no x is a shelf whose labels you can read but whose contents you cannot touch. A folder with x but no r is a room you can enter if you already know what you are looking for.

The second is that the kernel picks one triple and stops. If you are the owner, the owner triple decides, even when the group triple would have let you in. Owners can lock themselves out. A file owned by root with mode 600 cannot be read by the app user, no matter which group the app user belongs to, because the kernel never reaches the group triple at all.

Changing the bits is a two-verb job. chmod changes the mode, and you can write it either as letters or as an octal number, where r is 4, w is 2 and x is 1, added up within each triple. So 750 means owner 7 (rwx), group 5 (r-x), everyone else 0. The number is shorter but hides the meaning, so read it back in letters before you press enter. chown changes the owner and the group. Between the two you can describe almost any policy a building needs.

Most permission pain on a server comes from three places: the wrong owner, a group missing a member, or a mode that is too loose. Too loose is the one that makes headlines. A .env file that every user can read is a secret that every user can read, and we come back to that file later in this chapter.

Pick a default and keep it. For a small production box, a sane shape is this: application files are owned by the app’s user and readable by its group. Only the folders the app must write to are writable. Logs belong to the service that writes them. Configuration is owned by root or by your deploy user and readable by the app. Once you have a shape, you stop arguing with permissions every time something breaks, and you start asking which rule was broken.

Back to the lab. In Chapter 4 you got hello-pipes answering on port 3000, from every address the box has. That was the win. The problem was who it was answering as. If you look at the process list, its owner is root. Nobody chose that on purpose. It is what happens when a tutorial says to start the server with the manager’s keys, and you copy the line because it works.

Here is why it matters. A program is not a wall. It is a set of instructions, and instructions can be tricked into doing things nobody meant, through a bug in how they read a form, a filename or a header. Security is mostly about what happens after that first trick. If the program runs as root, the person who used the bug is root. Every file, every process and every key on the box is theirs. If the program runs as app, that person is app, and app can reach only what app owns.

That difference is called the blast radius, and it is the best argument in this chapter for a boring habit. You cannot prevent every bug. You can choose how far one bug is allowed to travel.

A widely told story says that early Unix machines had a handful of people sharing one room, and the rule was that you did your work under your own login. Services came later, and the easy path was to let each service run as root, because root could always open the file it needed. Convenience became habit, and the habit is still on a lot of servers.

The honest reason people give for running as root is ports. On Linux, listening on a port below 1024 has traditionally needed special privilege, and the web’s front doors, 80 and 443, sit below that line. That is a real constraint. It is not a reason to run the whole app as root. The pattern that avoids it is a receptionist in front. Chapter 8 covers reverse proxies in full, but the short version is this. A proxy, given the one privilege it needs, holds the front door on 80 and 443. It hands each visitor to your app, which listens quietly on a high port like 3000 or 8080 inside the building. The app never needs the master key, and the front door never has to be the app.

Blast radius as root and as an app userTwo panels show the same compromised app. In the root panel the app reaches system config, the database files, other apps and secrets. In the app-user panel it reaches only its own app folder and its data folder.Running as rootRunning as the app userapp as rootapp as app/etc configdatabaseother appsapp folderdata folderevery door on the box is openthe rest of the building is out of reach
Figure 5.3. The same compromised app in two buildings. As root, one bug opens every door on the box. As its own user, the bug reaches only the rooms that user owns.

Making the move is a three-part job, and the lab walks you through it. You create a user for the app, you give that user the app’s folder, and you tell the service manager to run the program as that user. Nothing about the code changes. Only the keys do.

You do not need the whole Linux filesystem in your head. You need six rooms, because most 2 a.m. questions start with “which room is the thing in?”

Linux has one tree. It starts at a single slash, /, and everything hangs off it. There is no C drive. When you attach a second disk, you attach it to a folder somewhere in the tree, and that folder becomes a door into the disk.

The Linux filesystem as one hallway with six roomsA long hallway labelled slash, the root. Six rooms hang off it: etc for settings, var log for logs, var www for web pages, home for people’s files, opt for hand-installed software, and tmp for scratch space that anyone can write to, shown in amber as untrusted./ the root, one hallway/etcsettings/var/logwhat happened/var/wwwweb pages/homepeople’s files/opthand-installed/tmpscratch, wipedOff the drawing: /var/lib, where programs such as PostgreSQL keep their data.
Figure 5.4. One hallway, six rooms. Most of your 2 a.m. questions are answered by knowing which room to walk into. /tmp is the only room everybody may write to by design.

Here are the rooms, roughly in order of how often you will visit them.

/etc holds configuration. Nginx’s settings, the SSH daemon’s settings and the service files you write for systemd all live here or close by. When someone says “the config is wrong”, the file is almost always under /etc.

/var/log holds logs for programs that write to files. Many modern services send their logs to the journal instead, which we will meet shortly, but Nginx still writes its access and error logs here.

/var/www is the traditional home for web content on Debian and Ubuntu. It is a convention rather than a rule, and your Next.js app may live elsewhere. You should still know it, because a default web server config may point here, and a stray file in this folder can end up on the public internet.

/home holds one folder per human user, such as /home/ops. Your personal SSH keys and shell history live there, so a home folder deserves more care than its cheerful name suggests.

/opt holds software that did not come from the operating system’s package manager: a vendor’s binary, a tool you compiled yourself, an app you copied over by hand. If you ever wonder whether you put something there or the OS did, /opt is the room where the answer is usually yes.

/tmp is the scratch pad. Any user can write to it, and it is cleared on a schedule, sometimes at every boot. Two rules follow. Never keep anything there you care about. And never trust a file you find there, because anyone on the box could have put it there.

Two more rooms deserve a mention. /var/lib holds the state that programs keep between runs, and PostgreSQL keeps its database files under it. /root is the home of the root user, which is why it is not under /home.

It is 2 a.m. and the site is down. Before you touch the code, you want one answer: is anything running at all?

A process is a program that is running right now. It has a number (its PID), an owner and some memory. A process you start by hand usually belongs to a terminal, and it dies when you close the window. A daemon does not depend on a terminal. It starts at boot, sits in the background and waits for work. By convention many daemon names end in a d: sshd is the SSH daemon, nginx is the web server.

Early Unix started its daemons from a pile of shell scripts in a fixed order. Most current servers, Ubuntu 24.04 included, use systemd instead, which has been the default on Ubuntu for roughly a decade. The world without a service manager is the one where you run node server.js in an SSH window, close the laptop, and the site goes down with it. Reboot the box and nothing comes back, and nobody remembers what was running.

systemd is process number 1, the first thing the kernel starts, and its job is to start and watch everything else. It learns what to run from unit files, small text files that each describe one thing. A service unit says which command to run, which user runs it, and what to do when it stops. Here is the shape, not the whole file:

[Service]
User=app
ExecStart=/usr/bin/node /opt/hello-pipes/server.js
Restart=on-failure
[Install]
WantedBy=multi-user.target

Read it as three questions. Who runs it? User=app, not root, which is the lesson from the last section. What runs? ExecStart, the exact command. What happens when it stops? Restart. The [Install] section answers the fourth question: should it come up at boot?

A unit file drives a process that systemd watches and logsAt the top, systemd as process 1 reads a unit file that names the user and the command. The process runs below it as user app. A loop arrow labelled restart returns from the process to the unit when it dies. A dashed arrow carries its output to the journal on the right.systemd, PID 1, starts everythinghello-pipes.serviceUser=app, Restart=on-failureprocess, runs as apppid 4312restartjournalstdout and stderrGreen in the status list, crashing in the journal: read the journal.
Figure 5.5. systemd starts the process as the user you name, restarts it on the policy you set, and keeps its output in the journal.

Starting now and starting at boot are two separate switches. A unit can be running right now and still stay down after a reboot, and the reverse is also true. Most “it worked until the reboot” mysteries come from confusing the two.

Restart policies decide what systemd does when the process ends. no never restarts it. on-failure restarts it after a crash or an error exit. always restarts it even after a clean exit. systemd also limits the damage: if a unit keeps dying in a short window, it gives up and marks the unit failed. That feels unhelpful at 2 a.m., and it is exactly what you want. Without the limit, a broken app restarts a thousand times a minute and eats the box.

journalctl is how you read the answer. systemd captures everything a service writes to standard output and standard error, and stores it in the journal, a log that systemd manages. You ask the journal three questions: which unit, since when, and whether you want it live. Nginx still writes its own files under /var/log, so the journal is one room of the house, not the whole house.

The best setup for a small production VPS is boring on purpose. One unit per app. Each app runs as its own user. It restarts on failure. It logs to standard output so the journal catches it. When something breaks, you read the journal before you read the code. Chapter 7 builds real app units in depth.

It is a Saturday, and a notice arrives about a bug in a library your server uses. You are not sure it touches you. The honest answer depends on three things: whether you installed that package, whether the vulnerable code runs on your box, and whether you have restarted what loaded it. Packages are where that story starts.

A package manager is the staff’s procurement desk. On Ubuntu it is apt. It keeps a list of repositories, which are catalogues of software signed by the distribution, and it records which files each package put where. That record is why you can upgrade or remove software without guessing. Before package managers, you copied a build onto the server, ran its install step, and hoped you would remember the paths. Many servers were hand-built museums that nobody could fully inventory.

Two steps are worth keeping apart. Refreshing the index asks, “what is available now?” Upgrading asks, “replace what I have with the newer version?” A server that never refreshes its index looks fine and quietly falls behind.

Then the subtle part. Upgrading a library replaces the file on disk. A process that loaded the old file keeps running the old code until it restarts. So “the package is updated” and “the vulnerability is gone” are different statements. The second one needs a restart of every service that used the library.

Unattended upgrades is the package that applies security updates on a schedule, without you. For a small production VPS, turn it on. The reason is not that machines are wiser than you. It is that the gap between an advisory and an attack has shrunk, and the update you planned for next month is the one you forget. Keep its scope to security updates. Let it reboot only at a time you chose, or not at all. Two limits matter. It patches the operating system’s packages. It does not update your app’s dependencies, and it does not update the base image of your Docker containers. Those stay yours.

Kernel updates need a reboot. The kernel is the core that every process talks to through system calls. When a new one installs, the running kernel keeps running. The new one waits on disk until the machine boots. Ubuntu leaves a marker file that says a reboot is required, and the login banner may mention it too. Read that as a to-do, not decoration.

The way to reboot without a 2 a.m. call is to do it in daylight, while you are watching. Confirm that your units are enabled, so the app returns by itself. Then check the site from outside the machine, not only from the shell. Chapter 7 covers the app side in depth.

Picture a database password sitting in a file called .env, next to the code, inside a folder the web server was happy to hand out. Nobody meant harm. The file was just in the wrong room.

Environment variables are settings handed to a process when it starts. Your app reads its database address, its payment key and its email key from its environment. A .env file is the common way to write those values down so they survive a restart. The app, or systemd on its behalf, reads the file at startup and passes the values along.

The values are secrets, and anyone who reads the file can use them. The file is the key, so the rules are about keeping keys. For a small server, there are four.

  1. Keep it out of the code. The file lives on the server, outside the repository, and is listed in .gitignore so it never reaches a commit.
  2. Owned by app, readable only by app. Mode 600 means the owner reads and writes, and nobody else touches it. The manager’s office does not keep spare copies of the master key. The resident who needs it holds it.
  3. Never under a web root. A web root is the folder your server hands out as plain files. If the project folder or /var/www is that root, a request for /.env is the first thing a doorknob-rattler tries. Scanners ask for it constantly. The request is weather. The file is your problem.
  4. Let systemd read it. systemd reads an EnvironmentFile= before it drops to the app user, so the file can stay mode 600 and owned by app. You do not need to make it readable by everyone so a service can see it.

Why not write the secrets into the code? Because code gets copied. It goes into git history, onto a laptop, into a backup and into a screenshot of a terminal. A file with the right owner and mode is one place, and you can rotate a key without touching the code.

Two quiet leaks deserve a warning. Printing the whole file into a shared terminal writes the secret into scrollback and maybe into a log. And an app that logs its entire environment at startup sends every secret to the journal, which has readers of its own. Log the names of the settings, never their values.

For the wider list of server habits, the hardening appendix has the checklist, and Chapter 7 shows how an app unit reads its environment file.

Three alarms cover most of the bad nights on a small server. Each one has an outside symptom, a first look, and a usual culprit. Learn the three rows, and you will stop guessing.

Triage table for disk, memory and CPUThree rows, one per alarm. Each row shows the outside symptom, the first thing to check, and the usual culprit. Disk full: uploads fail, check df then du, culprit logs and Docker images. Memory full: a process vanishes, check free then the journal, culprit a leak or the OOM killer. CPU pegged: everything is slow, check top sorted by CPU, culprit a runaway loop or a miner.Outside symptomFirst lookUsual culpritUploads fail, 500 errorsdf, then du on that folderLogs, Docker imagesA process just vanishesfree, then the journalA leak, the OOM killerEverything is slowtop, sorted by CPUA loop, or a miner
Figure 5.6. Three alarms, each with an outside symptom, a first look, and a usual culprit.

Nothing says “disk full” in plain words. What you see is uploads failing with a 500, a database refusing writes, apt complaining, or logins quietly breaking because sessions cannot be saved. Start with df, which tells you which filesystem is full. Then look inside the folder it points at, because df cannot tell you what is filling it.

The usual culprits are logs nobody rotated, Docker images and build cache piling up under the Docker data directory, and a database’s write-ahead log growing unchecked. Backups and core dumps dropped into /home or /tmp come a close fourth.

A memory problem often looks like a process that just disappears. There is no error in your app’s logs, because the app never got to write one. The kernel’s out-of-memory killer picked it, and systemd restarted it, which is why the service looks healthy again five seconds later. Swap fills first, the box slows, and SSH feels like it is typing through syrup.

Check total memory and swap first, then the journal for the kernel’s own words about killing a process. The usual culprits are a Node process with a slow leak, a database whose cache was tuned for a bigger machine, containers with no memory limit, and a build such as npm install or next build running on a 1 GB box that also serves your site.

A pegged CPU is quiet at first. Pages take ten seconds, the load average climbs past the number of cores, and memory and disk look fine. The load average counts runnable work, so a value of 4 on a 2-core box means half the work is waiting in line.

Start with a process list sorted by CPU, usually top, to find what owns the cores. Then decide whether the pressure comes from one process pinned at a hundred percent, which points at a bug or an intruder, or from many processes sharing the load, which points at traffic.

The usual culprits are a runaway loop in your own code that spins on one bad input, a cron job that overlapped with its own previous run, a bot hammering your most expensive endpoint, and the one nobody wants, a cryptocurrency miner that came in on a leaked credential. Scanners rattling doorknobs are weather, not personal. A scanner that found an open admin panel is a different conversation.

Goal. By the end, the app that has been running as root on your lab VPS runs as its own user, its secrets file is locked to that user, and the box patches its security holes without waking you up.

This lab uses the throwaway lab VPS from Chapter 4. It is never production, so break things on purpose if it helps you learn. The steps below are conceptual. The exact commands live in the exercises, so you can read this section once and then work through them.

Conceptual steps.

  1. Check your manager. Confirm that an ops user exists and can borrow the master key through sudo. From here on, you do your work as ops, not as root. If you already set this up in Chapter 4, confirm it instead of redoing it.
  2. Find the resident with the master key. Look at who owns the hello-pipes process and which addresses it listens on. Chapter 4 left it running as root on every address on the box. Confirm that before you change anything.
  3. Hire a user for the app. Create an app account that nobody logs into. It gets no login shell and owns the app’s folder. Its only job is to be the name on the process’s badge.
  4. Move the app into its own room. Give the app’s folder to app, then start the process as app instead of root. A running process keeps the user it started with, so the change has to happen at start-up, not by editing files.
  5. Tour the building. Visit /etc for the system’s settings, /var/log for the logs, and the journal for the app’s own history. Find the line that records the app starting. You want to be able to find that line on the next 2 a.m. call without help.
  6. Let the building patch itself. Turn on unattended security upgrades, then check whether the box is waiting on a reboot for a kernel update.
  7. Lock the drawer. Put a .env file in the app’s folder holding a fake secret. Make it owned by app and readable only by app. Then check that the app never hands that file to a visitor.

How you’ll know it worked.

  • The process listing shows hello-pipes owned by app, not by root.
  • The .env file shows owner app and permissions that only the owner can read and write.
  • A request to the app for /.env returns a normal not-found response, never the file’s contents.
  • The unattended upgrades service is enabled, and you can point to a log line that proves it has run.
  • You can name, without looking, where the app’s logs live and which command reads them.

If all five are true, you have run a small server the way a careful owner does. Most of the 2 a.m. calls in the world are people who skipped exactly these five checks.

You started this chapter as someone who knows that the app runs. You finish it knowing why it runs, who it runs as, and where to look when it stops.

  • A Linux server is a building with rules. Residents are users, wings are groups, and root holds the master keys. Sudo borrows those keys for one command at a time, and the borrowing is written down.
  • Permissions are three triples. Owner, group, everyone else. The first triple that matches the process decides the answer, and the rest are ignored.
  • Apps run as their own users. A bug in an app that runs as root is a bug in the whole box. A bug in an app that runs as its own user reaches only the rooms that user owns.
  • Every landmark has a job. /etc holds settings, /var/log holds the record, /var/www and /opt hold code, /home holds people, and /tmp is the one room everybody may write to.
  • Services are processes with a manager. systemd starts them, restarts them on the policy you choose, and keeps their output in the journal. Enabling a service makes it start at boot. Starting it makes it run now. You usually want both.
  • Updates are a habit, not an event. Security upgrades can run unattended. Kernel updates still need a reboot, and the box will tell you when it is waiting for one.
  • Secrets live in files with narrow permissions. A .env file belongs to the app, is readable only by the app, and is never something a web server should be able to hand out.
  • Triage starts from the outside symptom. Disk full, memory full, and CPU pegged each look different from outside, and each has a first command. Look before you guess.

You have also run the lab. The hello-pipes app is now a resident with its own badge, not a tenant with the master key. That is a real milestone, and it is the kind most people skip. Have a cup of chai. You earned it.

1. Sudo asks for your password. What is it actually lending you, and for how long?

It lends you root privileges for one command, not a permanent root session. Sudo also records who asked for what, which is why the borrowing is written down.

2. A file shows -rw-r----- app app. Who can read it?

The owner, app, can read and write it. Members of the group app can read it. Everyone else gets nothing, because the third triple is empty.

3. Why is running an internet-facing app as root a bad idea, even when it works fine?

Because a bug in that app becomes a bug in the whole box. An attacker who breaks into a root process gets every door on the server. An app running as its own user only reaches what that user owns.

4. Where should a service’s logs live, and why not in /tmp?

They belong in /var/log or in the journal, which systemd manages. /tmp is writable by everybody by design, is meant for scratch files, and is usually cleared at reboot, so it is the wrong place for a record you may need after a bad night.

5. What is the difference between enabling a systemd service and starting it?

Starting makes the service run now. Enabling makes it start automatically at boot. A service can be one, both, or neither, and a surprising number of outages come from a service that was started by hand but never enabled.

6. The disk is full. What is your first command, and what does it tell you?

Start with df -h, which shows which drive is full. Then use du on the busy path to find which folder inside it is growing. Logs, old Docker images, and forgotten backups are the usual culprits. Guessing at a folder without measuring it wastes the next hour.

7. Why does the kernel’s out-of-memory killer sometimes kill the database when the leak was in your app?

Because the killer picks the biggest process at the moment memory runs out, not the process that caused the growth. Read the journal to see what was killed, then ask who grew over the last hour. The guilty process and the victim are often different.

Warm-ups check that the ideas stuck. Lab exercises ask you to touch the real box. Staff exercises ask you to think like the person on call.

Warm-up 1. Read a permission string. A listing shows -rwxr-x--- app app for a file called server.js. Who can read it, who can write it, and who can run it? Who is not allowed to touch it at all?

Answer

The owner, app, can read, write, and execute it. Members of the group app can read and execute it, but not write it. Everyone else gets nothing, because the third triple is all dashes. Nobody outside the app’s own circle can even look at the file.

Warm-up 2. Find the evidence of a silent kill. Your app stopped at 3 a.m. with no error in its own log. You suspect the kernel killed it to save memory. Where would the kernel have written down what it did?

Answer

In the kernel’s messages, which systemd keeps in the journal alongside everything else. An out-of-memory kill leaves a line saying that the kernel killed a named process to free memory. The app’s own log stays silent because the app never got a chance to say goodbye. Check the kernel messages before you blame the code.

Warm-up 3. Update is not upgrade. On a Debian-family box, what is the difference between refreshing the package list and upgrading the installed packages? Which one should run before the other?

Answer

Refreshing the list tells the box what versions exist. Upgrading installs newer versions of the packages you already have. The list has to be refreshed first, or the upgrade works from stale information. Security upgrades depend on this order, which is why automated updates refresh first and then upgrade.

Lab 4. Move hello-pipes from root to app. On your lab VPS, find the process that runs hello-pipes and confirm it is running as root. Then give it a systemd unit that runs it as the app user, enable it at boot, and stop the careless root copy. Keep the app on port 3000 and check that it still answers.

Hint

Use ps to see the owner of the process, and ss with the listening flag to see who holds port 3000. Write the unit file under /etc/systemd/system/. The unit needs a [Service] section with User=app, a WorkingDirectory that points at the app folder, an ExecStart that runs Node on the entry file, and Restart=on-failure. Then run daemon-reload, enable --now, and check with systemctl status.

Model solution

A working unit file looks roughly like this. The paths are examples, so match them to your box:

[Unit]
Description=hello-pipes
After=network-online.target
[Service]
User=app
WorkingDirectory=/opt/hello-pipes
ExecStart=/usr/bin/node /opt/hello-pipes/server.js
Restart=on-failure
[Install]
WantedBy=multi-user.target

You should see the process owned by app, port 3000 still answering, and systemctl status hello-pipes reporting active. Reboot once if you like the satisfaction. The unit should come back by itself.

Lab 5. Tour the record and turn on unattended security upgrades. Look at what the box has been writing down, then make sure security patches install themselves.

Hint

Read the last lines of the hello-pipes journal with journalctl, filtered to the unit name. List /var/log to see the older text logs. Ubuntu usually ships the unattended-upgrades package already. If it is missing, install it with apt. Then turn it on with the reconfigure tool, and check that the unattended-upgrades service is active.

Model solution

You should be able to name at least three things in /var/log and say what each holds, and you should see your hello-pipes start-up lines in the journal. For the updates, the reconfigure step asks one yes or no question. Answer yes. A good check is a dry run of the upgrade logic, which lists what would be installed without installing it. If the dry run is quiet, the box is current, and that is the goal. Write one line in your notes saying when you last saw a kernel reboot pending.

Lab 6. Lock the secrets. Put a .env file in the hello-pipes folder with a fake database password in it. Make the app user the owner, set the mode so only that owner can read or write it, and prove that the web answer never hands the file out.

Hint

Create the file as a placeholder, not a real secret. Change its owner and group with chown, and its mode with chmod using the octal value for owner-only read and write. Check the result with stat, printing owner, mode, and name. Then ask the app for /.env over HTTP from your laptop and expect a 404, not the file.

Model solution

The listing should read owner app, group app, and mode 600. The HTTP request for /.env should come back as a 404 or a plain app response, never as the text of the file, because hello-pipes serves only the routes it defines. The lesson is two locks, not one. The file permissions stop other local users, and the app’s own routing stops the web from seeing the file at all. If you ever add static file serving, check that the folder is not the one holding .env.

Staff 7. The site slowed at 2 p.m. Customers say the site was sluggish from 2 p.m. to 2:20 p.m. and is fine now. Nothing crashed. What do you check, in what order, and what would you write in the incident note?

Model answer

Start from the outside symptom: slow, not down. Check load, memory, and disk in that order, since they are the three usual suspects. Load average tells you whether the CPU queue grew. Available memory and swap tell you whether the box was starved. Disk space and disk activity tell you whether writes stalled. Then read the journal and the app’s logs for the 2 p.m. window, because the window tells you what changed. A scheduled backup, a cron job, a log rotation, or a deploy is the usual culprit. The incident note should record the window, the three measurements, the suspected cause marked as a suspicion, and the one change that would have shown it earlier, such as a disk or load alert. Do not write “probably the database” without the evidence.

Staff 8. Protect the right process. The kernel’s out-of-memory killer took the database last night. The leak was in a report job that runs under the same user. How do you make sure the next squeeze hits the report job first, and what do you do about the job itself?

Model answer

The killer picks by size and by a score that you can influence. Systemd lets you set that score per service, and it lets you cap a service’s memory with a limit, so a runaway job hits its own wall before it starves its neighbours. Put the report job in its own unit with a memory limit, and give the database a lower kill preference so it is chosen last. Check the choice in the journal after the next test, not in theory. Then fix the leak, because a limit only moves the damage. Hedge on the exact option names and check your systemd version’s documentation before you trust a number. The team lesson is that the process that grew and the process that died are often different, and the unit file is where you say which one matters more.

Book 2 of a series. Book 1: The Glue.