Skip to content

6. SSH — The Key to the Building

“A password is something you say out loud. A key is something you prove.”

The story: the password that crossed the wire

Section titled “The story: the password that crossed the wire”

Picture a university computer room in the early nineties. Someone types a username and a password into a terminal, the login works, and they go back to their work feeling pleased. Somewhere on the same network segment, another machine is reading every packet that passes by. The password crossed the wire as plain text, and anyone curious enough could copy it down.

That was not a thought experiment. Telnet, the remote login tool that goes back to the early days of the ARPANET, sent everything in the clear: usernames, passwords, every keystroke. The Berkeley family, rsh and rlogin, did not do much better. They were built for a world of friendly machines that trusted each other by name, and a clever attacker can fake a name.

For a while the industry shrugged. Most people told themselves the network was basically friendly, and they picked passwords that were easy to type. Then, in early 1995, a password-sniffing attack hit the network at the Helsinki University of Technology. The widely told version of the story has a researcher named Tatu Ylönen among the people whose passwords were captured. I cannot check the details from here, so I will hedge them. The shape is the part that matters: the people who most needed a secure login were using an insecure one, and they knew it. Ylönen wrote a replacement that year and released it free. He called it SSH, for Secure Shell.

The design was almost stubborn in its simplicity. Encrypt the whole conversation before any secret is sent. Prove that the server is the one you meant to reach. Then prove that you are you. Those three promises still sit under everything you do with SSH, from the deploy script you run from your laptop to the git push you do before lunch.

The first version had its own flaws, so the protocol was rewritten as SSH-2 and standardized in the mid-2000s. Around the turn of the millennium, the OpenBSD team started OpenSSH, a free implementation that forked from the last freely licensed release of the original. That is the sshd running on most of the Linux servers you will ever touch, including the one you rented on Oracle Cloud. If memory serves, it has been the default on Ubuntu and Debian for as long as you have been deploying things.

Consider the world without it. You would either send passwords across networks you do not control, or walk into the server room and type on a console. Both options fail the chai-stall test. You would not shout your password across a chai stall, and you should not whisper it across a shared network either.

It is worth pausing on how strange this history is. SSH is not a grand architecture handed down from a committee. It is one annoyed person’s fix, released for free, that became the thing your deploys ride on. Most of the boring infrastructure you depend on has a story like that. The boring parts are the ones that survived.

If you finished Book 1, you already know that a packet can be read by anyone on the path between you and its destination. The Book 1 network chapter covers the path itself. This chapter is about what you put on top of that path so that the path stops mattering. Encryption does not stop the packets from travelling. It only makes them useless to the person copying them.

Ylönen’s trick is the reason you can log in to a stranger’s data centre without writing your password on a sticky note. Keep that thought, because the rest of this chapter is a tour of what that trick looks like in daily use: the handshake, the keys, the config file, the tunnels, the hardening, and the stories where it went wrong.

Every SSH login starts with a conversation that finishes before anyone types a password, and it always runs in the same order. Learn the order once and most SSH error messages stop being mysterious. Figure 6.1 is the whole conversation on one page.

The SSH handshake, step by stepA sequence diagram with your laptop on the left and the server on the right, time running down the page. The steps are: TCP connects to port 22, both sides exchange version strings, a key exchange creates an encrypted channel, the server shows its host key, the laptop checks known_hosts, the server issues a challenge, the laptop signs it, and a shell opens.Your laptopThe server1. TCP connects to port 22: the door opens2. Version exchange: both sides say SSH-23. Key exchange: fresh secrets, then encryption4. Server shows its host key, signed5. Laptop checks known_hosts6. Server asks: show me you hold your key7. Laptop signs the challenge. The key never leaves8. Accepted. A shell opens inside the encrypted channel
Figure 6.1. The SSH handshake: the door, the secret, the proof, and then you.

Read the figure as a visit to a building. Step one is the street. Your laptop opens a TCP connection to port 22, the numbered door that sshd stands behind. Book 1 covered how that connection gets built; this chapter starts once the door is open. Step two is a small courtesy: each side announces the protocol it speaks and the software behind it. In the logs you will see that as a line beginning SSH-2.0.

Step three is the one that matters. The two sides agree on a secret without ever sending it. Each generates a temporary key pair for this one visit, they swap public halves, and both compute the same shared secret, while someone watching every packet still cannot. Those temporary keys are called ephemeral, and throwing them away afterwards gives you forward secrecy. If an attacker records your session today and steals the server’s long-term host key next year, the recording still does not decrypt. You do not need the mathematics to use SSH. You do need the shape: a secret agreed in the open, then every byte encrypted with it.

From here on, nothing sensitive crosses the wire in the clear, including the username you typed. That is a small detail with a point to it, because the old Telnet login sent your name in plain text too.

Step four is identity. The server sends its host key and a signature, made with the matching private half, over the exchange you just finished. The signature proves the server holds that private key right now, and that the key belongs to this conversation, so an old recording cannot be replayed at you. Notice what it does not prove. It shows the server holds a key. It does not show that the key belongs to the machine you meant to reach. That is step five, and step five is your laptop’s job alone.

Steps six and seven are about you. The server sends a random challenge and asks you to sign it. Your client signs with your private key, which stays on your laptop, and the server checks the signature against the public key it holds for your user. The server never learns your private key. If key authentication is set up, no password was typed at any point. Step eight is the shell, and from then on your keystrokes and the server’s replies ride inside the same encrypted channel.

Trust on first use: host keys and known_hosts

Section titled “Trust on first use: host keys and known_hosts”

Step four gave you a server that proves it holds a key. Step five asks whether that key belongs to the machine you meant. Your laptop cannot work that out from first principles, so it does what a sensible person does with a stranger at the door. It remembers.

The first time you connect, you see something like this:

The authenticity of host '203.0.113.10 (203.0.113.10)' can't be established.
ED25519 key fingerprint is SHA256:Qx7n2Lw4fD9aTz1kRm0sYcUe6JhPgB8oVi3WnKtZ0aA.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

If you type yes, the key is written into ~/.ssh/known_hosts. Every later connection compares the key the server shows against that file. The industry calls this trust on first use, and it is an honest name. For one moment you trust without proof. That is why the first connection to a new server should happen from a machine and a network you already trust, and why you should check the fingerprint against something independent when you can. The provider’s console often shows it.

Host key decision: first visit, known match, or changed keyA decision diagram. Your client receives the server’s host key and checks known_hosts. Three outcomes: no entry yet, so you confirm and save it; a matching entry, so the connection continues; a different key, so the client stops and warns you.You connect. The server shows its host key.Is the host in known_hosts, and does the key match?First visitNo entry yet.You confirm once.Known and matchingThe key matches the file.Connection continues.Key differs: STOPRebuilt, or someonein the middle.The middle box is the everyday case. The right box is the one to respect.
Figure 6.2. Trust on first use. The left branch saves a key. The right branch is the alarm.

Most days you land in the middle box, and nothing you see is new. The alarm is the right box, and it has a banner you should learn by heart:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

The banner means the key on file no longer matches the key the server just showed you. There are two stories that produce that exact message, and your client cannot tell them apart. The first is that you rebuilt the machine. Reinstalling the operating system, restoring from a snapshot, or moving your domain to a fresh VPS gives the server new host keys. This is common and boring. The second is that someone sits between you and the server, presenting their own key and relaying your packets. That is rare on a clean network. It is not rare enough to ignore.

So the procedure is the same for both. Do not type yes. Do not delete the line out of irritation. Find out which story is true. Ask whoever rebuilt the box, check the provider’s console, or compare the fingerprint through a channel you trust. Only when you know can you remove the old entry, using the tool built for that job. The lab hint in the lab section shows which one.

The demo below walks the whole handshake one step at a time. Try the scenario where the key has changed. It is the one most people only ever meet at two in the morning.

Try it: walk the SSH handshake one step at a time

    Static fallback: an SSH login runs in order. First the TCP connection and version strings, both public. Then key exchange builds an encrypted channel. The server proves its host key, and your laptop checks it against known_hosts. Last, you prove who you are by signing a challenge with a private key that never leaves the laptop. If the host key changes, stop and find out why.

    A password proves you know a secret by typing it, and the secret crosses the wire every time. Public-key authentication proves something better: that you hold a secret, without ever sending it. That one distinction is most of why SSH feels safe, so it is worth a slow paragraph.

    A key pair is two files that belong together. The private key lives on your laptop, in ~/.ssh/, and you treat it like the key to your flat. The public key is its partner. It is safe to publish, and it goes on the server, in a file called authorized_keys. Knowing a public key gets an attacker nothing.

    The login, roughly: the server sends your client a random challenge. Your client signs that challenge with the private key. The server checks the signature against the public key it already has. Nothing secret crossed the network, and because the challenge is new every time, a recorded login is useless for replay.

    Public-key login in three movesdescYour laptopprivate keynever leaves this boxThe serverauthorized_keysholds the public key only1. public key, once2. random challenge3. signed reply
    Figure 6.3. The private key stays home. The server holds only a public key, and checks a signature.

    Why the private key never leaves: the signing happens inside your client, on your machine. Only the signature travels. If someone breaks into the server and copies everything in authorized_keys, they hold public keys, which open nothing anywhere else. That is the design win, and it is why a stolen server is not automatically a stolen identity.

    A private key sitting unencrypted on your disk is one stolen laptop away from being somebody else’s key. Give it a passphrase. The file on disk is then encrypted, and the passphrase is what unlocks it.

    Typing that passphrase on every connection gets old fast, which is what ssh-agent is for. The agent is a small background program that holds the decrypted key in memory and does the signing for your client. You unlock the key once per session, and after that your connections ask the agent to sign. The private key still never leaves your machine. It just lives in the agent’s memory instead of in a file.

    Key type matters less than the internet argues about, but there is a clear default. Ed25519 keys are small, fast and the modern choice, so make one today. RSA still works, and you will meet it on older servers and in old tutorials. If you ever have to make one, use at least 3072 bits.

    Most people’s first SSH is a long command with the details packed into it: the key, the port, the user, the address. You type it once, it works, and then you type it again forty times a week. ~/.ssh/config is where that memory lives. It is the cheapest quality-of-life upgrade in this whole chapter, and it costs you nothing to learn.

    It is a plain text file. Each Host block gives a short name, and the indented lines under it say what that name stands for. Here is the shape, with the lab’s values:

    Host lab
    HostName 203.0.113.10
    User ops
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

    After that, ssh lab means all four things at once. Two of those lines deserve a word. IdentityFile names the key to offer. IdentitiesOnly yes tells the client to offer only that key, and not every key your agent happens to hold. Without it, a laptop with a dozen keys knocks on the server with each one in turn, and the server gives up after a handful of wrong guesses with “Too many authentication failures” before it reaches the right one. The config file is not a security feature. It is a tidy desk, and a tidy desk also stops the wrong keys from being shouted at the server.

    The same file holds Port for a server that moved off port 22, ProxyJump for hops (you will meet that below), and ServerAliveInterval for keeping an idle connection from being dropped by a router. That last one is the Wi-Fi fix for the long job you started on the train.

    Three tools, one transport. All of them ride the same encrypted SSH channel with the same keys, so if ssh lab works, these work too.

    scp copies files the direct way: name a source and a destination, and it goes. It is fine for one file you need on the server right now. It has no memory. It does not skip what is already there, and it does not pick up an interrupted copy where it stopped. Recent OpenSSH releases run scp over the SFTP protocol underneath, so the protocol is not the thing to worry about. The lack of smarts is.

    rsync is the tool you reach for whenever a directory is involved. It compares source and destination, sends only what changed, and can be run again safely after an interrupted attempt. For pushing a built folder, or pulling a backup down, it is the right shape. Its sharp edges are two. The first is the trailing slash. “Copy the contents of this folder” and “copy this folder into that one” differ by a single character, and it is easy to get the wrong one at 2 a.m. The second is --delete, which removes files on the destination that are missing from the source. That flag is why you run rsync in dry-run mode first, every time. It is also a thread this chapter will pull on again in the GitLab story.

    SFTP is a file-browsing session over SSH. Use it when you want to look around a server, see what is there, and drag a file across. That means a graphical client, the sftp command, or a mount that makes the remote disk look local. It is interactive by nature. Scripts should prefer rsync or scp, which are built to run unattended.

    Port forwarding: tunnels with a door at each end

    Section titled “Port forwarding: tunnels with a door at each end”

    Here is a scene you have probably lived. Your Postgres database runs on the lab server, and it listens only on the server’s own loopback address, 127.0.0.1. Nobody on the street can reach it, which is exactly what you want. But your laptop GUI client cannot reach it either, and you would like it to.

    SSH has a trick for this called port forwarding. It is one of the most useful and least understood things the protocol does, because it turns the encrypted channel you already have into a set of private corridors. Once you see it as a door at each end, the three flags stop being magic.

    A local forward says: open a port on my laptop, and whatever arrives there, send it through the SSH channel and out the far end to a destination the server can reach.

    The part people miss is the word “destination”. The destination is resolved from the server’s point of view, not yours. So 127.0.0.1 in a local forward means the server’s own loopback, which is precisely where Postgres is listening. Your laptop’s 127.0.0.1 is a different place entirely.

    A local forward from a laptop port through an SSH tunnel to Postgres on the serverA box for the laptop has a door on its right edge, labelled 127.0.0.1 port 5432. A thick encrypted line runs from that door to a box for the server, ending at Postgres, which listens on the server’s loopback address 127.0.0.1 port 5432. The line is labelled as an encrypted SSH tunnel. A note at the bottom says the port is not open on the public address.Your laptoplistens on127.0.0.1:5432encrypted SSH tunnelbytes ride inside the channelServer (lab)Postgres127.0.0.1:5432bound to loopback onlyNo door on the public address. Port 5432 is never open to the street.
    Figure 6.4. A local forward. Your laptop's port is a door, the corridor runs through the encrypted tunnel, and the far door opens onto Postgres on the server's loopback.

    Notice what did not happen. Postgres did not get a new public door, and its configuration did not change. The only way in is the tunnel, and the tunnel only exists while your SSH session is alive and you are the one holding its key. That is the whole appeal.

    Remote forwarding reverses the direction. The server opens a port, and whatever arrives there comes back through the channel to a port on your laptop.

    This is how you show a local app to someone else without deploying it. Your laptop runs the app on port 3000, and a remote forward makes it appear on the server’s loopback port. The default matters here. The server binds that remote port to 127.0.0.1 only, so only processes on the server itself can reach it. To make it reachable from other machines, the server has to allow it with the GatewayPorts setting, and that is a decision, not a default. Turning that setting on quietly turns a private corridor into a door on the street. Keep it off unless you have thought hard about who can knock.

    Dynamic forwarding is different again. It does not name a single destination. It opens a SOCKS proxy on your laptop, and your browser or tool sends its traffic through the server. Each request leaves from the server’s network, so a website that only accepts visitors from the lab’s data centre now sees the lab, not your flat in Pune.

    It is a handy debugging tool, for example to check what the server sees when it fetches an external API. Treat it the same way as -R. A proxy that carries all your traffic is powerful, and you should know exactly which tunnel it is using at any moment.

    Real setups often have a private machine that only the bastion can see. The clean way to reach it is ProxyJump, which tells SSH to hop through the lab server on the way. Your config file can say it once:

    Host db
    HostName 10.0.0.9
    User ops
    ProxyJump lab

    After that, reaching the private host is one short name, and SSH does the hop, encrypting each leg separately. The jump host sees the connection pass through, but not your keys, because your keys never leave your laptop. This is the same agent trick from earlier, now doing the hopping.

    Forwarding is not a second network. It is the same encrypted channel, with a named door at each end and a corridor between them. Local puts a door on your side, remote puts one on the server’s side, and dynamic makes a whole proxy. Keep every door as private as the job allows, and remember that the destination is always judged from the far end.

    Picture this. You SSH into the lab, start a database dump that will take forty minutes, and then walk into a tunnel on the metro. The terminal freezes, then prints a line about the connection closing. When you reconnect, the dump is gone.

    It was never really running on the server. It was a child of your SSH session. When the session went away, the server sent a hangup signal to everything started inside it, and the dump went with it. The job lived inside the connection, not on the machine.

    tmux fixes that by moving your shell into a small server process on the VPS, one that does not care whether you are connected. You start a session, run the dump inside it, and detach. Close the laptop, go home, reconnect and attach again. The dump is still running, and its output is waiting for you.

    mosh attacks a different problem. SSH runs over TCP, and a TCP connection is tied to an IP address. Switch from office Wi-Fi to a phone hotspot and the session stalls, sometimes for minutes. Mosh runs over UDP and keeps its own session state, so it roams between networks and survives a laptop sleeping in your bag. It costs you something: it needs a range of UDP ports open on the server, it does not forward ports, and its scrollback behaves differently in some terminals.

    My opinion for a small production VPS: run long work inside tmux every time, and reach for mosh when your connection is flaky. Neither one survives a reboot of the server. For that, the job needs a service manager that starts it again at boot, which is the subject of Chapter 19.

    An SSH server open to the internet is a building with a door that every doorknob-rattler in the world can try. Scanners find a fresh port 22 within hours of it appearing. That is weather, not a personal attack, but it means the defaults matter.

    Harden in layers. Each layer is cheap, and each one still helps if another layer is missing.

    Keys only. Turn off password logins in the server’s SSH config. A password can be guessed, and bots try thousands of them a day. A good key cannot be guessed in any time frame you care about. Before you change this, keep your current session open, open a second one and confirm it logs in. A typo in the config can lock you out, and the open session is your rope back in.

    No root, and an allow-list. Refuse root logins, and list the users who may log in at all. On the lab VPS, ops is the door. app is a service account that never needs a shell, so it does not get one.

    fail2ban. This watches the logs and temporarily bans addresses that fail too often. It quiets the logs and slows the simplest bots. It is not a wall. A patient attacker holding a stolen key does not trip it, and it does nothing about a bug in sshd.

    The big move: no public SSH at all. The strongest version stops having a public SSH door. Your firewall (Chapter 14) drops port 22 from the internet, and you reach the server only over the tailnet from Chapter 16, where your own devices are the only things that can knock. Tailscale’s SSH can even use your tailnet identity as the login. Keep a recovery path that does not depend on the network, such as your provider’s web console, and write it down before you need it.

    SSH on the public internet, then SSH only over the tailnetLeft panel: scanners from the internet reach port 22 on the server directly. Right panel: the public side has no door, and a laptop on the tailnet reaches port 22 through an encrypted tunnel.Before: SSH on the public doorAfter: SSH only over the tailnetThe whole internet22 openno public door22 tailnet onlyLaptoptunnel
    Figure 6.5. Before and after: a public SSH door that the whole internet can knock on, and a private door that only your tailnet can reach.

    The figure shows the whole change in one picture. Nothing about the server got smarter. It just stopped standing on the street.

    SSH has a reputation for being boring and trustworthy. These three stories are why you should never fully trust that reputation, and why the habits in this chapter matter.

    The backdoor that was half a second slow (xz-utils, 2024)

    Section titled “The backdoor that was half a second slow (xz-utils, 2024)”

    In early 2024, Andres Freund, an engineer who works on PostgreSQL, was testing a Debian development machine and noticed that SSH logins were a little slower than they should be, and that the CPU was working harder for them. Most people would shrug. He chased it down.

    What he found was a backdoor in liblzma, the compression library shipped in the xz-utils package. Over roughly two years, a persona calling itself “Jia Tan” had made patient, helpful contributions to the project, earning trust until it could push a change into the release tarballs. On some Linux distributions, a patch to systemd meant that sshd loaded that library, so the backdoor sat right in the path of incoming SSH connections. As publicly reported, it was built so that a holder of a specific key could get in ahead of normal authentication.

    The attack targeted something you cannot audit by reading your own configuration: the trust we extend to maintainers and to the boring code underneath sshd. It was caught before, as far as public reports say, it reached most stable releases. The alarm was a performance number. Treat timing and CPU anomalies on your servers as security signals, not as noise.

    In mid-2024, the security firm Qualys published a vulnerability in OpenSSH’s server, nicknamed regreSSHion. It was a race condition in how sshd handled a signal when a connection had not yet logged in. The bug was an old one that had been fixed long before, then reintroduced by a later change. In principle it could allow code execution before authentication on some Linux systems. In practice, it was hard to exploit reliably.

    The lesson is not “patch faster”, although you should. It is that a daemon listening on the public internet has to be treated like a front door on the street. Exposure is the multiplier. A server that does not answer port 22 to the world is not safe because it is patched. It is safe because nobody can knock.

    GitLab, 2017: the delete on the wrong server

    Section titled “GitLab, 2017: the delete on the wrong server”

    In January 2017, a GitLab.com engineer, deep into a long and stressful night of recovery work on a database, ran a delete command on the primary production database when they meant to run it on a secondary. Over SSH, the two machines looked almost identical in the terminal. Several backup mechanisms turned out not to work as everyone had assumed. The team recovered from a snapshot that was, as widely told, about six hours old. GitLab’s public write-up of the incident became a classic.

    Chapter 19 tells the full story and the habits that would have caught it: naming hosts so they cannot be confused, restoring backups on purpose, and making the dangerous command hard to type by accident. For now, notice the bridge. The key was fine. The building was fine. The person was tired and in the wrong room.

    The lab: a hardened door on a throwaway server

    Section titled “The lab: a hardened door on a throwaway server”

    Goal. By the end, the lab VPS (hostname lab, Ubuntu 24.04, admin user ops) will accept you by key only, refuse root, keep its hello-pipes app private to the server, and keep a long job alive across a dropped connection. You will do this on the throwaway lab box, never on anything real.

    Conceptual steps.

    1. Make an ed25519 key pair on your laptop and protect the private key with a passphrase. Load the decrypted key into ssh-agent so you type the passphrase once per session, not once per command.
    2. Give the lab server a short alias in your ~/.ssh/config, with the user and key set once, so that the alias does all the remembering.
    3. Install your public key for ops on the server. Confirm you can log in with the key before changing anything else.
    4. Open a second terminal session to the server and keep it open for the rest of the lab. This is the one step you cannot skip. If you lock yourself out, the second session is your rope.
    5. Turn off password logins and root login on the server, reload sshd, and test from the second session before you close anything.
    6. Bind hello-pipes to the loopback address only, then reach it from your laptop through a local forward. Notice that the public address does not answer on that port.
    7. Start a long job inside tmux on the server, detach, drop your connection on purpose, reconnect, and reattach.

    How you’ll know it worked.

    • ssh lab connects with no password prompt. The agent or your passphrase is the only thing you type.
    • A deliberate password attempt against the server, with keys turned off on your side, is refused with Permission denied (publickey).
    • A login as root is refused, while ops still gets in.
    • The local forward lets your laptop’s browser load hello-pipes at a localhost address, while a direct attempt from the public address on that port times out or is refused.
    • The long job is still running after you reconnect, and tmux shows it where you left it.

    Commands live in the exercise hints below, not here. If you find yourself wanting a command, you are probably in the right place to go and read the hint.

    SSH is one door doing four jobs. It is an encrypted channel. It is how the server proves its identity to you. It is how you prove yours to it. And it is the pipe that your files and tunnels travel through. Most of what you do on a server rides on those four jobs.

    Here is the short list to carry out of this chapter:

    • Telnet and rsh sent passwords in plaintext. SSH encrypts the channel before anyone types a credential.
    • The host key is the server’s face, and known_hosts remembers it. When it changes, stop and find out why before you accept it.
    • Your private key never leaves your laptop. The server checks a signature, never a secret. ed25519 is the sensible default, and a passphrase plus ssh-agent makes it pleasant to live with.
    • Agent forwarding lends your key to whoever controls the machine in the middle. ProxyJump gets you there without the loan.
    • Local, remote, and dynamic forwards are the same idea pointed three ways: a door at each end, and a tunnel between them.
    • tmux keeps your work alive when your connection dies. mosh keeps your connection alive when your network does.
    • Hardening is layers: keys only, no root login, an allowlist of users, fail2ban for the noise, and then the big move, sshd reachable only over your tailnet (Chapter 16).

    The GitLab story in full waits for Chapter 19. The bridge is simple, though. An SSH session is only as safe as the command you type into it, and the most careful server in the world cannot tell a wrong database from the right one.

    You started this chapter able to follow a deployment tutorial. You now know why its commands work, what the warning banner is asking you, and which door to close first. That is real progress, so take a small victory before the quiz.

    1. Why did Telnet and rsh make the password a liability?

    They sent it in plaintext, so anyone who could watch the wire could read it. SSH encrypts the channel before the login step, so the credential travels inside a tunnel that nobody on the path can read.

    2. The banner says REMOTE HOST IDENTIFICATION HAS CHANGED. What are the two most likely explanations, and what do you do first?

    Either the server was rebuilt and got a new host key, or something on the path is intercepting the connection. Before you touch anything, confirm the new fingerprint out of band: ask whoever owns the server, or check the provider’s console. Only once you know which story is true should you remove the old line from known_hosts.

    3. Why does the server never need your private key to check you?

    It sends a challenge, your client signs it with the private key, and the server checks that signature against the public key stored in authorized_keys. The signature proves you hold the key without revealing it.

    4. Which way does each forward run: -L, -R, and -D?

    -L takes a port on your laptop and carries it to a destination the server can reach. -R takes a port on the server and carries it back to a port on your laptop. -D opens a SOCKS proxy on your laptop, and the server makes outbound connections on your behalf for whatever destinations your apps ask for.

    5. Why is agent forwarding risky even when you trust the host you are logging into?

    The forwarded agent socket lives on that host for as long as you are connected. Anyone with root there, or any process running as you, can ask your agent to sign things, and you would never see it happen. ProxyJump forwards the connection instead of the agent, so the key never travels at all.

    6. Your six-hour data migration died when your laptop went to sleep. What went wrong, and what would have saved it?

    The job ran in your login shell, attached to the SSH session. When the connection dropped, the shell got a hangup and the job went with it. Run long work inside tmux on the server, so the job belongs to the server’s session and you can reattach when you return. mosh helps on flaky networks, but tmux is what saves the job.

    7. Why is moving sshd to a different port a weak defense, and what is a stronger one?

    A new port cuts the noise in your logs, but scanners sweep every port often enough to find SSH wherever it hides. A stronger defense takes the door off the public internet: bind sshd to your tailnet address, or use Tailscale SSH, so that only your own devices can even attempt a connection.

    Warm-up 1. Read a fingerprint. A connection prompt shows SHA256: followed by a long string. What is it, and what do you do with it?

    Answer

    It is the fingerprint of the server’s host public key, a short hash that stands in for the whole key. Compare it with the value the server’s owner gives you through a channel you already trust, such as the provider console or a message from the team that runs the box. If they match, accept it. If they do not, stop and find out why.

    Warm-up 2. Name the forward. A database listens on 127.0.0.1:5432 on the server. You want localhost:5432 on your laptop to reach it. Which flag do you use, and which side opens the door?

    Answer

    -L, a local forward. Your laptop opens the port and carries each connection through the tunnel. The server then connects on to its own loopback address. Because the database listens only on loopback, this forward is the only way in from outside the box.

    Warm-up 3. Why turn off passwords? Give one reason the lab asks you to disable password logins.

    Answer

    Passwords can be guessed, and doorknob-rattlers try them around the clock. A key cannot be guessed, because the server only checks a signature made with a key you never share. Turning passwords off removes the guessable path entirely.

    Lab 4. Two sessions, one rule. Disable password logins and root login on lab, and keep your first session working the whole time.

    Hint

    Open a second terminal and log in with your key before you touch anything. Edit the server’s sshd settings in the first session only after the second one works. Check the syntax before you reload. The command for that check is sudo sshd -t.

    Model solution

    Set PasswordAuthentication no and PermitRootLogin no in the sshd config, test the syntax, reload the service, and then try a password login from a third terminal. It should fail with Permission denied (publickey). Keep the first session open until you have confirmed that a fresh key login still works.

    Lab 5. A job that survives a drop. Start a long job on lab, disconnect, and find it still running when you come back.

    Hint

    Use tmux. Create a named session, start the job inside it, and detach with the prefix key followed by d. Reconnect and list the sessions, then attach to the one you made.

    Terminal window
    tmux new -s job
    tmux ls
    tmux attach -t job
    Model solution

    The job runs inside a tmux session on the server, not in your login shell. When your connection drops, the session keeps running. After you reconnect, tmux ls shows the session still alive, and attaching shows the job’s latest output.

    Lab 6. One word to log in. Make ssh lab work with no flags, the right user, and your ed25519 key.

    Hint

    Add a Host lab block to your client config with HostName, User, and IdentityFile. Use the public address from the lab world. Check the result with ssh -G lab before you test the login.

    Model solution
    Host lab
    HostName 203.0.113.10
    User ops
    IdentityFile ~/.ssh/id_ed25519

    ssh lab now logs you in as ops with your key and no prompt for a password. The effective settings show in ssh -G lab.

    Lab 7. A door that only opens inward. Serve the hello-pipes app on 127.0.0.1:3000 on the server, and reach it from your laptop at localhost:8080 without opening port 3000 to the world.

    Hint

    Use a local forward from your laptop to the server’s loopback address. Then check two things: the forwarded address answers, and the public address on port 3000 does not.

    Model solution

    Run ssh -L 8080:127.0.0.1:3000 lab and leave it open. A request to localhost:8080 on your laptop reaches the app. A request to 203.0.113.10:3000 gets refused, because nothing listens there. The app is never exposed publicly, and the tunnel is the only way in.

    Staff 8. Bastion without the agent. Your team reaches a private database host only through a bastion. Design the access path so that no one forwards their agent to the bastion.

    Model answer

    Use ProxyJump in the client config, with a Host pattern for the database hosts that names the bastion. The client connects to the bastion and then runs a second SSH session through it, end to end to the database host. The bastion carries only encrypted bytes, and your agent socket never lands there. Set ForwardAgent no as the default. Put the bastion behind your tailnet or key-only auth, use one account per person so access can be revoked, and log the sessions.

    Staff 9. Argue the other side. A colleague says, “We run fail2ban, so we are safe.” Make the case in three sentences that fail2ban is one layer, not the wall.

    Model answer

    fail2ban cuts the noise and blocks repeat guessers, which is worth having. It is reactive, though, so it cannot stop an attacker who spreads guesses across many addresses or who presents a stolen key. Key-only logins, an allowlist of users, and reachability only over the tailnet remove the guessing surface before fail2ban ever needs to act.

    Book 2 of a series. Book 1: The Glue.