Skip to content

2. Packets, Routes and Handshakes

“Nobody sends a packet to Frankfurt. They hand it to the next router and hope.”

It is 2 a.m. and a student writes in to say the class-booking page on your dance-academy app “just spins.” This is a composite, but you have had some version of the call. You open a terminal, type ping followed by the VPS address, and watch the screen fill with nothing. Four packets sent, four lost, 100% packet loss. Your stomach drops. The server must be down.

Then you open the site in your own browser. It loads. The booking page renders in under a second, the login works, and the padlock sits where it always does. So either the server is down and somehow serving pages, or the tool you trust most just told you something false.

It did not lie, exactly. It answered a different question from the one you asked. Ping asked the server “are you there?” in a protocol the server had been told not to discuss with strangers. The website was asking on a different door, and that door was open.

That gap, between what a tool measures and what a user experiences, is where most networking confusion lives. This chapter is about the machinery underneath it: packets, the routes they take, the handshakes that make a conversation reliable, and the one budget you cannot negotiate, the speed of light. Ch 1 gave every machine an address. Ch 3 will give every conversation a door. In between, a packet has to get from one to the other, and that is the job of this chapter.

Book 1 covered the basics of these pieces, including DNS, TCP, TLS and HTTP, in its network chapter. If you read it, the nouns will feel familiar. Here we go one level down and ask operator questions: what does this cost, what is allowed to be blocked, and what will the tool actually tell me at 2 a.m.?

By the end you will be able to explain that ping result in one sentence, and you will know what to type next instead of panicking. Here is the sentence now, so you can check yourself later: the server answered on port 443 and refused to answer ping, so both facts are true, and neither one is an outage.

Packets: sealed envelopes with a destination on the outside

Section titled “Packets: sealed envelopes with a destination on the outside”

Start with something you already do without thinking. When you open a page, your browser does not send “the page” down a wire. It sends a request, and the network carries that request as a stack of small numbered parcels called packets. On an ordinary Ethernet or Wi-Fi link a packet carries at most about 1,500 bytes of payload. A large file becomes thousands of packets, and the machine at the far end has to put them back in order.

Think of each packet as a sealed envelope. The outside carries what the network needs: where it is going, where it came from, and how many more hops it is allowed to take. The inside carries your data. Routers only ever read the outside. That one fact explains most of what follows, including why a firewall can block a packet without knowing a single thing about the page inside it.

The textbook model has seven layers. You will meet it in old diagrams with names you will never need on a Tuesday. In practice four layers do the work for a web request, and each one answers exactly one question.

  • Application asks: what are we saying? This is HTTP, the request for the home page and the response that comes back.
  • Transport asks: which conversation is this, and did it arrive? This is TCP or UDP, and this is where port numbers live. Chapter 3 spends a whole afternoon on ports, so we will leave them there for now.
  • Network asks: where is the destination, and which way do I go? This is IP, with the addresses from Chapter 1, and it is where routing happens.
  • Link asks: who is the next machine on this particular wire or Wi-Fi network? This is Ethernet or Wi-Fi framing, and it only ever spans one hop.

Each layer wraps the layer above it, like a parcel inside a parcel. Your laptop wraps all four. The router in the middle opens only the outermost network envelope, reads the address, and wraps it again for the next wire. The server, at the far end, unwraps all four in reverse order. If you remember nothing else from this section, remember that the middle of the network is busy with the outside of the envelope and nothing else.

Follow one packet from a laptop in Pune to a server in Frankfurt. The route in this example is illustrative, not a map of any real provider, but the shape is right.

A packet hopping across five boxesFive boxes in a row, from your laptop in Pune on the left to a server on the right. Arrows show the packet moving left to right. Each box has a role, and a note under the row says the time to live drops by one at every router.Each box reads only the destination, then picks the next boxYour laptopwraps all fourHome routerNAT gateISP edgepicks next hopBackbonemany routersServeropens all fouroriginhop 1hop 2several morelast hopTime to live, the hop budget, drops by one at every router. At zero the packet is discarded.
Figure 2.1. A packet is re-addressed at the edges and only forwarded in the middle. Every box decides one hop at a time.

Your laptop wraps the request in a transport envelope, then a network envelope with the server’s address on the outside. Your laptop sends the packet over Wi-Fi to the home router. The home router is a gate guard of the most useful kind. It rewrites the packet’s return address to its own public address, and it remembers which device asked, so the reply can be sent back to you. That trick is NAT, and Chapter 1 has the gated-society version of it.

The packet then leaves your home through the ISP’s edge router. The edge looks up the destination in its table, chooses the next box, and forwards the packet. The packet crosses several backbone routers, each one doing the same small job: read the destination, choose a neighbour, decrement the time-to-live counter, forward. At the far end, the last router hands the packet to the server’s own network, and the server unwraps the four layers in reverse.

Two things are worth holding onto. First, every router makes a local decision. None of them holds the whole route, and none of them is in charge of your request. The path is what you get when many small decisions line up. Second, the reply does not have to come back the same way. Your request and the server’s answer can take different roads, because each direction is its own set of local decisions.

Routing: a table, a rumour and a lot of trust

Section titled “Routing: a table, a rumour and a lot of trust”

Every router you will ever meet does one job, very fast. It reads the destination address on a packet, finds the matching row in its routing table, and hands the packet to the next hop named in that row. It does not know the whole internet. It only needs to know who is one step closer.

The routing table is the thing to picture. Each row says, in effect, “for addresses in this range, send the packet to that neighbour.” When more than one row matches, the most specific row wins. A route for one host beats a route for a whole network, and a route for a network beats the default. The default route, written 0.0.0.0/0 for IPv4, is the catch-all: everything else goes to the upstream. Your VPS has, for practical purposes, one default route pointing at its provider’s edge.

A routing table lookupA destination address is compared with three routing table rows. The row for one host is the most specific match and wins; the default route is the fallback.Packet destination203.0.113.10203.0.113.10/32direct, wins (most specific)203.0.113.0/24via peer router0.0.0.0/0default, via upstream edge
Figure 2.2. A router picks the most specific matching row, and that row names the next hop, not the destination.

Routers fill their tables in two ways. Directly connected networks are free: if a cable goes into a network, the router knows that network exists. Everything beyond that comes from routing protocols, in which routers tell each other what they can reach.

Inside one organisation, protocols such as OSPF let routers share a map of their own network. Between organisations, the protocol is BGP, the Border Gateway Protocol. Each network under one operator’s control, an ISP, a cloud provider, a university, a large company, is called an autonomous system. Each one announces the address blocks it can reach, and its neighbours pass those announcements on, adding themselves to the path. A router that hears several paths to the same block chooses one. Path length counts, but operator policy usually counts for more: “prefer the paying customer over the free peer, and send my traffic out through this transit provider first.”

BGP is a rumour mill with a rulebook. It works because every neighbour is trusted to tell the truth, and it fails in the same way. An announcement is a claim, and the internet believes claims until someone who should know better says otherwise. The Glue Book’s networking chapter walks through the 2021 Facebook outage, when a bad change withdrew the company’s routes from the wider internet. Read that story at https://the-glue-book.pages.dev/part-1/ch05-network/ rather than here. What matters for you is the lesson: a route can vanish, and nothing on your own server will tell you why.

You will almost never run BGP yourself. Your VPS provider does, and what you control is the choice of provider and region, which decides your upstream. For a small production box, three habits are worth more than any clever network trick:

  • Ask the provider whether they offer a looking glass or a traceroute from their own network. A route that looks good from their data centre can look different from a Pune home connection.
  • Put a CDN or a proxy in front of anything public. It takes the long-distance routing problem off your box.
  • Remember that routes are not always symmetric. The path a request takes out can differ from the path the reply takes back, so a traceroute shows one direction only.

Remember the postcard from the packet section. IP gets it to the right building, and says nothing about whether it arrived, arrived in order, or arrived at all. Two protocols sit on top of IP and decide what happens next.

TCP (Transmission Control Protocol) makes promises. It numbers every byte, asks the other side to confirm what it received, resends what went missing, and hands your application one tidy, ordered stream. UDP (User Datagram Protocol) makes almost no promises. It sends each datagram and forgets it. If one is lost, nobody tells you, and nobody resends it unless the application builds that itself.

Neither is the better protocol. Pick by what a late or missing piece costs you. A checkout request, a database query or a webhook body needs every byte, in order, so it rides on TCP. A video frame that arrives 200 ms late is worthless, so a live call would rather skip it and move on. That is why it rides on UDP.

Before TCP sends any of your data, both sides must agree on starting sequence numbers and confirm that packets flow in both directions. That takes three messages, and the picture is the cheapest way to remember them.

The TCP three-way handshakeTime runs downward. The client sends SYN, the server replies SYN-ACK, the client sends ACK, and only then does the request travel.Client (Pune)Server (VPS)1. SYN (seq = x)2. SYN-ACK (ack = x+1, seq = y)3. ACK (ack = y+1)4. Request finally leavesAbout one round trip, roughly 230 ms from Pune to Ashburn.
Figure 2.3. The handshake costs one full round trip before the first byte of your request can leave.

The client sends SYN, short for synchronise, with its starting number. The server replies SYN-ACK, which acknowledges that number and offers its own. The client sends an ACK, and only then does the request go out. Each side has now shown the other that its messages arrive.

Notice what that costs. From Pune to a VPS in Ashburn, one round trip is roughly 230 ms on a good day, so the handshake alone spends a quarter of a second before your app has said a word. A request for a single page waits on that before anything useful happens. Keep the number in your head. The rest of this chapter keeps coming back to it.

Watch the handshake in slow motion. Pick a city, then press both buttons. The TCP run shows the extra round trips that encryption adds on top, and the QUIC run shows how HTTP/3 tries to save one of them.

Try it: count the round trips before the first byte

Handshake between client and serverTwo vertical lifelines: the client in Pune on the left, the server on the right. Arrows show each packet leg in order from top to bottom.Client (Pune)Server
Simulated clock: 0 msPick a mode to start.

Static fallback: TCP opens a connection in one round trip, TLS 1.3 adds another, and the request and its first byte take a third. QUIC folds transport and TLS into one exchange, so the same first byte takes two. At Frankfurt's roughly 140 ms, that is about 420 ms against 280 ms before the server does any work. Packet loss adds a retransmit timer, which is longer for TCP's starting value than for QUIC's.

Ping and traceroute: two thinking tools that tell half the truth

Section titled “Ping and traceroute: two thinking tools that tell half the truth”

Ping and traceroute are the first two tools most people reach for, and both are built on ICMP. ICMP (Internet Control Message Protocol) is neither TCP nor UDP. It is how the network talks about itself: echo request, echo reply, “destination unreachable”, “time exceeded”. Ping is just the friendliest use of it.

Ping sends an echo request and times the reply. That gives you two facts: whether something answered, and how long the round trip took. Run it from your laptop to a Frankfurt host and you will see a number near 140 ms. That number is the light budget at work, and we will come back to it.

Ping is a thinking tool, not a verdict. Plenty of servers drop ICMP on purpose, because they do not want strangers poking them. So “ping is blocked” means “this host does not answer ping”. It does not mean “this server is down”. Your web server can be serving thousands of HTTPS requests and still ignore every echo. Before you panic, check with the protocol the service actually speaks.

Every IP packet carries a TTL, a time to live. Each router that forwards it subtracts one. When the count reaches zero, the router throws the packet away and sends the sender an ICMP “time exceeded” message. TTL was invented to stop packets circling forever. Traceroute turns it into a map.

It sends a packet with TTL 1. The first router kills it and announces itself. It sends TTL 2, and the second router does the same. It keeps going until a packet survives to the destination, which replies normally. The routers that answered, in order, are the path.

Traceroute with rising TTLThree probes leave the laptop with TTL 1, 2 and 3. The first expires at router 1 and the second at router 2, and each router replies with time exceeded. The third reaches the server, which replies normally.Laptop (Pune)Router 1Router 2ServerTTL 1dies at router 1TTL 2dies at router 2TTL 3reaches the server, which repliesEach row is one hop further. Time-exceeded replies come back from each router on the way.
Figure 2.4. Traceroute is a row of packets that each die one hop further along than the last.

Read the output with two cautions. A line of stars means that router is set not to answer, though traffic still flows through it. Round-trip times are noisy, so judge the trend across hops, not one number. A hop that looks slow while every hop after it looks fast is usually a router that deprioritises ICMP, not a slow path. Trust the numbers that persist.

Who decides the path? Not your laptop. Each network announces to its neighbours which address blocks it can reach, using BGP (Border Gateway Protocol). The route you see in traceroute is the result of huge numbers of those announcements settling on a path. BGP is a story in its own right, and when it goes wrong it goes wrong in public. Book 1 tells the Facebook 2021 outage properly, so we will link rather than retell it: https://the-glue-book.pages.dev/part-1/ch05-network/

Priya opens the dance academy’s booking page on a Pune tower and each tap feels slightly sticky. Nothing is broken. The page is hosted in Frankfurt because the tutorial used the first region in the dropdown, and far is a physics problem, not a configuration problem.

Light in fibre travels at roughly 200,000 km per second, about two-thirds of its speed in a vacuum. That is about 5 microseconds per kilometre one way, or 10 for a round trip. You can buy bandwidth, a bigger VPS or a CDN. You cannot buy a shorter path than the glass allows.

Rough great-circle distances from Pune give you floors. Mumbai, about 150 km, so about 1.5 ms. Singapore, about 3,900 km, about 39 ms. Frankfurt, about 6,300 km, about 63 ms. Ashburn, about 13,000 km, about 130 ms. Treat these as minimums, never as forecasts. Cables do not run in great circles and every router adds its own delay, so real numbers land a couple of times above the floor.

Round trips from Pune against the speed-of-light floorFour rows: Mumbai, Singapore, Frankfurt and Ashburn. Each has a long amber bar for a rough observed round trip and a short teal stub for the physical minimum. The gap grows with distance, and in Mumbai the observed time is many times the floor because the last mile dominates.Mumbaiabout 150 kmabout 15 msfloor 1.5 msSingaporeabout 3,900 kmabout 70 msfloor 39 msFrankfurtabout 6,300 kmabout 140 msfloor 63 msAshburnabout 13,000 kmabout 230 msfloor 130 msBars drawn at 2 px per millisecond. Amber is observed, teal is the floor.
Figure 2.5. From Pune, the amber bar is what you feel and the teal stub is the physics it cannot beat.

The Mumbai row is the interesting one. Observed is about ten times the floor, because below a hundred kilometres or so, distance stops being the story. Your last mile, your Wi-Fi, your ISP and the data centre’s own network take over.

Round trips are what you actually pay in. Before a single byte of a page arrives over TCP with TLS 1.3, you need about three round trips: one for the TCP handshake, one for the TLS handshake, and one for the request and its first response byte. At 140 ms to Frankfurt, that is roughly 420 ms of waiting. QUIC merges the transport and crypto handshakes, so the same exchange takes about two round trips, around 280 ms. Connection reuse and 0-RTT resumption shave more. QUIC is not magic. It just stops charging you one extra round trip.

Where should the box live? For a Pune audience, a Mumbai VPS is the default. Frankfurt earns its place when your users or your data are in Europe, and Ashburn when your partners are in North America. Pick by the round trip your users feel, not by the first region in the dropdown. And if the app and its database sit in different cities, every query pays the fare too.

The lab: measure the light budget from your own laptop

Section titled “The lab: measure the light budget from your own laptop”

Goal. Measure round-trip times from your laptop to a Mumbai host and a Frankfurt host, then hold the numbers against the physics. You should finish with one floor you trust and one real time you can explain.

Conceptual steps.

  1. Pick two targets. Use a VPS in Mumbai (your lab box, if it lives there, or any small one you can rent for an afternoon) and a second one in Frankfurt.
  2. Measure the round trip to each many times, not once. Keep the minimum and the spread. The minimum is the closest you will get to the physics. The average is what your users feel.
  3. Compare each minimum with the floor from Figure 2.5: about 1.5 ms for Mumbai and about 63 ms for Frankfurt. Write down the gap, and guess what it is made of: queueing, cable detours, or the far end doing its own work.
  4. Trace the path to the Frankfurt host. Count the hops, find the one where the time jumps by tens of milliseconds, and say what kind of link you think it crosses. Ignore a single silent hop. Routers that refuse to answer are common and harmless.
  5. Find a host where ping gets no reply but HTTPS works. Pick a site you use every day, confirm it does not answer echo requests, then confirm a full HTTPS request still returns a status code. Explain both facts in one sentence.
  6. Write one verdict line per target: reachable and measured, or echo filtered but service fine.

How you’ll know it worked. Your Mumbai minimum sits within a few milliseconds of the floor. Your Frankfurt minimum is above 63 ms and plausibly somewhere near 100 to 150 ms once the cables take their scenic route. If a minimum comes in below the floor, your distance estimate or your clock is wrong, so recheck both before you trust anything else. You can point at the hop where the long-haul crossing happens, and you can say, without hedging, why a silent ping does not prove a dead server. If you can do all four, the lab has done its job.

You came in knowing that your VPS, your domain and your Next.js app were talking to each other. You now know why that talk takes the time it takes. A packet is a sealed envelope with a destination on the outside, and no router holds the whole route. Each router holds a table, makes one local decision and passes the envelope on. Routes are a rumour the internet has agreed to trust, and BGP is how the rumour travels.

TCP buys order and reliability with a handshake that costs a full round trip before your request leaves. TLS 1.3 adds another round trip on top. QUIC folds the transport and the crypto handshake into one, and it runs on UDP because TCP is baked into kernels and middleboxes, so changing it is nearly impossible. UDP lets QUIC evolve in user space.

Ping and traceroute are good thinking tools, and they tell you half the truth. A silent ping is far more often a filter than a dead server. The light budget is the one number you cannot negotiate. Pune to Frankfurt has a floor of about 63 ms round trip, and no tuning changes that.

Well done. You can now read a traceroute, explain why a handshake costs what it does, and stop blaming the server when a packet goes quiet.

1. Does a router know the full path a packet will take?

No. A router holds next-hop entries and picks the most specific match for the destination. The full journey is the result of many local decisions.

2. What does the TCP handshake cost before your first request byte?

One full round trip. The client sends SYN, the server answers SYN-ACK, and the client’s ACK can already carry the first data bytes. TLS 1.3 then adds one more round trip before the encrypted request.

3. Why does HTTP/3 run on UDP instead of TCP?

TCP lives in operating system kernels, and middleboxes expect its exact shape, so changing it is nearly impossible. UDP is a thin envelope, so QUIC can implement reliability, streams and encryption in user space. QUIC also avoids head-of-line blocking across streams, because loss on one stream does not stall the others.

4. How does traceroute learn the hops on a path?

It sends probes with an increasing TTL. Each router that decrements the TTL to zero drops the probe and returns an ICMP Time Exceeded message, which reveals its address. The destination answers differently, which tells traceroute it has arrived.

5. A server does not answer ping. What have you proven?

Only that no echo reply reached you. The host might be down, or it might be up and filtering ICMP, which is common. To learn whether the service works, test the service itself, for example with a full HTTPS request.

6. Roughly what is the round-trip floor from Pune to Frankfurt?

About 63 ms. Take roughly 6,300 km, multiply by about 5 µs per km for one way in fibre to get about 31 ms, then double it. Real paths are longer, so measured times sit above this number, never below.

7. Why is a handshake measured in round trips rather than milliseconds?

Because distance sets the duration of one round trip, and physics fixes that distance. A protocol that needs fewer round trips is faster on every link, while faster hardware only shrinks the processing part.

8. What does a ping that fails but a curl that succeeds tell you?

The host is up and serving the protocol you tested, but it does not answer ICMP echo, which is a filtering policy. Name the layer that failed, not the whole host.

Warm-up 1. Read the envelope. A packet leaves your laptop at 192.168.1.7 for 203.0.113.10. Name the field a router reads to pick the next hop, and the field it changes at every hop.

Answer

The router reads the destination IP address and looks it up in its routing table to choose the next hop. At every hop it decrements the TTL field by one. The source address and payload pass through unchanged. If the TTL reaches zero, the router drops the packet and sends an ICMP Time Exceeded message back to the source.

Warm-up 2. Draw the handshake. Sketch the TCP three-way handshake between your laptop and the lab VPS. Label each message and say which side is waiting for what.

Answer

The client sends SYN, the server replies with SYN-ACK, and the client answers with ACK. The client can send its first request right after that ACK. The server treats the connection as established once the final ACK arrives. Each message is one trip across the network, so the handshake costs one round trip before any application data moves.

Warm-up 3. Price the distance. Using the chapter’s rule of about 5 µs per km one way in fibre, work out the minimum round-trip time from Pune to Ashburn for a great-circle distance of about 13,000 km. Compare it with the roughly 230 ms you measured.

Answer

One way is about 65 ms, so the round trip floor is about 130 ms. The measured 230 ms is roughly 100 ms above that floor. The gap comes from routing detours, queuing, and time spent inside routers and endpoints. The lab asks you to find where it comes from.

Lab 4. Measure the Mumbai floor. From your laptop, measure the round-trip time to a host you know is in Mumbai. Compare the minimum of several samples with the light-budget floor for that distance.

Hint

Run ping with a count of ten against the host, using a placeholder HOST:

Terminal window
ping -c 10 HOST

Read the min value, not the average. Queues only add delay, so the minimum is the closest you will get to the physical floor.

Model solution

Mumbai is roughly 150 km from Pune, so the floor is about 1.5 ms. Your minimum will sit above that, because your home connection and the path out of India add their own delay. A gap of a few milliseconds is ordinary. A minimum several times the floor is worth a traceroute, which is Lab 5.

Lab 5. Find the long-haul hop. Run a traceroute to a Frankfurt host. Find the hop where the round-trip time jumps the most, and say whether that jump is distance or routing.

Hint

Use traceroute with a placeholder HOST:

Terminal window
traceroute HOST

On macOS, mtr shows the same path with repeated samples: mtr -rwc 10 HOST. Hops that show stars are routers that do not answer probes. That is not a broken link, so skip them when you read the numbers.

Model solution

The biggest jump usually comes where the path crosses a long-haul link, such as a submarine cable or an inter-city backbone. Compare the jump with the light budget. A jump near the distance floor is physics. A jump far larger than the distance explains is usually a detour through an extra exchange.

Lab 6. Find the silent host. Find a host where ping gets no reply but HTTPS still answers. Explain in one sentence why the two results disagree.

Hint

Try the two checks against the same HOST placeholder:

Terminal window
ping -c 3 HOST
curl -sI https://HOST

Try a few hosts that are known to filter ICMP. If ping times out and curl prints an HTTP status line, you have found one.

Model solution

Ping sends ICMP echo requests, and many servers and firewalls drop them on purpose. curl opens a TCP connection to port 443 and then speaks HTTPS. The server is up, and only one kind of message is being filtered.

Lab 7. Repeat at another hour. Run the Lab 4 measurement again at a different time of day. Record both minimums and say whether the floor moved.

Hint

Run the same ping command as Lab 4 at a second time. Compare only the minimum values. Averages move with load, but minimums should barely change.

Model solution

The minimum should stay nearly the same, because the floor is set by distance and the speed of light, not by the clock. If the average moved a lot while the minimum held, the extra time is queuing somewhere on the path.

Staff 8. Round trips before the first byte. A user in Pune opens a page from a Frankfurt server over TCP with TLS 1.3, then sends a request. Count the round trips before the first response byte arrives. Then say what QUIC changes and what 0-RTT costs you.

Model answer

TCP takes one round trip for the handshake and TLS 1.3 takes one more. The request then costs one more round trip before its first response byte returns, so about three round trips in total. At Frankfurt’s floor of roughly 63 ms, that is about 190 ms before the first byte. QUIC merges the transport and TLS handshakes, so the same answer costs about two round trips, near 130 ms. 0-RTT resumption can send the request in the first flight, roughly one round trip, but that early data can be replayed by an attacker. Servers should accept it only for requests that are safe to repeat.

Staff 9. The colleague’s message. A colleague writes: “Ping is blocked on the VPS, so the server is down. Let’s rebuild it.” Write a short reply that uses the chapter’s ideas, in under 100 words.

Model answer

“Before we rebuild, run curl against https on the VPS. Ping uses ICMP, and providers and firewalls often drop it on purpose. If curl gets a response, the app is reachable and the rebuild is not needed. If curl also fails, check DNS, then the firewall, then the process on the box, in that order.” The reply makes the key point: silent ping proves little, and a curl test asks the question you actually care about.

Book 2 of a series. Book 1: The Glue.