Everything That Goes Wrong, Explained by One Picture

The method is embarrassingly simple. Loss means a queue overflowed. Delay means a queue is too long or a timer fired. Everything else is working out which queue.

The ladder, in full

Stated as a table, receive path first:

And the transmit path:

The catalogue

Packets are dropped and the machine is idle

Where: the RX ring.

A queue does not overflow because of average load. It overflows because of the worst single gap between drains. A core that is sixty percent idle can still be unavailable for two hundred consecutive microseconds while it services a different interrupt, runs a long softirq for another queue, or executes a piece of your code that never yields.

Confirm: rx_missed_errors climbing while CPU is well under saturation.

Fix: a larger ring buys more tolerance for those gaps. Spreading receive work across more cores shortens them. Neither changes the average, because the average was never the problem.

Throughput falls as offered load rises

Where: softirq processing, spilling into ksoftirqd.

The receive path is consuming so much CPU that the work of actually responding cannot get a turn. It is the shadow of the livelock from Post 05 — the modern design prevents the total collapse but not the competition.

Confirm: column 3 of softnet_stat rising, ksoftirqd visible in top, and one core’s %soft far above the others.

Fix: get more cores involved in receive processing, or get the application off the cores doing it. If one core is pegged and the rest are idle, the card’s hashing is putting everything in one queue — which is common behind a proxy, where all traffic shares one source address.

Clients connect successfully, then time out

Where: the accept queue.

The most misleading failure in server operations, for the reason Post 06 gave: the handshake completes before your application is involved. The client has a working connection by every test it can perform. It sends a request into a socket that exists in your kernel and has never been handed to your code.

Your application logs nothing, because from its point of view nothing happened.

Confirm: TcpExtListenOverflows incrementing, and a non-zero Recv-Q on the LISTEN row of ss -tln.

Fix: accept faster, or raise the backlog to absorb bursts. Raising the backlog without accepting faster converts refusals into slow responses, which is sometimes better and sometimes much worse.

Memory climbs because of one slow client

Where: the socket send queue.

Your handler writes a response and returns. If the client reads slowly — a phone on a bad connection, or a deliberately hostile client — the bytes sit in that socket’s send queue, and the queue is kernel memory attributed to your connection. Multiply by a few thousand such clients.

Confirm: Send-Q large and persistent on many connections, with small cwnd values.

Fix: write timeouts, and limits on how much may be buffered per connection. The version of this attack that deliberately reads one byte at a time has a name and is old, and the defence has always been timeouts rather than buffers.

Exactly forty milliseconds

Where: a timer, not a queue.

Post 10’s collision: the sender holding a small write because earlier small data is unacknowledged, and the receiver holding the acknowledgement because it has nothing to piggyback on.

Confirm: the number is suspiciously constant. Real congestion is noisy; timers are not.

Fix: write the response in one call. Failing that, disable the sender-side rule for that socket.

The first request is slow, the rest are fast

Where: the congestion window, plus handshakes.

A new connection pays a round trip for the TCP handshake, one or two more for encryption setup, and then starts with a window of about fourteen kilobytes that must double a few times before it can deliver a large response in one go.

Confirm: compare time_appconnect and time_starttransfer from Post 09’s curl format on a cold connection and a warm one.

Fix: keep connections alive and reuse them. This is the single highest-value thing most services can do, and it is usually a configuration line.

Latency is terrible and nothing is being lost

Where: a buffer somewhere that is too large.

This one deserves its own paragraph because the signature is so distinctive. Queues absorb bursts, so the instinct when things go wrong is to make them bigger. But a queue that never overflows is a queue that is always full, and a packet entering a full queue waits behind everything already in it. You have traded a dropped packet — which TCP would have detected and repaired in one round trip — for a packet that is delivered very late, which TCP cannot detect at all, because as far as it can tell the path is simply slow.

Worse, TCP’s whole control loop depends on loss as its signal. Remove the signal by adding buffer and the sender keeps increasing its rate, filling the oversized buffer further, until latency is measured in seconds.

Confirm: rising round-trip time in ss -ti with retrans near zero, and non-trivial backlog in tc -s qdisc.

Fix: a queueing discipline that keeps itself short by dropping early and deliberately. This is exactly what the modern Linux default does, and its apparent drops are the mechanism working.

One slow response delays all the others

Where: head-of-line blocking, at one of two layers.

At the application layer, several requests sent on one HTTP/1.1 connection must be answered in order, because a stream has one dimension. HTTP/2 solves this by interleaving tagged chunks.

At the transport layer, TCP delivers one ordered stream, so a single lost segment stalls everything behind it — including, on an HTTP/2 connection, requests that have nothing to do with the lost data. Post 07’s out-of-order queue is precisely this: the bytes have arrived, and the kernel is forbidden from giving them to you because of a gap in front of them.

Confirm: several concurrent requests that all complete at the same instant, after a delay that matches a retransmission timer.

Fix: at the application layer, HTTP/2. At the transport layer, nothing — which is what Post 12’s last topic is about.

The p50 is fine and the p99 is dreadful

Where: everywhere. This is the worst-gap phenomenon from the first entry, generalised.

Every queue on the ladder is drained by something that runs intermittently. Add a garbage collection pause in your runtime, a scheduler delay because a core was busy in softirq, a retransmission timer, and a moment of accept-queue pressure — each individually rare — and the tail is the union of all of them.

And if your clients are on Wi-Fi, a large part of the tail is not on your machine at all. Post 04 left this owed: 802.11 acknowledges and retries every unicast frame individually, and renegotiates its data rate continuously against measured conditions. A frame that needed four attempts arrived four times later, and nothing on your server records that it happened — by the time it reached you it was an ordinary Ethernet frame with no history. Before spending a week on your p99, check whether it is your p99.

Confirm: it is not one thing, and looking for one thing is the mistake. Compare your handler’s own timing against the end-to-end timing; the gap is everything below you. Then compare your wired clients against your wireless ones, because if the distributions differ, the tail is theirs.

Fix: shorten the queues, reduce the variance in the drains, and stop reasoning about averages.

The method, stated plainly

Four questions, in order.

Is this loss or delay? They are different diseases with opposite treatments. Loss says something overflowed; look for a queue that is too small or drained too rarely. Delay says something waited; look for a queue that is too long, or a timer.

Which direction? Receive-path problems are about keeping up. Transmit-path problems are about permission and policy.

How far up did it get? Walk the ladder from the bottom. Each rung has a counter, and the first one that is unhappy is usually where the answer is — because a problem at one rung produces symptoms at every rung above it.

Is the number constant? Constant means a timer and therefore a protocol interaction. Noisy means contention.

See it for yourself

One command reads most of the kernel-side counters in the table.

TcpRetransSegs                  184203             0.0
TcpExtListenOverflows           1142               0.0
TcpExtListenDrops               1142               0.0
TcpExtTCPBacklogDrop            0                  0.0
TcpExtPruneCalled               0                  0.0
TcpExtTCPTimeouts               3311               0.0

Overflows and drops matching exactly, as above, is the accept-queue story and nothing else. Note that the SYN-queue counter splits in two depending on configuration: with syncookies enabled — which is the default almost everywhere — a full SYN queue does not drop the connection, it answers with a cookie instead, and increments a different counter. Looking only for the drop counter on a default system finds a zero and proves nothing. Retransmissions rising with timeouts flat is ordinary loss being repaired quickly. Retransmissions rising with timeouts is loss being repaired by the slow path, which costs hundreds of milliseconds each time.

And on the listening socket specifically:

State   Recv-Q  Send-Q   Local Address:Port
LISTEN  127     4096           0.0.0.0:8080

The six ideas, one last time

Everything in this post is an application of the same small set, which is what they were chosen for.

Every boundary is a queue, and every queue is finite. Work happens in execution contexts you do not control, which is why the boundaries exist at all. Buffers have owners, and handoffs are transfers of responsibility. Demultiplexing is one motion repeated at every altitude. Backpressure is how each stage says slow down, and each layer says it differently. Framing is destroyed by the transport and rebuilt by you.

If you hold those, you do not need to remember the counters. You can work out which one must exist and go looking for it, which is a considerably more durable skill than memorising a table.

One post remains, and it is a test rather than a lesson: four places where the picture changes, and what survives the change.