Memory Two Things Can Touch

This post is about the mechanism that explains it, which turns out to be the same mechanism that explains every copy in the networking stack, and the reason “zero-copy” is a meaningful phrase rather than marketing.

The whole thing reduces to one question: who owns this memory right now?

The symptom

The counter that tells you is on the card, not in the kernel.

     rx_missed_errors: 48213
   rx_no_buffer_count: 1102
   rx_fifo_errors: 48213
   rx_dropped: 0

Note what this is not. It is not the kernel deciding to drop something. It is not a firewall rule. It is not your application being slow to read. It happened below all of that, in hardware, and the only trace is a counter you have to ask for.

To understand it you need to know what the card was looking for and did not find.

The obvious design

The intuitive picture of packet reception goes something like: a frame arrives, the card tells the CPU, the CPU comes and fetches it.

Something has to move bytes from the card into main memory, and in this picture that something is the CPU — it reads the frame out of the card over the bus and writes it into a buffer. Then the kernel gets to work on it.

It is a perfectly reasonable design. It is also how some very early Ethernet controllers actually worked.

Where it breaks

Arithmetic, as usual.

A ten-gigabit link carrying full-size frames delivers about 812,000 frames per second. That is one frame every 1.2 microseconds. On a 3 GHz core, 1.2 microseconds is roughly 3,700 cycles — and that is your entire budget for everything, of which fetching the bytes is only the first step.

Worse, a 1500-byte copy is not free: it touches memory the CPU must then evict, and it does so in the middle of whatever else that core was doing. Spend the budget on moving bytes and there is nothing left for headers, sockets, TCP, or your application.

And the deeper problem is not throughput but timing. In this design, nothing can be received until the CPU gets around to it. The card must hold the frame somewhere until then. Its on-board memory is small — measured in hundreds of kilobytes, not megabytes, because fast memory on a card is expensive. So any delay in the CPU’s attention becomes a drop.

The fix is to stop involving the CPU in the movement at all, and to prepare the destination in advance.

What actually happens

Memory prepared before it is needed

Long before your frame arrived — back when the interface came up — the driver did two things.

First, it allocated a set of buffers in ordinary main memory. Each is big enough for a frame. There might be 256 of them, or 512, or 4096.

Second, it built a ring of descriptors: a small array, also in main memory, where each entry describes one of those buffers. A descriptor is not the data. It is a note that says here is the address of an empty buffer, it is this many bytes long, and it is yours.

Then the driver told the card where that ring lives, and the card began reading it.

The handoff, one descriptor at a time

Now a frame arrives, passes the card’s checks from Post 01, and sits briefly in the card’s internal memory.

The card looks at the descriptor its read index points to. If that descriptor is marked as available, the card takes the buffer address out of it and writes the frame’s bytes directly into main memory over the bus, with no CPU instruction executed anywhere. It then writes back into the descriptor how many bytes it wrote and some status flags, flips the ownership bit to say done, this one is yours again, and advances its index.

That is direct memory access. A device wrote your RAM.

Later — in softirq context, per Post 02 — the kernel walks the ring from its own index, finds descriptors the card has marked done, and for each one takes ownership of the buffer, wraps it in a packet struct, and sends it up the stack. Then, critically, it must put a fresh empty buffer back into that descriptor and mark it available again, or the ring slowly empties of usable slots.

Where the drops came from

Now the counter makes sense.

If the kernel does not get back to the ring quickly enough, descriptors are not returned to the card. The card’s read index catches up with the kernel’s and finds only descriptors it has already filled. It has nowhere to put the next frame. Frames back up in its small internal memory, and when that fills, they are discarded and rx_missed_errors increments.

The machine was thirty percent idle on average. The ring does not care about averages. It cares about the longest single interval during which nobody drained it — and a core can easily be unavailable for tens or hundreds of microseconds while it handles a different interrupt, or runs a long softirq for another queue, or is simply executing a piece of your code that does not yield.

Ownership is the rule

Look again at what the ownership bit in the descriptor is doing. It is not synchronisation in the locking sense — nobody waits. It is a protocol for transferring responsibility for a piece of memory between two parties that never speak directly.

Trace one buffer through its life:

1. The driver allocates it. The driver owns it.
2. The driver publishes it in a descriptor. The card owns it. The kernel must not read it — there may be nothing in it, or half a frame.
3. The card fills it and flips the bit. The kernel owns it. The card must not touch it again.
4. The kernel wraps it and passes it up the stack. Ownership moves along with the packet struct, one layer at a time.
5. Eventually the data is consumed, and the buffer is freed or recycled back into step 2.

At no point do two parties own it at once, and at every point exactly one of them is allowed to write.

So why is anything ever copied?

With that rule in hand, every copy in the stack becomes explicable, and there are fewer of them than you might think.

Between the card and the kernel: no copy. DMA straight into the destination.

Between layers of the kernel — Ethernet to IP to TCP: no copy. As Post 01 said, the packet struct holds pointers and a pointer moves forward. Stripping a header costs an addition.

Between the kernel and your process, when you call read: a copy, and here is why. That buffer is in kernel memory, in a different address space, under different protection. It may be shared with a device. The kernel needs it back to refill the ring. Its lifetime is not yours to extend. And your process is not trusted to be well-behaved with it. Every one of those reasons independently forces a copy, and together they make it unavoidable in the ordinary interface.

That last one is the copy that “zero-copy” techniques are about. Now that you know which copy is being removed and what the obstacles were, the techniques stop being magic and start being a menu of trades:

• Sending a file without reading it into your process first, so the payload never enters your address space at all.
• Asking the kernel to send directly out of your buffer, at the cost of you not being allowed to touch that buffer until told it is safe — an ownership transfer in the other direction, made explicit.
• Registering buffers with the kernel up front so the checks happen once rather than per call.
• Mapping the receive ring into your process and bypassing the kernel stack entirely, at the cost of implementing everything the rest of this series describes yourself.

Each removes one specific copy by solving one specific ownership problem. None removes the copy for free.

What the ring size actually buys

The ring is now the easiest thing in the stack to reason about, because it is just a bucket with a known drain.

Ring parameters for enp3s0:
Pre-set maximums:
RX:             4096
TX:             4096
Current hardware settings:
RX:             512
TX:             512

Convert the depth into time and it stops being an arbitrary number.

A 1024-entry ring on a saturated ten-gigabit link holds about 1.3 milliseconds of full-size frames. That is a lot of slack — a core can be distracted for over a millisecond and lose nothing.

But the same ring carrying minimum-size frames holds about 69 microseconds, because a saturated link at 64 bytes per frame is 14.9 million frames per second rather than 812,000. Same ring, same link speed, eighteen times less tolerance, entirely because of what the traffic looks like.

This is why sizing advice that ignores your traffic shape is worthless, and it is why a service handling many small packets is far more fragile here than one moving bulk data.

And the cost of a larger ring is the cost of every larger queue, which this series will keep meeting: a packet that arrives when the ring is nearly full waits behind everything ahead of it. You trade drops for delay. Sometimes that is right. Sometimes a dropped packet, retransmitted quickly, would have been better than a packet delivered very late — and Post 11 is largely about recognising which situation you are in.

What to carry forward

The ring is the first queue in the ladder, and it establishes the pattern for all the others: a fixed number of slots, a filler and a drainer that never run at the same time, an explicit handoff of ownership, and a failure mode that depends on the worst gap rather than the average rate.

You now have both halves of the lens. Post 02 gave you who is running. This one gives you who owns the memory. Every remaining post in this series can be read by asking those two questions at each step, and most of the confusing behaviour in the stack turns out to be one of them having a surprising answer.

Next, we go back to the beginning and do the physical layer properly: what the card decides entirely on its own, before software knows anything happened.