Who Is Running This Code?

Descend one level and that stops being true. The most important thing happening on a loaded server is work that runs on your machine, on your behalf, caused by your traffic — and that your process cannot see, is not charged for, and did not ask for.

This post is about who is actually running. It is short on networking and long on machinery, and it comes second in the series for one reason: it is the lens. Every remaining post is easier once you have it, and close to unreadable without it.

The symptom

Your service is slow. You do the responsible thing and profile it. The flame graph is unhelpfully healthy: the handler takes 1.8 milliseconds, the JSON serialiser is fine, nothing stands out. But the p99 latency is 40 milliseconds and you cannot find the other 38.

So you look at the machine instead.

CPU   %usr  %nice  %sys  %iowait  %irq  %soft  %steal  %idle
all   31.2   0.00   9.4     0.10   0.4   14.7    0.00   44.2
0   28.9   0.00   8.1     0.00   0.1    2.3    0.00   60.6
1   30.1   0.00   9.0     0.20   0.2    3.1    0.00   57.4
2    9.7   0.00   6.2     0.00   1.9   71.4    0.00   10.8
3   33.4   0.00  10.1     0.10   0.2    2.9    0.00   53.3

Core 2 is spending seventy-one percent of its time in something called soft, and nearly nothing in your application. No process on the machine is accountable for that time. It does not appear in any per-process accounting. If you had only your profiler, you would never know it existed.

That column is where almost all of your network receive processing lives. To understand what it is, you need the four answers to the title question.

The obvious design

Here is the model most application developers carry, usually without having articulated it: the kernel is a library.

You call into it — read, write, epoll_wait — it goes off and does something privileged, and it returns to you. It is synchronous, it is initiated by you, and while it is working, the thing running is still recognisably your thread, just in a different mode.

This model is not wrong. It is just extremely partial. It describes exactly one of the four contexts, and that one is the least interesting.

Where it breaks

A packet arrives when it arrives.

Nothing called anything. No thread asked for it. There is no scheduler decision waiting to be made, because the event happened in the physical world and the machine has to deal with it.

You might imagine solving this with a dedicated thread — one whose whole job is to sit in a loop waiting for packets. But then consider what “waiting” means. Either it spins, burning a core continuously whether traffic is arriving or not, or it sleeps, in which case something has to wake it, and waking a thread means running the scheduler, and the scheduler is software, and we are back to needing something to run at the moment the packet lands.

At some point the chain has to bottom out in a mechanism where hardware forces software to run. That mechanism is the interrupt, and everything above it is a structure for getting out of interrupt context as fast as possible.

The four contexts

Hardware interrupt

A device asserts an interrupt. On the core that takes it, whatever was running stops immediately — your process, someone else’s, kernel code, anything — and the CPU jumps to a handler the driver registered at boot.

The defining property is that there is no process behind this. It is not that some unfortunate process is running the handler; it is that the question does not apply. The handler borrowed a core mid-instruction-stream. The kernel notion of “the current process” points at whatever was interrupted, which is meaningless and, if you use it for anything, a bug.

From that follows everything else. This context cannot sleep — there is nothing to put to sleep and nothing that would wake it. It cannot wait on a lock that might sleep. It cannot do anything slow, because while it runs, at minimum its own interrupt is masked, and in the worst case others are queueing behind it.

So a good interrupt handler does the smallest possible thing. For a network card, that is roughly: note that the card has work, arrange for the real processing to happen shortly, return. Under a microsecond, start to finish.

Softirq

The interrupt handler needed somewhere to put the real work, and this is it.

A softirq is a piece of deferred kernel work with a fixed identity — there are about ten of them, and two are ours: NET_RX and NET_TX. Raising one is almost free: it sets a bit. Later, something checks the bits and runs the corresponding handlers.

“Later” is usually immediately — on the way out of the interrupt that raised it, on the same core, before returning to whatever was interrupted. So the sequence on a core is often: your process is running, a packet arrives, the hard interrupt fires and returns in a microsecond, the softirq handler then runs for rather longer, and only then does your process resume, having no idea any of it happened.

Softirq context is a strange middle ground. Interrupts are enabled again, so it is much freer than hard interrupt context. But it still has no process behind it. It still cannot sleep. And it can be preempted by hard interrupts but not, in general, by your process — so while it runs, your code does not.

This is where the network stack lives. IP header checks, firewall rules, connection tracking, TCP sequencing, socket lookup, the decision to append your bytes to a receive queue: essentially all of the machinery described in Post 01 runs here. The seventy-one percent on core 2 in that mpstat output is this.

Kernel thread

Now a problem. Softirq work runs on the way out of interrupts, and interrupts are caused by traffic. So under heavy load, a core can find itself doing softirq work almost continuously, and processes on that core stop making progress. Nothing about softirq context is fair, because fairness is the scheduler’s job and the scheduler is not involved.

The escape valve is a set of ordinary kernel threads — one per core, named ksoftirqd/0, ksoftirqd/1, and so on. When softirq work exceeds a budget, the kernel stops processing it inline and wakes the thread instead.

The thread is the interesting part. It is a real schedulable entity with a PID. It can sleep, it can be preempted, and — crucially — it competes with your application for CPU under the ordinary scheduler, rather than simply preempting it.

This is why seeing ksoftirqd near the top of top is a signal and not noise. It means network work has overflowed the fast path and is now contending for time with your server, on equal terms, as a peer.

kworker threads are the other family you will see, doing deferred work that genuinely needs to be able to sleep.

Process context

Finally, the familiar one. Your thread calls read. It traps into the kernel, the privilege level changes, but the schedulable entity does not: it is still your thread, accounted to your process, appearing in your profiler.

The capability that distinguishes it is the ability to sleep. Your thread can be marked not-runnable and set aside, and the core given to someone else, and later something can make it runnable again.

That single capability is what makes blocking possible. When read finds the socket’s receive queue empty, it can put your thread on a list attached to that socket and sleep. When a softirq later appends bytes to that queue, it walks that list and marks your thread runnable. This is the entire mechanism.

What this buys you

Three things fall out immediately.

Every boundary is a handoff between two contexts. And because two contexts run at different times, one cannot simply call the other. Something has to hold the work in between. That something is a queue, every time, without exception. The hard interrupt hands to the softirq through a list of devices needing attention. The softirq hands to your process through the socket receive queue. Your process hands back down through the send queue and the queueing discipline. Post 01 called this a ladder of queues; this is why it has to be one.

Your accounting is incomplete by construction. Time spent in softirq context is charged to a core, not to a process, because there is no process to charge it to. A server that looks like it is using 40% CPU may be on a machine where one core is saturated doing receive processing for it. Application-level profilers cannot see this, not because they are bad, but because the work genuinely happens outside every process.

Locality is not guaranteed. Nothing so far says the core that processes your packet is the core your application runs on. It usually is not. The receive path picks a core based on which hardware queue the card chose; your scheduler picks a core for your thread based on entirely different concerns. The bytes may well be written into a cache on one core and read from another. That gap is the subject of a great deal of tuning, and Post 04 shows where the choice is made — by the network card, in hardware, before the kernel is involved.

See it for yourself

The card’s interrupts are not abstract. They are numbered, they are named after the queues that raise them, and you can see exactly which cores have been taking them.

         CPU0     CPU1     CPU2     CPU3
124:       0        0  8471203        0  IR-PCI-MSI  enp3s0-rx-0
125:    3311        0        0  1204553  IR-PCI-MSI  enp3s0-rx-1
126:  892041        0        0        0  IR-PCI-MSI  enp3s0-tx-0

Then run top, press 1 to break out the cores, and watch the si column on whichever core owns the busiest queue. You are watching the second context in this post, doing the work of the whole receive path, for a process that has no idea.

The discipline

From here on, two questions get asked at every step of this series, and they are worth asking in your own head whenever something on this path confuses you.

Which context is running? Hard interrupt, softirq, kernel thread, or yours — because that determines what the code is permitted to do, how it is accounted, and whether it can wait.

Who owns this memory right now? Which is the next post, because the answer turns out to explain not just DMA but every copy in the stack.