Writing /

The node that lies: cordon, drain, recycle before it corrupts inference

10 min read 0 views
gpukubernetesreliabilitycontrollerssre

The incident that motivated all of this happened before I joined. I only heard the full version in my third week.

A single chassis started throwing NVLink faults. Two Xid 74 events about six hours apart, plus NVLink recovery and CRC counters climbing on one link. The node stayed Ready the entire time. The kubelet was delighted. The device plugin kept advertising eight healthy GPUs. Every pod on the box passed every probe it had.

The workload was a tensor-parallel inference deployment spread across all eight GPUs, which means a degraded link between two of them sits directly in the data path. It didn’t crash. It returned answers. Some fraction of those answers were wrong, in the specific way that’s worst: plausible, well-formed, and garbage.

Nobody caught it for most of a day. What finally surfaced it was a downstream quality metric drifting a couple of points, and the first three hypotheses were about the model, because of course they were. Somebody had shipped a prompt change that week. That’s where you look.

A node that fails is an incident. A node that lies is a research project.

Why the platform has no opinion about this

To be fair to Kubernetes, none of this is a Kubernetes bug. It’s a category mismatch.

Liveness and readiness are local and binary. The kubelet asks whether a container is running and whether a process answers on a port. Node conditions cover memory pressure, disk pressure, PID pressure, network availability. There is no condition for “the HBM on device 3 is retiring rows” and there shouldn’t be, because that’s a vendor concern.

Meanwhile the device plugin advertises nvidia.com/gpu: 8 and keeps advertising 8 until something explicitly tells it not to. And a process that returns wrong results is, to every layer above it, a perfectly healthy process with normal CPU usage and a good p99.

So the gap is ours to fill.

The shape of it

Signals in, one decision out, a ladder of actions with increasing cost.

The signals don’t come from raw dmesg parsing inside the controller. I tried that first and hated it within a day. Two sources feed it instead:

node-problem-detector runs a kernel monitor rule matching NVRM: Xid lines and turns them into node events and conditions. That gives us per-event fidelity with correct timestamps, and the Xid classification table lives in a ConfigMap anyone can read.

dcgm-exporter into Prometheus, and the controller queries a handful of recording rules rather than raw series. That means the definition of “unhealthy” is a set of PromQL expressions in Git that get reviewed like anything else, instead of a set of thresholds buried in Go. When we tune the policy, the diff is legible to people who don’t read Go.

From there the policy engine assigns every node a class, and each class maps to exactly one action:

stateDiagram-v2
  [*] --> Healthy
  Healthy --> Suspect: signal seen
  Suspect --> Healthy: clears in hysteresis window
  Suspect --> Degraded: persists, not fatal
  Suspect --> Failing: fatal signal
  Degraded --> Cordoned: existing pods keep running
  Failing --> Cordoned
  Cordoned --> Draining: safe boundary or budget expiry
  Draining --> Recycling: GPU reset, then reboot
  Recycling --> Probation: dcgmi diag passes
  Recycling --> Quarantined: diag fails
  Probation --> Healthy: 24h clean
  Probation --> Quarantined: recycled twice in 14d
  Quarantined --> [*]: human closes the ticket

The four actions, cheapest first:

Cordon applies a taint (platform.cogent/gpu-health=degraded:NoSchedule) so nothing new lands there. Existing pods keep running. This is nearly free and completely reversible, which is why the policy reaches for it aggressively and everything else conservatively.

Drain evicts, and this is where the interesting logic lives.

Recycle resets the affected GPU, verifies with dcgmi diag -r 2, escalates to a full node reboot if the reset didn’t clear it, verifies again, and returns the node to a probation pool.

Quarantine is terminal. No automatic return, a hardware ticket, and a human closes the loop.

Checkpoint-aware draining

This part exists because of a question I asked in week one. About a node with a memory fault, I’d said: why not just drain it. The answer was that the job’s last checkpoint was fifty minutes old across eight GPUs, and the restart cost more than the risk did for the next forty minutes. Someone had done that arithmetic deliberately.

So the controller does the same arithmetic, badly, on purpose.

Workloads declare a drain contract in pod annotations: a maximum grace budget, and an endpoint the controller can poll that answers one question, “are you at a safe boundary right now”. Two kinds of answer exist today. Training jobs answer based on steps since the last checkpoint, so they say yes right after writing one. Inference deployments answer based on in-flight requests and whether they’re mid-rollout.

The controller cordons, then polls, then evicts on the first yes.

Two honest caveats. The endpoint is advisory, not authoritative: a workload that always says no gets forced when its budget expires, and the budget is capped fleet-wide, because a node with failing memory holding a job hostage is a much worse outcome than a restart. And roughly half our workloads never declared a contract at all, so they run on the default budget and a preStop hook’s worth of politeness. That’s the weakest part of the design and I don’t have a clever fix. Advisory contracts work when teams care and rot quietly when they don’t.

Hysteresis, in four flavours

Almost every mistake I made in the first version was a missing damper.

A signal has to persist across several evaluation intervals before it escalates a class. This one change removed most of our noise and cost nothing but latency on genuinely fatal events, which are rarely subtle anyway.

Recovered nodes go to probation rather than straight back to Healthy: schedulable, but only for workloads that tolerate the probation taint, and watched at a higher scrape resolution for twenty-four hours.

A node that gets recycled twice in fourteen days doesn’t come back automatically. Repeat offenders are hardware problems wearing a transient costume, and I’d rather burn a node’s capacity than let it take three swings at corrupting output.

And there’s a fleet budget: the controller will never put more than two percent of the fleet into a non-serving state at once. If policy wants more than that, it stops and pages a human instead. That rule is the only reason the next section is funny rather than a postmortem.

The day I cordoned eleven nodes

A driver upgrade rolled through a rack. Correctable ECC counters jumped on a batch of cards more or less simultaneously, my first-pass policy treated a fast rise in single-bit errors as a pre-failure signal, and the controller started doing exactly what I’d told it to.

Eleven nodes cordoned in about nine minutes, and then the fleet budget tripped, the controller stopped, and it paged me. I un-cordoned them by hand while feeling several things at once.

The counters settled within the hour. Of course they did. A correctable ECC burst is precisely that: correctable, handled in hardware, nobody harmed. NVIDIA’s guidance says so if you read it properly. Xid 92, the high-correctable-rate signal, carries an immediate resolution bucket of ignore and an investigatory bucket of contact support. That pairing is the whole answer: don’t act now, do look into it later. I had collapsed those two into one action.

The other demotions from that week, most of which are covered in more detail in the field guide:

  • Xid 13 and Xid 31 came out of the fatal set entirely. Application faults, documented resolution restart the app. They get attributed to the owning pod and never touch the node’s class.
  • Xid 45 gets suppressed unless it’s the only Xid on the node. It fires when a job is torn down, so when something upstream kills a batch of work you get a synchronised storm that looks like carnage and is just cleanup.
  • Xid 94, the contained-error case, is attributed to the pod rather than the node. Containment worked. That’s the system succeeding.

The ones that looked benign and weren’t

The opposite direction was harder, and more interesting.

Xid 63, the memory remapping event, has a documented resolution bucket of ignore, and per event that’s correct. But a single GUID accumulating remap events over days is the clearest pre-death signature we have. So the rule is: ignore the event, alert on the trend, and treat DCGM_FI_DEV_ROW_REMAP_PENDING as an action-worthy signal on its own, because a remap only takes effect after a GPU reset.

NVLink CRC flit errors climbing with no Xid at all. The link retries and succeeds, so nothing errors, so nothing logs. All you get is a counter going up and a tail latency that’s slightly worse than last week. This is the signal that would have caught the original incident hours earlier, and it’s now weighted heavily for any node running tensor-parallel work.

Thermal. The dead fan I wrote about last month, the one showing up as continuous hardware thermal slowdown with every other signal green, now feeds a degraded class off DCGM_FI_DEV_CLOCKS_EVENT_REASONS. It cordons nothing, opens a ticket, and tags the node so capacity planning stops pretending it’s a full-speed box.

The policy file ended up roughly like this, which is about as complicated as I want it to get:

classes:
  failing:
    xids: [48, 64, 74, 79, 95]
    metrics:
      - expr: gpu_row_remap_pending
        for: 15m
    action: cordon_drain_recycle
  degraded:
    xids: [63, 92, 119, 120]
    metrics:
      - expr: gpu_nvlink_crc_flit_rate_high
        for: 30m
      - expr: gpu_hw_thermal_slowdown_active
        for: 1h
    action: cordon_and_ticket
  workload:
    xids: [13, 31, 94]
    action: attribute_to_pod
  ignored:
    xids: [45, 154]
    action: none
budgets:
  max_fleet_unavailable: 2%
  default_drain_grace: 20m
  max_drain_grace: 45m
  recycle_limit: 2 per 14d

Xid 119 and 120 sit in degraded on purpose. They’re GSP errors, the guidance points at software as much as hardware, and we’ve watched a cluster of them track one driver version. Resetting the GPU is the right immediate move; concluding the card is bad is not.

The first time it fired in anger

Late March, 02:41.

Xid 74 on one GPU, NVLink recovery errors climbing on a single link, and a second Xid 74 nineteen minutes later. The second event pushed the node from degraded to failing. The controller cordoned it, asked the inference deployment for a safe boundary, and got one in about forty seconds because the deployment happened to be mid-rollout anyway. It drained, reset the affected GPU, ran dcgmi diag -r 2, and the diag failed. So it escalated to a reboot, ran the diag again, passed, and put the node into probation.

03:04. Node back and serving, in a restricted pool, with a ticket open against the GUID.

I found out at 08:10 with a coffee in my hand, reading a Slack thread where the controller had narrated the entire thing to itself. Six messages, no replies.

I want to describe that honestly, because “I slept through it” sounds like a victory lap and it wasn’t at first. It was unsettling. I read the transcript twice looking for the moment it did something stupid, and then read it a third time looking for the moment it got lucky. I’ve spent most of a decade being the thing that gets woken up, and there’s an identity wobble in watching software be better at 02:41 than I am.

Then it happened again the following week and I barely looked. That’s when it started feeling like an achievement.

The failures that hurt most are the ones that keep passing health checks.

Where it’s still rough

The drain contract, as above. Advisory endpoints and half the fleet on defaults.

Recycle verification is slow. dcgmi diag -r 2 needs the GPU to itself for a couple of minutes, and the long run is closer to fifteen. Every recycle is a trade between confidence and capacity, and I don’t think we’ve picked the right point on that curve. Right now we run -r 2 on recycle and schedule -r 3 for anything that comes back twice.

And the original problem isn’t solved. We got much better at noticing hardware going bad. We’re still not good at noticing output that’s subtly wrong. Correctness canaries, a fixed set of inputs with known-good outputs run continuously against every serving replica, are the obvious next thing, and every version I’ve sketched runs into the same wall: for the workloads where this matters most, “known-good output” isn’t a fixed string.

That’s the next problem. I’ll write it up when I have something that works rather than something I believe in.