Xid 79 and friends: a field guide to dying GPUs

A couple of weeks ago I wrote about my first month on a 10 MW accelerator fleet and everything I got wrong coming from CPU-land. The single biggest gap in that first month was Xid errors. I could read them, sort of. I could not tell you which ones meant “this card is finished” and which ones meant “a data scientist indexed past the end of an array again”.

That distinction is most of my job now, so this is the field guide I wanted in week one. Everything factual here I checked against NVIDIA’s Xid catalog and the DCGM field definitions, because I got burned early by a blog post that had two codes backwards.

What an Xid actually is

An Xid is an error the NVIDIA kernel driver logs when something goes wrong on or around a GPU. It lands in the kernel log buffer, which means your distro flushes it to the journal and to /var/log/messages or /var/log/syslog. There’s exactly one way to find them:

sudo dmesg | grep "NVRM: Xid"

The format is worth knowing by eye. The first Xid for a device is preceded by a line carrying the GPU’s GUID, and every line after has the PCI bus address, the Xid number, and a payload specific to that error:

NVRM: GPU at 0000:03:00: GPU-b850f46d-d5ea-c752-ddf3-c4453e44d3f7
NVRM: Xid (0000:03:00): 14, Channel 00000001

Two things about the catalog first.

The catalog is split by architecture. NVIDIA publishes one Xid table for Ampere and newer, and points you at a separate document for Volta and older. The numbers overlap but the resolution guidance and some of the mnemonics differ, so read the table for the hardware you actually have. I wasn’t, for about a week.

And NVIDIA’s own hedge, which I’ve come to appreciate: hardware failures can cause almost any Xid, so the classification should be treated as a starting point for investigation rather than a verdict. An Xid is a symptom with a number attached, not a diagnosis.

The ones that mean the card is going

Xid 79, ROBUST_CHANNEL_GPU_HAS_FALLEN_OFF_THE_BUS. The driver tried to reach the GPU over its PCI Express connection and the GPU wasn’t there. NVIDIA’s guidance is contact support, and the doc notes it’s often caused by hardware failures on the PCIe link. In practice it’s the most decisive Xid in the catalog: the device is gone until somebody power cycles the chassis, and frequently it comes back and then leaves again. Twice on the same GUID inside a week and I stop trying to be clever.

Xid 48, ROBUST_CHANNEL_GPU_ECC_DBE. Double-bit ECC, meaning an uncorrectable memory error. This one gets reported back to the user application as well as logged, and it needs a GPU reset or a node reboot to clear. Newer drivers add companions that tell you where the fault was: Xid 171 (UNCORRECTABLE_DRAM_ERROR) and Xid 172 (UNCORRECTABLE_SRAM_ERROR), both explicitly additional to Xid 48 to differentiate DRAM from SRAM. If you’re on a recent driver, grab those too. Knowing it was SRAM rather than HBM changes what you do next.

Xid 74, NVLINK_ERROR. The GPU found a problem with a connection to another GPU or to an NVSwitch over NVLink. Needs a reset or reboot to clear, and the catalog says it may indicate a hardware failure. This is the one that scares me most, and the reason is in the next post: NVLink can degrade without stopping, and degraded interconnect on a multi-GPU inference deployment is not a crash, it’s bad output.

Xid 64, INFOROM_DRAM_RETIREMENT_FAILURE. The GPU tried to record a row-remapping entry into its InfoROM and failed. Resolution is reset the GPU and contact support. Practically, this is the card telling you its self-repair mechanism has broken, which is worse than any individual memory error.

Xid 95, ROBUST_CHANNEL_UNCONTAINED_ERROR. Error containment exists on A100 and newer: some memory errors get contained to the single application that hit them. Xid 94 is the contained case. Xid 95 is the uncontained one, where the error affects multiple applications and the GPU has to be reset before anything can use it again. Treat 95 as fatal for the node’s current workload set.

Xid 119 and 120, GSP_RPC_TIMEOUT and GSP_ERROR. These fire when something goes wrong in the code running on the GPU’s GSP core, or when the driver times out waiting for the GSP to answer an RPC. Resolution is a GPU reset or a node power cycle, and the catalog also flags them as worth investigating as software. We’ve seen a cluster of them track one specific driver version. Don’t RMA on GSP errors. Check what changed.

One more that isn’t a failure but shows up constantly: Xid 154, GPU_RECOVERY_ACTION_CHANGED. Informational, appears alongside other Xids, and summarises the recovery action the driver thinks is required. nvidia-smi surfaces the same thing as a GPU Recovery Action. If you’re parsing dmesg, treat 154 as metadata about the Xid next to it, not as an event of its own.

The ones that page you for nothing

This is the half nobody warns you about, and where most of the alert fatigue lives.

Xid 13, ROBUST_CHANNEL_GR_EXCEPTION (you’ll also see it as ROBUST_CHANNEL_GR_ERROR_SW_NOTIFY), described as a graphics engine exception. The catalog is blunt: this is logged for general user application faults, typically an out-of-bounds access where somebody walked past the end of an array, though it can also be an illegal instruction or illegal register. Resolution is restart the application. Not the node. Not the GPU. The application.

Xid 31, ROBUST_CHANNEL_FIFO_ERROR_MMU_ERR_FLT, a GPU memory page fault. A fault reported by the MMU, usually an illegal address access. Typically application-level bugs, says the catalog, though it can also be driver or hardware bugs. Same resolution: restart the app.

Between them, 13 and 31 were about eighty percent of the Xid volume on our fleet when I started building alerting, and every one was a workload problem. If you page a human on raw Xid count, this is what wakes them up.

Xid 94, ROBUST_CHANNEL_CONTAINED_ERROR. The containment mechanism did its job, the error is confined to one application, and the resolution is restart that application. It deserves a metric and a trend line. It does not deserve a phone call.

Xid 45, ROBUST_CHANNEL_PREEMPTIVE_REMOVAL. Logged when a user application aborts and the driver tears down its GPU work: Ctrl-C, a sigkill, a GPU reset. Which means that when something upstream kills a batch of jobs, you get a beautiful synchronised storm of Xid 45 that looks exactly like a catastrophe and is in fact just cleanup. The catalog even distinguishes the cases: seen on its own it’s worth a look, seen alongside another Xid you should follow the other Xid.

Correctable, uncorrectable, and the confusing middle

Single-bit errors are correctable. The hardware fixes them, your application never knows, and a small trickle of them is normal on HBM at scale. In DCGM they show up as DCGM_FI_DEV_ECC_SBE_VOL_TOTAL (volatile, resets on reboot) and DCGM_FI_DEV_ECC_SBE_AGG_TOTAL (aggregate, persistent). Double-bit errors are uncorrectable, they’re the Xid 48 case, and they have DCGM_FI_DEV_ECC_DBE_VOL_TOTAL and DCGM_FI_DEV_ECC_DBE_AGG_TOTAL.

The rate matters more than any single event on the correctable side. Xid 92, EXCESSIVE_SBE_INTERRUPTS, described as a high single-bit ECC error rate, exists precisely for that. Its immediate resolution bucket is ignore, and its investigatory bucket is contact support, which is a genuinely useful pair of signals: don’t act now, do look into it.

Row remapping, and why pending is the canary

Starting with Ampere, GPUs can permanently retire a bad memory row and use a spare from the same DRAM bank. This isn’t the older dynamic page offlining: the fix happens in hardware and leaves no software-visible hole in the address space.

The details that matter operationally:

• A remap can be triggered by correctable or by uncorrectable errors. When a bank’s reserved rows start running out, remapping done for correctable errors can be replaced by remapping for uncorrectable errors, because the uncorrectable case takes priority.
• There’s a ceiling. NVIDIA documents up to 512 remappings for the frame buffer.
• A remap requires a GPU reset to take effect, and once it does it persists for the life of the GPU.
• The counts you read from nvidia-smi are the entries recorded in the InfoROM, not the rows currently remapped in hardware. Alongside the counts, nvidia-smi reports remap pending and remap failure as booleans.

That third bullet is the whole thing. A GPU that has decided to remap a row is a GPU that is running with a known-bad row still in service until someone resets it. Which makes remap-pending the single best pre-death signal I’ve found: it is the card saying “I have already diagnosed myself, and I’m waiting on you”.

Worth knowing where the cliff is, too. NVIDIA’s RMA criteria for row remapping include a remap attempt for an uncorrectable error on a bank that already has eight uncorrectable rows remapped, a remap attempt for an uncorrectable error on a row that was already remapped, and 512 total uncorrectable remappings. Their field diagnostic tool is what actually decides. On Blackwell there’s an extra mitigation step, an attempt at HBM channel repair, before the remap failure flag gets set. So the same Xid 64 can mean different things depending on the generation in the slot, which is a fun thing to learn during an incident.

Getting this into Prometheus without lying to yourself

Here is the trap I fell into. DCGM_FI_DEV_XID_ERRORS (field 230) is documented as the value of the last Xid error encountered. It’s a gauge holding a code number. You cannot rate() it, you cannot count with it, and if two different Xids happen between scrapes you only ever see the second one. I wrote an alert on increase() of it, which is meaningless, and it took an embarrassingly long time to notice because it did fire sometimes.

The right tool is DCGM_EXP_XID_ERRORS_TOTAL, an exporter-derived counter that watches DCGM_FI_DEV_XID_ERRORS and emits a separate series per observed xid label. Xid value 0 counts as no error. Two caveats: it’s commented out in the default CSV so you have to opt in, and it resets when the exporter collector is recreated, including restarts and hot reloads. There’s also DCGM_EXP_XID_ERRORS_COUNT, a gauge covering only the last collection window, and a matching pair for clock events (DCGM_EXP_CLOCK_EVENTS_TOTAL, labelled by clock_event, edge-counted on transitions from inactive to active).

The fields I actually watch, beyond the Xid counters:

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, counter, uncorrectable ECC this boot
DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, counter, correctable ECC this boot
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, gauge, remaps from correctable errors
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, gauge, remaps from uncorrectable errors
DCGM_FI_DEV_ROW_REMAP_FAILURE, gauge, remap failure flag
DCGM_FI_DEV_ROW_REMAP_PENDING, gauge, remap awaiting a reset
DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL, counter, flit CRC errors
DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL, counter, NVLink replays
DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL, counter, NVLink recoveries
DCGM_FI_DEV_CLOCKS_EVENT_REASONS, gauge, why clocks are capped

Note that the remapped-row and remap-flag fields are gauges, so trend them with delta(), not increase(). Mixing those up is the second most common mistake I’ve made in this stack.

Two rules that earn their place

dcgm-exporter labels each series with gpu and UUID, plus container, namespace and pod when it’s running under Kubernetes, and Prometheus adds instance. That’s enough to route.

groups:
  - name: gpu-hardware
    rules:
      - alert: GpuFellOffTheBus
        expr: increase(DCGM_EXP_XID_ERRORS_TOTAL{xid="79"}[10m]) > 0
        labels:
          severity: critical
          gpu_action: quarantine
        annotations:
          summary: 'Xid 79 on GPU {{ $labels.gpu }} ({{ $labels.UUID }}) at {{ $labels.instance }}'
          runbook: 'PCIe link or device failure. Node needs a power cycle and a hardware ticket.'
 
      - alert: GpuRowRemapPending
        expr: DCGM_FI_DEV_ROW_REMAP_PENDING > 0
        for: 15m
        labels:
          severity: warning
          gpu_action: cordon
        annotations:
          summary: 'Row remap pending on GPU {{ $labels.gpu }} at {{ $labels.instance }}'
          runbook: 'Known-bad row still in service until a GPU reset. Drain at the next checkpoint boundary.'

The for: 15m on the second one isn’t caution about the signal, which is reliable. It’s caution about the action: cordoning a node is cheap, but the drain that follows it isn’t, and I don’t want a scrape blip to start that chain.

What we page on versus what we act on

Roughly where we’ve landed, after a month of tuning:

The thing I’d tell my week-one self is that the split in that table matters more than any individual code. Getting 13 and 31 out of the paging path was worth more to the team’s sleep than every clever alert I wrote afterwards.

Next post is about the piece that consumes all of this: a controller that decides on its own whether a node keeps taking work. Knowing a GPU is dying is only useful if something acts on it before the next batch of requests lands there.