# First month with a 10 MW GPU fleet: everything I had to unlearn > Notes from five weeks on an accelerator fleet: why compute measured in megawatts breaks CPU-land instincts, what DCGM and Xid and MIG mean day to day, and the naive question I asked in week one that reframed the whole job. Author: Karan Vijayakumar Published: 2026-03-08 Tags: gpu, platform-engineering, sre, observability Reading time: 9 min Canonical URL: https://karanvk.me/blog/first-month-gpu-fleet --- In my first week at Cogent Labs somebody described the fleet as "about ten megawatts" and I nodded, because that's what you do, and then spent the walk to the station trying to work out whether that was a lot. It is. But the number wasn't the interesting part. The interesting part was that power was the unit at all. For the two years before this I was a senior SRE at Sales Marker running a Kubernetes estate north of 2000 vCPUs, and I never once described it in watts. I described it in vCPUs, or in monthly spend, or in pods. Capacity was a number in a Terraform variable and a credit card that had already been approved. Here, capacity is a negotiation with a building. That single swap, from cores to watts, turned out to be the front door to about a dozen other things I had wrong. This is my list, roughly a month in, while I can still remember being confused. ## Megawatts as a unit of compute An eight-GPU SXM chassis pulls something in the neighbourhood of ten kilowatts flat out. NVIDIA quotes 10.2 kW max for a DGX H100, six power supplies, 3300 W each. Put a few of those in a rack and you have the draw of a small apartment building sitting in two square metres of floor. So ten megawatts isn't what we consume, it's what we're allowed to consume. It's a ceiling with a contract attached. You cannot scale out of a power envelope. There is no API for it. In cloud CPU-land, "we need more capacity" was a pull request and then it existed. Here it's a pull request plus a facilities conversation plus a delivery date plus a question about whether the cooling loop on that row can take it. The lead time on more compute is measured in weeks, and no amount of clever bin-packing changes that. Which means every workload that arrives is competing for watts with every other workload. I'd never thought about compute that way. ## Cattle, except every cow has a serial number I believed in cattle-not-pets. I still do, for CPUs. Nodes were interchangeable, I could not tell you the name of a single instance in the old estate, and that was the point: anonymity was the property that let me be aggressive about replacing things. GPUs don't work like that, and the driver tells you so immediately. Every accelerator has a GUID, globally unique and immutable, and the kernel log prints it right before the first error line for that device: ```text NVRM: GPU at 0000:03:00: GPU-b850f46d-d5ea-c752-ddf3-c4453e44d3f7 NVRM: Xid (0000:03:00): 14, Channel 00000001 ``` That's straight out of NVIDIA's own Xid documentation, and the shape of it is the lesson. The identity is per device, not per node. So we track accelerators individually, by GUID, with a history: which chassis, which slot, which faults it has thrown, how many times it's been reset, whether it's ever been through the RMA process. There's a table. People read the table. The instinct I had to actively suppress was "just replace the node". There is no replacing the node. There's a person, a cage, a screwdriver, and a lead time. The cheapest possible fix is a GPU reset, the next cheapest is a reboot, and everything after that involves hands. ## You cannot oversubscribe a GPU This one I knew intellectually and had absolutely not internalised. Most of my craft at the last job lived in the gap between requests and limits. Ask for 300m, burst to two cores, let the scheduler pack things tight and let the kernel sort out the contention. Half of the interesting capacity work in a CPU estate is exploiting the fact that nobody uses what they asked for. There is no such gap here. `nvidia.com/gpu` is an integer. No millicores, no fractional requests, and requests have to equal limits. A pod either owns the device or doesn't. MIG is the closest thing to partitioning, and it's static in a way that surprised me. You configure a geometry on the card, drawn from a fixed set of profiles (`1g.10gb`, `2g.20gb`, `3g.40gb`, `4g.40gb`, `7g.80gb` on the 80 GB parts), and those are the instances on offer until somebody reconfigures it, which needs the GPU idle. It's not a scheduler decision. It's closer to a partition table. So a node's shape gets decided before the workloads arrive. Guess high and you strand memory. Guess low and pods sit `Pending` on hardware that's technically free. Either way there's nothing to reclaim at runtime, because there was never any slack to begin with. ## Utilization stops being a comfort metric In CPU-land, a fleet averaging 25% was headroom, and headroom was safety. I would have defended that number in a review. Here, idle silicon burns capex on a schedule whether anyone is watching or not. A node sitting at 15% for a week isn't headroom, it's a slow withdrawal. That reframing alone changed which dashboards I care about. And then it turns out the obvious utilization number lies to you. `DCGM_FI_DEV_GPU_UTIL` tells you a kernel was resident on the device during the sample window. It does not tell you the device was doing anything useful. The honest fields are the profiling ones: `DCGM_FI_PROF_GR_ENGINE_ACTIVE`, `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`, `DCGM_FI_PROF_DRAM_ACTIVE`. I have now watched a node report cheerful high utilization while its tensor pipes were more or less asleep, which is the GPU equivalent of a process spinning on a lock and calling it work. Rebuilding our efficiency dashboard around that distinction was my second week, and it's the thing I've shipped that people still bring up. ## The vocabulary arrived faster than the understanding Roughly in the order I met them: - **DCGM**, NVIDIA's Data Center GPU Manager. Passive health watches (`dcgmi health -s a`, where the flags cover PCIe, memory, InfoROM, thermal and power, NVLink, driver) plus active diagnostics that need exclusive access to the device (`dcgmi diag -r 1` for quick deployment checks, `-r 2` for a couple of minutes including memory and interconnect, `-r 3` for the long stress run). Everything else in the stack reads DCGM's field IDs. - **Xid**, a numbered error the driver writes to the kernel ring buffer. `dmesg | grep "NVRM: Xid"` is now the first command in every GPU postmortem I've sat in. - **MIG**, the hardware partitioning above. - **NVLink** and **NVSwitch**, the fabric the GPUs use to talk to each other. Much faster than PCIe, with its own error counters and its own way of going bad without anything crashing. - **SXM**, the socketed module form factor as opposed to a PCIe card. Higher power, full NVLink, and firmly in the category of things you do not swap casually. - **InfoROM**, persistent storage on the board where the GPU keeps its own medical records, including which memory rows it has permanently retired. I kept a text file of these for the first two weeks. I'm not embarrassed about it. ## The question I asked in week one Someone walked me through an incident from before I joined. A node had thrown a memory fault, and a training job had been left running on it for another forty minutes. I asked, more or less: why not just drain it? The answer was that the job's last checkpoint was about fifty minutes old, spread across eight GPUs, and evicting it would throw that away and re-queue the whole thing behind everything else in the queue. So the cost of draining wasn't the drain. It was the restart. And for those forty minutes, someone had decided the restart cost more than the risk did. On purpose. With numbers. That answer rearranged my head more than any doc I've read since. In CPU-land, eviction is nearly free. Kill the pod, it lands somewhere else, the request retries, nobody files a ticket. I had built years of reflexes on top of that assumption, including a strong bias toward evicting early and asking questions later. Here, eviction has a price tag, and the price depends on what the workload was doing and how long ago it last wrote state to disk. Draining a node is a decision. It is not a reflex. My question wasn't stupid, it was just imported. Every instinct behind it was correct in a world where compute is fungible and cheap to interrupt, and I'd never worked anywhere that wasn't. ## Thermals and power are in the request path now Temperature used to be somebody else's dashboard. Facilities had it. I looked at it approximately never. Now it's a performance signal that sits directly in front of user-facing latency, because a GPU that gets hot clocks itself down rather than failing. `DCGM_FI_DEV_CLOCKS_EVENT_REASONS` (which used to be called `DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`, and you'll see both names in the wild) tells you which reason is active: software power cap, software or hardware thermal slowdown, hardware power brake. None of those is an error. Nothing pages. Throughput just quietly sags and your p99 goes soft in a way that looks like a bad model deploy. I found a chassis in week three where hardware thermal slowdown had been active more or less continuously, with every other signal green. A dead fan, expressed as a latency regression. Also worth knowing: HBM has its own thermal story, so `DCGM_FI_DEV_MEMORY_TEMP` and `DCGM_FI_DEV_GPU_TEMP` are separate fields and they don't always move together. ## node-exporter is now the least interesting exporter on the box I like node-exporter. It has told me the truth about a lot of machines. On these nodes it will tell you, cheerfully and accurately, that the box is fine. CPU mostly idle, memory free, disks healthy, load average boring. Meanwhile the component that costs more than everything else in the chassis put together is logging uncorrectable memory errors and quietly retiring rows. So dcgm-exporter is the exporter that matters, and I've been learning its fields the way I once learned node-exporter's. The ones I check without thinking about it now: `DCGM_FI_DEV_XID_ERRORS`, `DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`, `DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS`, the NVLink error counters, and the clock event reasons. The default counters CSV that ships with dcgm-exporter is a sensible starting point and not a complete one. `DCGM_FI_DEV_ROW_REMAP_PENDING` isn't in it, for instance, and that omission has consequences I want to write about separately once I've finished being wrong about it. ## Where I am Five weeks in, fewer certainties than I arrived with. My first real deliverable is a set of health checks that read DCGM signals and decide whether a node should keep accepting work, which sounds like a small thing and has been the most interesting problem I've had in years, mostly because "is this node healthy" turns out to have about nine answers. The estate I used to run forgave me. This one keeps receipts.