# MIG profiles are a bin-packing problem you solve with types

> GPU partitioning config is a combinatorial mess where a wrong answer does not crash, it just leaves very expensive silicon idle. Why Helm values files could not hold it, and what happened when I moved the whole thing into CUE constraints.

Author: Karan Vijayakumar
Published: 2026-05-23
Tags: cue, gpu, kubernetes, mig, platform-engineering
Reading time: 8 min
Canonical URL: https://karanvk.me/blog/mig-bin-packing-cue

---

The worst class of bug on this fleet is the one where nothing breaks.

A GPU node comes up. Node is Ready, driver loaded, DCGM reporting temperatures like everything's fine, nothing red anywhere. And a card that costs about seven dollars an hour on demand is advertising a resource nobody in the cluster asks for, so no pod ever lands on it, and it'll keep doing that until a human notices. Which on a fleet this size could be Monday.

Almost all of these trace back to one thing. Somebody described a GPU partition layout that isn't legal on that card, and every layer between the YAML and the silicon accepted it politely on the way down.

## MIG, briefly and correctly

Multi-Instance GPU is NVIDIA's hardware partitioning on A100 and H100. It's genuinely different from time-slicing or MPS: a MIG instance gets dedicated streaming multiprocessors, its own memory slice, its own slice of L2 cache and memory bandwidth, its own fault domain. A process wedging one instance doesn't touch the others and doesn't get to steal bandwidth from them. That isolation is the whole reason to bother. It's what lets you put seven unrelated inference tenants on one card and still make latency promises to each of them.

An H100 80GB has seven compute slices and eight memory slices to hand out. Profile names encode both halves, `Ng.MMgb`. What actually exists on that card:

| Profile | Instances per card |
|-|-|
| `1g.10gb` | 7 |
| `1g.10gb+me` | 1 |
| `1g.20gb` | 4 |
| `2g.20gb` | 3 |
| `3g.40gb` | 2 |
| `4g.40gb` | 1 |
| `7g.80gb` | 1 |

Sit with that table for a second, because its shape is the whole problem. Seven compute slices, eight memory slices, so the two axes don't divide evenly against each other. That asymmetry is why `1g.20gb` exists at all (one compute slice, two memory slices, four per card) and it's why arithmetic you do in your head is wrong more often than you'd like. The `+me` suffix means the instance also gets the media engines, and there's exactly one of those per card however you slice it. A100 is the same skeleton with different memory numbers, `1g.5gb` through `7g.40gb` on the 40GB part.

The profiles aren't freely composable either. A valid layout has to fit a fixed placement table, so it isn't just "does the sum come to seven." Two `3g.40gb` plus a `1g.10gb` is seven compute slices and 90 GB on an 80 GB card, which is nonsense, and you can land on it without noticing if you're only counting the `g` numbers.

## Where the combinatorics come from

Now multiply. Card generations, because we didn't buy all of this in one quarter. Seven-ish profiles each. Driver branches, because a qualified branch isn't something you change casually and different pools sit on different ones. Node pools, one per workload shape, each with its own opinion about how finely to carve. And MIG strategy, because the device plugin advertises instances as `nvidia.com/gpu` under the single strategy and as distinct resources like `nvidia.com/mig-1g.10gb` under mixed. So the resource names your workloads request are downstream of a configuration choice made somewhere else entirely.

The GPU operator's MIG manager reads a named layout out of a ConfigMap and a `nvidia.com/mig.config` label on the node. Good design, declarative and per node. It also means the label value has to match a ConfigMap entry, and that entry has to be legal for the card physically in that machine, and nothing checks either until a daemonset on that specific node tries to apply it.

So the failure ladder for one typo goes: values file renders, ConfigMap applies, node gets labeled, `nvidia-mig-parted` reads a layout it can't create and refuses. The node keeps its old geometry, the device plugin advertises accordingly, pods requesting the resource that was supposed to exist sit `Pending`. Nothing logs the word "invalid" at a severity anybody alerts on.

Sixteen nodes, eight cards each, one wrong profile name copy-pasted across a pool. That's arithmetic I did on a Tuesday and did not enjoy.

## Why the values files gave up

We started where everyone starts, a Helm chart with per-pool values files and a values schema. It held for a while.

JSON Schema can check that `profiles` is an array of strings matching a pattern. It cannot check that the compute slices across those strings sum to at most seven. It cannot know that `2g.20gb` exists on this card and not that one. It can't say "if MIG is on, the driver branch has to be one of these two." Every rule that matters here is a relationship between fields, and shape validation doesn't do relationships.

Then the merge. Three values files deep, base plus environment plus pool, and the effective config isn't something you read, it's something you render and then read. Reviewing a layout change meant running `helm template` and diffing output by hand, every time, because the diff in the PR was genuinely not informative about what would happen.

Then stringification. A profile name in a template is a string pasted into another string. `3g.40gb` and `3g.4gb` are equally valid as far as Go templates are concerned, and the difference between them is a full stop and about four hours.

And the one that actually decided it: there was nowhere to put the knowledge. Which profiles exist on which card is stable, published, and completely absent from a values file. It lived in three people's heads and a wiki page that was already wrong.

## Constraints instead

A type system is just a list of mistakes you've decided to stop making by hand.

CUE fit because its constraints and its values are the same thing, so the card's capabilities and the pool's request either unify or they don't. Here's the card, close to verbatim from what we run:

```cue
package gpu

// One creatable MIG instance. slices is the compute slice count the
// profile consumes, memGB is what the instance hands a pod, and max is
// how many of this profile a single card will accept.
#Profile: {
	slices: int & >=1 & <=7
	memGB:  int & >0
	max:    int & >=1
}

// H100 80GB: seven compute slices, and exactly these profiles exist.
// A name that isn't a field here is a typo, by construction.
#H100_80GB: {
	totalSlices: 7
	totalMemGB:  80
	profiles: {
		"1g.10gb": #Profile & {slices: 1, memGB: 10, max: 7}
		"1g.20gb": #Profile & {slices: 1, memGB: 20, max: 4}
		"2g.20gb": #Profile & {slices: 2, memGB: 20, max: 3}
		"3g.40gb": #Profile & {slices: 3, memGB: 40, max: 2}
		"4g.40gb": #Profile & {slices: 4, memGB: 40, max: 1}
		"7g.80gb": #Profile & {slices: 7, memGB: 80, max: 1}
	}
}
```

That's the wiki page, except it's executable and it's in the repo. Then what we want carved onto each card in a pool:

```cue
import "list"

#Carve: {
	card: #H100_80GB
	want: [...string]

	slots: [for p in want {card.profiles[p]}]

	usedSlices: list.Sum([for s in slots {s.slices}]) & <=card.totalSlices
	usedMemGB:  list.Sum([for s in slots {s.memGB}]) & <=card.totalMemGB

	counts: {for p in want {(p): len([for q in want if q == p {q}])}}
	_perProfileFits: [for p, n in counts {card.profiles[p].max >= n}]
	_perProfileFits: [...true]
}

serving: #Carve & {
	want: ["3g.40gb", "2g.20gb", "1g.10gb", "1g.10gb"]
}
```

Three rules, earning their keep differently. `card.profiles[p]` is the typo catcher: `#H100_80GB` is a definition, so its `profiles` struct is closed, and a name that isn't in it is an undefined field rather than an empty value that flows downstream. The `list.Sum` lines are the bin packing, capacity on both axes at once. `_perProfileFits` is the count ceiling, because summing to seven doesn't catch three `3g.40gb` on a card that takes two.

What you get when you're wrong is the part I actually care about:

```text
typo.slots.0: undefined field: "2g.24gb"
overcommitted.usedSlices: invalid value 8 (out of bound <=7)
overcommitted.usedMemGB: invalid value 90 (out of bound <=80)
tooMany._perProfileFits.0: conflicting values false and true
```

Cross-field rules use the same mechanism, which is the part that surprised me. Turning MIG on for a pool narrows what its driver branch is allowed to be:

```cue
#Pool: {
	name:         string
	migEnabled:   bool | *false
	driverBranch: "550" | "570" | "580"
	carve:        #Carve

	// MIG on these cards only runs on branches we've qualified, so
	// turning it on narrows what driverBranch is allowed to be.
	if migEnabled {
		driverBranch: "570" | "580"
	}
}
```

Set `migEnabled: true` alongside `driverBranch: "550"` and you get `conflicting values "570" and "550"` before anything renders. That combination was a real incident. It's now a line of config that refuses to exist.

## The honest limits

This model is an approximation and I want to be clear which parts. MIG's real rule is a placement table with fixed offsets, not a capacity sum, so there exist layouts my constraints accept that the hardware won't create. In practice the sums plus the count ceilings catch every mistake we've actually made, and for geometries going onto more than a handful of nodes we keep an explicit list of ones someone has run on real cards. If a placement ever bites us I'll encode the table properly and complain about it then.

CUE also only knows what I told it. It has no idea that a particular driver build misbehaves with a particular profile. That's still a human writing a comment next to a constraint, and the comment is load-bearing.

## What it was worth

The number that got this prioritized: at roughly seven dollars a card-hour, a 128 card pool carved wrong and left alone over a weekend is a five figure invoice for compute that ran nothing. We had a smaller version of that, caught on a Monday, and it bought me two weeks to do this properly.

The change I didn't expect was social. Reviewing GPU config used to mean a senior person squinting at rendered YAML and going on feel. Now the render either succeeds or it names the field and the bound it violated, so review is about whether a layout is a good idea rather than whether it's a legal one. Much better use of everybody.

Which leaves one thing open, and I've been circling it for a week. Argo CD reconciles this fleet and Argo CD has never heard of CUE. So all this checking currently happens in a CI job that runs `cue vet` and commits generated YAML, which works, and which I dislike for reasons I haven't finished assembling into an argument. Getting a renderer Argo CD trusts to run at sync time turns out to be its own story. I'll write that one once I've stopped being annoyed about it.