98% utilization without paging anyone

Our clusters run at about 98% allocated CPU. Availability has been 99.99% over the trailing year. Both of those are true at once, and when I mention them together in conversation the usual response is a pause, then a question about what I’m not telling them.

So here’s the whole thing, including what I’m not telling them.

I run SRE for a hybrid Kubernetes estate at Sales Marker, a bit over 2000 vCPUs across cloud and hardware we own. The interesting decision in that estate isn’t a technology choice. It’s that we decided not to buy headroom, and then had to actually earn the right to that decision.

Headroom is a feeling

Ask why a cluster runs at 60% and you’ll get an answer about safety. Push on it and the shape of the answer changes. It becomes about the traffic spike nobody predicted, the deploy that doubles memory for four minutes, the node that dies at the wrong moment. Push once more and you land somewhere honest: we don’t know exactly what’s going to happen, and space feels like insurance.

I have real sympathy for this. It’s not stupid, it’s just expensive in a way that never shows up as a decision anyone made. Nobody at any company I’ve worked at has ever approved a line item called “40% of our compute, held in reserve, indefinitely.” It arrives as a default and then it’s load-bearing culture.

What bothers me more than the money is what headroom does to feedback. Slack in a system absorbs signal. A memory leak in a service with 40% headroom around it takes three weeks to become visible instead of three hours, and when it finally does, the on-call engineer inherits a problem that’s been quietly compounding since before they were on rotation. Headroom doesn’t remove failure modes. It defers them, with interest, into a state where the person who has to fix them has the least context.

Running tight is uncomfortable in a specific and useful way. You find out about things.

Three mechanisms, one of which is unglamorous

Running at 98% without paging anyone comes down to three things, and I’d rank them in order of how much they matter, which is roughly the inverse of how interesting they are.

Preemption tiers, which do most of the work

The single most important reframe: not all pods deserve the same capacity guarantee, and Kubernetes has had a mechanism for saying so for years that most teams never touch.

Four tiers, in our case:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: revenue-critical
value: 1000000
globalDefault: false
description: 'Request path. Never preempted. Fits first, always.'

Below that, standard at 10000 for everything normal, which is the global default. Below that, deferrable at 100 for batch, ETL, model retraining, report generation, the whole family of work where “an hour later” is not a real problem no matter how the owning team feels about it. And at the bottom, capacity-reservation at negative one thousand, which is placeholder pods holding space for the capacity loop.

That last tier is what makes the top number honest. When someone hears “98% allocated” they picture a cluster with nowhere to put a new pod. What we actually have is a cluster where the last several percent is occupied by work that yields instantly. Preemption is a scheduler operation, not a hardware operation. A revenue-critical pod that needs room takes it in the time it takes to evict a pause container, which is not a duration anyone measures in minutes.

The hard part was never the YAML. It was the eight weeks of conversations where every team explained why their workload was actually critical. That’s a political problem wearing a technical costume, and I lost a couple of those arguments. What broke the deadlock was making the tiers visible: a dashboard per team showing what tier their pods ran in, how often anything got preempted, and how long preempted work waited. Once “deferrable” turned out to mean an average delay of forty seconds rather than the four hours people had imagined, the arguments mostly stopped on their own.

Bin-packing discipline, which is mostly about requests

You cannot pack a cluster tightly if the shapes are wrong, and the shapes are wrong because requests are guesses.

We didn’t solve this with a clever scheduler. We solved it with recommendations and social pressure. VPA in recommendation mode across everything, so every workload has a number derived from what it actually consumed at p95 over two weeks. Then a weekly report showing, per team, requested versus observed. Nobody gets forced. The report just exists, and it’s visible, and a service requesting eight cores to use one and a half stops being a private matter.

Two rules on top of that, both learned the annoying way. Keep the ratio of the largest pod to the node it lands on sane, because one pod that needs 60% of a node makes every packing decision around it worse. And treat memory and CPU limits differently, because CPU throttling is a latency problem you can see and OOMKill is a correctness problem you can’t argue with. We run CPU limits loose and memory limits tight, and I’d defend that asymmetry anywhere.

The capacity loop, briefly

The third piece is a small controller that watches Prometheus signals which move before saturation does: p99 latency as a fraction of its SLO budget, queue depth for async work, and the derivative of arrival rate. It scales the placeholder deployment, which gives Karpenter pending pods to provision against, so nodes arrive ahead of demand rather than in response to it. Hysteresis and asymmetric cooldowns keep it from oscillating, and that took a couple of iterations to get right.

It deserves its own writeup, which it’ll get. For here, the only claim that matters is that the buffer is dynamic and predictive. Static headroom is a bet that the future looks like the worst day you can remember. A loop is a bet on your signals.

The Tuesday it got tested

In March a database failover went wrong in the way failovers do, which is not the failover itself but everything downstream of it.

Primary went away, replica took over, took about ninety seconds longer than it should have. In that window every service in the request path was holding connections and retrying. Retries stacked. Our worker queues went from a few hundred deep to somewhere over forty thousand in about two minutes, because every failed job was going back on the queue and new work kept arriving on top.

On a cluster with 40% headroom, here’s what happens: the queue workers scale up, they get scheduled immediately, they hammer a database that is already unwell, and the recovery takes longer because you gave the stampede somewhere to run.

Here’s what actually happened. Queue workers are deferrable. They tried to scale, there was no room for them at their tier, and the scheduler declined. Meanwhile the capacity loop saw queue pressure spike and started provisioning, but node provisioning takes a couple of minutes, so for those two minutes the queue simply grew. Which was the correct behaviour. The queue was the right place for that work to sit.

The request path never lost capacity, because revenue-critical pods fit first and there was placeholder space to preempt. User-facing p99 went up because the database was slow, not because we’d run out of compute. When the replica finished promoting, the drain took about eleven minutes with the extra capacity that had by then arrived.

Total error budget consumed: a bit under two minutes of the month’s allowance. Nobody got paged outside the primary on-call, and the retrospective was about the failover, which is where it belonged.

The uncomfortable version of that story is that tightness helped. The absence of spare capacity applied backpressure that a roomier cluster would have absorbed and then paid for over a longer, worse recovery. I’m not going to over-claim this. It could have gone differently with a different failure shape. But I’ve stopped believing that headroom is uniformly protective, because I’ve now watched it be the thing that lets a stampede accelerate.

Where I’ve landed

Some caveats I’d want anyone to hear before copying this.

It needs signals you trust. Not dashboards, signals: metrics you’d bet capacity decisions on, with known latency and known failure modes. If your instrumentation has gaps, the headroom is doing a real job and taking it away is reckless.

It needs preemption tiers that reflect reality rather than negotiation. If everything is critical, nothing yields, and 98% just means you’re one bad deploy from a pile of unschedulable pods.

And it needs load with some shape to it. Ours is Japanese business hours and it repeats almost boringly. Genuinely unpredictable load is a different problem and I’d hold more slack in that world.

But the general case I’ll defend: headroom is what teams buy when they don’t trust their signals. It’s a reasonable purchase under that condition. The mistake is treating it as a permanent architectural property rather than what it actually is, which is the interest payment on not having built the feedback loop yet.

Build the loop. The capacity gets cheap on its own after that.