A Prometheus metric walks into Karpenter

Ask ten engineers what number they put in resources.requests.cpu and you’ll get ten variations of the same answer, which is that they copied it from a neighbouring service and then rounded up because a pod got throttled once.

I’m not being snide. I’ve done it. It’s a rational response to being asked a question nobody can answer, because the honest answer is “it depends on traffic I can’t predict, and if I guess low I get paged and if I guess high nobody ever notices.” The incentive gradient points one direction and it points at padding.

So here’s the thing that took me embarrassingly long to say out loud: if requests are fiction, then a scheduler decision made from requests is fiction, and a pending pod is fiction with a timestamp. Cluster autoscaling that waits for pending pods is reacting to a made-up signal after the fact.

That was the problem I spent a good chunk of 2025 on at Sales Marker, across a hybrid estate of a bit over 2000 vCPUs. What came out of it was a small control loop: real Prometheus signals feeding a controller that moved capacity before demand arrived rather than after. It held 99.99% availability at around 98% allocated utilization, which is the number I’m proudest of from that job. It also flapped so badly in its second week that I nearly deleted it.

Two lies stacked on each other

Pending-pod autoscaling has two independent problems and they compound.

The first is the fiction. A pod is pending because its requests don’t fit, and its requests are padding on top of a guess. So you provision hardware sized to a number that has, at best, a loose relationship to what the workload will actually consume. At our padding levels, allocating on requests meant real CPU utilization sat somewhere in the thirties while the scheduler insisted the cluster was full. You’re buying capacity for a spreadsheet.

The second problem is latency, and it’s the one that actually hurts. Walk the chain. Traffic rises. Pods saturate. HPA notices, but its metrics are averaged over a window and its sync period is 15 seconds by default. Replicas go up. New replicas don’t fit. Karpenter sees pending pods and launches nodes, which is fast by cluster autoscaler standards but still involves a cloud API, a boot, a kubelet registering, a CNI initialising, and then your image pull. Then readiness probes.

Add it up honestly and you’re a couple of minutes behind the traffic, minimum. Our morning ramp went from quiet to serious in under three minutes. A control loop whose response time is longer than the event it’s responding to isn’t a control loop, it’s a historian.

Signals that mean something

The fix wasn’t a better autoscaler. It was different inputs. I wanted signals that moved before saturation, not signals that described it.

Four, in the end.

p99 latency against the SLO budget, not against a fixed threshold. Latency ratio is the closest thing to a universal saturation signal, because it degrades before capacity runs out. Expressed as a fraction of the budget, it’s comparable across services that have wildly different absolute latencies.

Queue depth, where a queue exists. For anything async this is the best signal on the list by a distance. It’s an actual measurement of unserved work. No inference, no averaging, no padding. If the queue is growing, you are behind, and that statement needs no interpretation.

The derivative of arrival rate. Not the rate. The rate of change of the rate. This is the one that buys you the minutes back. Request rate tells you where you are; its derivative tells you where you’re going to be shortly, and traffic ramps have enough momentum that a 60 to 90 second extrapolation is usually right.

The daily curve. Our load was Japanese business hours and it was almost boringly repeatable. Sales tooling: quiet overnight, sharp climb from around 08:30 JST, plateau, dip at lunch, second smaller peak in the afternoon, decay from about 18:00. Once you accept that the shape repeats, the honest move is to stop pretending you’re surprised by it every morning. This was the least sophisticated input and probably the highest value one.

The recording rules, roughly as they ran:

groups:
  - name: capacity-signals
    interval: 30s
    rules:
      # p99 as a fraction of the latency budget. > 1 means we are
      # already spending error budget.
      - record: capacity:latency_pressure
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(http_request_duration_seconds_bucket{env="prod"}[2m]))
          ) / 0.4
 
      # unserved work, as a fraction of the depth we tolerate.
      - record: capacity:queue_pressure
        expr: |
          sum(worker_queue_depth{env="prod"}) / 400
 
      # where arrival rate is heading. positive means a ramp.
      - record: capacity:arrival_accel
        expr: |
          deriv(
            sum(rate(http_requests_total{env="prod"}[2m]))[10m:30s]
          )

Short windows on purpose. A 5 minute rate window on a 3 minute ramp is a smoothing function applied to the exact event you’re trying to catch.

The controller took the maximum of the pressure signals rather than a weighted blend. Two reasons. I never trusted myself to tune weights that wouldn’t rot, and blending lets a healthy signal mask a sick one, which is the specific failure mode where you find out during an incident that your composite metric looked fine the whole time. Max is crude and it’s legible at 3am, and I’ve come to weight legibility very heavily.

I also deliberately did not compute the max in PromQL. Element-wise max across unrelated series in PromQL is a puzzle rather than an expression, and a capacity controller is the last place you want a clever query. Three plain recording rules out, arithmetic in Go.

Talking to Karpenter

Karpenter has no “hold two spare nodes for me” knob, and I don’t think it should. Its job is to satisfy pending pods efficiently. So the way you ask it for capacity ahead of time is to give it pods, which is the placeholder pattern that’s been documented since the cluster autoscaler days and still works exactly as advertised.

A negative-priority class and a deployment of pause containers:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: capacity-reservation
value: -1000
globalDefault: false
description: 'Placeholder pods. First thing evicted, always.'

The controller scales that deployment’s replica count. Karpenter sees pending placeholders and provisions nodes for them the same way it would for anything else. When real traffic arrives, real pods at normal priority preempt the placeholders instantly, because preemption is a scheduler operation and doesn’t wait on hardware. The evicted placeholders go pending, Karpenter provisions more, and the buffer refills behind the workload.

The elegant part is that eviction and node provisioning happen in the right order. The workload never waits on a boot. The buffer does.

The NodePool it lands on wants a bit of care:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-burst
spec:
  weight: 20
  limits:
    cpu: '2400'
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 15m
    budgets:
      - nodes: '10%'
      # JST business hours. cron here is UTC, always.
      - schedule: '0 0 * * mon-fri'
        duration: 10h
        nodes: '5%'
  template:
    spec:
      expireAfter: 336h
      # nodeClassRef, requirements, taints omitted

Two fields carry most of the weight. consolidateAfter at 15 minutes stops Karpenter from reclaiming a node the moment the buffer shifts, because its timer resets whenever a pod lands on or leaves a node, and a placeholder buffer is by design a churning workload. Leave it near zero and you’ve built a machine for buying and returning EC2 instances.

The scheduled disruption budget throttles consolidation to 5% of nodes during the morning ramp. Karpenter’s consolidation is doing exactly its job when it looks at a half-empty node and decides to replace it with a smaller one, but during a ramp “half empty” is a measurement of a number that’s about to be wrong. Slow it down for the hours where you know the trend, let it run properly at night.

One detail worth the sentence: those cron schedules are UTC. Timezones aren’t supported. I lost an afternoon to a budget window I had confidently written in JST, which meant our morning protection was politely activating at six in the evening.

The week it flapped

The loop went in, worked beautifully for about ten days, and then started oscillating.

The mechanism was simple and I should have predicted it. The controller scaled the buffer up. Extra nodes appeared. Extra nodes made the average pressure signal drop, because the same load spread over more capacity looks calmer. Lower pressure, so the controller scaled the buffer down. Nodes got consolidated away. Pressure rose again.

Classic feedback without damping. The controller was measuring its own output and treating it as new information about the world.

Sawtooth on the node count graph, on maybe a nine minute period. Nothing broke, no SLO moved, which is precisely why it survived ten days before anyone looked. The cost was churn: node launches, image pulls, and a spot interruption rate that went up because we were rolling the dice on capacity far more often than we needed to.

Three things fixed it, all boring.

Asymmetric thresholds. Scale up when pressure crosses 0.7, scale down only below 0.45. That dead band is where the oscillation used to live. It also encodes the real asymmetry of the problem: being oversized costs money, being undersized costs error budget, and those are not the same currency.

Cooldowns, also asymmetric. 30 seconds before another scale-up, 10 minutes before any scale-down. Up fast, down slow. When we tried symmetric cooldowns the loop was still technically stable and still churned more than I wanted.

Rate limits on the delta. No more than 15% of current buffer capacity per decision. This one is less about stability and more about blast radius. A bad signal, and I have shipped bad signals, moves capacity by a survivable amount rather than the whole cluster.

None of this is novel. It’s the same hysteresis you’d put in a thermostat, and there’s something clarifying about admitting that a Kubernetes capacity controller is a thermostat with worse documentation.

What I’d keep

A year of running it, and the parts I’d rebuild without hesitating are: the arrival-rate derivative, because it’s the only input that genuinely bought time rather than reacting faster; the daily curve, because predictable load is a gift and refusing to use it is a kind of superstition; and taking the max of legible signals instead of blending them into something nobody can debug under pressure.

The part I’d do differently is that I built the controller before I built the observability for the controller. For its first two weeks the only way to know what it was thinking was to read Prometheus and infer. Instrument the decision, not just the outcome: current pressure, chosen buffer size, which signal won, which cooldown blocked what. Once I emitted that, the flapping diagnosis took ten minutes. Before it, I’d spent two days staring at node counts and guessing.

Autoscalers are software, and they deserve the same instrumentation you’d demand of anything else you put in the request path. Which a capacity controller absolutely is, whether or not it looks like it.