# Your autoscaler is a 300 GB file copy

> Picking a better autoscaling signal for GPU inference is polishing the wrong dial: the loop's response time is dominated by getting weights into HBM, so the controller cannot catch the incident that triggered it. A timed breakdown of where scale-up time actually goes on a 10 MW fleet, why this contradicts what I wrote about headroom and placeholder pods, and the warm-pool arithmetic I landed on instead.

Author: Karan Vijayakumar
Published: 2026-08-05
Tags: gpu, autoscaling, kubernetes, karpenter, sre
Reading time: 17 min
Canonical URL: https://karanvk.me/blog/autoscaler-is-a-file-copy

---

---
title: 'Six minutes of nothing: GPU autoscaling is a data-movement problem'
publishedAt: '2026-08-06'
summary: 'The autoscaler did its job in fifty seconds and the incident continued underneath it for six more minutes. Why the signal you scale on barely matters when scale-out latency is a property of node identity, and why that forced me to correct a post I published last year.'
tags: ['gpu', 'kubernetes', 'autoscaling', 'capacity', 'sre']
---

The graph is from a Tuesday morning in June and I have shown it to more people than any other graph I own.

At 11:04 the arrival rate on one model pool roughly doubled inside ninety seconds, because a customer had pointed a batch job at an interactive endpoint. That part is ordinary. The autoscaler noticed at 11:04:20. Karpenter had a `Ready` node at 11:05:12 — about fifty seconds from decision to schedulable hardware, which is genuinely good and which I would have defended in any review. The replica served its first token at 11:11.

Six minutes of nothing. The node-count panel went green and stayed green. p99 TTFT stayed above nine seconds until 11:13, and it came down because we shed the batch class by hand, not because the new replica arrived — which in practice meant telling a customer we had killed their job to protect everyone else's. The autoscaler did its job perfectly and the incident continued underneath it.

The part that took me longest to accept is that this was the *good* case. That node already had the runtime image and a copy of the weights on local disk. Everything that follows is about what happens when it doesn't.

I run SRE on a GPU fleet at Cogent Labs in Tokyo, roughly ten megawatts of contracted power, after two years on a 2000 vCPU Kubernetes estate at Sales Marker. Most of what I know about autoscaling came from the second one, and most of it doesn't survive here.

## The signal debate is the wrong dial

Every piece I read on this argues about the input. Queue depth. KV-cache utilization. The derivative of arrival rate, which is the one I picked when I [wired Prometheus into Karpenter](/blog/prometheus-karpenter-control-loop) at the last job and which I still think is the right primitive on CPUs. The [llm-d Workload Variant Autoscaler](https://github.com/llm-d/llm-d-workload-variant-autoscaler) reads request rate plus KV-cache utilization and queue depth, computes desired replicas from a capacity model, and hands the number to the scale target; [v0.5](https://llm-d.ai/blog/llm-d-v0.5-sustaining-performance-at-scale) refined that into explicit saturation detection against queue depth and KV-cache pressure. It is careful, well-argued work and I have no quarrel with any of it.

But all of those change *when the loop decides*. None of them change how long the decision takes to become capacity, and on this fleet the second number is roughly two orders of magnitude larger than the first.

Here is where the budget actually goes for a 70B-class dense model, tensor-parallel across eight GPUs on one node. These are rough numbers, because the precise ones aren't mine to publish, and they're medians over a few dozen scale-out events in Q2.

- Node provision, Karpenter decision to `Ready`: 45–70s
- Driver, fabric, device plugin, DCGM sidecar: 20–40s
- Container image pull, ~20 GB runtime image, cloud registry: 3–6 min
- Weight fetch onto the node: **8 min to over 40 min, or 0**
- Host to device transfer: 30–50s
- NCCL collective init across 8 ranks: 5–20s
- Engine init once the bytes are local — `torch.compile`, CUDA graph capture, KV allocation, warmup: 170–190s, near constant
- Prefix cache warm: never measured properly, which is its own admission

The terms people tune live in that list, and they are the small ones. Microsoft's teardown of vLLM cold start, [Dissecting LLM Container Cold-Start](https://techcommunity.microsoft.com/blog/linuxandopensourceblog/dissecting-llm-container-cold-start-where-the-time-actually-goes/4508831), measures that same near-constant bar independently and puts it at about 176 seconds regardless of how fast the image arrived. Their conclusion is that if your engine is vLLM, optimising the pull pipeline is the wrong lever — right up until the pull takes forty minutes.

Which it did for us. Weight fetch dominated by roughly an order of magnitude over everything else combined, and it dominated because of one property that has nothing to do with the model, the signal, or the controller: whether that specific node had seen that specific artifact before.

## Node identity is the variable, not the signal

A July preprint, [an empirical study of OCI-based model distribution](https://arxiv.org/abs/2607.16596), puts a number on the gap that I keep quoting to people. For a 70B-class artifact — 140 GB at fp16 — node-cached OCI delivery landed at 11.7 seconds. The same artifact re-downloaded from object storage: 40.7 minutes. That's 208x, and it is not a continuum. It's two regimes with almost nothing between them.

I reproduced the shape on our hardware and did not reproduce the magnitude. Our cached path came in nearer 90 seconds than 11, and our cold path nearer 25 minutes than 40, because our registry mirror sits on the same spine and our object storage is closer than theirs. Seventeen-fold, not two-hundred-fold. The ratio is a ceiling, not a constant. The **bimodality** is the transferable finding, and it's the one that matters: your scale-out time is not a distribution with a mean you can plan against, it's a coin flip between two very different numbers, and the coin is node identity.

The same paper names something I had wrong. containerd writes the blob and then unpacks it — a double pass — which makes the *first* cold pull cost up to twice a plain download of the same bytes. So for that 140 GB artifact you pay a 140 GB blob write, then a decompress-and-unpack pass producing another ~140 GB, then the loader's read on top, plus the ~20 GB runtime image. Call it 300 GB of local disk traffic before a single token exists. That is the whole event. Everything else in my budget list is rounding.

This is why the fast-loader work doesn't automatically save you. Run:ai's Model Streamer holds [a steady 80 Gbps from Azure Blob](https://devblogs.microsoft.com/azure-sdk/eliminate-llm-cold-starts-load-models-up-to-6x-faster-with-azure-blob-storage-and-runai-model-streamer/) where the default vLLM loader peaks around 40 and sags to 10 — 37 seconds against 226 on a 232 GiB model. Real, reproducible, and it works precisely because it skips the local disk hop. If you leave the weights baked into the container image, you've moved the bottleneck onto the node's own NVMe and its page cache, where no amount of network improves it.

One thing I'd argue is settled. The same study measured streaming hash verification at under 0.1% of delivery time, against a separate post-download verification pass costing up to 53%. Do that arithmetic against the *cached* path rather than the cold one and it gets worse: a standalone sha256 over 140 GB takes longer than the entire 11.7-second delivery it is verifying. When delivery is the whole budget, an integrity check priced at half of it is not a security posture, it's a decision to skip verification the first time someone is in a hurry at 03:00. Streaming verification is the only kind that survives contact with an incident, and surviving contact is the whole property you were buying.

## The number the benchmarks omit

AWS published a result I like a lot: [GPUDirect Storage on FSx for Lustre](https://aws.amazon.com/blogs/machine-learning/accelerate-llm-model-loading-and-increase-context-windows-with-gpudirect-on-amazon-fsx-for-lustre-and-turboquant/) loading Llama 3.1 70B in 1.28 seconds against roughly three minutes for a standard vLLM load, and 405B in 6.4 seconds against about eighteen minutes. The filesystem was 96 TiB Persistent_2 delivering around 94 GiB/s, and all eight GPUs read their own shards straight into HBM.

Read the method before you quote the headline, though, because I did it the other way round first. Those two figures are the FP8 path: weights quantized offline *and* pre-sharded per GPU rank, so each shard holds exactly the slices that GPU needs. The BF16 numbers from the same post are 2.17 and 10.4 seconds — still extraordinary, and still what you should compare against if you serve BF16. Pre-sharding is also not free in the way a benchmark makes it look: the artifact is now keyed by model *and* quantization *and* tensor-parallel degree, so changing TP from 8 to 4 doesn't reshape a cache entry, it misses one. Every variant you serve multiplies the surface over which node identity is a coin flip.

AWS is honest about the rest of the framing — the post says plainly that each node loads independently from the shared filesystem. So do the arithmetic they didn't. One BF16 replica at 141 GB is about 1.4 seconds of the filesystem's *entire* throughput. Thirty replicas cold-starting inside the same ninety-second window want 4.2 TB, which is roughly 45 seconds of the whole storage tier doing nothing else — and it is doing something else, because your training jobs checkpoint to it. At the FP8 shard size actually measured, halve it. Twenty seconds of total storage monopoly is still a training incident wearing an inference incident's clothes.

That's the condition a scale-out event creates by construction. The trigger for scaling and the trigger for simultaneous cold starts are the same event. A loader benchmarked on one idle node is being measured under exactly the circumstances that will never hold when you need it.

Our own tail says the same thing more cheaply. Time from scale decision to first token, across Q2: p50 around four minutes, p95 over eleven. Same image, same model, same instance type. The spread came from cache-miss nodes and from availability-zone placement, and a capacity plan built on that p50 is a plan that works on the days that don't need one.

A benchmark is a measurement of one node on a good day. A capacity plan is a claim about your worst node in your worst minute.

## I have to correct myself here

In June last year I published [a defence of running an estate at 98% allocated](/blog/against-headroom) with no headroom. The line I'd stand by is that idle capacity is organizational anxiety billed hourly, and the mechanism that made it work was a `capacity-reservation` priority class at −1000 holding placeholder pause pods, with `deferrable` batch work above it yielding in about forty seconds on average.

That worked because preemption is a scheduler operation. Evicting a pause container costs nothing, and the replacement pod is a process start.

On GPUs that trick does not transfer, and it fails at a specific joint. Preemption gives you the *device*. It does not give you the device's memory. A scheduled node with cold HBM is not capacity — it is a promise to become capacity in three to twenty minutes, which is the same promise the autoscaler was already making and failing to keep. The placeholder pod bought me scheduling latency, and scheduling latency was never the GPU problem.

So which post is wrong? Neither, and the resolution is narrower than either. What I argued against last year was *empty* headroom: silicon held in reserve doing nothing, absorbing signal, deferring feedback into the lap of whoever is on call three weeks later. I'd defend that today. What I got wrong was assuming the only alternative to empty headroom is no headroom. On GPUs there's a third state — capacity that is warm, meaning weights resident and graphs captured, and simultaneously *occupied* by work you are willing to destroy without ceremony.

Warm capacity is defensible. Empty warm capacity is not. That is a different claim from both of my previous ones and it took me a quarter to say it out loud.

The occupant has to be preemptible batch: eval sweeps, offline scoring, embedding jobs, low-priority fine-tunes. Every GPU fleet has this work. But there's a constraint that has no CPU analogue and it's the part that makes this hard.

> [!WARNING]
> On CPUs the placeholder held a *slot*, so any placeholder worked. On GPUs the occupant has to hold the *model*. A batch job running different weights on that node warms the driver and the image and nothing else — you keep the cheap terms of the budget and lose the expensive one. The preemptible occupant must be running the same checkpoint, at the same quantization and the same tensor-parallel degree, as the replica that will replace it: offline eval against the production model, batch scoring on the same weights, or a shadow replica chewing replayed traffic.

## Sizing the pool as arithmetic, not a percentage

Three inputs, all measured on your own fleet.

First, burst statistics over your time-to-warm window rather than over a minute — the standard mistake, and one I made. Measure the distribution of arrival-rate increase across the interval the loop actually needs. On our interactive pool, over a six-minute window, p99 of the increase was about 1.8x the trailing median and p99.9 about 3.1x.

```promql
# Burst ratio over the warm window, not over a scrape interval.
# Count arrivals at the gateway. vllm:request_success_total is the
# obvious metric and the wrong one: it counts what you served, so it
# flattens exactly when you saturate — the burst erases its own evidence.
# The denominator is the trailing hour, so business-hours ramp
# does not read as a burst. Subquery needs the parens or it will not parse.
  sum(rate(gateway_inference_requests_total{pool="interactive"}[6m]))
/
  quantile_over_time(0.5,
    (sum(rate(gateway_inference_requests_total{pool="interactive"}[6m])))[1h:1m]
  )
```

Second, time-to-warm per pool, as a measured histogram. Ours was p50 four minutes and p95 eleven on the cold path, dropping to roughly 90 seconds and three minutes once weights were node-local.

Third, per-replica capacity at your TTFT SLO, from your own load test. Not the vendor's throughput number, which was measured at a concurrency you will never run at.

Then the pool size is the ceiling of the p99 burst arrival rate minus provisioned capacity, divided by per-replica capacity at SLO. No percentage anywhere. For us that resolved to 0.8 times the trailing median of a busy hour, over per-replica capacity at the TTFT SLO, which came out at four warm replicas — half a rack, permanently occupied by eval work.

Being wrong small costs you the incident I opened with: you eat the cold path, and the SLO pays. Being wrong large costs less than people assume, because the pool is occupied. List price for an 8-way node-hour is a two-digit number and nobody at this scale pays list, so I'll leave the invoice out of it — the number that actually governs the decision is what the preemptible tenant loses to being killed. We measured that at roughly 15–20% of that pool's batch throughput, which is a real number and a much smaller one than the invoice suggests.

And there's a case where none of this is the right shape. Our load is Japanese business hours and it repeats almost boringly, which I said in the headroom post and which is still true. If you already know 09:00 JST is coming, a reactive loop against a multi-minute warm path can only ever apologise. Buy the nodes at 08:40 with a reservation controller and let the reactive loop handle the residual. A controller whose dead time exceeds the disturbance it's correcting is not a controller.

## The fast axis is admission control

So demote the scale-out loop. It is the slow axis, it is a cost-efficiency mechanism, and it is structurally incapable of catching a burst. The fast axis is what happens in the first two hundred milliseconds, and it's admission control: priority tiers on *requests* rather than pods, a queue-time budget per tier, and the willingness to shed or defer the class that can wait. Interactive gets 500 ms of queue budget, standard gets five seconds, batch is unbounded and gets stopped first.

The blunt version, which cost us a quarter to learn: the scale-out loop failing should not page. The admission controller failing should. We had it backwards — an alert on desired-replicas-pinned-at-max that woke someone up to watch a graph they could not affect for six minutes.

You cannot scale your way out of a burst that arrives faster than your weights. You can only choose who waits.

## Your image store is now load-bearing state

Here's the consequence nobody mentions. Once weights are cached node-locally, the node's image store is state, and Karpenter's entire job is deleting nodes.

`consolidationPolicy: WhenEmptyOrUnderutilized` will happily replace a node holding 140 GB of warm artifact with a cheaper empty one and book it as a saving. It *is* a saving, on the invoice. It is also a debit against your next scale-out, drawn at the worst possible moment, and nothing in the cost model can see it.

```yaml
# GPU NodePool. The CPU cost model assumes nodes are fungible
# and repacking is cheap; both are false once the node holds weights.
spec:
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30m
    budgets:
      - nodes: "10%"
      # Recycle outside JST business hours, when a cold start
      # is a batch problem rather than a TTFT problem.
      - nodes: "0"
        schedule: "0 8 * * 1-5"
        duration: 10h
  template:
    spec:
      # Node lifetime traded against cache value. Default is 720h;
      # we shortened it because driver drift and row remaps accumulate.
      expireAfter: 336h
```

`expireAfter` is the real trade and I don't think there's a general answer. Long-lived nodes accumulate warm artifacts, and they also accumulate driver drift, retired memory rows and every failure mode in [the node that lies](/blog/the-node-that-lies). We landed on 336h with the health controller holding override authority.

Warm-pool pods carry `karpenter.sh/do-not-disrupt`, and that annotation has no expiry — it's a bare `"true"` or nothing. I wanted a TTL on it and there isn't one, so a controller of ours strips it on a schedule. Without that, one forgotten annotation pins an expensive node out of the fleet indefinitely and the only symptom is a bill.

## Instrument the decision, not the outcome

This is the lesson I paid two days for on the previous controller. I had beautiful dashboards of what the fleet *did* and nothing at all about what the controller *decided*, so an incident review turned into two days of reconstructing intent from effects.

Two SLIs, and they're the first thing I'd build now rather than the last.

Time-to-warm: from scale decision to the replica passing a probe that generates an actual token. Not `Ready`, not the readiness gate, not "the pod is running" — a real inference. A histogram, per pool, per model, because those distributions are not the same shape.

And a measure of how much queueing happened while the loop believed it was helping:

```yaml
# scale_in_flight is our own gauge, set by the controller between the
# scale decision and first token. Nothing upstream publishes this:
# the HPA considers itself finished when the pod goes Ready, which is
# the exact moment the expensive part of the budget starts.
- record: pool:queued_during_scale:rate5m
  expr: |
    sum by (pool) (rate(gateway_requests_queued_total[5m]))
      and on (pool) (pool:scale_in_flight == 1)
```

Integrate that over a week and compare it against total queueing on the same pool. If almost all of your queueing happens inside scale-in-flight windows, the loop is not absorbing bursts, it is narrating them. Ours was a historian for two quarters and the dashboards were green the whole time.

## Where I've landed

Scale-to-zero is still a promise I can't make on the interactive path. llm-d v0.5 ships it with an activator that holds requests through the cold sequence instead of dropping them, and that's the right architecture — but against a floor of roughly three minutes of engine init *before any delivery at all*, the honest statement to a product team is "your first request after idle takes minutes." That isn't a cold start, it's an outage with a good excuse. We run it on batch endpoints, where it's fine and saves real money, and nowhere else.

Who should ignore all of this. If your traffic is steady, you don't have this problem; you have a bin-packing problem, and [MIG geometry](/blog/mig-bin-packing-cue) is a better use of your quarter. If your models are small — a 7B at 14 GB warms in seconds — the entire essay collapses and you should just scale on queue depth like a normal person. And if your fleet is under a few dozen replicas, reserved capacity is almost certainly cheaper than the engineering: this took two of us most of a quarter, and I've done the sum both ways more than once since.

What I'd do differently is the ordering. We spent six weeks arguing about queue depth versus KV utilization, and it was not a friendly six weeks — two teams, one of whom had built the thing being questioned. Nobody measured time-to-warm until the end. The measurement took an afternoon and ended the argument, and I still owe someone an apology for the five weeks before it.