Teaching deploys to read the SLO first

Fifty minutes. That’s how long it took one deploy to spend about a third of a service’s monthly error budget.

The change itself was three lines. Somebody adjusted a retry policy so a downstream timeout would be retried twice instead of once, which is a sensible-looking change and was in fact the exact wrong one, because the downstream in question was already saturated and the retries turned a slow path into a 5xx path. Green CI. Green canary metrics, because the canary’s own error rate looked fine while it hammered a shared pool. Deployed at 14:10 on a Tuesday. Somebody noticed at 15:00 because a customer asked.

Here’s what makes that story annoying rather than just bad. Prometheus knew at 14:14. The data was there, correct, queryable, four minutes in. Nothing was configured to ask.

That’s the gap I spent the last few weeks closing, and this is the writeup, including the parts I’m not proud of.

The estate, for context

We run a hybrid Kubernetes setup, a bit over 2000 vCPUs, and we run it hot. Around 98% utilization, 99.99% platform uptime. That combination is deliberate and it’s also why detection speed matters so much here: there’s no slack in the system to absorb a bad deploy. A service that starts misbehaving doesn’t get quietly cushioned by spare capacity, it just misbehaves.

Metrics live in a self-hosted LGTM stack taking north of 2 TB an hour. Which matters for one practical reason: I am not going to build a gate that fires an expensive query at that stack on every deploy. The burn-rate arithmetic has to be precomputed.

Burn rate, and why multi-window

If you’ve read the SRE workbook chapter on alerting on SLOs you can skip this. Briefly: error budget burn rate is how fast you’re consuming your allowance relative to spending it evenly over the window. A 99.9% SLO gives you 0.1% errors over 30 days. Burn at 1x and you exhaust it exactly at the end of the month. Burn at 14.4x and you’re done in about 50 hours.

The workbook’s move is to alert on two windows at once, a long one for signal and a short one so the alert resets quickly when the burn stops. The recommended pair for a page is 14.4x over 1 hour with a 5 minute short window, and 6x over 6 hours with a 30 minute short window. Those numbers aren’t arbitrary: an hour at 14.4x is 2% of a 30 day budget, six hours at 6x is 5%.

The recording rules are the boring half. One per window, evaluated in the ruler so nothing has to compute them at query time:

groups:
  - name: slo_signal_api
    interval: 30s
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="signal-api", code=~"5.."}[5m]))
            /
          sum by (job) (rate(http_requests_total{job="signal-api"}[5m]))

Then the same expression again at 30m, 1h, and 6h, each recorded under the matching name. Four rules per service, generated from a template because writing them by hand is how you end up with a 6h rule that secretly uses a 1h range.

The alert on top of them:

- alert: SignalApiErrorBudgetFastBurn
  expr: |
    (
      job:slo_errors_per_request:ratio_rate1h{job="signal-api"} > (14.4 * 0.001)
      and
      job:slo_errors_per_request:ratio_rate5m{job="signal-api"} > (14.4 * 0.001)
    )
    or
    (
      job:slo_errors_per_request:ratio_rate6h{job="signal-api"} > (6 * 0.001)
      and
      job:slo_errors_per_request:ratio_rate30m{job="signal-api"} > (6 * 0.001)
    )
  for: 2m
  labels:
    severity: page

The 0.001 is the budget, so it’s the one number you change per service. We keep it in the same file that defines the SLO, because two sources of truth for “what does 99.9 mean here” is a bug I have already shipped once.

Wiring it into the pipeline

Two pieces. A gate that runs before promotion, and a watcher that runs after.

The gate is the easy one conceptually. Before promoting to production, ask whether the service is currently burning. My first version was a single line in the workflow:

promtool query instant "$PROM_URL" \
  'ALERTS{alertname="SignalApiErrorBudgetFastBurn", alertstate="firing"}' \
  | grep -q . && exit 1

That shipped, and it had a bug I want to point at because it’s the classic one. No result and a dead Prometheus look identical. If the query endpoint was unreachable, promtool printed nothing, grep found nothing, and the gate cheerfully concluded that the error budget was in great shape. So it did nothing at all in precisely the situation where you’d most want it doing something.

So it became a script that has opinions about its own failures:

import json, os, sys, urllib.parse, urllib.request
 
prom = os.environ["PROM_URL"]
expr = os.environ["BURN_EXPR"]
url = prom + "/api/v1/query?" + urllib.parse.urlencode({"query": expr})
 
try:
    with urllib.request.urlopen(url, timeout=10) as resp:
        body = json.load(resp)
except Exception as err:
    sys.exit(f"cannot reach prometheus ({err}), refusing to promote")
 
if body.get("status") != "success":
    sys.exit(f"prometheus returned {body.get('status')}, refusing to promote")
 
firing = body["data"]["result"]
if firing:
    sys.exit(f"fast burn active on {len(firing)} series, refusing to promote")
 
print("error budget looks fine, promoting")

There’s a second hole in the same family and it took an incident to find. A relabel change renamed a metric label, the recording rules quietly started producing no series at all, so there was nothing for the alert to evaluate and nothing for the gate to see. It passed everything for most of a day while the SLI was completely blind. The fix is to check for that absence directly, rather than inferring health from a silent alert:

absent(job:slo_errors_per_request:ratio_rate5m{job="signal-api"})

If that returns anything, the SLI is missing and the gate refuses. Cheap, and it has since caught two more instances of the same class.

The watcher is the after half, and it’s cruder. The deploy step pushes a rollout marker, then a job polls the fast-burn expression every 30 seconds for 20 minutes. If it trips inside that window, the job runs kubectl rollout undo on the deployment and pages whoever shipped it. Twenty minutes is not a principled number. It’s how long our slowest rollout takes to fully replace pods plus a bit, and it’s short enough that a burn starting at minute 25 belongs to somebody else’s shift, which we know and accept.

What it caught

The retry change that started this post. We reproduced it in staging, then let a similar one through in production on purpose during a low-traffic window. Fast burn tripped at just under four minutes, revert completed at five and a half. Total budget spend: a bit under half a percent of the month instead of thirty.

A cache TTL change that looked completely harmless and turned out to double origin load enough to push a downstream over its own SLO. Caught by the 6h window rather than the 1h one, which is the case for keeping both.

And, unexpectedly, two blind SLIs. Not deploy problems at all, just the absent() check earning its place.

What it misses, and this list is longer

Slow burns. The workbook has a ticket-severity pair at 3 days and 6 hours, and it’s genuinely useful for humans and completely useless as a gate, because nobody is going to hold a promotion for three days waiting on a window to clear. So the gate only ever sees fast breakage. Gradual degradation, the kind where p99 creeps and error rate ticks from 0.02% to 0.08%, is still found by a person looking at a dashboard on Thursday.

Anything that skips the pipeline. Feature flags flipped in an admin UI. A ConfigMap edited by hand at 2am during an incident. An index dropped in a migration that ran outside the deploy. Our worst budget burn last quarter was a feature flag, and the gate never had a chance to have an opinion about it.

Low-traffic services. A 5 minute window on something serving four requests a minute is noise, and after a few false reverts we exempted them. Which means the services with the least observability pressure on them are also the ones with no gate, and I don’t have a good answer for that yet.

Cross-service blast radius. The gate watches the SLO of the thing being deployed. Deploy A, break B, and nothing intervenes until B’s own alerting fires, by which point A’s watcher window has closed and A looks innocent.

The part that bothers me

All of this is CI duct tape and I want to be clear about that rather than dress it up.

It’s a composite action, a Python script, and a polling job that holds a CI runner for twenty minutes doing almost nothing. If someone cancels the workflow, the watcher dies and nothing reverts, because there’s no state anywhere outside the running job. The revert path is the least exercised code in the whole pipeline, which is a wonderful property for the code you only run when things are already bad.

And the revert itself is only half a revert. kubectl rollout undo moves the cluster back. It doesn’t move Git. So the desired state still says the bad image, and the next deploy, or any reconcile, puts it right back. We handle that with a page and a checklist item that says “revert the commit,” which is exactly the sort of task humans are worst at under pressure.

A rollback that doesn’t move Git is a lie with a timer on it.

Which is the thing I keep coming back to. The gate works. It has paid for itself several times over already and I’d build it again tomorrow. But it’s bolted onto the side of the deploy, and the decision it’s making, is this change still worth keeping, is fundamentally a continuous-delivery decision, not a CI one. The system that owns the desired state should be the system that decides to abandon it. It should know the SLO the same way it knows the image tag, and reverting should mean changing what it’s converging toward, not fighting it.

I don’t have that. What I have is a Python script and a twenty minute timer. But I’m increasingly sure that’s where this ends up, and I’d like to be the one who builds it.