# Five nines is a boring number > Five nines is 5.26 minutes of downtime a year, which rules out heroics by arithmetic. Looking back two years later at what actually moved the number, what only looked like reliability work, and why almost all of it came from changing client behavior rather than adding capacity. Author: Karan Vijayakumar Published: 2026-06-27 Tags: sre, reliability, slo, engineering-culture Reading time: 9 min Canonical URL: https://karanvk.me/blog/five-nines-boring --- Five nines is 5.26 minutes of downtime a year. Twenty-six seconds a month. One bad afternoon spends the annual budget and borrows against next year's. I helped take a system-wide API protection layer to that number at FourKites, and it has never once felt like an achievement. It felt like an audit result. Every time somebody congratulated me I had the same deflating thought, we didn't build our way there, we subtracted our way there. Very little of the work was clever. Most of it was deleting things, lowering things, and telling people no. I'm two years and three jobs downstream now, on a GPU fleet whose failure modes have nothing in common with that job: thermals, driver versions, a scheduler that will leave two hundred thousand dollars of silicon idle because a node label has a typo. Enough distance to be honest about which parts mattered and which parts I was just enjoying. ## The arithmetic that rules out heroics Page fires. Phone buzzes. Human surfaces from sleep, finds a laptop, gets a VPN up, loads a dashboard, forms a hypothesis. On a good night you are eight minutes in before you have done anything, and you have already overspent the annual budget by half. Add a second person. Add the two minutes where everyone agrees the graph looks weird. There is no version of five nines where a human is in the loop for a common failure. So the target stops being "respond quickly" and becomes "don't have the failure," or failing that, "have the failure absorbed by something that doesn't need to wake up." ## What actually moved the number Four things, and three of them look like nothing on a resume. ### Load shedding, which means protecting a system from its own customers The rate limiter came first: Rust, gRPC, around 100K requests a second. What it was *for* is the part I'd get wrong in an interview. Not fairness. Not billing. It existed so no single client, including well-intentioned internal ones, could take the platform down by being enthusiastic. It took me embarrassingly long to accept the underlying fact, which is that nearly all our outages were caused by our own users' traffic, at volumes the system had been sized for, in shapes it hadn't. A mobile client retrying on a fixed timer with no jitter, so every phone on the planet retried in the same second. A partner polling a list endpoint every 200ms because someone hardcoded a value in 2019 and then left. Capacity does not fix that. A system that collapses at three times nominal load collapses at three times whatever you provision, so buying servers just moves the cliff. Shedding fixes it, because a 429 served in three hundred microseconds is a successful outcome for the system even though it's a failure for that request. Which is why the real design constraint was that rejection had to be cheaper than service. The limiter needed to say no faster than the backend could say yes, or you've built a slower way to fall over. Everything else was negotiable. That wasn't. ### The router that failed itself over Second, a layer-4 router with automated failover, which I've written about separately. It took the most common infrastructure failure, one degraded machine or one sick availability zone, and made it resolve itself in seconds without a page. The routing was never the hard part. It took a year of health scoring and hysteresis work before anyone trusted it unsupervised, and unsupervised is where the availability was. ### Killing retry storms If I started over I'd do this first, and I'd expect it to be the largest win. We found a client library configured for five retries at a fixed 100ms interval with no jitter. Work through it. A service near capacity briefly errors on 10% of requests. Those become six, synchronized to the same 100ms grid, so offered load jumps by half at exactly the moment the system has the least headroom. Which raises the error rate. Which produces more retries. That's not a retry policy, it's a denial of service attack with good intentions and full credentials. The fixes are textbook and none of them are the hard part. Exponential backoff with full jitter, so recovery attempts spread out instead of arriving as a wall. A retry budget, capping retries at a fraction of successful request volume over a rolling window, which most people skip and which bounds the amplification. Deadlines propagated from the caller's remaining budget rather than a per-attempt constant, so a request already out of time doesn't get retried at all. ```text # Full jitter, per the AWS backoff writeup sleep = random(0, min(cap, base * 2 ** attempt)) ``` The hard part was social. That logic lived in eleven client libraries across five languages owned by six teams, and my pitch was "please make your code slightly slower to recover, for the benefit of a system you don't operate." Six months of pull requests, and nobody's performance review mentioned it. ### The timeout audit The dumbest item on the list and probably the highest value per hour spent. We inventoried every timeout in the request path and put each one next to the objective it was supposedly protecting. A spreadsheet. Two weeks of grepping for anything that looked like a duration, then a month of arguing over rows. What we found was worse than I'd guessed. A 30 second HTTP client timeout inside a request path with a two second latency objective. gRPC calls with no deadline at all, which is the default and functionally infinite. And best of all, a retry policy stacked on one of the 30 second timeouts, so one logical request could hold a worker for 90 seconds while the client that made it gave up 88 seconds earlier. A timeout longer than the objective it protects is not a timeout, it's a comment. It documents an intention while permitting exactly the behavior it claims to prevent. Fixing those rows removed more tail latency than any optimization I have written, and none of it required reading business logic. ## What looked like reliability work and wasn't Dashboards nobody opened. North of two hundred Grafana dashboards, and I instrumented view counts on them, which felt petty and was the most useful thing I did that quarter. Nine accounted for nearly all views, and about a hundred and forty hadn't been opened in ninety days. They weren't neutral: query load on Prometheus, search made useless so people rebuilt panels they already had, and a manufactured sense of coverage. "We monitor that" was true in the sense that a panel existed somewhere. Chaos experiments that confirmed things we knew. We ran a chaos program for about a year, and the first two months found real problems. Then it settled into terminating pods and confirming that Kubernetes reschedules pods. Every experiment had a hypothesis we were confident about and we were right every time, which should have been the tell. Meanwhile the failures that actually hurt were never "a thing died." They were "a thing got slow," "a thing lied about being healthy," "a thing came back and stampeded." All much harder to inject, which is precisely why we didn't. A runbook wiki that rotted. Three hundred pages, median last-edited date well over a year old, describing a system rebuilt twice underneath it. The mechanism is obvious once you say it out loud: nobody reads a runbook until 3am, and nobody edits one at 3am. Following confident instructions for an architecture that no longer exists is worse than having nothing, because you'll trust it for the first four steps. What survived was short pages attached to a single alert, each with a "verified on" date an owner re-signed quarterly. Putting an expiry date on documentation was unpopular and correct. ## The part that still surprises people Almost all of the availability we gained came from changing client behavior, not from adding server capacity. Not one of the four added a machine. The rate limiter reduced effective load. The retry work reduced it dramatically, precisely when it counted. The timeout audit freed up workers. The router moved existing load around. We spent two years making the system serve *less* traffic more carefully, and the number went up. That inverts everyone's intuition, which makes it a hard sell. Reliability feels like it should be additive: more replicas, more zones, more headroom. Redundancy matters, it's table stakes. But past a point each additional server is one more thing that can be slow, one more pool to exhaust, one more cache to go cold. Adding capacity to absorb pathological client behavior is paying rent on a problem instead of fixing it. Meanwhile the demand side is unbounded and almost entirely unowned. Every service has an owner. Every dashboard has an owner. Nobody's objectives say "our clients behave reasonably," so the thing with the largest effect on availability sits in the gap between teams, which is where an SRE function should be useful and mostly isn't. ## Two years on The GPU work is a different world. My failure domain is driver versions and MIG profiles and a scheduler making expensive decisions on the strength of a label. But the pattern transferred without modification. My highest-value contribution this quarter is render-time validation that turns an invalid GPU pool spec into a failed pull request instead of a 3am page about pods pending on a quarter million dollars of hardware, plus a gate that reverts a bad rollout without asking me. Both are subtraction: both remove a path by which a human becomes necessary. The awkward thing about five nines is that once you're there the work becomes invisible, which makes it a strange thing to aim at. Nobody thanks you for the outage that didn't happen. The graph is flat, the pager is quiet, and if you're not careful somebody looks at the quiet and concludes the team is overstaffed. I don't have a fix for that. I just know the number is boring by construction, and the day it stops being boring is the day something has gone badly wrong.