The router that failed over by itself

The failover happened at 3:47 in the morning. I found out at 10:15, holding a coffee, standing behind someone at standup who had pulled up a traffic-by-zone graph for an unrelated reason. There was a step function in it. Two zones went up, one went to nothing, and forty seconds later everything was flat again.

Six and a half hours between a production failover and a human noticing. That gap is the whole point of the project, and it took about eight months to earn.

Some context. FourKites, 2023, SRE work on the API tier. We built a layer-4 router that sat under the API fleet and steered traffic between backends, with automated failover nobody had to approve. It’s the infrastructure I’m still most attached to, mostly because so little of the difficulty was where I expected.

Why layer 4

The question I got in every design review: why not another L7 proxy?

Long-lived gRPC streams, first. Our traffic wasn’t independent requests, it was a population of HTTP/2 connections that stayed open for hours multiplexing thousands of RPCs each. An L7 proxy terminates those connections, so a config reload becomes a decision about streams that are mid-flight: kill them (customers see it) or wait (indefinitely, because some never end). Draining an L7 proxy with hour-long streams on it isn’t a solved problem, it’s a choice between two bad afternoons.

Then TLS passthrough. Terminating at the routing layer means that layer holds private keys and becomes another fleet you rotate certificates on, and it breaks the end-to-end mTLS several integrations relied on. At layer 4 the bytes pass through untouched and the backend handshakes with the actual client. Fewer secrets in fewer places is a property I’ll take for free.

What actually decided it was the latency budget. Every hop spends part of your objective, and a proxy that parses headers, evaluates rules, and re-encodes does real per-request work. You can make that fast. You can’t make it free. Our routing budget was tens of microseconds, and at L4 a forwarding decision is a hash and a table lookup. Every millisecond I didn’t spend there was one available for somebody’s actual database query.

The L7 proxy stayed, by the way. Never a replacement, just a thing underneath.

What it actually did

Traffic arrived at a virtual IP announced from several router nodes. A connection table kept each connection on one backend for its whole life, consistent hashing over a ring picked that backend, and a backend going away drained rather than dropped: weight to zero for new connections, existing ones allowed to finish or hit a deadline.

The decision worth explaining: backends were never up or down. Each carried a health score from 0 to 100 that set its weight in the ring. Binary health is a lie that costs money, because the realistic failure isn’t a machine that stops answering, it’s one that answers slowly, or answers wrong 4% of the time, or is fine except on one code path. A score lets you pull 30% of traffic off something clearly unwell without committing to the claim that it’s dead.

The forwarding path was a few hundred lines and I barely touched it after the first month. Four fifths of the codebase was about deciding what “healthy” means, and every hard bug for the next two years lived in that four fifths.

Health checks that don’t lie

The failure mode that keeps me up isn’t a check that fails when things are fine. That’s annoying and it’s visible. It’s a check that passes while the thing is broken, because now automation is steering traffic into a hole.

Every health check is a model of what working means, and every model is wrong in the optimistic direction. A TCP connect succeeds against a process deadlocked in its worker pool. An HTTP 200 comes back from a handler that returns a static string and never touches the database.

So we ran two independent signals and required them to agree.

The synthetic probe was a real gRPC call to a real method with an assertion on the response body, over the same path production traffic took. Not a dedicated health endpoint: the value of a synthetic is that it exercises the machinery, and an endpoint that short-circuits it exercises nothing.

The real-traffic signal came from what the router could already see without parsing anything: connection resets, TLS handshake failures, time to first byte, setup latency. Per backend, effectively free, since it handled those bytes anyway.

A backend needed both signals healthy to hold a full score, and disagreement between them degraded the score on its own. That last part was the good idea. Synthetics passing while real error rates climb means the probe tests something customers don’t do. Synthetics failing on quiet traffic means it really is broken. Either way your model of working has drifted from reality, so we alerted on sustained disagreement.

It paid off almost immediately. Our probe hit a method that had quietly been excluded from a feature flag rollout, so it passed happily for three weeks while one specific client path returned 500s. The probe never noticed. The disagreement did.

Split brain, and the bias toward doing nothing

The scary part of automated failover isn’t failing over. It’s two nodes concluding they own the VIP, or two observers each deciding the other is dead.

Ownership went through a lease with a fencing token, and a node that lost its lease stopped announcing inside a window strictly shorter than the lease TTL. That inequality is the entire safety argument, so we tested it with SIGSTOP rather than SIGKILL. A dead node is easy. A node that resumes after eleven seconds still believing it’s in charge is the one that hurts.

More importantly, a router that couldn’t reach the coordination store didn’t get to make failover decisions. It kept forwarding on last known state and stopped changing anything. A partitioned router doing what it was doing five seconds ago is mostly harmless. One with opinions can move all your traffic somewhere terrible on the strength of a view only it holds. Demotion needed agreement from more than one observer, for the same reason.

Failback hysteresis, or: flapping is worse than failing

Failing over quickly is the easy half. Failing back is where I got hurt.

A backend that just recovered isn’t the backend you removed. Cold caches, empty connection pools, deoptimized JIT, stale DNS. Return it to full weight the moment a check goes green and it falls over under a third of the load it handled fine an hour ago, and a flap is worse than the original problem, because every cycle drops connections.

So the thresholds were deliberately lopsided. Three consecutive bad results to demote, twelve good ones to promote. Then a ramp instead of a jump: 5% of nominal weight, 20%, 50%, 100%, each step gated on the score not sagging. Plus a cooldown that doubled per flap in a rolling window, capped at an hour. A backend that flapped three times in ten minutes got pinned out and paged a human, on the grounds that the router had run out of useful opinions.

One more rule came out of a humbling incident. A dependency got slow rather than unavailable, so every backend degraded at once, in the same way. Scores oscillated in lockstep and an earlier version of the router spent twenty minutes shuffling traffic between machines that were all equally miserable, achieving nothing but connection churn. Moving traffic can’t help when the problem is downstream of everybody. So now, if more than half the fleet degrades at once, the router stops rebalancing and says so. That isn’t a failover situation, it’s an outage, and treating it as one makes it worse.

The night it earned its keep

The 3:47 event was an availability zone brownout, the version of an AZ failure that doesn’t do you the courtesy of being obvious. Instances stayed up. The status page was green. But packet loss climbed and latency inflated inside the zone, so backends there were reachable and slow rather than gone.

The real-traffic signal caught it before the synthetic probes did, which is the opposite of what I expected when we built the thing. The probes were still passing. Slower, but passing, and a probe that passes slowly is a probe that passes. Meanwhile time to first byte in that zone roughly tripled and resets climbed, so scores decayed, weight shifted to the other two zones over about eleven seconds, and open connections drained instead of dying. API p99 bumped for under a minute and settled.

Nobody paged. The burn rate alert never fired because there wasn’t meaningfully any burn. Hence the coffee and the graph.

My honest first reaction wasn’t pride, it was a small cold feeling of what else has this thing been doing that I don’t know about. Which turned out to be the right instinct: we added a weekly digest of every automated action the router took, mailed to the team whether or not anything hurt. Automation you can’t audit isn’t reliability, it’s just a faster way to be wrong.

What it doesn’t do

It can’t fix a bad request, because at layer 4 there is no request. No header inspection, so no per-endpoint routing and no request-level retries, since a retry needs to know an operation is idempotent and bytes on a socket won’t tell you.

Connection granularity is coarse for long-lived streams, which is ironic, since streams are why we went to L4 at all. A stream opened an hour ago is pinned until it ends, so draining means waiting out the longest one or breaking it. We capped stream lifetime at thirty minutes to bound the wait, which pushed reconnection handling onto every client, a six-month conversation with six teams.

It also put network topology inside the application’s failure domain. When the router misbehaved, everything did.

The limit I actually worry about is the dozen or so tunables in the health scoring, fitted to a traffic shape from 2023. Two years on I’d bet none have been revisited, because nothing is broken and nobody re-derives thresholds on a system that isn’t paging. The failure mode of a well-tuned automated system is quietly becoming a badly-tuned one.

Where I landed

The advice I’d give is about where the time goes. The routing is a couple of weeks. The trust is a year. Every hour I spent making failover faster returned less than every hour spent making sure it wouldn’t fire when it shouldn’t, because a failover that fires wrongly at 3am doesn’t merely fail to help. It spends the credibility you need to be allowed to keep the automation at all.