Blog.

Notes from wiring CMP sidecars into a GPU fleet rollout: what plugin.yaml actually does, a SOPS plugin and a CUE plugin that earn their keep, and the ways all of this bites in production.

Five nines is 5.26 minutes of downtime a year, which rules out heroics by arithmetic. Looking back two years later at what actually moved the number, what only looked like reliability work, and why almost all of it came from changing client behavior rather than adding capacity.

Notes on a Rust and Aya agent that captured scheduler, disk, and socket metrics across a fleet of warehouse robots: why devices are the worst observability target, how the CPU budget became a test, and the map that filled up and made everything look wonderful.

GPU partitioning config is a combinatorial mess where a wrong answer does not crash, it just leaves very expensive silicon idle. Why Helm values files could not hold it, and what happened when I moved the whole thing into CUE constraints.

Two weeks into a lights-out GitOps rewrite for a GPU fleet, the hard question is not which tool to use, it is which tool owns what. The line I drew, the gray zone where every argument actually happens, and why the bootstrap handoff is a real design decision.

A retrospective on the gRPC gateway I ran at Sales Marker: the p99 budget that shaped every decision, the Tokio habits that held it, and the discovery that our worst tail latency lived in HTTP/2 flow control rather than in our code.

A GPU node with intermittent NVLink faults quietly returned wrong inference results for most of a day without ever failing a health check. This is the controller we built so it cannot happen twice: DCGM signals in, a policy engine deciding cordon versus drain versus recycle, checkpoint-aware draining, and the false positives that cost me eleven nodes.

What Xid errors actually are, which ones mean a GPU is about to die and which ones are just an application bug that wakes you up for nothing, how correctable and uncorrectable ECC differ, why row-remap-pending is the canary, and the dcgm-exporter metrics and alert rules I settled on.

Notes from five weeks on an accelerator fleet: why compute measured in megawatts breaks CPU-land instincts, what DCGM and Xid and MIG mean day to day, and the naive question I asked in week one that reframed the whole job.

CPU requests are fiction, so scaling on pending pods is scaling on fiction. Notes on a small controller that fed real Prometheus signals into Karpenter capacity, held 99.99% at 98% utilization, and flapped badly before it did.

The renewal quote that started it, the order we migrated in (Loki, then Mimir, then Tempo), everything that broke on the way, and the honest ledger with hindsight.

A year-ender written in an empty Tokyo office: the numbers that held, the two things I shipped that mattered, the failure I did not see coming, and what has been pulling at my curiosity all year.

Puzzles unlock at 2pm in Tokyo, which makes December a lunch-break sport. Notes on the rig, the day 15 run that finished second globally, and why unreadable code is fine for exactly one month a year.

A year of running Loki, Mimir and Tempo ourselves at over 2 TB an hour: what the infrastructure actually costs, how much upgrade toil is real, what it feels like to be on call for your own observability, and what I would do differently.

Two years on the hiring side of an SRE loop in Tokyo: the questions I dropped, the four I kept, and what a candidate correcting themselves twenty minutes late taught me about signal.

A single deploy ate a third of a monthly error budget in fifty minutes. So I wired multi-window burn-rate queries into the pipeline as a promotion gate, plus a watcher that reverts on fast burn. Honest notes on what it catches and where it is still duct tape.

Idle capacity is organizational anxiety, billed hourly. How priority classes, bin-packing discipline and a predictive capacity loop let us run a 2000 vCPU estate near full without trading away availability.

The stock objections to Rust in infrastructure, answered from three systems I actually shipped: a rate limiter, an eBPF agent on a robot fleet, and the gRPC gateway I run today. Including the places I still reach for Go and Python.

Why we put a layer-4 router under long-lived gRPC traffic instead of another L7 proxy, and why the routing was the easy part. Health scoring, split-brain protection, failback hysteresis, and the AZ brownout we found out about from a graph at standup.

An exporter is production code with a scrape target stapled to it, and most teams audit neither half. Notes from running a 50K samples/second metrics pipeline: the cardinality incident, drop rules that actually work, and the quality bar I ended up enforcing.

Contests gave me habits that broke production code and instincts that now save it. A story about a rate limiter I could not read, and what I kept after I stopped writing like a contestant.

A comprehensive guide to Kubernetes networking and all service types with detailed explanations, use cases, and architecture diagrams.

A comprehensive look at the evolution of state management in React applications, from Redux to React Server Components.

Why Rust is taking over the JavaScript tooling ecosystem, and what this means for web developers.

Introducing the new blog - A journey into AI, ML, and software engineering.