https://karanvk.me/blog Technical articles on Kubernetes, infrastructure, machine learning, and software development by Karan Vijayakumar. en-us Sun, 26 Jul 2026 20:08:40 GMT https://karanvk.me/blog/argocd-cmp-plugins https://karanvk.me/blog/argocd-cmp-plugins Sat, 25 Jul 2026 00:00:00 GMT Notes from wiring CMP sidecars into a GPU fleet rollout: what plugin.yaml actually does, a SOPS plugin and a CUE plugin that earn their keep, and the ways all of this bites in production. https://karanvk.me/blog/five-nines-boring https://karanvk.me/blog/five-nines-boring Sat, 27 Jun 2026 00:00:00 GMT Five nines is 5.26 minutes of downtime a year, which rules out heroics by arithmetic. Looking back two years later at what actually moved the number, what only looked like reliability work, and why almost all of it came from changing client behavior rather than adding capacity. https://karanvk.me/blog/ebpf-robot-fleet https://karanvk.me/blog/ebpf-robot-fleet Sat, 13 Jun 2026 00:00:00 GMT Notes on a Rust and Aya agent that captured scheduler, disk, and socket metrics across a fleet of warehouse robots: why devices are the worst observability target, how the CPU budget became a test, and the map that filled up and made everything look wonderful. https://karanvk.me/blog/mig-bin-packing-cue https://karanvk.me/blog/mig-bin-packing-cue Sat, 23 May 2026 00:00:00 GMT GPU partitioning config is a combinatorial mess where a wrong answer does not crash, it just leaves very expensive silicon idle. Why Helm values files could not hold it, and what happened when I moved the whole thing into CUE constraints. https://karanvk.me/blog/terraform-argocd-boundary https://karanvk.me/blog/terraform-argocd-boundary Sat, 09 May 2026 00:00:00 GMT Two weeks into a lights-out GitOps rewrite for a GPU fleet, the hard question is not which tool to use, it is which tool owns what. The line I drew, the gray zone where every argument actually happens, and why the bootstrap handoff is a real design decision. https://karanvk.me/blog/tokio-tail-latency https://karanvk.me/blog/tokio-tail-latency Sat, 25 Apr 2026 00:00:00 GMT A retrospective on the gRPC gateway I ran at Sales Marker: the p99 budget that shaped every decision, the Tokio habits that held it, and the discovery that our worst tail latency lived in HTTP/2 flow control rather than in our code. https://karanvk.me/blog/the-node-that-lies https://karanvk.me/blog/the-node-that-lies Sat, 11 Apr 2026 00:00:00 GMT A GPU node with intermittent NVLink faults quietly returned wrong inference results for most of a day without ever failing a health check. This is the controller we built so it cannot happen twice: DCGM signals in, a policy engine deciding cordon versus drain versus recycle, checkpoint-aware draining, and the false positives that cost me eleven nodes. https://karanvk.me/blog/xid-field-guide https://karanvk.me/blog/xid-field-guide Sun, 22 Mar 2026 00:00:00 GMT What Xid errors actually are, which ones mean a GPU is about to die and which ones are just an application bug that wakes you up for nothing, how correctable and uncorrectable ECC differ, why row-remap-pending is the canary, and the dcgm-exporter metrics and alert rules I settled on. https://karanvk.me/blog/first-month-gpu-fleet https://karanvk.me/blog/first-month-gpu-fleet Sun, 08 Mar 2026 00:00:00 GMT Notes from five weeks on an accelerator fleet: why compute measured in megawatts breaks CPU-land instincts, what DCGM and Xid and MIG mean day to day, and the naive question I asked in week one that reframed the whole job. https://karanvk.me/blog/prometheus-karpenter-control-loop https://karanvk.me/blog/prometheus-karpenter-control-loop Sat, 21 Feb 2026 00:00:00 GMT CPU requests are fiction, so scaling on pending pods is scaling on fiction. Notes on a small controller that fed real Prometheus signals into Karpenter capacity, held 99.99% at 98% utilization, and flapped badly before it did. https://karanvk.me/blog/leaving-datadog https://karanvk.me/blog/leaving-datadog Sat, 07 Feb 2026 00:00:00 GMT The renewal quote that started it, the order we migrated in (Loki, then Mimir, then Tempo), everything that broke on the way, and the honest ledger with hindsight. https://karanvk.me/blog/2025-in-review https://karanvk.me/blog/2025-in-review Tue, 30 Dec 2025 00:00:00 GMT A year-ender written in an empty Tokyo office: the numbers that held, the two things I shipped that mattered, the failure I did not see coming, and what has been pulling at my curiosity all year. https://karanvk.me/blog/advent-of-code-leaderboard https://karanvk.me/blog/advent-of-code-leaderboard Sun, 21 Dec 2025 00:00:00 GMT Puzzles unlock at 2pm in Tokyo, which makes December a lunch-break sport. Notes on the rig, the day 15 run that finished second globally, and why unreadable code is fine for exactly one month a year. https://karanvk.me/blog/lgtm-one-year-ledger https://karanvk.me/blog/lgtm-one-year-ledger Sat, 08 Nov 2025 00:00:00 GMT A year of running Loki, Mimir and Tempo ourselves at over 2 TB an hour: what the infrastructure actually costs, how much upgrade toil is real, what it feels like to be on call for your own observability, and what I would do differently. https://karanvk.me/blog/interviewing-sres-tokyo https://karanvk.me/blog/interviewing-sres-tokyo Sat, 20 Sep 2025 00:00:00 GMT Two years on the hiring side of an SRE loop in Tokyo: the questions I dropped, the four I kept, and what a candidate correcting themselves twenty minutes late taught me about signal. https://karanvk.me/blog/slo-gated-deploys https://karanvk.me/blog/slo-gated-deploys Sat, 09 Aug 2025 00:00:00 GMT A single deploy ate a third of a monthly error budget in fifty minutes. So I wired multi-window burn-rate queries into the pipeline as a promotion gate, plus a watcher that reverts on fast burn. Honest notes on what it catches and where it is still duct tape. https://karanvk.me/blog/against-headroom https://karanvk.me/blog/against-headroom Sat, 28 Jun 2025 00:00:00 GMT Idle capacity is organizational anxiety, billed hourly. How priority classes, bin-packing discipline and a predictive capacity loop let us run a 2000 vCPU estate near full without trading away availability. https://karanvk.me/blog/rust-infrastructure-language https://karanvk.me/blog/rust-infrastructure-language Sat, 17 May 2025 00:00:00 GMT The stock objections to Rust in infrastructure, answered from three systems I actually shipped: a rate limiter, an eBPF agent on a robot fleet, and the gRPC gateway I run today. Including the places I still reach for Go and Python. https://karanvk.me/blog/l4-failover https://karanvk.me/blog/l4-failover Sat, 05 Apr 2025 00:00:00 GMT Why we put a layer-4 router under long-lived gRPC traffic instead of another L7 proxy, and why the routing was the easy part. Health scoring, split-brain protection, failback hysteresis, and the AZ brownout we found out about from a graph at standup. https://karanvk.me/blog/exporter-liability https://karanvk.me/blog/exporter-liability Sat, 22 Feb 2025 00:00:00 GMT An exporter is production code with a scrape target stapled to it, and most teams audit neither half. Notes from running a 50K samples/second metrics pipeline: the cardinality incident, drop rules that actually work, and the quality bar I ended up enforcing. https://karanvk.me/blog/competitive-programming-engineer https://karanvk.me/blog/competitive-programming-engineer Sat, 18 Jan 2025 00:00:00 GMT Contests gave me habits that broke production code and instincts that now save it. A story about a rate limiter I could not read, and what I kept after I stopped writing like a contestant. https://karanvk.me/blog/kubernetes-networking-services https://karanvk.me/blog/kubernetes-networking-services Sun, 08 Dec 2024 00:00:00 GMT A comprehensive guide to Kubernetes networking and all service types with detailed explanations, use cases, and architecture diagrams. https://karanvk.me/blog/react-state-management https://karanvk.me/blog/react-state-management Sun, 10 Mar 2024 00:00:00 GMT A comprehensive look at the evolution of state management in React applications, from Redux to React Server Components. https://karanvk.me/blog/rust https://karanvk.me/blog/rust Tue, 20 Feb 2024 00:00:00 GMT Why Rust is taking over the JavaScript tooling ecosystem, and what this means for web developers. https://karanvk.me/blog/style-guides-component-libraries-design-systems https://karanvk.me/blog/style-guides-component-libraries-design-systems Mon, 15 Jan 2024 00:00:00 GMT A deep dive into creating scalable design systems and component libraries that help teams build consistent, maintainable, and beautiful user interfaces. https://karanvk.me/blog/hello https://karanvk.me/blog/hello Thu, 23 Mar 2023 00:00:00 GMT Introducing the new blog - A journey into AI, ML, and software engineering.