eBPF on a robot fleet at under 1% CPU

The requirement I was handed was one sentence long: don’t make the robots worse.

That was 2024, at Rapyuta Robotics, a couple of jobs ago now. I was on the platform side, which mostly meant a GKE control plane and the pipelines around it, but the interesting problem was down on the devices. We had a fleet of warehouse robots and essentially no visibility into what their operating systems were doing. When a unit started behaving oddly, and they did, the diagnosis was somebody physically walking over to it with a laptop. On a good day.

What came out of that was a Rust agent built on Aya, capturing kernel scheduling latency, block I/O, and socket behavior across the fleet, inside a CPU budget of one percent. Two years of distance has clarified which parts were good ideas and which parts I got away with.

Robots are the worst observability target I’ve had

I’ve since worked on estates of a couple of thousand vCPUs and I’m currently on a GPU fleet measured in megawatts, and neither is as hostile to instrumentation as a warehouse robot.

The CPU is shared with control loops that have real deadlines. Not soft deadlines in the SRE sense where a slow response annoys someone. A control loop that misses its cycle produces a robot that stops in an aisle, and in the bad case a robot that stops in an aisle while carrying something. So the usual observability posture, where an agent takes what it needs and you tune it later if someone complains, is unavailable. The complaint arrives as a floor incident.

The uplink is flaky and metered. Robots roam, warehouse WiFi coverage is a physics problem with shelving in the way, and the fallback link had a bill attached. Raw logs or per-event traces were never on the table. Even per-second metrics for a few hundred series per device add up to a number somebody notices.

The kernels were a zoo. Multiple hardware generations, each with whatever kernel the vendor’s board support package shipped, ranging from a genuinely elderly 4.x through several 5.x. Nobody was going to let me upgrade kernels on a working fleet to make my metrics nicer, and they were right not to.

And there was no development toolchain on the devices. A few gigabytes of eMMC, an image close enough to immutable that adding to it was a project, and no Python. That constraint shaped the design more than any of the others.

What the agent actually was

One statically linked binary, musl, about 4 MB, running as a systemd unit inside a cgroup with a hard CPU ceiling.

The kernel side used tracepoints where they existed and kprobes where they didn’t. Scheduler runqueue latency came from sched:sched_wakeup and sched:sched_switch: stash a timestamp when a task becomes runnable, take the delta when it’s actually scheduled. That number turned out to be the most valuable thing we collected, because it directly measures “is something else stealing time from the control loop,” which was the question we always actually had. Block I/O latency came from block:block_rq_issue and block:block_rq_complete, and mattered more than I expected because two failing eMMC modules announced themselves in the disk tail long before anywhere else. Socket retransmits came from tcp:tcp_retransmit_skb, which is how we learned one warehouse’s WiFi was much worse in one specific aisle.

The important architectural decision was that nothing left the device as an event. Everything was folded into log2 histograms in kernel maps, read on an interval, and shipped as a pre-aggregated payload of a few hundred bytes per metric. The kernel side looks roughly like this, and this is close to the real thing:

use aya_ebpf::{
    helpers::bpf_ktime_get_ns,
    macros::{map, tracepoint},
    maps::{HashMap, PerCpuArray},
    programs::TracePointContext,
};
 
const NEXT_PID_OFFSET: usize = 44;
const SLOTS: u32 = 27;
 
#[map]
static WAKEUP_TS: HashMap<u32, u64> = HashMap::with_max_entries(10_240, 0);
 
#[map]
static RUNQ_LAT: PerCpuArray<u64> = PerCpuArray::with_max_entries(SLOTS, 0);
 
#[tracepoint(category = "sched", name = "sched_switch")]
pub fn on_switch(ctx: TracePointContext) -> u32 {
    let Ok(pid) = (unsafe { ctx.read_at::<u32>(NEXT_PID_OFFSET) }) else {
        return 0;
    };
    let Some(&queued_at) = (unsafe { WAKEUP_TS.get(&pid) }) else {
        return 0;
    };
    // log2_slot() is a six line leading-zeros bucket, omitted here.
    let delta_us = (unsafe { bpf_ktime_get_ns() } - queued_at) / 1_000;
    if let Some(slot) = RUNQ_LAT.get_ptr_mut(log2_slot(delta_us)) {
        unsafe { *slot += 1 };
    }
    let _ = WAKEUP_TS.remove(&pid);
    0
}

That NEXT_PID_OFFSET is the part I’d get letters about, deservedly. A hardcoded byte offset into a tracepoint’s format record is exactly the portability trap CO-RE exists to eliminate. We resolved it properly in the end, from the tracepoint format description at load time, passed in as global data. I’m showing the const because it’s what shipped to the first bench robot, and pretending otherwise would make this post less useful.

CO-RE is why one binary worked across that kernel zoo at all. Aya emits BTF relocations, so where the target kernel has BTF, field offsets get fixed up at load time rather than compile time. Combined with musl static linking that gives you the property which made the project possible: build once on a laptop, copy one file onto any robot, and it works. The oldest kernel had no BTF at all and we carried a generated blob for it, a footnote that consumed about a week of my life.

Why Aya and not bcc or bpftrace

This is usually where somebody points out that bcc has had a runqueue latency tool for years and asks why I wrote one.

Because bcc wants Python, clang, and kernel headers on the device at runtime. It compiles your program when you run it. On a machine with a few gigabytes of storage and an image change process that involved a meeting, that isn’t a tradeoff, it’s a wall. bpftrace is better on that axis, one binary and no Python, but it’s a language for interactive investigation at a shell prompt. What I needed was a daemon: attach on boot, aggregate on an interval, buffer when the uplink is down, back off, retry, expose its own health, respond to a remote kill switch. That’s a program, not a one-liner, and wrapping bpftrace in a supervisor to fake it is the kind of decision you regret in month four.

libbpf and C is the mainstream answer and it would have worked. What Aya bought was that the kernel side and the userspace side lived in one cargo workspace, in one language, sharing type definitions for the map values so a struct layout change couldn’t silently disagree across the boundary. One cargo build, one artifact, one cross-compile target per board. For a team of not very many people that was the whole argument.

The honest counterpoint is that Aya in 2024 was rougher than libbpf. Fewer examples, verifier errors that took real experience to decode, a couple of map types that weren’t there yet. The ecosystem has moved a long way since, but I was not on a paved road, and roughly a fifth of the project’s time went into problems a C implementation would not have had.

The one percent was a test, not a hope

The budget was the requirement, so it had to be enforced by something that could fail a build.

We ran it on real hardware. One robot on a bench in the lab plus a matching single board computer, both wired into CI. The job ran a synthetic control loop, a fixed-period task doing a fixed amount of work, and measured two things over ten minutes: the agent’s own CPU time, read as a delta from its cgroup’s cpu.stat rather than from anything resembling top, and the p999 cycle jitter of the loop. The build failed if agent CPU exceeded one percent of a core, or if jitter regressed past a threshold baselined with the agent absent entirely.

That second check is the one that mattered, and I nearly didn’t write it. CPU percentage is a proxy. What the robotics team cared about was whether their loop ran on time, and it’s entirely possible to consume very little CPU while still ruining someone’s timing, by holding a lock or thrashing a cache or generating interrupts. Measuring the thing they cared about directly meant the conversation was over before it started, which is worth more than any number of green dashboards.

Belt and braces on top: the userspace half ran at low scheduling priority, and the cgroup had a hard cpu.max ceiling. If the agent ever tried to win an argument with a control loop, the kernel settled it, and I’d rather lose metrics than lose a cycle. That also made sampling rate a safe knob, because the worst case of turning it too far up was throttling rather than a floor incident.

The map that filled up and made everything look wonderful

Look at WAKEUP_TS in that snippet again. Entries go in on wakeup, come out on switch. It’s a balanced ledger, and it’s balanced only if both sides of it are running.

On one hardware generation with the oldest kernel in the fleet, they weren’t. The sched_switch program failed to attach, for BTF reasons that took me a while to untangle, and the agent logged the error and carried on starting up, because past me thought a partial agent was better than no agent. So on those units, wakeups were recorded forever and switches never removed them. The map has 10,240 slots, so nothing crashed and nothing ran out of memory. It filled, and then every insert started failing with a return code nobody was checking, and the only entries left in the map were the stale ones already there.

The metric didn’t break. It got beautiful. Runqueue latency on those robots looked better than anywhere else in the fleet, because the handful of surviving measurements were whatever happened to make it through, and I remember looking at that dashboard and feeling briefly good about our scheduler tuning.

What caught it was a colleague asking why six specific robots were so much healthier than the rest. Not an alert. A person being suspicious of good news, which is a skill I now try to hire for.

An observability agent that lies is worse than no agent, because you will act on it.

The fixes were all things that should have been there from the start. The agent now refuses to start unless every expected program attaches, loudly, rather than degrading quietly. WAKEUP_TS became an LRU hash so unbounded growth evicts instead of jamming, with a periodic userspace sweep as a second line. Every map got an occupancy gauge and every failing helper call got a counter, shipped in the same tiny payload as the real metrics. And we added a check for series that should never be zero and distributions that should never be that clean, because “suspiciously good” is a failure mode and nothing in our alerting could express it.

What I’d change now

Self-instrumentation first, before any real metric. The agent should have been able to describe its own health on day one, and adding it after an incident meant we’d been flying partly blind for months without knowing it.

More tracepoints, fewer kprobes. We learned this twice, which is once more than necessary. A kprobe on an internal kernel function is a bet on that function’s name and signature staying put, and across a five-year span of kernel versions that bet loses. Every kprobe we replaced with a tracepoint made the fleet quieter.

More aggregation in the kernel, less in userspace. A few paths shipped events up to be folded into histograms in userspace, and every one got cheaper once the folding moved into the eBPF program. Doing the arithmetic where the data already is turns out to be the whole point, and I only half believed that at the start.

And I’d ask harder whether we needed to build it. In 2024 I don’t think there was an off-the-shelf answer that fit a metered uplink and a one percent budget on an ancient kernel with no toolchain, and I’d probably still write it today. But I’d spend a week proving that instead of a day assuming it.

The one thing I’d change nothing about is the kill switch. We built a remote off button before we built the second metric, and over the following months it got used more than any other feature, mostly by robotics engineers during unrelated debugging who wanted to eliminate us as a variable. Every time they flipped it and the problem stayed, our credibility went up. That turned out to be the thing that let the agent stay on the fleet at all.