# Every exporter is a liability > An exporter is production code with a scrape target stapled to it, and most teams audit neither half. Notes from running a 50K samples/second metrics pipeline: the cardinality incident, drop rules that actually work, and the quality bar I ended up enforcing. Author: Karan Vijayakumar Published: 2025-02-22 Tags: prometheus, observability, sre, rust Reading time: 9 min Canonical URL: https://karanvk.me/blog/exporter-liability --- The worst outage I ever helped cause was a label. Not a bad deploy. Not an expired certificate, not a schema migration gone sideways at 2am. A label. Somebody on a product team added `request_id` to a counter so they could line a metric up against a trace, a reasonable instinct served by completely the wrong mechanism, and forty minutes later the Prometheus instance scraping that service went from a comfortable four million active series to something north of nine. The head block stopped fitting in memory. The OOM killer did what it does. The replacement pod then spent twenty minutes replaying a write-ahead log it was going to outgrow the second it finished, so we got to watch the same death twice, with the dashboards everyone needed for the *actual* incident sitting blank the whole time. That was at FourKites, two jobs ago now, back when I owned the metrics pipeline. It's the incident I replay in my head every time someone asks me to add an exporter. Here's what I didn't understand when I started that job: an exporter is not monitoring. It's a piece of software running in production, on the hosts you care about, with a network listener open and a scrape target stapled to the side. It has a memory profile, a lock profile, a latency curve under load, a failure mode when its dependencies get slow, and a blast radius. Almost nobody reviews any of that. They review whether the metric name reads nicely in Grafana. I have approved exporters that would have failed a normal service code review inside five minutes, because it was "just metrics." ## The cardinality bill Everybody has heard the word cardinality. Very few have priced it. Every unique combination of label values is its own time series, with its own chunk in the head block, its own entry in the inverted index, its own memory floor. A counter with three labels at ten values each is 1,000 series and nobody notices. Add a fourth label with a hundred values and you're at 100,000, from one metric, from one service. The bill arrives late, too. You ship the label on a Tuesday and the instance dies three weeks later when retention catches up, long after anyone connects the two events. The `request_id` incident wasn't really a cardinality problem though. It was churn, which is the nastier cousin. A high-cardinality label with stable values (say, 50,000 customer IDs) is expensive but predictable: you can provision for it, the index settles, the series get reused. A request ID appears exactly once, ever. So every single scrape mints a fresh set of series and orphans the previous set, and the index grows without bound while your dashboards go blank because nothing has enough consecutive samples to draw a line through. Volume you can buy your way out of. Churn eats the index and there's nothing to buy. The first thing I run when a Prometheus starts breathing hard is still `/api/v1/status/tsdb`, which hands you total head series plus the top offenders by series count without paying for a query. After that, three PromQL expressions: ```promql # Top 20 metric names by active series. Expensive: it touches everything, # so don't put it on a dashboard, run it by hand. topk(20, count by (__name__) ({__name__=~".+"})) # Which targets are actually feeding the beast, counted after relabeling. topk(20, scrape_samples_post_metric_relabeling) # Churn, which hurts more than volume: new series minted per scrape. topk(20, scrape_series_added) ``` That third one is the query I wish I'd known about a month earlier. `scrape_series_added` would have pointed at the offending job within a minute, because a normal service adds approximately zero new series per scrape once it's warm, and a service leaking UUIDs into labels adds thousands, forever. ## Drop it at the scrape, not in the query The instinct after an incident like that is to write a wiki page telling people not to do the bad thing. We wrote it. It changed nothing, because the person who ships the next bad label has not read your wiki, has never heard of you, and is under deadline. What worked was making the pipeline refuse the data. Prometheus gives you two relabeling stages and the distinction matters: `relabel_configs` runs against discovered *targets* before a scrape happens, and `metric_relabel_configs` runs against *samples* after the scrape and before ingestion. Cardinality control lives in the second one. ```yaml title="prometheus.yml" scrape_configs: - job_name: shipment-api scrape_interval: 30s scrape_timeout: 10s sample_limit: 20000 metric_relabel_configs: # A histogram nobody has queried in six months. Gone before the TSDB sees it. - source_labels: [__name__] regex: 'shipment_internal_queue_latency_seconds_bucket' action: drop # Any label whose *name* smells like an identifier gets removed. # Note: labeldrop matches label names, so there is no source_labels here. - regex: '(request_id|trace_id|session_id|user_email)' action: labeldrop # Path is fine at low arity and unbounded the moment someone forgets # to template it. Rewrite rather than drop, so the series survives. - source_labels: [path] regex: '/v1/shipments/[0-9a-f-]{36}(/.*)?' target_label: path replacement: '/v1/shipments/:id' action: replace ``` `sample_limit` is the seatbelt, and it deserves a warning label. It's evaluated post metric relabeling, and when a scrape exceeds it Prometheus rejects the *entire* scrape and drops `up` to 0 for that target. I learned this by setting a limit too tight on a job that had legitimately grown, which converted a slow cardinality problem into an immediate total-blindness problem for that service. Worse trade. So: set it generously above observed volume, and alert on `prometheus_target_scrapes_exceeded_sample_limit_total` increasing rather than waiting to notice the gap in a graph. The limit is there to stop a catastrophe, not to enforce hygiene. ## Recording rules, and the hourly rule The rule I eventually wrote down: if a query appears on a dashboard that anyone looks at more than once an hour, it becomes a recording rule. The reasoning is dull and correct. A dashboard panel re-runs its query per viewer per refresh, and during an incident that means fifteen people hammering the same `histogram_quantile` over a five minute range across a few hundred thousand series, at the exact moment Prometheus is least able to spare the cycles. A recording rule runs it once per evaluation interval, writes a tiny series, and all fifteen get an instant answer off a single-series read. ```yaml title="rules/shipment-api.yml" groups: - name: shipment-api-slo interval: 30s rules: - record: job:shipment_api_requests:rate5m expr: sum by (job, code) (rate(shipment_api_requests_total[5m])) - record: job:shipment_api_request_duration:p99_5m expr: | histogram_quantile( 0.99, sum by (job, le) (rate(shipment_api_request_duration_seconds_bucket[5m])) ) ``` The `level:metric:operations` naming convention looks fussy until you have three hundred of these, at which point it's the only thing keeping them navigable. Also, and this took an embarrassing number of repeats to internalize: aggregate with `sum by (job, le)` before `histogram_quantile`, not after. Quantiles don't average. ## Budgets, because taste doesn't scale The structural fix was a cardinality budget per team. Series count attributed by an owner label, a number each team got, and a monthly report showing usage against it. Going over didn't trigger an outage or a block, it triggered a conversation with me. It's unglamorous social machinery and the highest-leverage thing I built that year, because it converts an argument about taste into an argument about a number. "That label is too high cardinality" is a matter of opinion and I will lose that argument to a senior engineer with a deadline. "That label costs 400,000 series and you have 250,000 left" is arithmetic, and it puts the trade-off where it belongs, with the team that wants the metric. ## The quality bar I ended up enforcing Three questions, asked of every new exporter, in a PR template. **What does `/metrics` do under load?** A distressing number of exporters build the entire response in memory, holding a global mutex the whole time. That's tolerable at 200 series and a disaster at 200,000, and the disaster shows up as scrape latency creeping toward the interval until Prometheus starts timing out on a target that is perfectly healthy. **What happens on timeout?** `scrape_timeout` has to be shorter than `scrape_interval`, but the interesting case is what the exporter does when Prometheus gives up. Many of them keep generating. So you get an exporter burning CPU producing a response nobody will read, every interval, forever, and no data in the TSDB. Blind and expensive at the same time. **Does it tell the truth about freshness?** This is the one that scared me most. An exporter that polls a backend, caches the last known values, and keeps serving them when the backend goes away looks *identical to a healthy system*. Prometheus has staleness handling for targets that disappear, and an exporter that lies about freshness defeats it completely. If you cache, publish the age of the cache as its own gauge, and alert on the age. ## Why the hot ones ended up in Rust Two exporters in that pipeline carried most of the weight, sustaining somewhere over 50,000 samples per second between them. Both started life in a garbage-collected language and both had the same signature failure: scrape latency looked fine at p50, and p99 had a fat shelf where a collection cycle happened to land during response encoding. Which is exactly the moment allocation pressure is highest, because you're building a large response. Rewriting them in Rust wasn't a language argument, it was a tail latency argument. No GC pauses, memory inside a predictable envelope under load so you can set a real container limit, and an encode path that's essentially a buffer you write bytes into. The p99 shelf disappeared and stayed gone. Part of the motivation was that I wanted to write Rust and this was a defensible excuse. It happened to also be the right call. ## Sometimes the answer is a log line The thing I'd say to the engineer who added `request_id`, if I could go back and be less annoyed about it: you were asking a legitimate question with the wrong tool. A metric answers "how much" and "how fast." It is a counter or a distribution, aggregated across a population, and its whole efficiency comes from *not* keeping the individuals. A log line answers "which one." That's what it's for. Wanting to find the specific slow request is normal and important, and the answer is a structured log with the request ID in it, or a trace with a proper sampling policy, and then an exemplar linking the histogram bucket to the trace if you want the metric and the individual to shake hands. Reaching for a label to answer a "which one" question is the most common way people accidentally destroy their own monitoring, and it happens because metrics feel cheaper than logs. They're not cheaper. They're cheap in a specific shape and unboundedly expensive outside it. ## Where this leaves me I still care about metrics. I've built more of this stuff since. But the framing I carry now, two years on, is that adding an exporter isn't adding visibility, it's adding a dependency. It gets the same review as a service: resource limits, an owner, a cardinality estimate the author has to actually compute, and a documented answer to what happens when it's slow. You don't get to run less code by calling it observability. The pipeline that watches your system is part of your system, and it will take you down just as thoroughly as anything you're proud of.