Leaving Datadog at 2 TB/hour: the bill that made us build our own

The renewal quote showed up as a PDF on a Thursday afternoon in June 2024. I remember the day because I was eating a konbini onigiri at my desk and I put it down.

Some context. I’d just come into Sales Marker as a senior SRE. The estate was a hybrid Kubernetes setup, a bit over 2000 vCPUs split between cloud and hardware we owned, and we were shipping somewhere north of two terabytes an hour of logs, metrics and traces into Datadog. The number at the bottom of that PDF was seven figures a year, in USD. Not high seven figures. But seven. For a company our size that wasn’t a line item anymore, it was a strategy question.

I left that job a few weeks ago, so I can finally write the whole thing down without hedging. What follows is the migration off Datadog and onto a self-hosted Grafana stack, in the order it actually happened, including the parts where I was wrong.

What Datadog was genuinely good at

I want to get this out of the way first, because most posts in this genre are revenge pieces and they’re all slightly dishonest.

Datadog was very good. Not “fine.” Good. Somebody could join the company on a Monday, be handed a laptop, and be usefully reading a service map by Wednesday. The correlation between a trace, the logs from that request, and the host metrics underneath it worked without anyone thinking about it, which sounds small until you’ve spent an afternoon manually lining up timestamps across three systems you built yourself. Their agent auto-discovered integrations we didn’t know we had. Log pipelines with grok parsing in a UI meant a backend engineer could reshape their own log format at 2am without filing a ticket against my team.

And critically: none of it was my pager. When their ingest lagged, someone in New York got woken up and I got a status page. I signed away that privilege voluntarily and I want to be clear-eyed that it was a privilege.

So the decision wasn’t “Datadog is bad.” It was that we were paying a premium tuned for companies with a tenth of our data volume, and the premium scaled with the thing we were least willing to reduce.

The shape of the bill

Datadog bills on independent axes that all move at once. Per host for infrastructure. Per host again for APM. Per GB to ingest logs, then again per million events to index them, with the price stepping up by retention window. Custom metrics get a per-host allotment and everything past it is priced per time series per month.

That last one is the trap for anyone running Prometheus-flavoured instrumentation. A single histogram with a handful of labels is not one custom metric. It’s buckets times label cardinality, and the engineer who added a customer_id label to a request counter in good faith on a Tuesday costs you real money by Friday. We had a Slack channel for this. It was mostly me being unpleasant about labels.

The uncomfortable part when I actually modelled it: our biggest single spend was on log data that nobody had queried in ninety days. We were paying indexing prices for insurance. That’s a legitimate thing to pay for. It just isn’t worth what we were paying.

Why we thought we could do it

I’d run Prometheus and its exporters at serious volume at FourKites before this, on the order of 50 thousand samples a second, and I’d done GKE and Argo CD work at Rapyuta. So the stack wasn’t unfamiliar. That’s not the same as knowing what two terabytes an hour does to a compactor, and I’d like the record to reflect that I did not know that yet.

The pitch to leadership was deliberately unglamorous. Object storage is cheap and S3-compatible storage on our own hardware is cheaper. Loki, Mimir and Tempo all separate compute from storage, so ingest cost and query cost stop being the same number. And we get to stop rationing cardinality, which was quietly making our instrumentation worse every quarter.

The pitch I did not make, but should have: this converts a predictable vendor bill into an unpredictable engineering commitment, and you are trading dollars for my team’s attention.

The order we did it in, and why the order mattered

Logs first, then metrics, then traces. If you take one thing from this post, take the ordering.

Loki first, because logs were the biggest number on the bill and the easiest thing to run in parallel. Log shipping is fan-out by nature. You point a second sink at the same stream and nobody downstream notices. We ran double-shipping into Loki and Datadog for about six weeks, which meant we could compare query results on real production data before anything depended on it.

We started on Promtail, because Alloy had only just been announced and I didn’t want a two-month-old collector in the critical path of a migration I was already nervous about. We moved to Alloy the following year when Promtail went into long-term support, and that migration was genuinely uneventful, which is not a sentence I get to write often.

The one thing I’d tell anyone starting here is to get your schema right before you have data, because fixing it afterwards means living with two index periods forever:

# loki: tsdb + v13 from day one. structured metadata needs both,
# and retrofitting a schema on live data means you keep the old
# period around for as long as its retention lasts.
schema_config:
  configs:
    - from: 2024-07-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

Mimir second. Metrics are harder than logs in one specific way: alerts. Log queries are mostly interactive, so a slightly different result is an annoyance. An alert that doesn’t fire is an outage nobody knows about. We had a few hundred Datadog monitors and every single one had to be rewritten as a Prometheus rule, because Datadog’s query language and PromQL don’t map onto each other cleanly. anomalies() has no PromQL equivalent and never will. Neither does a monitor whose real logic lived in a UI dropdown labelled “evaluate over the last 15 minutes.”

That rewrite took two of us most of a month and it was the single most valuable part of the whole migration, for a reason I didn’t expect. Roughly a third of those monitors were dead. Alerting on things that didn’t exist anymore, routing to people who’d left, or so noisy that the on-call rotation had trained itself to ignore them. You cannot find that out by reading a list. You find it out by being forced to explain, one monitor at a time, what condition it’s supposed to detect.

Tempo last, on purpose. Traces were where Datadog’s product lead was widest, and I wanted the team’s confidence banked from two successful migrations before I took away the service map. TraceQL is good and I like it more than I expected to, but the honest comparison isn’t the query language, it’s the fifty small conveniences around it. We lost some of those. Some we rebuilt. Some we decided we didn’t need, which was occasionally true and occasionally us coping.

What broke

Dashboard parity was a trap. My original plan was to reproduce every existing Datadog dashboard in Grafana. This was stupid and I want to say clearly why. Half of those dashboards were nobody’s. Built for one incident in 2022, never opened since, kept because deleting things feels risky. Rebuilding them faithfully meant importing that debt into a system we hadn’t finished learning yet. We stopped after the twenty or so that people actually opened during incidents, and rebuilt the rest on request. Very few requests came.

Cardinality surprised us from the opposite direction. I’d budgeted a lot of worry for series explosion, and we did hit it, but the real problem was that we’d removed the price signal. Under Datadog, adding a label had a dollar cost that showed up on a bill someone would ask about. Under Mimir it had an ingester memory cost that showed up as a pod restart at 3am with no obvious author. Freedom is great. Freedom without a feedback loop is how you end up with an ingester carrying a million series for one team’s debug label. We ended up doing what we should have done first, which is drop the labels we know are worthless at the collector:

# scrape config: kill the known offenders before they reach the ingester.
metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'go_memstats_.*|promhttp_metric_handler_.*'
    action: drop
  - regex: 'pod_template_hash|controller_revision_hash'
    action: labeldrop

Plus a per-tenant series limit with an actual owner attached to each tenant, so the page goes to whoever caused it. Limits without attribution just move the pain to whoever is on call.

Muscle memory was the slowest thing to fix. This is the cost nobody models. Engineers had years of Datadog reflexes. They knew where the button was. For about two months after each cutover, “I can’t find X” tickets were a meaningful fraction of my week, and most of them weren’t missing features, they were missing habits. I ran a Friday afternoon office hour and screen-shared PromQL and TraceQL until people stopped needing me. Budget for that. It’s real engineering time and it looks like nothing on a plan.

The compactor. I’ll spare you the details here because I wrote them up separately, but the short version is that Loki’s compactor at our ingest rate was the single component that taught me the most about my own overconfidence.

The ledger

Rough numbers, because the precise ones aren’t mine to publish.

Against a seven-figure annual quote we landed at low six figures of annual infrastructure, most of it storage and the compute for query paths, helped considerably by owning hardware. So on paper, an order of magnitude.

On paper. The rest of the ledger: roughly two engineers for four months to migrate, then a standing ongoing cost of something like a quarter of an engineer forever, for upgrades, capacity planning, and being the person paged when the thing that tells you about outages is the outage. Loki upgrades in particular are not a helm upgrade and a coffee. Add the alert rewrite month, add my Friday office hours, add the on-call load of a stack that has genuinely woken me up.

Still hugely net positive at our volume. I’d do it again at that volume. I wrote the fuller accounting a year in, with the parts that only show up after four seasons of operating it, in the one-year ledger.

But the honest way to say what we did is this: we didn’t save money on observability, we swapped a vendor bill for a small database company that we now run on the side.

When you should not do this

I’ve had this conversation maybe a dozen times since, and my answer keeps getting more boring.

If your bill is growing faster than your headcount and the growth is coming from data volume rather than host count, start modelling it now, because that curve doesn’t bend on its own.

And if you do it, do it because you have a volume problem, not because self-hosting feels more serious. It isn’t more serious. It’s just yours.

The thing I keep coming back to is that the migration’s biggest win wasn’t the money. It was that a third of our alerts turned out to be fiction and we only found out because something forced us to read them. You can get that win for free, this afternoon, on whatever stack you’re already paying for.