Self-hosting LGTM, one year in: the honest ledger

We finished migrating off Datadog around this time last year. Logs into Loki first, then metrics into Mimir, traces into Tempo last, and by late autumn 2024 the Datadog agents were gone from every node.

A year is roughly the shortest interval where I trust my own opinion about infrastructure. Six months in, you’re still enjoying the parts you built. Twelve months in, you’ve upgraded it twice, been paged by it, watched it behave badly in a way that wasn’t in any runbook, and had to explain a bill to someone. So here’s the ledger, as honestly as I can write it while still working here.

Scale, for context, because none of these numbers mean anything without it: a hybrid Kubernetes estate at Sales Marker, a bit over 2000 vCPUs across cloud and our own hardware, ingesting north of two terabytes an hour across logs, metrics and traces.

The infrastructure bill

Low six figures a year, USD, all in. Against a Datadog renewal quote that was seven figures.

That gap is the headline and it’s real, but the composition of it is more interesting than the total. Object storage is genuinely cheap and it’s the smallest surprise on the list. What actually costs money is the query path: store-gateways and queriers holding index and cache in memory, and the compactor, which is the component I’ve thought about more than any other this year and which I’ll get to.

Two things flatter our number and I want them on the record. We own hardware, so a meaningful share of the compute is capex we’d already committed to, and self-hosted S3-compatible storage on our own disks costs a fraction of cloud object storage per terabyte. And we’re in one region with no cross-region replication of observability data, which is a deliberate accepted risk rather than a design achievement. If we lose the region we lose our telemetry along with everything we’d want it for. I’ve raised it twice. It’s still on the list.

A company doing this on pure cloud with cross-region durability should expect a materially worse ratio. Still probably good at our volume. Not an order of magnitude.

Upgrade toil, which is the line item nobody quotes you

This is where the ledger stops being flattering.

Loki is not a helm upgrade and a coffee. Over the past year I’ve done four Loki version bumps and exactly one of them was uneventful. The 3.x line has been good software and it has also had real breakage in it. Structured metadata being on by default requires both the TSDB index and the v13 schema, and Loki simply will not start if you’re on an older schema, which is a correct design decision and also a Saturday if you find out from a crashlooping pod. shared_store disappeared as a config key. A default max line size of 256KB started rejecting log lines from a service that logged entire API payloads, and I got to be the person explaining to that team that the truncation was, in fact, an improvement.

None of these are unreasonable. Every one of them is documented in the upgrade notes, and I’ve read those notes more carefully with each release. The point is that the notes exist because there is something in them, and reading them properly, in a staging environment, with a rollback plan, is a day of work per upgrade minimum. Times three components. Mimir and Tempo have both been calmer than Loki, but “calmer” is not “free.”

Call it fifteen to twenty engineer-days a year on version maintenance alone, and that’s with the discipline to stay current. Falling behind is worse, because upgrade paths compound and skipping schema migrations turns into a project.

The compactor

I’ll give this its own section because it’s the single thing that taught me the most.

At two terabytes an hour, retention is not a config value, it’s a workload. Loki’s compactor handles both index compaction and retention deletion, and at our ingest rate it needs to keep up continuously, forever, with no slack. When it falls behind, the failure is quiet. Nothing pages. Nothing errors. Your index just gets slowly worse, queries that took two seconds start taking nine, and object storage keeps growing past the retention you believe you configured.

We found ours three weeks behind. Three weeks. The symptom that finally got attention was a storage bill that didn’t match the retention policy, which is an embarrassing way to discover a broken component.

The fixes were unremarkable: give it more resources than you think it needs, run it on fast local disk rather than network storage, and most importantly alert on compaction lag directly rather than inferring health from whether the pod is running. I now watch loki_boltdb_shipper_compact_tables_operation_duration_seconds and the timestamp of the last successful compaction, and if that timestamp is older than a couple of hours I want to know about it while I’m awake.

The general lesson generalises past Loki: in any system where a background maintenance process is load-bearing, the maintenance process needs its own SLO. Components that fail loudly are a luxury. It’s the quiet ones that cost you.

Being on call for your own observability

I knew this would be uncomfortable. I underestimated the specific shape of the discomfort.

It isn’t that the stack goes down. It’s been up. The uncomfortable part is the ordering problem during an incident. When something is wrong, your first instinct is to look, and your second thought, now, is whether the thing you’re looking at is telling you the truth. That doubt costs seconds at exactly the moment seconds matter. With Datadog, if the dashboard was weird it was their problem and I moved on. Now “the dashboard is weird” is a hypothesis I have to eliminate.

We’ve had two incidents this year where telemetry was degraded at the same time as the thing it was measuring, both times because they shared an underlying dependency. That’s a category of failure a SaaS vendor gives you for free, and I want to be honest that we bought it back at full price.

Two mitigations that have earned their keep. We run a small, boring, completely separate Prometheus that scrapes the LGTM components themselves and alerts through a different path, so the thing that tells us the observability stack is sick is not the observability stack. And we keep a short list of things you can check without Grafana at all, which sounds primitive and has been used in anger twice.

The freedom, which is the actual payoff

Here’s what I didn’t properly appreciate before we did this, and it’s the reason I’d make the same call again.

Under Datadog, cardinality had a price per time series and query volume had a bill attached, so both were rationed. Not by policy, by instinct. Engineers learned not to add labels. I learned not to run the expensive query. That rationing was invisible and it was making our instrumentation worse every quarter, because the cheapest metric is the one that answers the least.

Now a team can add a label because the label is useful. I can run a query across three months of logs at 4am to test a theory that’s probably wrong, and the only cost is queriers doing work they’re already provisioned for. Trying a hypothesis went from a thing with a price to a thing you just do, and the number of hypotheses tested per incident went up in a way I can feel even if I can’t cleanly measure it.

The freedom is not unlimited and that’s fine. Cardinality still costs ingester memory, so per-tenant series limits exist, each tenant has a named owner, and the page goes to them rather than to me. Mimir’s cardinality API means I can answer “which label is doing this” in about a minute instead of guessing. The constraint moved from a vendor’s price list to physics we can see, which is a much better place for a constraint to live.

What I’d do differently

Get the schemas right on day one. TSDB and v13 from the first byte, in every component that has an equivalent decision. Retrofitting a schema on live data means keeping the old index period around for as long as its retention lasts, and we’re still carrying one.

Alert on the pipeline before the pipeline carries anything. Compaction lag, ingester append failures, distributor rejections, queue depth on every hop. We built alerts for our services on this stack for months before we built proper alerts for the stack itself. Backwards.

Budget the alert rewrite as a project. Translating a few hundred Datadog monitors into Prometheus rules was the most valuable month of the migration, because roughly a third of those monitors turned out to be dead, but I scheduled it as a task inside a larger task and it deserved its own name and its own timeline.

Take Alloy earlier. We started on Promtail because Alloy was new and I didn’t want an unproven collector in a migration I was already nervous about. That was defensible at the time. When Promtail went to long-term support this February we moved, and the migration was so uneventful that I think my caution cost us more in split documentation than it saved in risk.

The verdict

At two terabytes an hour, self-hosting LGTM is clearly the right call and I’d do it again without much deliberation.

The number I’d want a reader to take away isn’t the cost ratio, it’s the standing commitment. Roughly a quarter of an engineer, permanently, on top of the four-month two-person migration. That’s upgrades, capacity planning, being paged by your own instruments, and the ongoing work of being the team that owns three distributed databases nobody at the company thinks of as databases.

If that quarter of an engineer is the difference between shipping your roadmap and not, buy the SaaS. It’s good software and the people who wrote it are better at running it than you will be for at least a year. Below roughly a quarter million a year in spend, the arithmetic doesn’t work no matter how much you’d enjoy the project.

Above it, at some point the vendor stops selling you a tool and starts selling you a tax on your own data volume. That’s the moment the ledger flips, and the only way to know where your moment is, is to actually do the accounting instead of the vibes.