Where Terraform ends and Argo CD begins

Two weeks into this rewrite and I’ve written surprisingly little Terraform. What I’ve done is argue.

Not about Terraform versus Argo CD. Nobody sane argues that anymore, they’re not competing for the same job. The arguments are all about the seam. Every design review this fortnight has bottomed out in the same question, phrased differently each time: which of these two systems owns this object, and what happens the first time they disagree about it?

Some context. I’m building a lights-out GitOps setup for a GPU fleet, roughly 10 MW of it. Terraform owns the AWS estate. Argo CD reconciles every rollout. The ambition, still very much an ambition, is that an SLO gate decides whether a deploy stays or gets reverted with nobody watching. That last piece is barely started. I’m writing this from the middle of the thing, which is the only moment you still remember which parts were hard.

This is my third Argo CD estate. Rapyuta Robotics was the first, GKE, 2024, and I got this boundary wrong there in a way that took months to unpick. Sales Marker was the second, a hybrid setup north of 2000 vCPUs. So the opinions below have mileage on them. They aren’t principles I reasoned my way to. They’re scars I eventually got around to labeling.

The half nobody argues about

The easy version of the line is lifecycle duration.

If a thing’s lifecycle is measured in months, Terraform owns it. VPCs and their subnets and route tables. The EKS control planes. IAM roles and policies. S3 buckets, KMS keys, the Route 53 zones, the RDS instance holding fleet inventory. These are objects you create once, touch a handful of times a year, and want a human to read a diff before changing. They also have real dependency graphs across AWS services, which is exactly what Terraform’s graph engine is for, and they have no controller in the cluster that could possibly reconcile them.

If a thing has to continuously converge, Argo CD owns it. Deployments, the DCGM health check daemonset, the degraded-node controller, operators, ConfigMaps, network policies, every per-service config knob. These change weekly or hourly. They drift, because pods get evicted and nodes get replaced and somebody runs a kubectl edit at 2am under pressure. You want a controller noticing that and putting it back, forever, without anybody asking it to.

The compressed version I keep saying out loud in reviews: if it needs a plan, Terraform. If it needs to converge, Argo.

A plan is a promise about one moment. A controller is a promise about every moment after it.

Why I stopped letting Terraform hold manifests

The Terraform kubernetes provider will happily manage Kubernetes objects. kubernetes_manifest takes arbitrary YAML. helm_release installs charts. It works, in the demo sense of working, and every estate I’ve walked into had some of it.

It’s state-drift hell and I’d rather not do it again.

Start with the plan-time problem, because it’s the one that bites first and hardest. kubernetes_manifest needs API access during plan, not just apply, because it resolves your YAML against the cluster’s OpenAPI schema to figure out the GroupVersionResource. So the cluster has to already exist and be reachable when you type terraform plan. Which means you cannot create a cluster and put objects on it in the same apply. The documented workaround is targeted applies, cluster first, manifests second, and once you’re doing targeted applies as routine practice you’ve given up the one property that made Terraform worth the ceremony.

Then there’s the drift itself, which is the part that grinds. Kubernetes mutates objects after you submit them. Admission webhooks inject sidecars and fields. Defaulting fills in things you didn’t set. An HPA rewrites the replica count on the Deployment you declared. Terraform’s model is that state file plus config equals reality, so every one of those mutations reads as unexplained drift, and you get a plan that wants to undo them. Every time. Forever. The genuinely dangerous version is the HPA one, where an apply run for an unrelated reason quietly scales a service back down to whatever integer is in the HCL.

And the fundamental thing underneath both: Terraform reconciles when a human or a pipeline invokes it. Kubernetes objects need reconciling continuously. You can paper over that with a cron running apply every ten minutes, and I have watched someone do it, but at that point you’ve built a worse Argo CD with a state file and a lock.

I keep helm_release for exactly one chart. I’ll get to which.

The gray zone

The line above is clean enough to fit on a slide, and the slide is useless, because all four of the recurring arguments live in the space it doesn’t cover.

CRDs. A CRD is cluster-scoped, versioned, and breaks everything downstream if it’s wrong, which makes it feel like Terraform’s kind of object. It also ships as part of the operator it belongs to and changes whenever that operator upgrades, which makes it Argo’s. We landed on Argo, but split from the operator. Karpenter actually documents this pattern: install the karpenter-crd chart as its own Application and set skipCrds on the main chart, so CRD upgrades are a visible, separately-syncable step instead of a side effect of bumping an app version. Do that and you get one specific piece of pain for free, which is the 262144 byte annotation limit. Argo CD’s default client-side apply stuffs the whole object into kubectl.kubernetes.io/last-applied-configuration, and a fat CRD (Prometheus Operator’s is the famous one) blows past the API server’s limit and just fails. The fix is ServerSideApply=true as a sync option, because server-side apply tracks field ownership on the server and never writes that annotation. Set it on the CRD Applications and forget it.

Cluster addons. VPC CNI, kube-proxy, CoreDNS, EBS CSI. I put these in Terraform as aws_eks_addon resources and I don’t feel great about it, but the reasoning holds up: these are versioned against the control plane, AWS gates compatibility on the cluster version, and a broken CNI means Argo CD can’t reach anything to fix itself. They’re part of what “the cluster exists” means. Everything else that arrives as a Helm chart, including the GPU operator, goes to Argo.

Karpenter NodePools. This one I got wrong twice before landing. A NodePool looks like infrastructure. It’s about instance types and capacity and AMIs. But it isn’t infrastructure, it’s the input to a control loop, and we tune it constantly. On GPU pools that means taints, consolidation policy, capacity type mix, instance families as availability shifts under us. At Sales Marker I had a Prometheus-driven loop feeding Karpenter, and the NodePool spec was something we edited weekly. Argo owns NodePools and EC2NodeClasses. The IAM role Karpenter assumes, its instance profile, the interruption queue: Terraform. Split down that seam and both halves stay boring.

IRSA. The most annoying one, because a single logical thing genuinely straddles the line. The IAM role and its trust policy against the cluster’s OIDC provider are unambiguously Terraform. The eks.amazonaws.com/role-arn annotation on the service account lives in a manifest, which is unambiguously Argo. So the binding is two commits in two repos and a role ARN copy-pasted between them, and yes, someone has already shipped a typo’d ARN and spent an hour on a very confusing AccessDenied. We’re evaluating EKS Pod Identity partly to fix this, because aws_eks_pod_identity_association puts the whole binding on the AWS side and takes the annotation out of the manifest entirely. That collapses the straddle instead of managing it. Not fully migrated. It’s on the list, below things that are on fire.

The handoff

Terraform creates the cluster. Something has to install Argo CD. Then Argo CD takes over. That transition is not plumbing, it’s a design decision, and where you put it determines how much Terraform you’re stuck with forever.

Ours is deliberately as thin as I could make it. Terraform builds the VPC, the EKS control plane, the node groups that host the platform itself, the IAM. Then exactly one helm_release for argo-cd, and exactly one kubernetes_manifest for a single root Application. That Application points at a directory in the platform repo, and everything else in the cluster, including Argo CD’s own configuration, gets pulled in from there by app-of-apps. Two Kubernetes-shaped resources in the entire Terraform estate. That’s the budget.

The point of the thin version is that the interesting questions become Git questions instead of state file questions. Onboarding a new region is a Terraform run and a directory. Argo CD’s own values are reconciled by Argo CD, which reads as a paradox for about a day until you notice that the failure mode is fine: if you break Argo with a bad sync, Terraform can still reinstall the chart, and the chart’s values come from Git either way. There’s no bootstrap loop, just a slightly odd-looking arrow.

The temptation I’m resisting, hard, is letting the root Application list grow. It wants to. Every new subsystem arrives with a reason why its Application should be created by Terraform, usually because someone needs an output from a Terraform module and passing it through the Application is the shortest path. That’s the trap, and it’s exactly how Rapyuta ended up with a state file that had opinions about forty Kubernetes objects. What actually works is publishing those Terraform outputs somewhere the workloads themselves can read (SSM Parameter Store, in our case, fetched at pod startup) and keeping the handoff at exactly one object.

Still open

The SLO gate is the part I can’t yet tell you works, because it doesn’t yet exist. The shape I’m sketching is that Argo CD syncs, a controller watches burn rate against the service’s budget for some minutes after the rollout marker, and reverts by pinning the Application back to the previous revision if the fast window trips. Terraform has no role in that and shouldn’t. It has no concept of an SLO and no business acquiring one.

The honest summary two weeks in: this boundary isn’t a technical constraint, it’s a maintenance bet about which changes get a human reading a diff and which get a controller acting without one. Moving something from Argo to Terraform buys you the review and costs you the convergence, and I’ve spent enough of my career on the wrong side of that trade to be pretty stingy about it now.