# Argo CD config management plugins: the escape hatch I keep reaching for > Notes from wiring CMP sidecars into a GPU fleet rollout: what plugin.yaml actually does, a SOPS plugin and a CUE plugin that earn their keep, and the ways all of this bites in production. Author: Karan Vijayakumar Published: 2026-07-25 Tags: argocd, gitops, kubernetes, platform-engineering Reading time: 11 min Canonical URL: https://karanvk.me/blog/argocd-cmp-plugins --- Two months ago I hit a wall with Argo CD, and the way out has quietly become my favorite part of the platform. Some context first. The project I'm on right now is a lights-out GitOps setup for a GPU fleet: Terraform owns the AWS estate, Argo CD reconciles every rollout, and an SLO gate decides whether a deploy stays or gets reverted, no human in the loop. I'm still in the middle of it. Around week three, the rollback controller needed every image reference pinned to a digest at render time, resolved against what the registry actually held rather than whatever a values file claimed. Helm can't do that. Helm doesn't know registries exist. Kustomize flat out refuses to run arbitrary code, which is normally the right call, just not that day. The fix ended up being a config management plugin. Forty lines of shell in a sidecar container, and Argo CD treats it as a first-class renderer sitting right next to Helm and kustomize inside the repo server. I'd bounced off the CMP docs twice before at previous jobs and assumed the feature was for people with weirder problems than mine. Turns out I just hadn't hit my weird problem yet. So this post is the writeup I wish I'd had: how the thing actually works, two plugins I now run for real, and the parts that hurt. ## What a CMP actually is The repo server has one job. Take a Git revision, produce a stream of Kubernetes objects. Helm, kustomize, Jsonnet, and plain YAML directories are built in, and a config management plugin is how you add your own renderer to that list. Since Argo CD 2.8 there's exactly one supported way to do it, which is a sidecar container on the `argocd-repo-server` pod. The moving parts, roughly in the order they matter: - Your plugin is a container image you own. Whatever binaries the render needs (sops, cue, helm, some in-house CLI) get baked into it. - Its entrypoint is `argocd-cmp-server`, a small gRPC shim that ships in the repo server image and gets shared into your sidecar through the `var-files` volume. You don't write this part. - On startup the shim reads `/home/argocd/cmp-server/config/plugin.yaml` and opens a unix socket in `/home/argocd/cmp-server/plugins/`, a directory it shares with the repo server. - When an Application needs manifests, the repo server checks out the revision and asks each registered plugin over its socket: is this yours? Whichever plugin matches runs its generate command inside the checkout, and whatever lands on stdout had better be valid Kubernetes YAML or JSON. ```mermaid flowchart LR A[Application sync] --> R[argocd-repo-server] R -->|git checkout| W[working dir] R -->|gRPC over unix socket| S[argocd-cmp-server shim] S -->|discover, then generate| P[your plugin commands] P -->|manifests on stdout| R ``` That's the entire contract. Stdout is manifests, stderr is logs, non-zero exit fails the render. It's basically CGI, twenty-five years later, wearing a GitOps trenchcoat. And the smallness of the interface is exactly why it works. My digest pinner is a shell script that calls skopeo and sed. There's no SDK to learn because there's nothing an SDK would do. ## Why the sidecar model won If you ran Argo CD before 2.8 you probably remember the old way, a `configManagementPlugins` block in the `argocd-cm` ConfigMap with your commands executing directly inside the repo server container. Deprecated in 2.4, removed in 2.8. I don't miss it. The dependency story alone was miserable. Your plugin ran on the repo server's filesystem, so every binary it needed had to be baked into a custom repo server image, or smuggled in with an init container. Every Argo CD upgrade meant rebuilding that image and praying. And because plugin commands shared the repo server's process space, a leaky Jsonnet render or a runaway template loop was starving manifest generation for everything else on the instance. I watched a variant of this happen at a previous job, on a hybrid estate north of 2000 vCPUs, and the postmortem was not a fun read. One team's plugin, everyone's outage. It also ran with the repo server's privileges against every repository the instance could see, which in hindsight should've scared me more than it did at the time. Sidecars fix this by making the plugin an actual container. Own image, own resource limits, own security context, own lifecycle. The repo server talks to it through a socket and that's the whole relationship. A few weeks ago our digest pinner wedged itself against a slow registry mirror; it burned its own CPU limit and its own timeout budget, and Helm renders for everything else carried on like nothing happened. That's the property I'm paying for. ## Anatomy of plugin.yaml Here's a complete spec, annotated: ```yaml title="plugin.yaml" apiVersion: argoproj.io/v1alpha1 kind: ConfigManagementPlugin metadata: name: digest-pinner spec: version: v1 init: # Runs first, in the checked-out source directory. # Output is ignored; use it to fetch dependencies. command: [sh, -c, "helm dependency build 2>&1"] generate: # The actual render. Stdout must be K8s YAML or JSON. command: [sh, -c, "./render.sh"] discover: # How Argo CD decides this plugin owns a directory. fileName: ".digest-pinner.yaml" preserveFileMode: false ``` Despite the `kind`, this isn't a CRD and you never `kubectl apply` it. It lives inside the sidecar, mounted from a ConfigMap or baked into the image. If you ever see "no matches for kind ConfigManagementPlugin" in an error, someone applied it to the cluster. The CRD went away with the old plugin system. `discover` is the field worth slowing down on. Three matchers, and if you specify more than one, only the first in this order counts: `fileName` (a glob against the app's source directory), `find.glob` (same but with `**` for nested paths), and `find.command` (a command run in the repo, matches if it exits 0 and prints anything). My rule, learned the annoying way, is that discovery should key on an explicit marker file like the `.digest-pinner.yaml` above. I'll get to why. ## Wiring it into the repo server The plugin ships as a patch on the `argocd-repo-server` Deployment, and every line of this shape matters: ```yaml title="repo-server-patch.yaml" containers: - name: digest-pinner image: registry.internal/platform/digest-pinner:1.4.2 command: [/var/run/argocd/argocd-cmp-server] securityContext: runAsNonRoot: true runAsUser: 999 resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: "1", memory: 512Mi } volumeMounts: - mountPath: /var/run/argocd name: var-files # provides the cmp-server binary - mountPath: /home/argocd/cmp-server/plugins name: plugins # shared socket directory - mountPath: /home/argocd/cmp-server/config/plugin.yaml subPath: plugin.yaml name: digest-pinner-config - mountPath: /tmp name: digest-pinner-tmp # never share /tmp with the repo server volumes: - configMap: name: digest-pinner-config name: digest-pinner-config - emptyDir: {} name: digest-pinner-tmp ``` `runAsUser: 999` isn't a style choice. The shared volumes belong to the argocd user and the shim just fails without it, with an error message that won't point you anywhere useful. The dedicated `emptyDir` on `/tmp` gives the plugin scratch space that no other container in the pod can see. The image itself stays boring, since the shim arrives through the shared volume rather than the image: ```dockerfile title="Dockerfile" FROM alpine:3.20 RUN apk add --no-cache bash curl jq skopeo COPY render.sh /usr/local/bin/render.sh # No entrypoint needed: the Deployment sets # command: [/var/run/argocd/argocd-cmp-server] USER 999 ``` On the Application side you either name the plugin or let discovery pick it up: ```yaml spec: source: repoURL: https://git.internal/platform/workloads path: payments/production plugin: name: digest-pinner-v1 # metadata.name plus spec.version env: - name: REGISTRY value: registry.internal ``` ## The SOPS plugin The highest-value CMP I run is also the dumbest one. I've never loved Sealed Secrets (re-encrypting everything for a key rotation is misery) and the external secret operators add a runtime dependency I really didn't want sitting in the rollback path. SOPS with age keys is the compromise I actually like: secrets live encrypted in Git next to the manifests that use them, a diff shows you which keys changed without showing values, and decryption happens exactly once, at render time, inside the plugin. ```yaml title="sops-plugin.yaml" apiVersion: argoproj.io/v1alpha1 kind: ConfigManagementPlugin metadata: name: sops-render spec: version: v1 generate: command: [sh, -c] args: - | for f in *.enc.yaml; do sops -d "$f" echo "---" done # plain manifests pass through untouched for f in *.yaml; do case "$f" in *.enc.yaml) ;; *) cat "$f"; echo "---";; esac done discover: find: glob: "**/.sops-render" ``` The age private key mounts into the sidecar from a Kubernetes Secret, and sops picks it up through its usual env var: ```yaml env: - name: SOPS_AGE_KEY_FILE value: /sops/age/keys.txt volumeMounts: - mountPath: /sops/age name: sops-age-key readOnly: true ``` The operational shape of this is what sold me. The private key exists in one place, the sidecar, and nowhere else. App authors encrypt against the public recipient key and never see the other half. We ran a key rotation drill a few weeks into the project, and it came down to one Secret update plus a repo server restart. The SLO gate didn't blink. If your teams live in kustomize, ksops gets you the same trust model. Generate becomes `kustomize build --enable-alpha-plugins --enable-exec .` and the decryption moves inside a kustomize generator. One more layer of indirection, same idea. ## CUE for the GPU fleet GPU node configuration is the kind of combinatorial mess that YAML templating quietly falls apart on. MIG profiles crossed with driver versions crossed with node pools, and an invalid combination doesn't fail loudly, it just leaves very expensive silicon idle. We moved that config into CUE because CUE checks constraints at render time. A MIG profile that doesn't exist for the card in that pool stops being a 3am page and becomes a type error. Argo CD doesn't support CUE and honestly it doesn't need to: ```yaml title="cue-plugin.yaml" apiVersion: argoproj.io/v1alpha1 kind: ConfigManagementPlugin metadata: name: cue-render spec: version: v1 generate: command: [sh, -c] args: - cue cmd dump ./... discover: find: glob: "**/cue.mod" ``` `dump` is a CUE tool file in the repo that marshals every concrete object into a YAML stream. Standard pattern, straight out of CUE's own Kubernetes tutorial. What I care about is the failure mode. Before this, a typo'd pool spec rendered fine, synced fine, and showed up hours later as pods stuck in `Pending` on a quarter million dollars of hardware. Now the render fails, the app goes `ComparisonError`, the sync never starts, and the PR diff already told the author what they broke. Same information, four hours earlier, before anything touched a cluster. ## Parameters, which everyone skips Most CMP writeups stop at env vars, and I get it, env vars are enough to ship. But `spec.parameters` is what makes a plugin feel native in the UI. Static announcements declare which knobs exist, and a dynamic command can compute them per application: ```yaml parameters: static: - name: environment title: Target environment required: true string: staging dynamic: # Runs in the app source dir, prints JSON in the same schema command: [sh, -c, "./list-overlays.sh"] ``` At generate time each parameter shows up as a `PARAM_` variable (`PARAM_ENVIRONMENT=staging`), plus the whole set as JSON in `ARGOCD_APP_PARAMETERS`. Add the standard build env (`ARGOCD_APP_NAME`, `ARGOCD_APP_NAMESPACE`, `KUBE_VERSION`, that family) and you can put together an envsubst-style stamping plugin in an afternoon. That's literally how our per-environment DNS and quota values get injected today. One design detail I want to call out because it's easy to resent until you understand it: env vars set in an Application spec reach your commands prefixed as `ARGOCD_ENV_*`. That prefix is injection safety. App authors are usually less trusted than plugin authors, and without it, anyone who can edit an app spec could set `PATH` or `SOPS_AGE_KEY_FILE` or `LD_PRELOAD` and quietly own your plugin's execution. Don't work around the prefix. And quote every `ARGOCD_ENV_` value you interpolate into shell like it's hostile, because it might be. ## The parts that hurt The repo is untrusted input. I keep repeating this to myself because it's easy to forget when the plugin is forty lines of shell you wrote at your own desk. Your generate command executes against files controlled by whoever can push to the repo, so on a multi-team instance a CMP is a code execution service with a Git-shaped front door. > [!WARNING] > Quote everything, never `eval`, leave `preserveFileMode` off unless you can say out loud why you need executable bits from Git, and treat `provideGitCreds` as the loaded gun it is. My worst CMP incident so far wasn't a crash, it was silence. An early version of the CUE plugin discovered on `find.glob: "**/*.cue"`, which felt reasonable right up until a team vendored a dependency that happened to contain one stray `.cue` file into a perfectly normal Helm chart repo. Argo CD handed the whole directory to the CUE plugin instead of Helm. Nothing crashed. The app just sat `OutOfSync` with a diff that made no sense, while two of us stared at a chart that had nothing wrong with it. When several renderers could plausibly claim a directory, you're depending on an evaluation order you don't control. Marker files ended that entire class of problem for us, permanently, and I'm never going back to clever globs. Timeouts stack in a genuinely unhelpful way, and this one cost me an hour. The shim kills any plugin command that runs past `ARGOCD_EXEC_TIMEOUT`, default 90 seconds, set on the sidecar. Separately, the repo server's client-side timeout (`controller.repo.server.timeout.seconds` in `argocd-cmd-params-cm`) defaults to 60. So when our `helm dependency build` crawled past 60 seconds against a slow chart mirror, the error surfaced from the repo server, not from the plugin, and I went digging in exactly the wrong place. If you raise one, raise both, and keep the repo server's below the exec timeout so failures blame the right component. Give the sidecar real limits. It's a container, treat it like one. A render that OOMs inside its own cgroup is a clean, attributable failure; a limitless sidecar fighting the repo server for memory on a packed node is a mystery you get to solve at night. Debugging, for what it's worth, has settled into a routine for me. Check the repo server pod and confirm the sidecar actually registered (the extra container should be Ready). Your stderr lives in the sidecar's logs, nowhere else. `argocd app manifests ` prints exactly what generate produced, which beats squinting at the UI diff every single time. And rendered manifests are cached in Redis, so a stale render wants a hard refresh on the Application, not a repo server restart. Changing a ConfigMap-mounted plugin.yaml does need the pod restart though. I've mixed those two up more than once. ## Where I've landed There's a respectable school of thought that says everything belongs in Helm, and that stretching a chart until it screams still beats running custom code in the render path. I held some version of that opinion myself until the digest pinning problem, and I've come around to thinking it's exactly backwards. The render pipeline is where platform opinions belong, because it's the last moment a human-readable diff exists and the first moment policy can reject a change before it touches a cluster. Two of the renderers in my production Argo CD right now are plugins we wrote ourselves, with a third on the way. Each one replaced something more fragile than itself. That's about the strongest endorsement I know how to give a Kubernetes feature.