GitOps Strategy¶

ArgoCD manages the cluster state declaratively: if it is not in Git, it is not in the cluster, and if you put it there anyway the next reconcile puts it back.

Structure¶

flowchart LR
    subgraph "Bootstrap (once, by hand)"
        PJ[payload/platform/argocd-projects/projects.yaml]
        AA[payload/argocd/application.yaml]
    end

    AA --> AR[Application argocd]
    AR --> |"argo-cd chart + values.yaml"| SELF[ArgoCD itself]
    AR --> |"payload/argocd/applicationset-platform.yaml"| AS[ApplicationSet platform]
    AS --> |"payload/platform/*/application.yaml"| INFRA[Every component]
    AS --> |"payload/workloads/application.yaml"| WA[Application workloads]
    WA --> |"payload/workloads/applicationset.yaml"| APPS[ApplicationSet apps]
    APPS --> |"homelab-apps: */application.yaml"| WL[Every workload]

argocd is the only Application not generated by the ApplicationSet and the only one applied by hand. It syncs the chart that runs ArgoCD and, through a second source, the platform ApplicationSet, so changes to the generator arrive through Git. Its chart values are described in Platform → ArgoCD, along with the recovery commands for the day ArgoCD syncs a broken ArgoCD config.

Everything else is a component directory with one application.yaml, including ArgoCD's own supporting resources:

The AppProjects cannot arrive through ArgoCD alone: an Application naming a project that does not exist is rejected. payload/platform/argocd-projects/projects.yaml is applied once at bootstrap, ahead of the argocd Application that names system, and argocd-projects adopts it on its first sync. A broken project definition is fixed with kubectl apply, not a sync.

Each application.yaml is a complete Application, not a fragment: the generator copies its labels, annotations, finalizers and spec unchanged, and Renovate and the Makefile read it like any other manifest.

Sync policy¶

Every Application, generated or not, carries the same syncPolicy: automated sync with prune and selfHeal, and a retry block of ten attempts with a backoff from 30s to 5m. That is the whole delivery contract:

• A commit is live within one polling interval. ArgoCD runs on a private address, so no GitHub webhook can reach it; the intervals are in Platform → ArgoCD.
• Drift is reverted. A resource edited or deleted by hand is put back on the next reconcile, in the platform as much as in the workloads.

• A failed sync retries itself. ArgoCD never re-attempts a failed sync of the same revision unless the Application has a retry block, which is why every application.yaml carries one. The ten attempts span about forty minutes; after that the Application stays OutOfSync until its revision changes or it is synced by hand:

  argocd app sync <application>
  

• Patching a generated Application's spec achieves nothing. The next ApplicationSet reconcile copies the file back over it. That includes switching automated sync off: the only rollback is a revert commit — see Rolling back a bad sync.

• One exception to prune, and one annotation. kube-vip syncs with prune: false, because its objects are the only thing holding the API VIP; copy the syncPolicy from any other sibling. Elsewhere a resource whose deletion would take more than itself carries a per-resource argocd.argoproj.io/sync-options: Prune=false instead: a Namespace (everything in it), an AppProject (every Application in it is orphaned), a CloudNativePG Cluster and the bucket its backups land in. A removed policy or manifest still prunes; the container it lived in stays.

Bootstrap convergence¶

Nothing orders the Applications. On a fresh cluster the ApplicationSet generates all of them at once and they converge on their own: a sync that fails because a kind does not exist yet is retried once the chart that ships it has landed, and a resource that applies but cannot go Ready waits without failing anything. What each component waits for is in Platform → Rollout order.

Two placement rules keep this convergence short. A resource lives with what it needs, not with what it configures: the ClusterSecretStore is in openbao, not external-secrets, and cert-manager's issuers and certificates are their own certificates Application, so the controllers report Healthy on their own and the resources that wait for OpenBao are the ones that say so. A resource that fails to apply for longer than the retry budget belongs elsewhere: the budget covers a chart arriving minutes late, not a human arriving hours late.

Bootstrap pauses at OpenBao¶

A fresh cluster looks stuck at OpenBao, and that is the design. OpenBao's pods report Ready while sealed, but the ClusterSecretStore beside it stays Degraded until OpenBao is initialised, unsealed and has the Kubernetes auth role, and with it every ExternalSecret in the platform. make bao-init clears that, and make bao-unseal after every later restart; certificates then waits for make bao-secrets to write the Route53 credentials, and the Gateways for the certificates. Nothing fails and nothing times out: every one of those resources applied cleanly and is waiting on status.

Workloads live in a second repository¶

The applications the cluster exists to run are in JanWelker/homelab-apps, one directory per application; the apps ApplicationSet generates an Application from each */application.yaml there. The split keeps a platform change with a cluster-wide blast radius apart from a workload change with a one-app one, and keeps the reusable half of the cluster in a repository that could build somebody else's.

The dependency runs one way: workloads reference the platform (the apps-gateway, the rook-ceph-block StorageClass, the openbao ClusterSecretStore, the CloudNativePG operator), and the platform references the workloads repository exactly once — the repoURL in payload/workloads/applicationset.yaml.

What the split costs¶

• Two repositories to clone, and a change that spans both is two pull requests that cannot merge atomically; merge the platform half first.
• The diff preview covers only this repository. Both ApplicationSets carry argocd-diff-preview/ignore because the tool renders the repository it is given.
• A workload can reference a platform resource that does not exist, and nothing catches it until ArgoCD tries to sync.

Workloads sync alongside the platform¶

The apps ApplicationSet ships inside the workloads Application, which the platform ApplicationSet generates like any component. That is why payload/workloads/ is a sibling of payload/platform/, and why the platform ApplicationSet names it as a second generator path rather than a glob:

        files:
          - path: payload/platform/*/application.yaml
          - path: payload/workloads/application.yaml

On a fresh cluster the workloads therefore start syncing before the platform under them is Healthy, and converge the same way it does: a workload application.yaml carries the same retry block, its HTTPRoute waits for the apps-gateway, its volumes for Rook, its secrets for OpenBao. Ordering within a workload uses resource-level sync waves — the CloudNativePG Cluster is in an earlier wave, and ArgoCD's built-in health check for postgresql.cnpg.io/Cluster waits for it before applying the application.

Moving a resource to another Application¶

Moving a manifest between directories is not a move to ArgoCD: the old Application sees a tracked resource vanish and, with prune: true, deletes it; the new one recreates it, possibly seconds later. For a ClusterSecretStore that is an outage of every secret; for a child Application with the resources finalizer it deletes everything that Application deployed. A move takes two merges:

1. Annotate the resource where it is, and let that sync:

   annotations:
     argocd.argoproj.io/sync-options: Prune=false
     argocd.argoproj.io/compare-options: IgnoreExtraneous
   

   Both are read off the live object. Prune=false stops whichever Application prunes first; IgnoreExtraneous keeps the old Application Synced while it still sees the resource. 2. Move the file, keeping the annotations. The new Application takes over the tracking annotation.

The annotations can go once the second change has synced everywhere.

Version pins¶

Each chart version is pinned once, in the application.yaml that owns the component; the node-level versions (Flatcar, Kubernetes, syslinux) are pinned once in ansible/inventory.yaml. The Makefile reads targetRevision out of the manifests for the bootstrap installs (make install-cilium, make install-argo), so what bootstrap installs is what ArgoCD then reconciles. kube-vip is the one double pin: kube_vip_version in the inventory for the bootstrap static pod, and the image in payload/platform/kube-vip/daemonset.yaml for what runs after the first sync. Renovate tracks all of them; its automerge policy is in Maintenance.