Skip to content

Argo CD

Argo CD owns everything in the cluster. It is installed from the argo-cd Helm chart, rendered by kustomize in k8s/talos/infra/argocd/, and manages itself: the argocd directory is one of the Applications it syncs. Every Application tracks HEAD of main, so a merge is a deploy and a direct kubectl apply is reverted by self-heal.

Applications from directories

Two ApplicationSets use the git directory generator on this repo. Each directory one level below the path becomes one Application named after the directory.

ApplicationSet Directories Project Destination namespace Application sync wave
infra k8s/talos/infra/* infra none, manifests set their own 0
apps k8s/talos/apps/* apps the directory name 10

Adding an app is adding a directory; deleting the directory deletes the Application and, through the resources-finalizer.argocd.argoproj.io finalizer, everything it created. Neither set uses CreateNamespace, so a directory ships its own Namespace manifest. A manifest that names a namespace explicitly wins over the apps destination, which is how the <app>-stage directories deploy to stage-<app>.

Argo CD builds each directory with kustomize build --enable-helm (kustomize.buildOptions in values.yaml), so a directory is either plain manifests with a kustomization.yaml or a helmCharts: entry with a local values.yaml.

Both sets share the same sync policy:

Setting Effect
automated.prune, automated.selfHeal Removes what left Git and reverts drift
retry.limit: 1 One retry, backoff 10 s doubling up to 3 min
ApplyOutOfSyncOnly=true A sync only applies the resources that differ
PruneLast=true Pruning runs after every other resource is healthy
ServerSideApply=true No last-applied-configuration annotation, so large CRDs and ConfigMaps fit
RespectIgnoreDifferences=true Fields listed in ignoreDifferences are left alone on sync too

The apps set adds one ignoreDifferences entry: /spec/replicas on every Deployment. KEDA owns the replica count of the apps that scale to zero (keda.md), and without this entry self-heal would reset it. The same entry means a replicas change in Git is not applied to a Deployment that already exists; scale it by hand or recreate it.

The apps set also has a templatePatch that stamps kargo.akuity.io/authorized-stage on the Applications Kargo promotes. It needs goTemplate: true, so its fields use the {{.path.*}} syntax while infra keeps the older {{ path }} form. Onboarding and the offline check for the patch are in kargo.md.

Projects

project-apps.yaml and project-infra.yaml define the AppProjects apps and infra. Both allow only this repo as a source and allow every destination and every resource kind, cluster scoped included. They separate the two sets in the UI and in RBAC; they do not restrict anything.

Sync waves

Waves order resources inside one Application. The ones in use:

Wave Used for
-3 Authentik's identity namespace
-2 Authentik's ExternalSecret; everything in cert-manager and Cilium (commonAnnotations)
-1 Authentik blueprint ConfigMaps, the Kargo Project namespaces, External Secrets Operator, the Grafana OIDC ExternalSecret
0 Default; most app kustomizations set it explicitly; Kargo Projects
1 Some ExternalSecrets, ReferenceGrants, Traefik, Kargo ProjectConfigs and Warehouses
2 HTTPRoutes, Traefik Middlewares, several namespaces and ExternalSecrets, Kargo AnalysisTemplates and Stages
201 cert-manager Certificates, after the rest of cert-manager

The Application-level waves in the two templates (0 and 10) are not an ordering guarantee: the ApplicationSet controller creates all Applications at once, and each one syncs on its own. Anything that must exist first, such as CRDs, is either in the same Application or tolerated with SkipDryRunOnMissingResource=true, as the VolSync ReplicationSources do.

Per-resource sync options

Where Option Why
dashboards/kustomization.yaml ServerSideApply=true The SPOG dashboard JSON is over the 256 KiB annotation limit of client-side apply
volsync.yaml in each backed-up app SkipDryRunOnMissingResource=true The VolSync CRDs come from the volsync Application
talos-serviceaccount.yaml SkipDryRunOnMissingResource=true The talos.dev CRD exists only once Talos kubernetesTalosAPIAccess is on
cilium/namespace.yaml Prune=false The namespace is never pruned

Cilium CiliumIdentity objects are excluded from tracking through resource.exclusions.

Access

Item Value
UI https://argocd.local.bigd.no through traefik-gateway-private (httproute.yaml)
LoadBalancer 10.3.10.100 (Cilium LB IPAM)
SSO Authentik OIDC, issuer https://auth.local.bigd.no/application/o/argocd/
Admin group argocd-admins maps to role:admin; any other SSO user gets no permissions
Break-glass The built-in admin account stays enabled and bypasses RBAC

The OIDC client secret comes from Bitwarden through the ExternalSecret argocd-oidc (argocd-oidc-secret.yaml), the same item Authentik reads for its side. The provider, application slug and group are declared in argocd-blueprint.yaml; see identity. Dex is disabled.

The admin password of a fresh install is in argocd-initial-admin-secret; the command is in talos.md.

Components

Component Replicas PDB
server 2 maxUnavailable: 1
repo-server 2 maxUnavailable: 1
applicationset-controller 2 maxUnavailable: 1
application-controller 1 none

Redis runs as a single instance (redis-ha off). Notifications are enabled. server, repo-server and application-controller expose ServiceMonitors for kube-prometheus-stack.

Bootstrap

Terraform installs the chart with the same values.yaml and applies the two AppProjects and the infra ApplicationSet (k8s-argocd.tf). The infra set then creates the argocd Application, which brings in the apps set. The helm_release ignores all changes after creation, so upgrades go through the chart version in kustomization.yaml, not Terraform. The full cluster bootstrap is in talos.md.

Gotchas

The argocd namespace runs under a default-deny CiliumNetworkPolicy (ciliumnetworkpolicy.yaml). Egress to the internet is limited to GitHub, ghcr.io, *.github.io and a fixed list of Helm repositories, which is what repo-server renders from. A chart from a new repository fails to render until its host is added to the toFQDNs list.

The OIDC issuer needs its trailing slash, and its path segment is the Authentik application slug, not the provider name. A mismatch gives oidc: issuer did not match. Authentik has no groups scope; the claim rides on profile, and requesting groups returns invalid_scope.

argocd-server only resolves $argocd-oidc:clientSecret from a Secret labelled app.kubernetes.io/part-of: argocd, which the ExternalSecret template sets.

Argo Rollouts

k8s/talos/infra/argo-rollouts/ installs the argo-rollouts chart only for its AnalysisTemplate and AnalysisRun CRDs and controller. Kargo uses them for Stage verification: each Stage's smoke test is an AnalysisTemplate that runs a curl Job (kargo.md). No workload uses the Rollout kind, and the dashboard is off.