Skip to content

Homelab technical spec

This file describes the current shape of the homelab: what runs where, and how the parts depend on each other. It describes the present only. Plans and work in progress live in docs/projects/<name>/. The detail for each area lives in the folders in the map at the end. The manifests and the Terraform are what runs. When this file and the code disagree, the code is correct and this file gets a fix.

Layers

Layers of the homelab, from the internet down to the Synology NAS

The overview and the network flow of the whole homelab, and how to read and change them: diagrams.md.

Physical and virtual

  • Three Lenovo nodes run Proxmox VE: hyper1, hyper2, hyper3. The datacenter and node configuration is in terraform/proxmox/hyper-cluster/datacenter.
  • The Talos VMs are in terraform/proxmox/hyper-cluster/k8s/talos. genesis-worker-01 on hyper1 has the iGPU passed through. Plex and Tdarr use it for Quick Sync (../platform/cluster/gpu-passthrough.md).
  • The Synology DS1522+ NAS holds all persistent data. It also runs the Proxmox Backup Server VM (terraform/proxmox/pbs).
  • Home Assistant runs on its own mini PC, outside the cluster. Only its ingress lives in this repo. Home Assistant itself holds its configuration.
  • Upgrades: ../platform/cluster/talos-upgrade.md.
  • Every Terraform stack, its Azure state and the laptop-only files it needs: ../platform/cluster/terraform.md.

Network

Delivery

  • Argo CD runs two ApplicationSets, apps and infra, in k8s/talos/infra/argocd/. They generate one Application per directory under k8s/talos/apps/ and k8s/talos/infra/, synced from main. Argo CD reverts a direct kubectl apply (../platform/delivery/argocd.md).
  • k8s/talos/infra/crds installs the CRDs that no other Application ships: the prometheus-operator CRDs, the VolumeSnapshot CRDs that VolSync needs, and Traefik's. Traefik's own chart renders with includeCRDs: false.
  • Kargo promotes every image built here or in an app repo, except innestemme's. That tag is bumped by hand. Portfolio and blog go from stage to prod. headroom demo goes to prod. logeverylift, verksted and reelsmith go straight to prod. Every prod promotion is a pull request (../platform/delivery/kargo.md). CI only builds and pushes tags (../platform/delivery/ci.md).
  • KEDA with the HTTP add-on scales idle apps to zero. The interceptor wakes them (../platform/delivery/keda.md).
  • kro turns a ResourceGraphDefinition into a new Kubernetes kind. One object of that kind creates a group of objects (../platform/delivery/kro.md).
  • GitHub Pages publishes docs/ to https://docs.nordbye.it (.github/workflows/docs.yaml). The backlog stays in the repo only.
  • Renovate proposes chart, image and provider bumps. It does not propose Postgres major upgrades. Those are done by hand (../platform/data/postgres.md).

Secrets

No secret is committed. Values live in Bitwarden Secrets Manager (Homelab project). External Secrets Operator turns an ExternalSecret that names the item's UUID into a Kubernetes Secret (../platform/secrets/README.md). Reloader (k8s/talos/infra/reloader) restarts a workload annotated reloader.stakater.com/auto: "true" when a ConfigMap or Secret that it uses changes. Outside the cluster, AI agents and scripts reach the same project with their own access token. bws-token (ai/skills/secrets/) serves that token on each machine from a copy of a Bitwarden vault item. The vault itself holds human sign-ins only. Agents never unlock it.

Policy

Kyverno checks every Pod for pinned image tags, the baseline and restricted Pod Security Standards, and resource requests. It writes PolicyReports. All policies are enforced: the API server rejects a Pod that breaks one, unless the workload has a named exception. CI attests provenance for the portfolio image. A further policy audits that provenance on stage. Policy Reporter exports the reports to Prometheus for the SPOG dashboard (../platform/policy/README.md). trivy-operator scans the image of every running workload for fixable high and critical CVEs and exposed secrets. It writes a report per container (../platform/policy/README.md#vulnerability-scanning).

Storage and data

Storage classes and CSI drivers: ../platform/storage/README.md.

  • proxmox-local (Proxmox CSI) for block volumes such as app config.
  • syno-nfs-csi for shared volumes. A new NFS volume is root-only for non-root uids. An app that creates directories in it needs an initContainer chown.
  • syno-nfs-csi-scratch: the same share with onDelete: delete, for the temporary volumes of the restore tests.
  • Postgres 18 for logeverylift is the CloudNativePG Cluster logeverylift-db: a primary and a standby, each on an NFS PVC. WAL and nightly base backups go to Garage (../platform/data/postgres.md).
  • Outline has its own CloudNativePG Cluster outline-db: one instance, no WAL archive, a nightly pg_dump to the NAS.
  • The CloudNativePG operator (cnpg-system) with its Barman Cloud plugin, and a single-node Garage (garage) as the S3 target for base backups and WAL. Garage keeps its data on NFS and its LMDB metadata on proxmox-local, snapshotted to NFS (../platform/data/cloudnative-pg.md).
  • The media share /volume1/shared-data/media is one mount, so imports can hardlink (../apps/media-stack/README.md).
  • Backups: etcd snapshots, Postgres dumps, VolSync restic per PVC, Home Assistant, NAS snapshots, offsite copy, and PBS for whole VMs (../platform/backups/README.md, restores in ../platform/backups/restore.md).

Observability

The stack is kube-prometheus-stack, Grafana, Loki with Alloy for logs, and Tempo for traces. Traefik and logeverylift send traces over OTLP through Alloy. Alloy filters out monitoring and probe traffic and strips client IPs and query strings. Alertmanager sends only actionable, critical alerts to Discord. A Cloudflare Worker reports a missing Alertmanager heartbeat. Falco watches syscalls on every node and posts to Discord itself through falcosidekick (../platform/observability/README.md). Past incidents are in ../platform/observability/incidents.md. OpenCost estimates electricity cost per namespace from Prometheus usage (costs). Pyrra keeps SLOs and error budgets for the public sites over a 28-day window. It pages on the burn rate of the probe SLO. Kargo reads those budgets from Prometheus before it opens a prod promotion PR. A low budget stops the auto-merge (../platform/observability/slos.md). A k6 CronJob in synthetics signs in to logeverylift in Chromium every 10 minutes and walks the log-a-set flow. It remote-writes the result of each step to Prometheus. A flow that breaks behind a page that still answers 200 therefore pages Discord (../platform/observability/README.md). Argo CD notifications writes a Grafana annotation for every finished sync. The Homelab dashboards draw them as deploy markers (../platform/delivery/argocd.md). Grafana reads pull requests from the GitHub API through a GitHub App (../platform/observability/README.md). The same App lets Git Sync load the dashboards under grafana/ and commit each UI save to a branch for a PR (../platform/observability/README.md). Grafana also reads Cloudflare's analytics from its GraphQL API with a read-only token (../platform/observability/README.md). Outside the cluster, Prometheus scrapes the Proxmox API (pve-exporter), node-exporter on the hosts (CPU temperatures and NVMe health), Proxmox Backup Server (pbs-exporter), the NAS over SNMP, and the UniFi gateway. Inside the cluster, CloudNativePG's exporter reports logeverylift's database, and electricity-price supplies the NO1 spot price for the cost panels of the SPOG (../platform/observability/README.md). metrics-server (k8s/talos/infra/metrics-server) serves the resource metrics API for kubectl top.

Workloads

24 Argo CD Applications live under k8s/talos/apps/, one per directory. These groups have apps that depend on each other:

Other apps, one directory each under k8s/talos/apps/:

  • headroom: headroom.local.bigd.no, SQLite on NFS. Do not add runAsUser. The image starts as root and falls back to root when its chown fails on NFS. A non-root start breaks SQLite. ALLOWED_HOSTS must list every hostname it is served on, or it answers
  • headroom-demo is the same image on headroom.nordbye.it with fictional data.
  • trek: travel planner on trek.bigd.no. The admin in trek-secret is created only on a boot with no users. ENCRYPTION_KEY must never change, or the stored secrets become unreadable.
  • home-assistant: no pod. A Service with hand-written Endpoints points at the HA mini PC, 10.3.10.15:8123, behind ha.local.bigd.no.
  • audiobookshelf: audiobookshelf.local.bigd.no, private gateway. Reach it from outside through remote access.
  • ollama and open-webui: local LLM chat on open-webui.local.bigd.no. ollama is pinned to the hyper3 zone. open-webui reaches it through the ollama-wake Service, so its requests wake ollama from zero. The model cache is not backed up.
  • innestemme: the voice assistant for the Voice PE in the living room. It replaces Assist. It dials the device on IoT and serves spoken answers on the VIP 10.3.10.99:9090. One UniFi firewall policy lets the device reach that VIP. Its skills page is on jarvis.local.bigd.no (../apps/innestemme/README.md).
  • mealie: recipe manager on mealie.local.bigd.no, private gateway, SQLite, sign-up off.
  • outline: notes wiki on docs.local.bigd.no, private gateway. Sign-in is Authentik OIDC only, limited to the outline-users group. Postgres in outline-db, Valkey with no persistence, attachments on an NFS PVC. Its MCP endpoint is /mcp.
  • homepage: the hub on hub.bigd.no, behind Authentik forward-auth.
  • bigd: the static bigd.no landing page. nginx serves index.html from a ConfigMap.
  • omni-tools: a stateless browser tool collection on omni-tools.bigd.no. It is a kro StaticSite instance, so kro owns its objects and Argo CD owns only the instance.
  • isponsorblocktv: skips sponsor segments and ads in the YouTube app on the TVs through YouTube's lounge API. It has no route and no volume. The paired screens live in Bitwarden (../apps/isponsorblocktv/README.md).

Doc map

docs/ has one folder per kind of work and one subfolder per area. The README.md of each area is its current state. Runbooks sit beside it.

Folder Contents
platform/backups backup layers, watching them, restores
platform/cluster Proxmox and Talos, upgrades, GPU passthrough, Terraform stacks
platform/data in-cluster Postgres, major upgrades, CloudNativePG, Garage
platform/delivery Argo CD, Kargo promotion, KEDA scale to zero, CI
platform/identity Authentik SSO
platform/network VIPs, Traefik, Cilium, certificates, DNS, Cloudflare, UniFi, remote access
platform/observability monitoring, alerting, incidents, the reliability practice
platform/post-quantum crypto inventory, Shor on a real quantum computer
platform/policy Kyverno policies, Trivy image scanning, their reports
platform/secrets Bitwarden to cluster secrets
platform/storage storage classes, CSI drivers
apps/innestemme the voice assistant for the Voice PE
apps/isponsorblocktv SponsorBlock for the TVs, pairing a screen
apps/media-stack requests, storage, 4K, seeding, subtitles, re-encoding
apps/portfolio the portfolio site, brand decisions, the fun room
apps/reelsmith the reelsmith publishing gateway
apps/verksted the agent session sandbox
projects/<name> work in progress, created when needed and deleted when it ships
backlog known gaps agreed to leave for later
assets diagrams, logo, social preview