Building My Two-Site K3s Homelab — GitOps, Tailscale & Monitoring
I’m building a small Kubernetes homelab to learn how the pieces fit together: deployment, private networking, monitoring and recovery.
The setup has two independent K3s clusters, one at home and one in the cloud. I manage both with Flux, connect them through Tailscale and use Zabbix and Grafana to see what is happening across the two sites.
I wanted something small enough to understand, with enough moving parts to practise the work around running applications.
The setup
The home cluster runs on a small Intel N100 machine with Debian and NVMe storage. The cloud cluster runs on an ARM VM in Oracle Cloud Infrastructure (OCI), with Ubuntu and a separate data volume.
Both are single-node clusters. The home side now runs Paperless-ngx with PostgreSQL and Valkey, alongside lab experiments. The cloud side hosts Zabbix, Grafana, Vikunja and the Homepage portal. Both sites have source-encrypted application backup jobs.
1
2
3
4
5
6
7
8
9
10
11
12
13
Git repository
|
Flux GitOps
/ \
Home cluster Cloud cluster
\ /
Tailscale
|
Private management
Home: Paperless-ngx
Cloud: Zabbix + Grafana + Vikunja + Homepage
Backups: encrypted copies in R2
The names and paths in the examples below are illustrative.
Why I kept the clusters separate
Each site has its own control plane and etcd data. Tailscale connects the machines, but I haven’t stretched one Kubernetes cluster across the two locations.
That keeps the home internet connection out of the cloud cluster’s control-plane path. I can work on the home node without also taking down the cloud workloads, and each cluster has its own upgrade and recovery process.
This is a learning setup with one node per site, so the separation is useful even though neither cluster provides node-level high availability.
Deploying with Flux
Flux watches the Git repository and reconciles each cluster from its own directory:
1
2
3
4
5
6
7
8
clusters/
├── home-cluster/
│ ├── flux-system/
│ └── infrastructure/
└── cloud-cluster/
├── flux-system/
├── infrastructure/
└── applications/
My usual workflow is to change a manifest, review the diff and push it. CI checks the configuration, then Flux applies it to the relevant cluster.
I can check the result with:
1
2
3
flux get kustomizations --context home-cluster
flux get helmreleases --context cloud-cluster
kubectl --context cloud-cluster get pods -A
Keeping the desired configuration in Git makes it easier to see what changed and rebuild the application definitions later. I still back up application data separately.
Keeping secrets out of plaintext Git
I use SOPS with age for encrypted secret manifests. Each cluster has its own age key pair, and Flux decrypts the manifests during reconciliation.
The repository holds the encrypted files. Private identities stay outside Git, with recovery copies stored separately.
A typical application directory might contain:
1
2
3
release.yaml
config.yaml
credentials.sops.yaml
CI handles encrypted manifests separately from ordinary Kubernetes YAML. That lets me validate the configuration without treating encrypted values as deployable plaintext.
Private access and HTTPS
Tailscale connects my management machine and both nodes. I use it for cluster access, SSH and monitoring traffic between sites.
Public SSH is closed at both the host firewall and cloud ingress layers. Before changing those rules, I verified that a fresh private session worked and kept a recovery route available.
For application HTTPS, I use Traefik, cert-manager and Let’s Encrypt, with Cloudflare DNS-01 validation. DNS validation lets certificate issuance work without exposing a separate HTTP challenge endpoint.
That gives me a repeatable ingress and certificate setup as I add services.
Monitoring both sites
Zabbix and Grafana run in the cloud cluster. That gives me a place to observe the home node while I’m restarting it or experimenting locally.
The host monitoring covers CPU, memory, filesystems, network traffic, uptime and agent health. Kubernetes monitoring adds pod health, restart counts, deployment replicas, node capacity and control-plane metrics. Kubelet metrics also provide measured pod CPU usage.
The two clusters have separate pod networks. For home-cluster metrics, Zabbix uses the authenticated Kubernetes API service proxy over Tailscale to reach kube-state-metrics. A dedicated read-only service account supplies access, with the cluster CA used for TLS verification.
Grafana brings those metrics into infrastructure, storage, network, control-plane and workload views. I keep resource requests and limits separate from measured usage so the dashboards show both allocation and activity.
Alerts go to Discord. Dashboard exports and restoration notes are saved with the lab configuration so I can recreate the monitoring views.
Backing up application data
I keep cluster snapshots and application backups as separate layers. An etcd snapshot preserves cluster state; database contents and files on persistent volumes need their own backup path.
The cloud host runs a daily application backup for the task-management and monitoring services. It captures PostgreSQL custom-format dumps, application attachments and a manifest with checksums and version information.
Before publishing a backup, the database dumps are restored into disposable PostgreSQL instances. That checks that the archives can be read and restored. The temporary instances and their storage are removed afterwards.
The resulting archive is encrypted with age, then uploaded to a private Cloudflare R2 bucket using rclone:
1
2
3
4
5
6
7
8
9
Database dumps + application files
|
Isolated SQL restore check
|
Encrypted archive
|
R2 upload
|
Download and compare SHA-256
After upload, the encrypted object is read back and its checksum compared with the local archive. I keep 14 successful local generations and apply a 30-day remote expiration rule.
An hourly job retries an upload that hasn’t been verified yet. It reuses the daily archive rather than creating a new backup every hour. The scheduled workflow runs on the cloud host, so it doesn’t depend on my desktop being online.
Storage-usage alerts help me watch growth in the bucket. The recovery identity is stored separately from the encrypted archives.
Making recovery repeatable
Git holds the Kubernetes configuration, but rebuilding the lab also needs data, keys and the host-level backup jobs. I’m keeping those recovery steps documented alongside the configuration.
The next exercise is a complete isolated restore: decrypt an archive, restore the application, sign in and check that its attachments are present. That will extend the current SQL restore and remote-copy checks into an application recovery test.
Backup-freshness, private HTTPS and certificate-expiry checks are now active for the task board and document service. I still want an independent watchdog. Monitoring in the cloud helps with a home outage; a separate check is needed to alert if the cloud monitoring stack itself stops responding.
Where I’m taking it next
The foundation now covers deployment, private access, HTTPS, monitoring and encrypted off-node application backups.
My next priorities are:
- complete the end-to-end recovery exercise
- extend collection-staleness, backup-failure and independent monitoring alerts
- extend off-node protection for cluster snapshots and their recovery material
- extend recovery protection before adding more optional applications
The homepage portal now runs in the cloud cluster, and Paperless-ngx runs at home with private access, monitoring and daily encrypted backups. An isolated Paperless check verified login and a synthetic document download; restoring from a decrypted remote archive remains a separate test. Important workloads also have reviewed resource budgets and health probes, with a documented update and rollback procedure.
That is what I’m enjoying about this lab: each application gives me a reason to work on the deployment, monitoring and recovery around it.