How and why I built a portable HA k3s cluster on a free-tier cloud

The other posts here are war stories — single bugs, dug out one at a time. This one is the map they hang on: what the cluster actually is, and why each piece is the way it is. If you (or the agent reading on your behalf) are about to build something similar, start here, then follow the links into the details.

The whole design falls out of three constraints, so I'll state them first because every decision below is downstream of them:

1. It's a learning vehicle. The point is to understand the machinery, not to rent it by the click. That biases every choice toward "do it by hand in code" over "click the managed button."
2. The stakes are low. Everything running on it is self-used or friends-and-family. Nobody is paying. So "what if the cloud reclaims it" turns from a business risk into a feature to practise — rehearse the rebuild.
3. The budget is ~zero. It targets a free-tier ARM allocation. That ceiling (cores, RAM, and a fixed storage pool) shapes the node count, the storage choice, and why it's k3s and not something heavier.

Why k3s, and why three servers

k3s over managed Kubernetes: a managed control plane hides exactly the parts I wanted to learn, and costs money the budget doesn't have. k3s over kubeadm: it's lightweight, arm-native, and fits free-tier RAM — kubeadm's footprint fights the ceiling for no learning gain at this size.

Three server nodes with embedded etcd, not one. A single node is simpler, but control-plane HA is one of the main things I wanted to learn, and the free pool is just big enough to carve three small servers (one per availability domain) with etcd quorum. The honest scope line: stateless apps reschedule cheaply, so HA there is free; a HA database is genuinely advanced, so stateful services run single-replica and lean on backups. Chasing stateful HA in a learning lab is where you burn weeks for little.

In front of the three servers sits a small L4 load balancer (HAProxy on a tiny node) so there's a single API/ingress entrypoint that doesn't pin to one server. It's currently a single box — a known SPOF, deliberately deferred; HA of the LB itself is a later exercise.

Two of the nastiest surprises bringing this up are their own posts: the API server advertised a public IP and the cloud image's firewall silently killed pod networking.

The IaC split: provision / configure / deploy / reconcile

Four layers, each with one job, all living in one git repo as the source of truth:

The split is the point. OpenTofu owns what exists; Ansible owns what's on the box; manifests own what runs; Argo CD owns staying that way. The decision that made this tractable: secrets never live in git — the cluster token is generated on the primary at bootstrap and handed to the joiners in memory; app credentials are Kubernetes Secrets created out of band. Git holds the shape, not the keys.

Two ingress planes — and why the dashboards aren't public

This is the design choice I'm happiest with. There are two ways into the cluster, and which one an app gets depends on who it's for:

• Public plane — Traefik (the k3s default) + cert-manager + Let's Encrypt, with the L4 LB out front and a wildcard subdomain pointing at it. TLS is per-host via the HTTP-01 challenge solved through Traefik — no DNS-01, no DNS API token to hold. This is for things genuinely meant for the internet (the blog you're reading).
• Private plane — a Tailscale ingress class. Anything I apply with it becomes reachable only on my tailnet, never on the public internet, with no port open at the edge.

Why bother with two? Because the admin tooling has weak or no authentication, and the safest auth is not being reachable. The cluster dashboards — the Kubernetes GUI, the storage UI, the GitOps UI, the uptime dashboard — all go on the private plane. The storage UI in particular ships with no login at all; tailnet-only is its entire security model. Putting an unauthenticated admin panel on the public internet behind a "nobody will find it" URL is exactly the mistake this avoids.

The cutover from old infra onto the public plane had its own trap, worth reading before you migrate anything stateful with TLS: the cluster kept resolving the old IP and one config slip took out all DNS.

Storage: replicated where it must be, free where it can be

Two tiers, chosen against that fixed storage pool:

• Longhorn for anything that needs to survive a node dying — it replicates volumes across nodes so a stateful pod can reschedule with its data. That's the whole reason it's here: single-replica DBs plus replicated storage is a reasonable durability story without chasing database-level HA.
• Node-local space for anything reproducible or disposable, because the free storage pool is capped and external block volumes pile billable storage on top of a pool that's already maxed (the free-tier storage floor is its own surprise).

Longhorn also taught me that its "used" gauge is not filesystem usage — a story about thin provisioning and TRIM.

Observability, alerting, identity, failover

• Observability: I ran kube-prometheus-stack for metrics, with alerts pushed to a notification topic so the cluster could page me without an in-cluster notifier to babysit — and learned the hard way how fast a tiny cluster fills a metrics volume (its own post). That cost is exactly why I tore the whole stack out for a leaner Beszel + Gatus setup; full Prometheus is more observability than a friends-and-family cluster needs to carry.
• Identity: a single self-hosted IdP (Zitadel) as the one place accounts live, rather than per-app logins scattered around.
• Failover: a documented manual runbook, not automation — provision the standby with OpenTofu against a second cloud, repoint DNS. For friends-and-family stakes, manual is the right scope; auto-failover is a lot of machinery for no payers. The stability model is fast designed recovery, not "it never fails."

The bill, and the north star

The point of all this is to collapse a handful of paid VMs down to a free-tier cluster plus cheap object storage for backups — landing the running cost near the storage floor, a euro or so a month. The rule that keeps it there: only ever provision free-tier shapes, and set a near-zero budget alert as a tripwire, because the free tier has no hard spend cap.

If you're building this too

The decisions that mattered most, distilled:

1. Let your constraints pick your architecture. Free-tier + low-stakes + learning is what justifies k3s, manual failover, single-replica DBs, and doing it all in code. Different constraints, different cluster.
2. One git repo, four layers, no secrets in it. Provision / configure / deploy / reconcile each own one thing; keys are injected, never committed.
3. Default admin tooling to "not reachable." A private network plane for anything with weak auth beats a public URL you hope nobody guesses.
4. Replicate storage only where data can't be regenerated. Everything else runs on disposable local space — especially when the storage pool is capped.

Everything above is the why. The linked posts are the what broke and how I fixed it. Together they're the version of this project I wish I'd been handed on day one.