This was the single worst bug of the whole build, so it goes first for the next person — or the next agent — standing up HA k3s on dual-homed cloud VMs.

The cluster came up. Three server nodes, all Ready, etcd healthy, kubectl get nodes clean. And then nothing that needed the API from inside the cluster worked. CoreDNS CrashLooping. Service accounts failing. Controllers timing out talking to kubernetes.default. The control plane was fine; the cluster's own clients couldn't reach it.

The symptom, precisely

Every cloud VM here has two addresses: a public IP on the internet-facing NIC and a private IP on the internal virtual network (the 10.x range the nodes share). Pods route to the API server through the in-cluster Service kubernetes.default — a ClusterIP whose Endpoints are the real addresses of the API servers.

I looked at those Endpoints. They were the nodes' public IPs.

That is the whole bug. Pods live on the pod/private network. The cloud's security rules and the lack of NAT hairpin mean a pod cannot reach a node's public IP from inside. So every in-cluster client was being told "the API server is at <public-ip>:6443" — an address it has no route to. External kubectl worked (it comes in from the internet); in-cluster traffic died.

Why k3s does this

By default k3s picks its advertise address from the node's default route. On a dual-homed cloud node the default route goes out the public NIC, so k3s advertises the public IP — and that address is what lands in the kubernetes Service Endpoints that every pod uses. On a single-homed box you'd never see it. On a cloud VM with a public NIC it's the default, and it's wrong for in-cluster traffic.

The fix: pin every address explicitly

Don't let k3s guess. Spell out which address is for what. On both the cluster-init node and every joining server:

--node-ip            <private-ip>   # what this node IS on the cluster network
--advertise-address  <private-ip>   # what goes into the kubernetes Service — THE fix
--node-external-ip   <public-ip>    # public reachability, kept out of in-cluster routing
--tls-san            <public-ip>    # so the API cert is valid for external kubectl too

The decision in one line: private IP for everything in-cluster (--node-ip, --advertise-address); public IP only for things that face outward (--node-external-ip, --tls-san). After re-installing with those flags, the kubernetes Endpoints showed the 10.x addresses, CoreDNS settled, and the controllers reconnected.

Takeaways for anyone (or anything) building this

  1. On any multi-homed cloud node, never let the CNI/distro auto-detect the advertise address. The default route is the public NIC, and the public IP is exactly the wrong thing to put in the kubernetes Service.
  2. The tell is the Endpoints, not the node status. Nodes go Ready because the kubelet reaches the API fine over localhost/public. The breakage is purely in pod → API routing. Check what kubernetes.default's Endpoints actually resolve to.
  3. Separate the two jobs of an IP. Internal identity (node-ip, advertise-address) and external reachability (node-external-ip, tls-san) are different concerns. Conflating them is what bites you.

If your freshly-built cluster is green but its own pods can't talk to the API, this is almost certainly it. Look at the Endpoints first.