Migrating apps onto the cluster meant cutting their public DNS over to the new ingress. From the outside, instant and clean — every name resolved to the new IP. From inside the cluster, two problems surfaced that, between them, cost a real outage. Writing both down because they're the kind of thing you only debug once if someone tells you first.

Problem 1: in-cluster DNS lags the cutover

I pointed the public records at the new ingress and watched the new certificates fail to issue. cert-manager's HTTP-01 flow includes an in-cluster self-check: before asking the CA to validate, it resolves the hostname from inside the cluster and confirms the challenge is reachable. That self-check kept resolving the old IP.

Why: in-cluster resolution goes through CoreDNS, which forwards to an upstream resolver — and upstream caches and propagation lag behind your authoritative change. Externally you've cut over; internally CoreDNS is still handing back the previous address for the length of the TTL (and then some). The self-check hits a dead endpoint and the cert never issues.

Problem 2: no hairpin to your own public IP anyway

Even once the old record expired, in-cluster clients pointed at the new public IP still couldn't reach the ingress. Many clouds don't provide NAT hairpin — a pod cannot reach its own cluster's public ingress IP by going "out and back in." From inside, the only address that actually works is the ingress's private/internal IP.

So both problems have the same answer: in-cluster clients should resolve these hostnames to the internal ingress address, not the public one.

The fix: a CoreDNS override pointing at the internal IP

k3s ships CoreDNS with a coredns-custom ConfigMap for exactly this. Resolve the affected hostnames to the ingress's internal address for in-cluster lookups, while the public DNS keeps serving the public IP to the outside world:

app-example.server: |
  app.example.com:53 {
    hosts {
      10.0.0.10 app.example.com
      fallthrough
    }
    forward . /etc/resolv.conf
  }

fallthrough means anything that isn't one of these names falls through to normal forwarding, so you only override the specific hostnames.

The mistake that turned a fix into an outage

The coredns-custom ConfigMap recognizes two kinds of keys, and the difference is not cosmetic:

  • <name>.server — adds a new, dedicated zone block. Self-contained.
  • <name>.override — appends its content into the main server block.

I first wrote it as .override. That appended a second hosts plugin into CoreDNS's main block — and you may only have one hosts plugin per block. Invalid config. CoreDNS refused to load and CrashLooped cluster-wide. Not "the override didn't work" — all DNS in the cluster went down, because the main block is what every normal lookup uses too.

The rule I now follow: anything that can't legally appear twice in a block (hosts, and friends) must go in a .server zone block, never .override. .override is only safe for directives that are additive to the main block. Switching the key from .override to .server fixed both the outage and the original cutover problem in one move.

Takeaways

  1. In-cluster DNS does not cut over when public DNS does. It trails by the upstream cache/TTL. If an in-cluster self-check (cert-manager HTTP-01, health probes, service-to-service calls) breaks right after a cutover, suspect stale internal resolution first.
  2. Don't assume hairpin NAT exists. On many clouds a pod can't reach its own public ingress IP. Point in-cluster clients at the internal address deliberately, via CoreDNS — don't wait for propagation to "fix" it.
  3. .server vs .override is a sharp edge. .override mutates the main CoreDNS block; duplicating a single-instance plugin like hosts there takes down all cluster DNS. Use a .server zone block for anything self-contained.

Two subtle networking facts and one config footgun, stacked. Each is obvious in hindsight; together they made "I just changed a DNS record" cost an afternoon.