dune.at / notes

The cloud image shipped an iptables rule that silently kills all pod networking

2026-06-04T00:00:00+00:00

Companion to the API-address bug: same cluster, same morning, different way for a "healthy" node to have completely broken pod networking. This one is the stock cloud OS image fighting your CNI, and it's invisible until pods try to talk to each other.

The symptom</h2>
k3s installed cleanly. Nodes `Ready</code>. But pods couldn't reach Services, DNS lookups timed out, and anything multi-pod CrashLooped. Node-level networking (SSH, the API on the host) was perfectly fine. Only forwarded traffic — pod-to-pod, pod-to-Service — was dead.`

The cause: a default-REJECT host firewall</h2>
The cloud's Ubuntu image ships a preconfigured host firewall. The FORWARD</code>
chain ends in a single blanket rule:</p>
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
</code></pre>
Here's why that's fatal to Kubernetes specifically. Every pod-to-pod and
pod-to-Service packet is routed</strong>, not delivered locally — so it traverses the
FORWARD</code> chain. A CNI like flannel assumes it owns FORWARD</code> and inserts its own
ACCEPT rules. But the image's blanket REJECT is already there, and on a fresh
boot it wins: each forwarded packet hits "host prohibited" and is dropped. The
node looks healthy because nothing the host</em> does is forwarded; only the pods
suffer.</p>
The fix</h2>
Two parts, and the reasoning behind each matters more than the commands:</p>
1. Remove the blanket FORWARD REJECT</code>.</strong> Let k3s / flannel / kube-proxy manage
the FORWARD</code> chain, which is what they expect to do:</p>
# idempotent — only acts if the rule is present
iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited
</code></pre>
2. Accept intra-cluster traffic on INPUT</code></strong> so node-to-node control traffic
isn't blocked either — the internal VCN range, the pod CIDR, the service CIDR,
and flannel's VXLAN port:</p>
iptables -I INPUT -s <vcn-cidr>      -j ACCEPT   # node↔node: etcd, kubelet, …
iptables -I INPUT -s <pod-cidr>      -j ACCEPT   # pod network
iptables -I INPUT -s <service-cidr>  -j ACCEPT   # service network
iptables -I INPUT -p udp --dport 8472 -j ACCEPT  # flannel VXLAN
</code></pre>
Then persist (netfilter-persistent save</code>) so a reboot doesn't reinstate the
breakage.</p>
The decision behind it: don't run two firewalls</h2>
It's tempting to keep the host firewall and</em> the cloud one. Don't. The cloud
provider already gives you a network security layer (security lists / security
groups) at the VCN edge — that is your real perimeter, and it's the one you can
reason about centrally. The host iptables rules the image ships are redundant
with it and actively conflict with the CNI. Keep the cloud security list as the
perimeter; strip the host rules that collide with Kubernetes.</strong> One firewall, in
one place, that the CNI is allowed to manage.</p>
Takeaways</h2>

A green node says nothing about pod networking.</strong> Node Ready</code> only proves
the kubelet is happy. If multi-pod workloads fail, check iptables -L FORWARD</code>
before you touch the CNI config.</li>
Default-REJECT host firewalls and CNIs are incompatible by construction.</strong>
Any image (cloud or otherwise) that ends FORWARD</code> in REJECT will break
forwarding the moment a CNI relies on it. This isn't Oracle-specific; it's
"preconfigured host firewall meets Kubernetes."</li>
Let the cloud layer be the firewall.</strong> Centralize the perimeter at the
provider's security list and let kube-proxy/flannel own the host chains.</li>
</ol>
The frustrating part is how healthy</em> everything looks. The fix is one deleted
rule — but only once you know to suspect the OS image rather than your manifests.</p>

After a DNS cutover, my cluster kept resolving the old IP — and one config slip took out all DNS 2026-06-04T00:00:00+00:00 Migrating apps onto the cluster meant cutting their public DNS over to the new ingress. From the outside, instant and clean — every name resolved to the new IP. From inside the cluster, two problems surfaced that, between them, cost a real outage. Writing both down because they're the kind of thing you only debug once if someone tells you first. Problem 1: in-cluster DNS lags the cutover</h2> I pointed the public records at the new ingress and watched the new certificates fail to issue. cert-manager's HTTP-01 flow includes an in-cluster self-check: before asking the CA to validate, it resolves the hostname from inside the cluster and confirms the challenge is reachable. That self-check kept resolving the old IP. Why: in-cluster resolution goes through CoreDNS, which forwards to an upstream resolver — and upstream caches and propagation lag behind your authoritative change. Externally you've cut over; internally CoreDNS is still handing back the previous address for the length of the TTL (and then some). The self-check hits a dead endpoint and the cert never issues. Problem 2: no hairpin to your own public IP anyway</h2> Even once the old record expired, in-cluster clients pointed at the new public IP still couldn't reach the ingress. Many clouds don't provide NAT hairpin — a pod cannot reach its own cluster's public ingress IP by going "out and back in." From inside, the only address that actually works is the ingress's private/internal IP. So both problems have the same answer: in-cluster clients should resolve these hostnames to the internal ingress address, not the public one. The fix: a CoreDNS override pointing at the internal IP</h2> k3s ships CoreDNS with a coredns-custom</code> ConfigMap for exactly this. Resolve the affected hostnames to the ingress's internal address for in-cluster lookups, while the public DNS keeps serving the public IP to the outside world: app-example.server: | app.example.com:53 { hosts { 10.0.0.10 app.example.com fallthrough } forward . /etc/resolv.conf } </code></pre> fallthrough</code> means anything that isn't one of these names falls through to normal forwarding, so you only override the specific hostnames. The mistake that turned a fix into an outage</h2> The coredns-custom</code> ConfigMap recognizes two kinds of keys, and the difference is not cosmetic: <name>.server</code> — adds a new, dedicated zone block. Self-contained.</li> <name>.override</code> — appends its content into the main server block.</li> </ul> I first wrote it as .override</code>. That appended a second hosts</code> plugin into CoreDNS's main block — and you may only have one hosts</code> plugin per block. Invalid config. CoreDNS refused to load and CrashLooped cluster-wide. Not "the override didn't work" — all DNS in the cluster went down, because the main block is what every normal lookup uses too. The rule I now follow: anything that can't legally appear twice in a block (hosts</code>, and friends) must go in a .server</code> zone block, never .override</code>. .override</code> is only safe for directives that are additive to the main block. Switching the key from .override</code> to .server</code> fixed both the outage and the original cutover problem in one move. Takeaways</h2> In-cluster DNS does not cut over when public DNS does. It trails by the upstream cache/TTL. If an in-cluster self-check (cert-manager HTTP-01, health probes, service-to-service calls) breaks right after a cutover, suspect stale internal resolution first.</li> Don't assume hairpin NAT exists. On many clouds a pod can't reach its own public ingress IP. Point in-cluster clients at the internal address deliberately, via CoreDNS — don't wait for propagation to "fix" it.</li> .server</code> vs .override</code> is a sharp edge. .override</code> mutates the main CoreDNS block; duplicating a single-instance plugin like hosts</code> there takes down all cluster DNS. Use a .server</code> zone block for anything self-contained.</li> </ol> Two subtle networking facts and one config footgun, stacked. Each is obvious in hindsight; together they made "I just changed a DNS record" cost an afternoon. How and why I built a portable HA k3s cluster on a free-tier cloud 2026-06-04T00:00:00+00:00 The other posts here are war stories — single bugs, dug out one at a time. This one is the map they hang on: what the cluster actually is, and why each piece is the way it is. If you (or the agent reading on your behalf) are about to build something similar, start here, then follow the links into the details. The whole design falls out of three constraints, so I'll state them first because every decision below is downstream of them: It's a learning vehicle. The point is to understand the machinery, not to rent it by the click. That biases every choice toward "do it by hand in code" over "click the managed button."</li> The stakes are low. Everything running on it is self-used or friends-and-family. Nobody is paying. So "what if the cloud reclaims it" turns from a business risk into a feature to practise — rehearse the rebuild.</li> The budget is ~zero. It targets a free-tier ARM allocation. That ceiling (cores, RAM, and a fixed storage pool) shapes the node count, the storage choice, and why it's k3s and not something heavier.</li> </ol> Why k3s, and why three servers</h2> k3s over managed Kubernetes: a managed control plane hides exactly the parts I wanted to learn, and costs money the budget doesn't have. k3s over kubeadm: it's lightweight, arm-native, and fits free-tier RAM — kubeadm's footprint fights the ceiling for no learning gain at this size. Three server nodes with embedded etcd, not one. A single node is simpler, but control-plane HA is one of the main things I wanted to learn, and the free pool is just big enough to carve three small servers (one per availability domain) with etcd quorum. The honest scope line: stateless apps reschedule cheaply, so HA there is free; a HA database is genuinely advanced, so stateful services run single-replica and lean on backups. Chasing stateful HA in a learning lab is where you burn weeks for little. In front of the three servers sits a small L4 load balancer (HAProxy on a tiny node) so there's a single API/ingress entrypoint that doesn't pin to one server. It's currently a single box — a known SPOF, deliberately deferred; HA of the LB itself is a later exercise. Two of the nastiest surprises bringing this up are their own posts: the API server advertised a public IP</a> and the cloud image's firewall silently killed pod networking</a>. The IaC split: provision / configure / deploy / reconcile</h2> Four layers, each with one job, all living in one git repo as the source of truth: Layer</th> Tool</th> Job</th></tr></thead> Provision</td> OpenTofu</td> create the VMs, network, firewall rules</td></tr> Configure</td> Ansible</td> OS prep, host firewall, install k3s, bootstrap HA</td></tr> Deploy</td> k8s manifests</td> the apps and platform add-ons</td></tr> Reconcile</td> Argo CD</td> keep the cluster matching git (GitOps)</td></tr> </tbody></table> The split is the point. OpenTofu owns what exists; Ansible owns what's on the box; manifests own what runs; Argo CD owns staying that way. The decision that made this tractable: secrets never live in git — the cluster token is generated on the primary at bootstrap and handed to the joiners in memory; app credentials are Kubernetes Secrets created out of band. Git holds the shape, not the keys. Two ingress planes — and why the dashboards aren't public</h2> This is the design choice I'm happiest with. There are two ways into the cluster, and which one an app gets depends on who it's for: Public plane — Traefik (the k3s default) + cert-manager + Let's Encrypt, with the L4 LB out front and a wildcard subdomain pointing at it. TLS is per-host via the HTTP-01 challenge solved through Traefik — no DNS-01, no DNS API token to hold. This is for things genuinely meant for the internet (the blog you're reading).</li> Private plane — a Tailscale ingress class. Anything I apply with it becomes reachable only on my tailnet, never on the public internet, with no port open at the edge.</li> </ul> Why bother with two? Because the admin tooling has weak or no authentication, and the safest auth is not being reachable. The cluster dashboards — the Kubernetes GUI, the storage UI, the GitOps UI, the uptime dashboard — all go on the private plane. The storage UI in particular ships with no login at all; tailnet-only is its entire security model. Putting an unauthenticated admin panel on the public internet behind a "nobody will find it" URL is exactly the mistake this avoids. The cutover from old infra onto the public plane had its own trap, worth reading before you migrate anything stateful with TLS: the cluster kept resolving the old IP and one config slip took out all DNS</a>. Storage: replicated where it must be, free where it can be</h2> Two tiers, chosen against that fixed storage pool: Longhorn for anything that needs to survive a node dying — it replicates volumes across nodes so a stateful pod can reschedule with its data. That's the whole reason it's here: single-replica DBs plus replicated storage is a reasonable durability story without chasing database-level HA.</li> Node-local space for anything reproducible or disposable, because the free storage pool is capped and external block volumes pile billable storage on top of a pool that's already maxed (the free-tier storage floor</a> is its own surprise).</li> </ul> Longhorn also taught me that its "used" gauge is not filesystem usage — a story about thin provisioning and TRIM</a>. Observability, alerting, identity, failover</h2> Observability: I ran kube-prometheus-stack for metrics, with alerts pushed to a notification topic so the cluster could page me without an in-cluster notifier to babysit — and learned the hard way how fast a tiny cluster fills a metrics volume (its own post</a>). That cost is exactly why I tore the whole stack out for a leaner Beszel + Gatus setup</a>; full Prometheus is more observability than a friends-and-family cluster needs to carry.</li> Identity: a single self-hosted IdP (Zitadel) as the one place accounts live, rather than per-app logins scattered around.</li> Failover: a documented manual runbook, not automation — provision the standby with OpenTofu against a second cloud, repoint DNS. For friends-and-family stakes, manual is the right scope; auto-failover is a lot of machinery for no payers. The stability model is fast designed recovery, not "it never fails."</li> </ul> The bill, and the north star</h2> The point of all this is to collapse a handful of paid VMs down to a free-tier cluster plus cheap object storage for backups — landing the running cost near the storage floor, a euro or so a month. The rule that keeps it there: only ever provision free-tier shapes, and set a near-zero budget alert as a tripwire, because the free tier has no hard spend cap. If you're building this too</h2> The decisions that mattered most, distilled: Let your constraints pick your architecture. Free-tier + low-stakes + learning is what justifies k3s, manual failover, single-replica DBs, and doing it all in code. Different constraints, different cluster.</li> One git repo, four layers, no secrets in it. Provision / configure / deploy / reconcile each own one thing; keys are injected, never committed.</li> Default admin tooling to "not reachable." A private network plane for anything with weak auth beats a public URL you hope nobody guesses.</li> Replicate storage only where data can't be regenerated. Everything else runs on disposable local space — especially when the storage pool is capped.</li> </ol> Everything above is the why. The linked posts are the what broke and how I fixed it. Together they're the version of this project I wish I'd been handed on day one. I built the full observability stack, verified it, then deleted it 2026-06-04T00:00:00+00:00 A few days ago I stood up the textbook self-hosted observability stack on my k3s cluster</a>: Prometheus for metrics, Grafana for dashboards, Loki for logs, alerts wired to push notifications. It worked. Dashboards rendered, a test alert hit my phone, I marked the issue done. Then I tore the whole thing out and replaced it with two small tools. This is the reasoning, because "I built the impressive thing and then deleted it" is a more useful story than most build logs. What pushed me over</h2> Two days after it went live, the Prometheus volume filled and ingestion stopped — the cluster went quietly blind right when I'd have wanted it most. That incident is its own post</a>; the short version is that the stack was the heaviest tenant on the cluster, running on CPU-bound single-core nodes, and its appetite was the direct cause of the outage. Two days of firefighting to keep the monitoring from being the thing that needed monitoring. Then the honest question: how much of this depth do I actually use? Almost none. I barely opened the deep metric explorers; I'd never written a serious Grafana query against my own data. I was paying — in CPU, in disk, in incidents — for capability I didn't touch. That fails the only test that matters in a self-hosted lab: don't run what you don't use. The replacement: two engines, not a platform</h2> I split "monitoring" back into the two questions it actually answers, and picked the lightest tool for each: Resource vitals — CPU, memory, disk, temperature, per host and container — handled by Beszel: a lightweight agent on each host reporting to a small hub. Node-level, cheap, always-on.</li> Availability + heartbeats — Gatus: a black-box prober that hits the real URLs of the things I care about and pages me when one stops answering, plus dead-man checks for jobs that should run on a schedule.</li> </ul> Delivery for both is a single push channel — ntfy — one templated line per alert: what broke, and where. No dashboards-as-a-product, no query language, no log warehouse. If I need deep diagnosis, I open the cluster GUI or reach for kubectl</code> on demand — I don't keep a metrics firehose running 24/7 on the off chance. The insight that reshaped it: a monitor can't certify itself</h2> This is the part worth stealing. My in-cluster stack could go falsely green — and on the incident day, an in-cluster heartbeat would have, because the thing checking was inside the thing that was failing. A monitor that lives with what it monitors can't tell you the patient is dead; it dies with it. So the availability prober runs off the cluster, on a different provider, and checks the public URLs from outside — the same path a real user takes. That black-box, outside-in check is the actual fix for the blind spot, not a bigger in-cluster stack. I run two such watchers in different places and have them cross-watch each other, with a free external dead-man service as the apex backstop for the rare case where both are down at once. Pull-first, with exactly one exception</h2> A principle that fell out of this: prefer pulling over pushing. A watcher that reaches out to probe needs no inbound hole in anything it watches — it fits a "no open ports" posture by construction. The only push in the design is backup heartbeats: a cron job has no endpoint to probe, so it has to announce "I ran." Those ride a private network, and the semantics are simply missed = noticed. One deliberate exception, justified; everything else pulls. What I gave up, on purpose</h2> Honesty about the trade-off, because it's real: I lost per-pod metrics inside the cluster. The lightweight agent sees the node, not each container, on this runtime. I decided that's acceptable: user-facing health is covered by the outside-in app probes;</li> node pressure is covered by the vitals agent;</li> deep, occasional diagnosis is on-demand via the cluster GUI / kubectl</code>.</li> </ul> And it's recoverable: if I ever genuinely want long-term cluster metrics back, the move is a hosted free-tier metrics backend (Grafana Cloud's free tier) fed by a curated, low-cardinality remote-write — the lesson from the incident baked in — not a full self-hosted stack returning to eat the cluster again. Deferred until I actually want it. Things I considered and rejected</h2> Netdata (the one tool that does see per-pod via cgroups): ruled out — too heavy for single-core nodes, and it had nearly taken down another box of mine before. The per-pod visibility wasn't worth the weight for something I rarely use.</li> Folding host vitals into the cluster Prometheus (my own first instinct): rejected — it keeps the heavy stack and couples the host vitals to the very cluster that goes blind when it's sick.</li> A second alerting brain off-cluster with alert inhibition: cut. It adds a public silence API (another hole) and, worse, inhibition can silently swallow a real page. Plain grouping, severity, and long repeat intervals get the "don't spam me" outcome without a rule that can eat an alert you needed.</li> </ul> Takeaways</h2> Match the monitoring to what you'll actually look at. Impressive ≠ useful. The heavy stack failed the "do I use this" test; deleting it was the upgrade.</li> A monitor can't be inside what it monitors. Outside-in, black-box probes from a different failure domain are the only checks that don't go falsely green.</li> Pull by default; justify every push. Probing needs no inbound holes. The one push here (job heartbeats) earns its exception because cron has no endpoint.</li> Name your trade-off out loud. I gave up per-pod metrics knowingly, with a documented path to get them back cheaply if I ever care. That's a decision, not an accident.</li> </ol> The fanciest version of this cluster ran a full observability platform. The better version runs two small tools and a notification topic — and is no longer at risk of being knocked over by its own monitoring. k3s advertised a public IP for the API server, and every pod lost the cluster 2026-06-04T00:00:00+00:00 This was the single worst bug of the whole build, so it goes first for the next person — or the next agent — standing up HA k3s on dual-homed cloud VMs. The cluster came up. Three server nodes, all Ready</code>, etcd healthy, kubectl get nodes</code> clean. And then nothing that needed the API from inside the cluster worked. CoreDNS CrashLooping. Service accounts failing. Controllers timing out talking to kubernetes.default</code>. The control plane was fine; the cluster's own clients couldn't reach it. The symptom, precisely</h2> Every cloud VM here has two addresses: a public IP on the internet-facing NIC and a private IP on the internal virtual network (the 10.x</code> range the nodes share). Pods route to the API server through the in-cluster Service kubernetes.default</code> — a ClusterIP whose Endpoints are the real addresses of the API servers. I looked at those Endpoints. They were the nodes' public IPs. That is the whole bug. Pods live on the pod/private network. The cloud's security rules and the lack of NAT hairpin mean a pod cannot reach a node's public IP from inside. So every in-cluster client was being told "the API server is at <public-ip>:6443</code>" — an address it has no route to. External kubectl</code> worked (it comes in from the internet); in-cluster traffic died. Why k3s does this</h2> By default k3s picks its advertise address from the node's default route. On a dual-homed cloud node the default route goes out the public NIC, so k3s advertises the public IP — and that address is what lands in the kubernetes</code> Service Endpoints that every pod uses. On a single-homed box you'd never see it. On a cloud VM with a public NIC it's the default, and it's wrong for in-cluster traffic. The fix: pin every address explicitly</h2> Don't let k3s guess. Spell out which address is for what. On both the cluster-init</code> node and every joining server: --node-ip <private-ip> # what this node IS on the cluster network --advertise-address <private-ip> # what goes into the kubernetes Service — THE fix --node-external-ip <public-ip> # public reachability, kept out of in-cluster routing --tls-san <public-ip> # so the API cert is valid for external kubectl too </code></pre> The decision in one line: private IP for everything in-cluster (--node-ip</code>, --advertise-address</code>); public IP only for things that face outward (--node-external-ip</code>, --tls-san</code>). After re-installing with those flags, the kubernetes</code> Endpoints showed the 10.x</code> addresses, CoreDNS settled, and the controllers reconnected. Takeaways for anyone (or anything) building this</h2> On any multi-homed cloud node, never let the CNI/distro auto-detect the advertise address. The default route is the public NIC, and the public IP is exactly the wrong thing to put in the kubernetes</code> Service.</li> The tell is the Endpoints, not the node status. Nodes go Ready</code> because the kubelet reaches the API fine over localhost/public. The breakage is purely in pod → API routing. Check what kubernetes.default</code>'s Endpoints actually resolve to.</li> Separate the two jobs of an IP. Internal identity (node-ip</code>, advertise-address</code>) and external reachability (node-external-ip</code>, tls-san</code>) are different concerns. Conflating them is what bites you.</li> </ol> If your freshly-built cluster is green but its own pods can't talk to the API, this is almost certainly it. Look at the Endpoints first. Longhorn said 3 GB used. The filesystem said 0.9. Nobody was lying. 2026-06-04T00:00:00+00:00 Yesterday I fixed a real disk-full on my cluster's Prometheus volume — expanded it to 3 GB, trimmed the metrics flooding it. Today I opened the Longhorn UI and saw the volume at 2.97 GB of 3 GB used again. Less than a day later. Stomach drop. It wasn't full. Here's the gotcha, because it'll catch anyone running thin-provisioned storage. actualSize is not disk usage</h2> The number Longhorn shows you — actualSize</code> — is how many blocks the volume's replicas have touched on disk, including Longhorn's own internal snapshots. It is not how full the filesystem is. I checked what Prometheus actually had on the filesystem: blocks (persisted): 619 MB write-ahead log: 233 MB head chunks: 27 MB ----------------------------- real data: ~880 MB of a 3 GB volume (29%) </code></pre> 29%. The pod had 24 hours of uptime, zero restarts, no write errors. Nothing was wrong with the filesystem at all. So where did 2.97 GB come from? Two hoarders</h2> A leftover snapshot. When you expand a Longhorn volume it auto-creates a system snapshot (expand-<size></code>). Mine was ~1 GB and Longhorn hadn't coalesced it — it just sat in the replica chain, counting against actualSize</code>. Thin-provisioning lag. This is the one worth internalising. When Prometheus deletes data — old blocks aged out by retention, the high-cardinality series I trimmed yesterday — the filesystem marks those blocks free. But the block layer underneath doesn't get told. Those blocks stay "allocated" as far as Longhorn is concerned until something issues a TRIM (fstrim</code>) to hand them back. Another ~1 GB of "used" that was actually free. So: ~880 MB real data + ~1 GB stale snapshot + ~1 GB un-trimmed free blocks ≈ the 2.97 GB that scared me. None of it filesystem pressure. The fix is to take out the trash, not buy a bigger bin</h2> Delete the leftover expand snapshot.</li> Turn on Longhorn's "remove snapshots during filesystem trim" so a trim also coalesces removed snapshots in one pass.</li> Run a filesystem trim.</li> </ol> actualSize: 3.19 GB → 0.98 GB </code></pre> One pass, ~2.2 GB reclaimed, Prometheus didn't even notice (online trim, no restart). 0.98 GB matches the real data — the hoarders are gone. Then the part that matters more than the cleanup: a weekly Longhorn RecurringJob with task filesystem-trim</code>, so freed blocks get released automatically and I never watch this gauge again. The takeaway for thin-provisioned storage</h2> When your storage layer screams "nearly full," check the filesystem before you add disk: actualSize</code> (or equivalent) ≠ filesystem usage. It counts snapshots and un-reclaimed blocks. Look at what the workload actually wrote.</li> Thin provisioning doesn't shrink on its own. Deleting data frees it for the filesystem, not for the block layer. Schedule periodic fstrim</code> — or your "usage" only ever goes up.</li> Volume operations leave snapshots. Expansions and rebuilds drop system snapshots that linger unless you (or a job) clean them.</li> </ul> Yesterday's disk-full was real and I gave it more disk. Today's was the storage layer hoarding free space — and the answer was a trim job, not a bigger volume. Worth knowing which one you're looking at before you reach for the disk slider. A static blog on Kubernetes with no registry, no database, and no persistent volume 2026-06-04T00:00:00+00:00 This is the post about the site you're reading. I wanted to publish a static blog on the cluster without three things the "obvious" path drags in: a container registry to push a custom image to, a CI pipeline to build that image, and a persistent volume to hold the rendered site. None of them are necessary when the content is fully reproducible from git. Here's the shape, and why each piece is missing on purpose. The idea: the pod builds itself at startup</h2> There is no custom image. The running Pod assembles the site from git every time it starts, using a chain of initContainers over a shared scratch volume, then a stock web server serves the result: clone — a stock git</code> image shallow-clones the repo into an emptyDir</code>.</li> build — a stock static-site-generator image renders the site, in place, inside that clone.</li> copy — a tiny image copies the rendered output into a second emptyDir</code>.</li> serve — a stock nginx</code> image serves that second volume, read-only.</li> </ol> initContainers: git-clone → ssg-build → copy-output └──── emptyDir: src ────┘ │ emptyDir: site container: nginx (mounts emptyDir: site, readOnly) </code></pre> Publishing is just git push</code>. CI does one thing: kubectl rollout restart</code> the Deployment. New Pods come up, re-run the init chain, and serve the latest commit. The "build" happens in the Pod, at start, from upstream images. Why each thing is absent — the actual decisions</h2> No registry. You only need a registry if you're baking a custom image. Here every image is stock and upstream (git</code>, the SSG, nginx</code>); the only thing that varies — your content — is injected at runtime by cloning it. Nothing to build, nothing to push, nothing to store, nothing to keep patched yourself. No persistent volume. emptyDir</code> is exactly right when the data is disposable and reproducible. The site is regenerated from git on every start, so there is nothing worth persisting. The Pod is genuinely stateless: kill it, it rebuilds identically. PVs exist to keep data a Pod can't regenerate; this Pod can regenerate everything. No CI image build. Because the build runs in the Pod, CI has no image step. It triggers a rollout and stops. The cluster, not the CI runner, is the build host — which also means the build environment is the same upstream image every time, pinned by tag. Gotchas worth keeping</h2> Build into the SSG's default output dir, not a mounted path. Some generators wipe the output directory before writing (a --force</code>-style clean). If that directory is a mount point, the wipe fails. So build inside the cloned tree and copy</code> the result into the served volume as a separate step — that's why there are two emptyDir</code>s and a copy stage, not one shared mount.</li> Replicas each rebuild independently. With two replicas and a rolling restart, each Pod clones and builds on its own. There's a few seconds of version skew mid-rollout. For a blog that's fine; for anything transactional it wouldn't be.</li> Private repo = read-only token in a Secret. The clone uses a scoped, read-only credential pulled from a Kubernetes Secret. No write access, nothing baked into an image.</li> </ul> The honest trade-offs</h2> This is the right tool only when the build is cheap and the content is git-native. What you pay for the simplicity: Slower Pod start — every start pays clone + build, instead of pulling a prebuilt image.</li> No build gate — a broken commit produces a broken build inside the Pod. If that matters, also build in CI (purely as a validation step) or gate the rollout on a readiness probe so a failed build never serves.</li> Doesn't scale to heavy builds or huge sites — a multi-minute build on every Pod start is the wrong place to be; that's when a real image pipeline earns its keep.</li> </ul> Takeaway</h2> When your artifact is fully reproducible from git, you can collapse the whole registry + PV + CI-image stack into stock images + initContainers + rollout restart</code>. The Pod becomes the build host and git becomes the only source of truth. It's not the answer for every workload — but for a static site it removes three moving parts you'd otherwise own, patch, and debug forever. My 3-node lab filled 1GB in hours. My old monitoring box would've taken years. 2026-06-03T00:00:00+00:00 I come from monitoring, not observability. Fifteen years of Nagios, then Zabbix, then check_mk. In that world a gigabyte of disk is a lot. I've run boxes that watched a few hundred hosts and didn't fill 1GB in a year. So when I gave Prometheus a 1GB volume on my three-node k3s lab, it felt generous. It was empty by lunchtime. Not "getting full" — full, write errors, ingestion stopped, my brand-new alerting blind. Hours, not years. Here's what I learned pulling it apart, written for the version of me who still thinks in checks. The old world: you store the answers</h2> In monitoring, you write the questions up front. "Is disk over 90%?" "Is the service responding?" "Is latency above 200ms?" The system runs your checks and stores the results — often just a state (OK / WARN / CRIT), or one pre-aggregated number per check, frequently into a round-robin database that overwrites old data at a fixed resolution. Your footprint is bounded by how many checks you bothered to define. That's why 1GB lasts years: you only ever wrote down what you chose to ask. The new world: you store the raw material</h2> Prometheus inverts it. The philosophy is collect everything at full detail now, decide the questions later. You don't store "API latency" — you store API latency broken down by HTTP verb × resource type × response code × scope × instance, and pre-bucketed into a dozen-plus latency ranges, every combination kept as its own stream, so that six months from now you can ask a question you haven't thought of yet without having pre-defined the check. The unit of cost is the series: one unique metric name plus its exact set of labels. These are three different series: apiserver_request_duration_seconds_bucket{verb="GET", resource="pods", le="0.1"} apiserver_request_duration_seconds_bucket{verb="GET", resource="pods", le="0.5"} apiserver_request_duration_seconds_bucket{verb="POST", resource="cm", le="0.1"} </code></pre> Change any label — a different verb, a different bucket boundary le</code> — and it's a brand-new stream that gets a fresh value written to disk every scrape. My lab had 300,000 of them. Every 60 seconds it wrote 300,000 numbers to disk. That's the gigabyte. 71% of it was histograms I never looked at</h2> When I asked Prometheus what was actually in there, the answer was blunt: 92% of all series came from just two scrape jobs (the kubelet and the apiserver).</li> 71% of the entire database was histogram buckets.</li> A single metric — apiserver_request_duration_seconds_bucket</code> — was 44,608 series on its own.</li> </ul> Histograms are the multiplier. To hand you a p99 at query time without storing every individual request, Prometheus pre-counts: requests faster than 5ms, than 10ms, than 25ms… one counter per boundary, per label combination. One logical metric becomes dozens of series, times every verb and resource. There were 397 distinct bucket boundaries live in my tiny cluster. And nothing I'd built — not one dashboard, not one alert — ever read them. They were there because the default install ships dashboards and SLO rules that might want them, on the assumption you're running a cluster big enough to care. The k3s twist: I was collecting it all several times over</h2> Then the part that actually surprised me. That 44,608-series histogram? It was being scraped from six different endpoints. k3s is famous for collapsing the Kubernetes control plane — apiserver, etcd, scheduler, controller-manager, and the kubelet — into a single process per server node. Convenient. But kube-prometheus-stack is built for "real" clusters where those are separate things on separate endpoints. So it scrapes the apiserver on :6443</code>, and it also scrapes each node's kubelet on :10250</code> — and on k3s the kubelet endpoint serves the whole shared process registry. The apiserver and etcd histograms come out of the kubelet port too. Three server nodes × two endpoints each = the fattest metric set in Kubernetes, collected six times. I'd even "disabled" the etcd and scheduler scrape jobs earlier — did nothing, because those metrics were never coming from those jobs. They were leaking in through the kubelet. The mental flip</h2> The thing I had backwards: in monitoring, the discipline is add the checks you need. In observability, the firehose is on by default, and the discipline is drop the dimensions you don't. The master resource isn't "number of metrics" or "number of hosts" — it's cardinality, the count of distinct label combinations. One metric can be 1 series or 50,000 depending entirely on its labels. So the fix wasn't a bigger disk (I bumped it to 3GB in the heat of the incident; that only bought time). The fix was a scrape-time drop list: throw away the control-plane histogram buckets nothing reads, and stop the kubelet endpoint from re-serving the apiserver's metrics. Keep the cheap stuff — the request counts for rates and error ratios, the node and pod and volume gauges I actually alert on. That cut ~70% of the series. On the trimmed set, the original 1GB would have been fine. If you're coming from monitoring too</h2> Three things I wish I'd known on day one: A "series" is the billable unit, and labels mint them. Before you keep a metric, multiply its label cardinalities together. That's how many streams it costs.</li> Histograms are not one metric. Every _bucket</code> is a series per boundary per label combo. Keep them only where you'll genuinely open a percentile graph.</li> Match the scrape to your topology. On k3s, the collapsed control plane means the stock chart double-counts. Trim it, or you pay for the same data many times.</li> </ol> The observability world gives you answers to questions you didn't know to ask. It's genuinely powerful. It just bills you up front, in disk, for the privilege — and unlike the monitoring box in the corner, it will absolutely take you up on a gigabyte by lunchtime. The log shipper that broke every other pod 2026-06-02T00:00:00+00:00 I added monitoring to my k3s cluster — kube-prometheus-stack for metrics, Loki + Alloy for logs. Textbook LGTM-light, sized down for tiny nodes. It came up clean. Within the hour, apps I hadn't touched started throwing errors. The symptom</h2> First one app, then another, then a third — all logging the same line: failed to create fsnotify watcher: too many open files </code></pre> "Too many open files" reads like a file-descriptor leak. But three unrelated apps — none of which I'd deployed or changed — all hitting it at the same moment? A leak lives in one process. This was something they share running out. What they share</h2> inotify. It's the Linux mechanism for watching files and directories for changes; anything that does config hot-reload or log tailing uses it. Creating a watcher calls inotify_init()</code>, which consumes one inotify instance, and the kernel caps instances per user: $ cat /proc/sys/fs/inotify/max_user_instances 128 </code></pre> 128 - Here's the part that bites on Kubernetes: instances are counted per real UID on the host kernel, not per container or per pod. Most containers run as root, so every root container on a node draws from the same 128-instance budget. I counted what was actually in use on the node: $ sudo find /proc/[0-9]*/fd -lname 'anon_inode:inotify' | wc -l 138 </code></pre> 138 — over the ceiling. New inotify_init()</code> calls were failing with EMFILE</code>, which Go's fsnotify library surfaces, unhelpfully, as "too many open files." What pushed it over</h2> The log shipper. Alloy — like Promtail, Fluent Bit, Filebeat — tails log files, and file-tailers lean hard on inotify to notice new lines and new files. Dropping a log-shipper DaemonSet onto every node added a hungry consumer of a resource the whole node shares, and tipped a node already sitting near 128 over the edge. The monitoring stack didn't break itself — it looked perfectly healthy. It starved everything else, and the errors landed on innocent pods. The fix</h2> Raise the ceiling. 128 is an ancient default sized for a desktop, not a container host running dozens of pods: # /etc/sysctl.d/90-inotify.conf fs.inotify.max_user_instances = 1024 fs.inotify.max_user_watches = 524288 </code></pre> sudo sysctl --system </code></pre> Errors stopped cluster-wide within a minute. Then I put it in the Ansible base role, so a rebuilt node inherits it instead of rediscovering this at 9pm. The lesson</h2> "Too many open files" isn't always about files. For inotify_init()</code> it means you've hit max_user_instances</code> — a per-UID kernel limit, not the per-process fd limit (ulimit -n</code>) you'd reach for first.</li> A log shipper is a node-shared-resource hog, and inotify is the resource. Before you roll a tailer onto nodes with the stock max_user_instances=128</code>, raise it. The failure mode is mean: the thing you deployed looks fine while everything around it quietly fails to start watchers.</li> </ol> Bump the limit before the log shipper, not after the pager goes off. Oracle Cloud's invisible 47 GB floor 2026-05-31T00:00:00+00:00 Oracle Cloud's Always-Free tier is genuinely generous: 4 ARM cores, 24 GB RAM, and "200 GB of block storage." I used it to build a 3-node HA k3s cluster plus a small load-balancer node. Then my budget alert started twitching. The surprise</h2> I had 5 instances. I checked my block usage: 47 GB server-1 47 GB edge 47 GB server-2 47 GB oracle-monitor 47 GB server-3 ────────────────────────────────────────── 235 GB total / 200 GB free → 35 GB OVER </code></pre> Every instance — even the tiny 1 GB AMD micros — has a 47 GB boot volume, and boot volumes count against the same 200 GB pool. The OS uses ~5 GB; the other ~42 is just… there. Five instances and you're over budget, paying a euro or so a month for storage you're not using. So the obvious question: can I shrink those boot volumes? Dead end #1 — shrink the boot volume</h2> No. OCI volumes can only grow, never shrink. And the docs are explicit: For Linux images, the custom boot volume size must be larger than the image's default boot volume size or 50 GB, whichever is higher. </blockquote> So if you customize, the floor is 50 GB — bigger, not smaller. The only way to sit below 50 is to take the image's own default (47 GB) and not touch it. You literally cannot request less. Dead end #2 — the famous qemu-img</code> trick</h2> The community classic: attach a blank block volume, write a tiny cloud image onto it with qemu-img convert</code>, boot from that. Clever — but OCI block volumes have a 50 GB minimum (that's why every guide says "add a 50 GB volume"). So this hands you a 50 GB volume, which is bigger than the 47 GB default. It lets you run a different/leaner OS, but it reclaims zero quota. Dead end #3 — import a minimal image (the one that should work)</h2> Here's where it got interesting. That same rule has an asymmetry: a boot volume can be under 50 GB if the image's own default is under 50. Real minimal cloud images are tiny — the Debian genericcloud</code> arm64 image is a 3 GiB virtual disk (326 MB download). So: import that, and surely you get a ~3 GB boot volume? I tested it instead of guessing. Imported the 3 GiB Debian image via Object Storage and asked the API what size OCI assigned it: imported_image_size_mb = "47694" </code></pre> 47694 MB ≈ 46.6 GB — the exact same number as Oracle's own Ubuntu image. OCI pads every imported image up to its floor. The 3 GB image became a 47 GB image on import. Dead end confirmed, by experiment. The conclusion</h2> There is a universal ~47 GB floor on OCI instances. No knob, no trick, no slim image gets under it: Approach</th> Result</th></tr></thead> Custom boot volume size</td> floored at 50 GB</td></tr> qemu-img</code> → block volume</td> block volumes floored at 50 GB</td></tr> Import a 3 GB minimal image</td> clamped to 47 GB on import</td></tr> </tbody></table> What to actually do</h2> Plan your instance count around it. Each instance ≈ 47 GB. The 200 GB free pool realistically fits ~4 instances. Want more (I wanted 5, for HA)? Budget ~€0.025/GB-month for the overage — about €1/month. It's a floor, not waste.</li> Or run fewer, bigger boxes. This is why most "ultimate free-tier" guides build one instance with all 4 cores and 200 GB — one boot volume, lots of headroom. You trade HA for €0.</li> For Kubernetes PVs, use local-path</code>, which carves from the ~42 GB of free space already inside each boot volume. Don't add OCI block-volume PVs — those pile billable storage on top of a pool you've already maxed.</li> </ol> The free tier is still a fantastic deal. Just know that "200 GB" really means "~4 instances," and that the 47 GB floor is real, universal, and — now — empirically proven.