Companion to the API-address bug: same cluster, same morning, different way for a "healthy" node to have completely broken pod networking. This one is the stock cloud OS image fighting your CNI, and it's invisible until pods try to talk to each other.

The symptom

k3s installed cleanly. Nodes Ready. But pods couldn't reach Services, DNS lookups timed out, and anything multi-pod CrashLooped. Node-level networking (SSH, the API on the host) was perfectly fine. Only forwarded traffic — pod-to-pod, pod-to-Service — was dead.

The cause: a default-REJECT host firewall

The cloud's Ubuntu image ships a preconfigured host firewall. The FORWARD chain ends in a single blanket rule:

-A FORWARD -j REJECT --reject-with icmp-host-prohibited

Here's why that's fatal to Kubernetes specifically. Every pod-to-pod and pod-to-Service packet is routed, not delivered locally — so it traverses the FORWARD chain. A CNI like flannel assumes it owns FORWARD and inserts its own ACCEPT rules. But the image's blanket REJECT is already there, and on a fresh boot it wins: each forwarded packet hits "host prohibited" and is dropped. The node looks healthy because nothing the host does is forwarded; only the pods suffer.

The fix

Two parts, and the reasoning behind each matters more than the commands:

1. Remove the blanket FORWARD REJECT. Let k3s / flannel / kube-proxy manage the FORWARD chain, which is what they expect to do:

# idempotent — only acts if the rule is present
iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited

2. Accept intra-cluster traffic on INPUT so node-to-node control traffic isn't blocked either — the internal VCN range, the pod CIDR, the service CIDR, and flannel's VXLAN port:

iptables -I INPUT -s <vcn-cidr>      -j ACCEPT   # node↔node: etcd, kubelet, …
iptables -I INPUT -s <pod-cidr>      -j ACCEPT   # pod network
iptables -I INPUT -s <service-cidr>  -j ACCEPT   # service network
iptables -I INPUT -p udp --dport 8472 -j ACCEPT  # flannel VXLAN

Then persist (netfilter-persistent save) so a reboot doesn't reinstate the breakage.

The decision behind it: don't run two firewalls

It's tempting to keep the host firewall and the cloud one. Don't. The cloud provider already gives you a network security layer (security lists / security groups) at the VCN edge — that is your real perimeter, and it's the one you can reason about centrally. The host iptables rules the image ships are redundant with it and actively conflict with the CNI. Keep the cloud security list as the perimeter; strip the host rules that collide with Kubernetes. One firewall, in one place, that the CNI is allowed to manage.

Takeaways

  1. A green node says nothing about pod networking. Node Ready only proves the kubelet is happy. If multi-pod workloads fail, check iptables -L FORWARD before you touch the CNI config.
  2. Default-REJECT host firewalls and CNIs are incompatible by construction. Any image (cloud or otherwise) that ends FORWARD in REJECT will break forwarding the moment a CNI relies on it. This isn't Oracle-specific; it's "preconfigured host firewall meets Kubernetes."
  3. Let the cloud layer be the firewall. Centralize the perimeter at the provider's security list and let kube-proxy/flannel own the host chains.

The frustrating part is how healthy everything looks. The fix is one deleted rule — but only once you know to suspect the OS image rather than your manifests.