I added monitoring to my k3s cluster — kube-prometheus-stack for metrics, Loki + Alloy for logs. Textbook LGTM-light, sized down for tiny nodes. It came up clean. Within the hour, apps I hadn't touched started throwing errors.

The symptom

First one app, then another, then a third — all logging the same line:

failed to create fsnotify watcher: too many open files

"Too many open files" reads like a file-descriptor leak. But three unrelated apps — none of which I'd deployed or changed — all hitting it at the same moment? A leak lives in one process. This was something they share running out.

What they share

inotify. It's the Linux mechanism for watching files and directories for changes; anything that does config hot-reload or log tailing uses it. Creating a watcher calls inotify_init(), which consumes one inotify instance, and the kernel caps instances per user:

$ cat /proc/sys/fs/inotify/max_user_instances
128

128 - Here's the part that bites on Kubernetes: instances are counted per real UID on the host kernel, not per container or per pod. Most containers run as root, so every root container on a node draws from the same 128-instance budget.

I counted what was actually in use on the node:

$ sudo find /proc/[0-9]*/fd -lname 'anon_inode:inotify' | wc -l
138

138 — over the ceiling. New inotify_init() calls were failing with EMFILE, which Go's fsnotify library surfaces, unhelpfully, as "too many open files."

What pushed it over

The log shipper. Alloy — like Promtail, Fluent Bit, Filebeat — tails log files, and file-tailers lean hard on inotify to notice new lines and new files. Dropping a log-shipper DaemonSet onto every node added a hungry consumer of a resource the whole node shares, and tipped a node already sitting near 128 over the edge. The monitoring stack didn't break itself — it looked perfectly healthy. It starved everything else, and the errors landed on innocent pods.

The fix

Raise the ceiling. 128 is an ancient default sized for a desktop, not a container host running dozens of pods:

# /etc/sysctl.d/90-inotify.conf
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches  = 524288

sudo sysctl --system

Errors stopped cluster-wide within a minute. Then I put it in the Ansible base role, so a rebuilt node inherits it instead of rediscovering this at 9pm.

The lesson

  1. "Too many open files" isn't always about files. For inotify_init() it means you've hit max_user_instances — a per-UID kernel limit, not the per-process fd limit (ulimit -n) you'd reach for first.
  2. A log shipper is a node-shared-resource hog, and inotify is the resource. Before you roll a tailer onto nodes with the stock max_user_instances=128, raise it. The failure mode is mean: the thing you deployed looks fine while everything around it quietly fails to start watchers.

Bump the limit before the log shipper, not after the pager goes off.