A few days ago I stood up the textbook self-hosted observability stack on my k3s cluster: Prometheus for metrics, Grafana for dashboards, Loki for logs, alerts wired to push notifications. It worked. Dashboards rendered, a test alert hit my phone, I marked the issue done. Then I tore the whole thing out and replaced it with two small tools. This is the reasoning, because "I built the impressive thing and then deleted it" is a more useful story than most build logs.

What pushed me over

Two days after it went live, the Prometheus volume filled and ingestion stopped — the cluster went quietly blind right when I'd have wanted it most. That incident is its own post; the short version is that the stack was the heaviest tenant on the cluster, running on CPU-bound single-core nodes, and its appetite was the direct cause of the outage. Two days of firefighting to keep the monitoring from being the thing that needed monitoring.

Then the honest question: how much of this depth do I actually use? Almost none. I barely opened the deep metric explorers; I'd never written a serious Grafana query against my own data. I was paying — in CPU, in disk, in incidents — for capability I didn't touch. That fails the only test that matters in a self-hosted lab: don't run what you don't use.

The replacement: two engines, not a platform

I split "monitoring" back into the two questions it actually answers, and picked the lightest tool for each:

  • Resource vitals — CPU, memory, disk, temperature, per host and container — handled by Beszel: a lightweight agent on each host reporting to a small hub. Node-level, cheap, always-on.
  • Availability + heartbeats — Gatus: a black-box prober that hits the real URLs of the things I care about and pages me when one stops answering, plus dead-man checks for jobs that should run on a schedule.

Delivery for both is a single push channel — ntfy — one templated line per alert: what broke, and where. No dashboards-as-a-product, no query language, no log warehouse. If I need deep diagnosis, I open the cluster GUI or reach for kubectl on demand — I don't keep a metrics firehose running 24/7 on the off chance.

The insight that reshaped it: a monitor can't certify itself

This is the part worth stealing. My in-cluster stack could go falsely green — and on the incident day, an in-cluster heartbeat would have, because the thing checking was inside the thing that was failing. A monitor that lives with what it monitors can't tell you the patient is dead; it dies with it.

So the availability prober runs off the cluster, on a different provider, and checks the public URLs from outside — the same path a real user takes. That black-box, outside-in check is the actual fix for the blind spot, not a bigger in-cluster stack. I run two such watchers in different places and have them cross-watch each other, with a free external dead-man service as the apex backstop for the rare case where both are down at once.

Pull-first, with exactly one exception

A principle that fell out of this: prefer pulling over pushing. A watcher that reaches out to probe needs no inbound hole in anything it watches — it fits a "no open ports" posture by construction. The only push in the design is backup heartbeats: a cron job has no endpoint to probe, so it has to announce "I ran." Those ride a private network, and the semantics are simply missed = noticed. One deliberate exception, justified; everything else pulls.

What I gave up, on purpose

Honesty about the trade-off, because it's real: I lost per-pod metrics inside the cluster. The lightweight agent sees the node, not each container, on this runtime. I decided that's acceptable:

  • user-facing health is covered by the outside-in app probes;
  • node pressure is covered by the vitals agent;
  • deep, occasional diagnosis is on-demand via the cluster GUI / kubectl.

And it's recoverable: if I ever genuinely want long-term cluster metrics back, the move is a hosted free-tier metrics backend (Grafana Cloud's free tier) fed by a curated, low-cardinality remote-write — the lesson from the incident baked in — not a full self-hosted stack returning to eat the cluster again. Deferred until I actually want it.

Things I considered and rejected

  • Netdata (the one tool that does see per-pod via cgroups): ruled out — too heavy for single-core nodes, and it had nearly taken down another box of mine before. The per-pod visibility wasn't worth the weight for something I rarely use.
  • Folding host vitals into the cluster Prometheus (my own first instinct): rejected — it keeps the heavy stack and couples the host vitals to the very cluster that goes blind when it's sick.
  • A second alerting brain off-cluster with alert inhibition: cut. It adds a public silence API (another hole) and, worse, inhibition can silently swallow a real page. Plain grouping, severity, and long repeat intervals get the "don't spam me" outcome without a rule that can eat an alert you needed.

Takeaways

  1. Match the monitoring to what you'll actually look at. Impressive ≠ useful. The heavy stack failed the "do I use this" test; deleting it was the upgrade.
  2. A monitor can't be inside what it monitors. Outside-in, black-box probes from a different failure domain are the only checks that don't go falsely green.
  3. Pull by default; justify every push. Probing needs no inbound holes. The one push here (job heartbeats) earns its exception because cron has no endpoint.
  4. Name your trade-off out loud. I gave up per-pod metrics knowingly, with a documented path to get them back cheaply if I ever care. That's a decision, not an accident.

The fanciest version of this cluster ran a full observability platform. The better version runs two small tools and a notification topic — and is no longer at risk of being knocked over by its own monitoring.