I come from monitoring, not observability. Fifteen years of Nagios, then Zabbix, then check_mk. In that world a gigabyte of disk is a lot. I've run boxes that watched a few hundred hosts and didn't fill 1GB in a year.

So when I gave Prometheus a 1GB volume on my three-node k3s lab, it felt generous. It was empty by lunchtime. Not "getting full" — full, write errors, ingestion stopped, my brand-new alerting blind. Hours, not years.

Here's what I learned pulling it apart, written for the version of me who still thinks in checks.

The old world: you store the answers

In monitoring, you write the questions up front. "Is disk over 90%?" "Is the service responding?" "Is latency above 200ms?" The system runs your checks and stores the results — often just a state (OK / WARN / CRIT), or one pre-aggregated number per check, frequently into a round-robin database that overwrites old data at a fixed resolution.

Your footprint is bounded by how many checks you bothered to define. That's why 1GB lasts years: you only ever wrote down what you chose to ask.

The new world: you store the raw material

Prometheus inverts it. The philosophy is collect everything at full detail now, decide the questions later. You don't store "API latency" — you store API latency broken down by HTTP verb × resource type × response code × scope × instance, and pre-bucketed into a dozen-plus latency ranges, every combination kept as its own stream, so that six months from now you can ask a question you haven't thought of yet without having pre-defined the check.

The unit of cost is the series: one unique metric name plus its exact set of labels. These are three different series:

apiserver_request_duration_seconds_bucket{verb="GET",  resource="pods", le="0.1"}
apiserver_request_duration_seconds_bucket{verb="GET",  resource="pods", le="0.5"}
apiserver_request_duration_seconds_bucket{verb="POST", resource="cm",   le="0.1"}

Change any label — a different verb, a different bucket boundary le — and it's a brand-new stream that gets a fresh value written to disk every scrape. My lab had 300,000 of them. Every 60 seconds it wrote 300,000 numbers to disk. That's the gigabyte.

71% of it was histograms I never looked at

When I asked Prometheus what was actually in there, the answer was blunt:

  • 92% of all series came from just two scrape jobs (the kubelet and the apiserver).
  • 71% of the entire database was histogram buckets.
  • A single metric — apiserver_request_duration_seconds_bucket — was 44,608 series on its own.

Histograms are the multiplier. To hand you a p99 at query time without storing every individual request, Prometheus pre-counts: requests faster than 5ms, than 10ms, than 25ms… one counter per boundary, per label combination. One logical metric becomes dozens of series, times every verb and resource. There were 397 distinct bucket boundaries live in my tiny cluster.

And nothing I'd built — not one dashboard, not one alert — ever read them. They were there because the default install ships dashboards and SLO rules that might want them, on the assumption you're running a cluster big enough to care.

The k3s twist: I was collecting it all several times over

Then the part that actually surprised me. That 44,608-series histogram? It was being scraped from six different endpoints.

k3s is famous for collapsing the Kubernetes control plane — apiserver, etcd, scheduler, controller-manager, and the kubelet — into a single process per server node. Convenient. But kube-prometheus-stack is built for "real" clusters where those are separate things on separate endpoints. So it scrapes the apiserver on :6443, and it also scrapes each node's kubelet on :10250 — and on k3s the kubelet endpoint serves the whole shared process registry. The apiserver and etcd histograms come out of the kubelet port too.

Three server nodes × two endpoints each = the fattest metric set in Kubernetes, collected six times. I'd even "disabled" the etcd and scheduler scrape jobs earlier — did nothing, because those metrics were never coming from those jobs. They were leaking in through the kubelet.

The mental flip

The thing I had backwards: in monitoring, the discipline is add the checks you need. In observability, the firehose is on by default, and the discipline is drop the dimensions you don't. The master resource isn't "number of metrics" or "number of hosts" — it's cardinality, the count of distinct label combinations. One metric can be 1 series or 50,000 depending entirely on its labels.

So the fix wasn't a bigger disk (I bumped it to 3GB in the heat of the incident; that only bought time). The fix was a scrape-time drop list: throw away the control-plane histogram buckets nothing reads, and stop the kubelet endpoint from re-serving the apiserver's metrics. Keep the cheap stuff — the request counts for rates and error ratios, the node and pod and volume gauges I actually alert on. That cut ~70% of the series. On the trimmed set, the original 1GB would have been fine.

If you're coming from monitoring too

Three things I wish I'd known on day one:

  1. A "series" is the billable unit, and labels mint them. Before you keep a metric, multiply its label cardinalities together. That's how many streams it costs.
  2. Histograms are not one metric. Every _bucket is a series per boundary per label combo. Keep them only where you'll genuinely open a percentile graph.
  3. Match the scrape to your topology. On k3s, the collapsed control plane means the stock chart double-counts. Trim it, or you pay for the same data many times.

The observability world gives you answers to questions you didn't know to ask. It's genuinely powerful. It just bills you up front, in disk, for the privilege — and unlike the monitoring box in the corner, it will absolutely take you up on a gigabyte by lunchtime.