Yesterday I fixed a real disk-full on my cluster's Prometheus volume — expanded it to 3 GB, trimmed the metrics flooding it. Today I opened the Longhorn UI and saw the volume at 2.97 GB of 3 GB used again. Less than a day later. Stomach drop.

It wasn't full. Here's the gotcha, because it'll catch anyone running thin-provisioned storage.

actualSize is not disk usage

The number Longhorn shows you — actualSize — is how many blocks the volume's replicas have touched on disk, including Longhorn's own internal snapshots. It is not how full the filesystem is.

I checked what Prometheus actually had on the filesystem:

blocks (persisted):  619 MB
write-ahead log:     233 MB
head chunks:          27 MB
-----------------------------
real data:          ~880 MB   of a 3 GB volume  (29%)

29%. The pod had 24 hours of uptime, zero restarts, no write errors. Nothing was wrong with the filesystem at all. So where did 2.97 GB come from?

Two hoarders

A leftover snapshot. When you expand a Longhorn volume it auto-creates a system snapshot (expand-<size>). Mine was ~1 GB and Longhorn hadn't coalesced it — it just sat in the replica chain, counting against actualSize.

Thin-provisioning lag. This is the one worth internalising. When Prometheus deletes data — old blocks aged out by retention, the high-cardinality series I trimmed yesterday — the filesystem marks those blocks free. But the block layer underneath doesn't get told. Those blocks stay "allocated" as far as Longhorn is concerned until something issues a TRIM (fstrim) to hand them back. Another ~1 GB of "used" that was actually free.

So: ~880 MB real data + ~1 GB stale snapshot + ~1 GB un-trimmed free blocks ≈ the 2.97 GB that scared me. None of it filesystem pressure.

The fix is to take out the trash, not buy a bigger bin

  1. Delete the leftover expand snapshot.
  2. Turn on Longhorn's "remove snapshots during filesystem trim" so a trim also coalesces removed snapshots in one pass.
  3. Run a filesystem trim.
actualSize: 3.19 GB  →  0.98 GB

One pass, ~2.2 GB reclaimed, Prometheus didn't even notice (online trim, no restart). 0.98 GB matches the real data — the hoarders are gone.

Then the part that matters more than the cleanup: a weekly Longhorn RecurringJob with task filesystem-trim, so freed blocks get released automatically and I never watch this gauge again.

The takeaway for thin-provisioned storage

When your storage layer screams "nearly full," check the filesystem before you add disk:

  • actualSize (or equivalent) ≠ filesystem usage. It counts snapshots and un-reclaimed blocks. Look at what the workload actually wrote.
  • Thin provisioning doesn't shrink on its own. Deleting data frees it for the filesystem, not for the block layer. Schedule periodic fstrim — or your "usage" only ever goes up.
  • Volume operations leave snapshots. Expansions and rebuilds drop system snapshots that linger unless you (or a job) clean them.

Yesterday's disk-full was real and I gave it more disk. Today's was the storage layer hoarding free space — and the answer was a trim job, not a bigger volume. Worth knowing which one you're looking at before you reach for the disk slider.