Longhorn Volume a lot bigger than mounted drive

Question

I have a small InfluxDB database running inside my K3S cluster.
As Storage Class I use Longhorn.
I know it's not optimal to run a database in Kubernetes, but this is only for some metric logging for Telegraf.
The problem is that in the pod the mounted volume is 200 MB big, but in Longhorn it's 2.5 GB big as actual size. The volume is only 1 day old. At this speed, my disk storage will be full soon.

Why is this? And is this something I can fix?

Answer 1

I suspect the reason for this is snapshots.

Longhorn volumes have different size "properties":

• The volume's size - this is what you define in your manifest. The actual filesystem contents can't exceed that
• The amount of storage currently used on the volume head - this is essentially how full the volume is. run df -h inside an attached pod or use a tool like df-pv to check usage (this is relevant when your volume is getting full)
• snapshot size: how big a snapshot is, building incrementally on top of the last one. this can be viewed in the snapshots section of longhorn UI
• actual size: how much space the volume is really using on your host machine. This can be larger than the volume's "defined" size due to a number of reasons - the most common of which being snapshots

Longhorn keeps a history of previous changes to a volume as snapshots. you can either create them manually from the UI or create a RecurringJob that does that for you automatically.

Having many snapshots is problematic when a lot of data is (re-)written to a volume. Imagine the following scenario:

1. Write a 1GB file to volume
2. take snapshot (this snapshot is now 1GB big)
3. delete the file (volume head only contains the "file deleted" info, previous snapshot size is unaffected)
4. write a new 1GB file. volume head is now 1GB (new file)+the info from 3. big, BUT your previous snapshot is another GB. That way, your actual size is already 2x as big as the space currently used inside the volume

There's also an ongoing discussion about reclaiming space automatically