Longhorn using more than 50% of storage as reserved space

Question

I'm running Longhorn v1.2.3 on RKE1 cluster (provisioned by rancher), this cluster has 5 nodes with dedicated 20GiB disks mounted on /var/lib/longhorn, with ext4 filesystem and 0% reserved blocks for root user/group.

In the dashboard, i see the following stats:

Type	Size
Schedulable	33.5 Gi
Reserved	58.1 Gi
Used	6.18 Gi
Disabled	0 Bi
Total	97.8 Gi

I changed Storage Minimal Available Percentage in settings to 5 (from 25 as i recall), but that haven't changed anything. When i open "nodes" tab, i see the following in "Size" tab:

7.86 Gi
+11.7 Gi Reserved

exact size varies for different nodes, but it's around 8Gi.

These dedicated disks were added after provisioning longhorn in cluster, and system disks are 40 GiB in size, so possibly the reason for this overuse is because reserved size was calculated at the time, when longhorn was deployed alongside operating system, and haven't adjusted when i mounted this folder to new disk.

Why do i have more than half of my space "reserved"? What can i do to get more usable space from longhorn? Thanks!

Answer 1

After digging deeper and finding that it was one day possible to adjust these values from UI (i wasn't able to find it), i've searched for longhorn CRDs, and came across nodes.longhorn.io. And inside definition i've found exactly what i searched for:

spec:
  allowScheduling: true
  disks:
    default-disk-fd0000000000:
      allowScheduling: true
      evictionRequested: false
      path: /var/lib/longhorn/
      storageReserved: 536870912
      tags: null

Here i changed storageReserved to 536870912 (512 MiB) on all nodes, just in case, and longhorn applied this change immediately. This is ok in my case, because those disk are dedicated, and, per docs

Reserved: The space reserved for other applications and system.

Now i have my storage back, hope it helps

Edit: I've found a reason why i wasn't able to find a GUI setting - simply because, due to sidebar in my browser, it was hidden by not-that-obvious horizontal scroll bar.

Answer 2

@UI go to Node -> (Operation) Edit node and disks -> Storage Reserved

(do this for all nodes)

Answer 3

https://github.com/longhorn/longhorn/issues/4773 added a storage-reserved-percentage-for-default-disk setting for this situation.