What consumes the space difference between a zpool and its contents?

Question

I have a zpool 'zoo' on mirrored 2 * 1TB drives:

[develop@silversurfer /mnt/zoo]$ zfs list -r zoo
NAME                  USED  AVAIL  REFER  MOUNTPOINT
zoo                   829G  69.9G   996M  /mnt/zoo
zoo/beyond            807G  69.9G   114G  /mnt/zoo/beyond
zoo/officetemplates  48.8M  69.9G  46.2M  /mnt/zoo/officetemplates
zoo/overflow          152K  89.9G    96K  /mnt/zoo/overflow

'overflow' is an empty dataset with (now) minimum size 20 GB. The other datasets do not have fixed sizes or quotas. I encountered two disk-full errors in the past weeks which I solved through shrinking 'overflow'.

As you see the zpool allocates 829 GB but its contents only add up to 117 GB.

'du' confirms the smaller size:

[develop@silversurfer /mnt/zoo]$ ls -al
total 75
drwxrwxrwx  7 root     wheel   6 Nov 18 14:46 .
drwxr-xr-x  5 root     wheel   5 Nov 12  2021 ..
dr-xr-xr-x+ 3 root     wheel   3 Sep  3  2019 .zfs
drwxrwxr-x  7 develop  2B      6 Nov 17  2021 beyond
drwxrwx---  6 www      2B     75 May 23 17:30 officetemplates
drwxrwxrwx  2 root     wheel   2 Nov  1 10:32 overflow
drwxrwxr-x  4 popeye   2B      8 Nov 13 10:37 scans
[develop@silversurfer /mnt/zoo]$ sudo du -hs /mnt/zoo
117G    /mnt/zoo
[develop@silversurfer /mnt/zoo]$ sudo du -hs /mnt/zoo/.zfs
4.9G    /mnt/zoo/.zfs
[develop@silversurfer /mnt/zoo]$ sudo du -hs /mnt/zoo/beyond
116G    /mnt/zoo/beyond
[develop@silversurfer /mnt/zoo]$ sudo du -hs /mnt/zoo/officetemplates
45M /mnt/zoo/officetemplates
[develop@silversurfer /mnt/zoo]$ sudo du -hs /mnt/zoo/scans
994M    /mnt/zoo/scans
[develop@silversurfer /mnt/zoo]$ sudo du -hs /mnt/zoo/overflow 
512B    /mnt/zoo/overflow

What is eating up the space between reported 829 GB and 117 GB? How can I reclaim this space?

Edit: it may be noteworthy that the pool is used for hosting four FreeBSD (12.1) jails. Also, please note that the question is not directed at differences between different ways of determining used/available space but at space differences in one and the same command.

Answer 1

I finally found an answer to this problem. The size difference came from snapshots. Use the -o space flag:

[develop@silversurfer ~]$ zfs list -r -o space zoo
NAME                 AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zoo                  82.2G   817G      196K    996M              0       816G
zoo/beyond           82.2G   795G      681G    114G              0          0
zoo/officetemplates  82.2G  48.8M     2.60M   46.2M              0          0
zoo/overflow          102G   152K       56K     96K              0          0

Wow, 681 G of snapshot space for payload data of 117 G!

List the snapshots and consumed space for each:

zfs list -r -o space -t snapshot zoo

It turned out there were 1000+ snapshots, automatically taken over a period of more than three years. A snapshot may use little space but sizes do add up.

I piped the snapshot names into a file and, after checking and editing the file, used it for batch-destroying snapshots:

zfs list -r -o name -t snapshot zoo > myfile
for x in `cat myfile`; do echo $x; sudo zfs destroy $x; done

Note that looping over a sudo command is generally asking for trouble. Use at your own risk.

While the script is running I can see the disk usage level slowly drop:

[develop@silversurfer ~]$ zfs list -r -o space zoo
NAME                 AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zoo                   680G   219G      196K    996M              0       218G
zoo/beyond            680G   198G     84.1G    114G              0          0
zoo/officetemplates   680G  48.8M     2.60M   46.2M              0          0
zoo/overflow          700G   152K       56K     96K              0          0

Destroying ~1000 snapshots took me about 10 minutes, your mileage may vary.

Hope this helps anybody out there facing a mysteriously filled disk.