btrfs df

[jlittle]

Just did my first btrfs send backup. The output of btrfs filesystem df is confusing me:

! sudo btrfs fi df /mnt/backup
[sudo] password for john: 
Data, single: total=13.01GiB, used=12.12GiB
System, DUP: total=8.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=318.00MiB
GlobalReserve, single: total=33.20MiB, used=0.00B

The "used" is reasonable, but the btrfs filesystem mounted on /mnt/backup is 1.5 TB.
My root system shows total = 209 GiB, used 70 GiB. (I have snapshots going back to last year.)

Is the data total figure just a high water mark?

[oshunluvr]

I can explain the btrfs output: The "total" you see with btrfs fi df is total allocated not total available. Total available comes from btrfs fi show.

As an example, here's mine:

$ df
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc3       460G  166G  289G  37% /


$ suso btrfs fi df /
Data, RAID0: total=367.88GiB, used=161.10GiB
System, RAID1: total=64.00MiB, used=44.00KiB
Metadata, RAID1: total=5.00GiB, used=2.18GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

$ sudo btrfs fi sh /
Label: 'ssd840pro'  uuid: 8f0c1661-4e84-4512-b875-23bcfd5be1d8
        Total devices 2 FS bytes used 163.29GiB
        devid    4 size 229.97GiB used 189.00GiB path /dev/sdc3
        devid    5 size 229.97GiB used 189.00GiB path /dev/sdd3

You can see (two 230GB devices) 460GB of total space but only 368GB has been allocated and 161GB in use for data and 2.3GB for the other stuff. BTRFS, unlike other filesystems, doesn't pre-allocate (write a format to) the entire device. It allocates more space as needed. Honestly, the amount of space allocated is of little use to a user (vs a developer). Plus, it would be less confusing if the word allocated was included in the fi df output.

But as to why your df shows 70GB used and btrfs has only claimed 12GB I can't fathom. Are your snapshots incremental vs. whole subvols? Is /mnt/backup a subvol itself or a sholw filesystem, and then are there other subvols on the same device as /mnt/backup ?

[oshunluvr]

I re-read your original post just now - I suspect your "root" system has several other subvols within it and don't forget - assuming you used the default install - your home resides on the same filesystem as your root install but within a separate subvol. So it would be normal for df to report the same amount of used space for / and for /home. At a minimum you have those two subvols on the same filesystem along with any snapshots.

AFAIK, the only ways to determine individual subvol size is to enable quota groups or do a send|receive to a separate file system with nothing else on it.

[GreyGeek]

And, as a point of comparison, here are my listings:

[FONT=monospace][B][COLOR=#000000]$ df[/COLOR][/B]
Filesystem     1K-blocks     Used Available Use% Mounted on
...
/dev/sda1      728667580 96152288 627465616  14% /
...
/dev/sda1      728667580 96152288 627465616  14% /home
...
[COLOR=#000000]
[B]$ sudo btrfs fi df /[/B][/COLOR]
Data, RAID1: total=98.00GiB, used=90.75GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=2.00GiB, used=803.70MiB
GlobalReserve, single: total=161.42MiB, used=0.00B
[COLOR=#000000]
[B]$ sudo btrfs fi sh /[/B][/COLOR]
Label: none  uuid: 12980ae8-4117-4cc5-bbb8-8065e82af93d
        Total devices 2 FS bytes used 91.54GiB
        devid    1 size 691.19GiB used 100.03GiB path /dev/sda1
        devid    2 size 698.64GiB used 100.03GiB path /dev/sdc

[/FONT]

I have two 750Gb HD's in a RAID1 configuration

[#]BTRFS[/#]

[vinnywright]

AFAIK, the only ways to determine individual subvol size is to enable quota groups or do a send|receive to a separate file system with nothing else on it.

if you mount the partition somewhere

vinny@vinny-Bonobo-Extreme:~$ sudo mount -t btrfs /dev/sda1 /mnt/sda1
[sudo] password for vinny: 
vinny@vinny-Bonobo-Extreme:~$

then you can view the subvolumes in dolphin ,,,,and right click>properties ,,,,,and let it calculate the size .



VINNY

[jlittle]

I can explain the btrfs output: The "total" you see with btrfs fi df is total allocated not total available.

Thank you. IMO, then, the df subcommand
is badly named.

BTRFS, unlike other filesystems, doesn't pre-allocate (write a format to) the entire device. It allocates more space as needed.

Thanks again. I've read a bit about btrfs but if I saw that I forgot it.

But as to why your df shows 70GB used and btrfs has only claimed 12GB I can't fathom.

I've got lots of snapshots. Most of them by snapper, including 10 monthly ones. I suppose what counts most is the earliest.

Are
your snapshots incremental vs. whole subvols?

I haven't done an incremental backup yet.

Is /mnt/backup a subvol itself or a sholw filesystem, and then are there other subvols on the same device as /mnt/backup ?

There's a whole file system, as created by gparted. The only subvolumes on it are the ones created by btrfs receive, for / and /home. I've run du -h on my / and /home, and their backups.

They're slightly smaller on the backups, but diffing the output only shows differences on things like /tmp, /var/tmp, and ~/.cache, so I think I've got all the data. I assume that the extra 70 - 13 = 57 GiB is from all those snapshots and their metadata.
The send/receive was super quick; I think I'll do that to one of the old snapshots then delete them all except the current one, and report what the used comes down to. I wonder if there are any housekeeping commands I should run after the deletions.
Something I've found confusing is that the commands only work with the mount points; the only place the names "@" and "@/home" are used is in /etc/fstab.

Regards, John Little

[oshunluvr]

if you mount the partition somewhere
~~~
then you can view the subvolumes in dolphin ,,,,and right click>properties ,,,,,and let it calculate the size .
VINNY

True, but not an accurate measurement. This gives to the total byte count if the files were uncompressed and does not include overhead (meta, system, global reserve) space used or reserved but unoccupied blocks (for example, a 22k file uses 32k if the file system is using 32k blocks).

Granted the info is useful at a certain level or for some uses like determining if a thumb drive is large enough to hold all the files, but it is not an actual subvolume size.

I have to point out that you don't actually mount a partition, you mount a file system. This distinction is especially important with btrfs as you can also mount parts of the file system as subvolumes or the entire file system.

I think I've got all the data. I assume that the extra 70 - 13 = 57 GiB is from all those snapshots and their metadata.

Yeah, that makes sense. I didn't notice at first you were referring to two separate btrfs file systems. I assume you're aware that snapshots, when made, take zero data space. They grow as changes occur to the original subvolume.

The send/receive was super quick

This will differ greatly depending on subvol size and transfer speeds - obviously. I see some long send|receives when I backup my home, which is much larger than my installation.

I think I'll do that to one of the old snapshots then delete them all except the current one, and report what the used comes down to. I wonder if there are any housekeeping commands I should run after the deletions.

You will see a lot of variation depending of the state of the file system. Here's a good experiment: do df, btrfs fi df, and btrfs fi sh, and note the sizes. Then delete a subvol (snapshot) with the "commit" flag (-c) or sync after the deletion. Then do the df's and show commands again. Then run a balance and another round of df's and show commands and see what if difference is.

Something I've found confusing is that the commands only work with the mount points; the only place the names "@" and "@/home" are used is in /etc/fstab.

Takes a while to get used to this, but if you consider that most things are done using btrfs while it's still accessible. It's the only file system I've used that allows you to add or remove devices (drives or partitions) while still using the computer - you don't even have to reboot. Consider that the information you seek or the action you're taking is on the file system not the device. Really, the way btrfs references itself it is more technically correct than other file systems IMO. For example, you're not actually checking a partition for errors you're checking a file system. Why refer to the partition device name then? What about multi-device btrfs file systems? Would you check only the first drive and not the second? Thinking about it this way makes it seem more logical to me.

Yes, the subvolume names are only used to mount or to do snapshots and backups. You can address (use) the entire file system without subvolumes, but you lose a lot of the most useful features without subvolumes.

[jlittle]

I said:

I think I'll do that to one of the old snapshots then delete them all except the current one, and report what the used comes down to.

Well, better late than never. Used went down to 16 GiB, and I've just done the exercise again (deleting all but a recent snapshot on / and /home, after doing a backup), and used=16.86 GiB.

I managed to screw up snapper (trying to suspend it) so I've purged it. I need to learn systemd if I want to have control of my computer, it seems.

[GreyGeek]

I came to the conclusion that snapper was too snap happy and purged it as well. I messed around with a script be decided that was overkill since the manual comands to create backup snapshots, deletes and rollbacks are so simple. So, when I "sudo -i" then I just use the up arrow to recall previous commands and change the mm/dd part and hit the Return Key. Drop dead easy.
Systemd has a gui component you can install and it appears at the bottom of the systemsettings5 GUI. I do most of my systemd work from it.

[oshunluvr]

or install my Dolphin servicemenus...

...blatant self-promotion

[vinnywright]

or install my Dolphin servicemenus...

...blatant self-promotion

I will +1 this ,,,,,it works very well ,, makes it dead easy to take/or delete snapshots .

VINNY

[jlittle]

or install my Dolphin servicemenus...

...blatant self-promotion

Ok, I'll give them a try. However, twice after installing snapper, I found myself glad of the automatic hourly snapshots, for files I didn't anticipate the need.

Regards, John Little

[oshunluvr]

Ok, I'll give them a try. However, twice after installing snapper, I found myself glad of the automatic hourly snapshots, for files I didn't anticipate the need.

Regards, John Little

Here's the d/l link to the current set of service menus. Let me know if you have any issues. https://www.dropbox.com/s/wc42xmqeih...us.tar.gz?dl=0