BTRFS for new users. Why you should consider it.

[oshunluvr]

This confuses me; can't quite wrap my head around it. As I'm "visually oriented"; I need to 'see' things (actual, ideas, concepts, etc). So let me ask this: I take a snapshot. As you say, it is "initially empty". I immediately delete EVERYTHING except the created snapshot (this is a mental exercise, so go with me here). What good is the snapshot now? I can't recover from my self-inflicted gun shot with it, can I? I'm expecting the answer is "No, you cannot."

To be technically correct, snapshots do take space, but it's just metadata space, not file space. Just like when you delete a file from an EXT4 file system, the file isn't actually deleted. The space it occupies is marked as available and the metadata (the part that your file system uses to locate the file) is deleted. BTRFS works the same way.

Snapshots are not copies of files. The files are already there. It copies the metadata. So a "pointer" to a file now exists in two places. If you "delete" a file from subvolume A, the metadata in that subvolume is deleted, but subvolume B remains untouched so the file is still there in B.

So to your example Snowhog:

Subvolume A contains 1000 files using 10GB.
A snapshot of A is made and it's named subvolume B.
All 1000 files that are in A are also in B.
The used space on your drive does not change - still 10GB in use.
You delete subvolume A.
All 1000 files that formerly were in A are still in B.
The used space on your drive does not change - still 10GB in use.

The changes occur when you change a file that is part of a snapshot. So given that A and B still exist:

You delete a file from subvolume A of 500MB.
Subvolume A now contains 999 files totaling 9.5GB.
All 1000 files that formerly were in A are still in B.
The used space on your drive does not change - still 10GB in use.
You add 10 files totaling 2GB to subvolume B.
Subvolume A still contains 999 files totaling 9.5GB.
Subvolume B now contains all the original 1000 files plus the 10 new ones and totals 12GB.
The used space on your drive grows to 12GB.

As you add and subtract files from each subvolume, they expand and contract accordingly, but the total space occupied on the file system (remember a snapshot must be co-located on the same file system as it's source) is the total amount of SHARED files (not two times the amount, just once) plus the total amount of DIFFERENT files.

In the EXT4 world, if you copied an entire partition, they have no relationship to each other. Therefore, the amount of used space doubles. Snapshots share filesystem space until a difference is introduced into one or both of the subvolumes.

A real world example:

Prior to a new kernel install, take a snapshot.
Zero bytes added to the file system.
Install the new kernel and delete an old kernel.
New kernel adds 100MB and old kernel removes 90MB.
The file system grows by 100MB. It does not reduce by 90MB because the old kernel is still alive and well in the snapshot.
All is well? - Delete the snapshot. file system shrinks by 90MB.
All is not well? - Reboot to the snapshot and delete the source subvolume, file system shrinks by 100MB.

Deleting the snapshot removes the old kernel from the file system. Deleting the source subvolume removes the new kernel. ALL the other files remain in place and unchanged.

Does this help at all?

[GreyGeek]

And when Oshunluver says he “re-snapshots the snapshot” he’s taking a rw snapshot of a ro snapshot so that when he makes it a new @ or @home or whatever, it can be read and written to.


Sent from my iPhone using Tapatalk

[oshunluvr]

And when Oshunluver says he “re-snapshots the snapshot” he’s taking a rw snapshot of a ro snapshot so that when he makes it a new @ or @home or whatever, it can be read and written to.

My primary reason for the re-snapshot-ing is to preserve the snapshot in it's current state in case I need it again. Generally, and specifically in the case I outlined above, I intend to re-attempt whatever went wrong - like a kernel installation - and I want to be able to try again and again until I figure it out. Think of it like restoring from a backup, if you have to restore from a backup you don't so it and then delete the backup. You restore and preserve the backup until it's replaced by a newer one. Same principal here.

I personally don't bother with ro snapshots unless I'm preparing to send a snapshot to another file system - then ro status is required. I don't mount or access any of my snapshots except to restore one like I did here. I figure as long as I'm not mucking about with them, they're safe. However, in this example the snapshot could be ro or rw and I still would go through the same process. There's certainly nothing wrong using ro as a default status for snapshots.

When I do my automated backups, I have the script automatically re-snapshot the received snapshot so it has rw status, then I (again automated) replace the UUIDs within the snapshot so that it's bootable from the backup location as is. I actually haven't had to test this, but I should one day just to be sure it works.

[Snowhog]

...
Does this help at all?

To be brutally honest, only a bit. I'm a tough nut to crack!

Okay, let's add to the scenario I outlined. After I immediately delete everything except the snapshot, I then (again, a mental exercise) zero out the space occupied by the contents I deleted (don't worry about how I did that; again, an exercise), but again, leave the snapshot intact. What happens to the 'contents' of the snapshot now? Is it still able to restore what I nuked, or is it now also, and empty container?

[oshunluvr]

To be brutally honest, only a bit. I'm a tough nut to crack!

Okay, let's add to the scenario I outlined. After I immediately delete everything except the snapshot, I then (again, a mental exercise) zero out the space occupied by the contents I deleted (don't worry about how I did that; again, an exercise), but again, leave the snapshot intact. What happens to the 'contents' of the snapshot now? Is it still able to restore what I nuked, or is it now also, and empty container?

Well, yeah - if you nuke your file system with dd you get what you paid for, but if you mean you zero out all the unused space in the file system - No you don't lose a file. The space is still used by the snapshot, therefore it's not unused. Point being - deleting the subvolume but not the snapshot deletes nothing with regards to data. It just removes the metadata.

So your illustration here doesn't hold water - by deleting the subvolume that has a snapshot you haven't released any space formerly occupied by the subvolume, therefore there is no space to zero out.

[oshunluvr]

I'm struggling to come up with a parallel example...

Your files are in a file cabinet.
The file system is a manager that is responsible to maintain the contents of the file cabinet.
The metadata is an index that tells the manager where to find a file.

A snapshot makes a copy of the index, so now the manager has index A and index B but the files haven't changed and the cabinet is the same size.

You go to the manager and tell him to delete a file. But you are only accessing index A. The manager draws a line though the file name on index A but index B remains unchanged and the file remains in the cabinet.

Now you got to the manager again and hand him a new file to put in the cabinet. He does so and adds the file to index A - but not to B. The cabinet got fuller, but index B is remains unchanged.

Then you go to the manager and tell him to throw away the entire file cabinet. He throws away index A and pulls the one file out of the cabinet that you added to index A and throws it away also.

The remainder of the files, as listed on index B, remain in the file cabinet.

[Snowhog]

I sort of understand that. When I said 'zero out' I meant writing zeros over the spaces the files that were flagged as deleted occupy. I do understand that a deleted file only has first bit/character replaced with a deletion flag, telling the system to no longer display it 'normally', and to let the system know that future disk writes can use that space when needed. So if I were to accomplish that, but didn't touch the snapshot 'file', it's metadata (would that still exist?) would point to locations that no longer contained useful data, yes? Truly not trying to be difficult, but these type of examples, and the answers to the questions I raise in them, help me to "see".

[oshunluvr]

Well, the point is, if a snapshot exists the space the file occupies isn't flagged as available for use just because you deleted it in one place. This doesn't happen until all subvolumes (snapshots are subvolumes) that the file belongs in have released the space - either by deleting the file from every subvolume OR deleting all the subvolumes containing the file.

This is super easy to demonstrate yourself if you have a BTRFS file system to play with.

Mount the root BTRFS file system and make a subvolume: sudo btrfs su cr A

You now have an empty subvolume named "A". Enter the subvolume and add a folder named "1" and put some files in it, then make another folder named "2" in the same subvolume and copy all the files from 1 into 2. You now have a single subvolume with 2 folders containing identical files.

Note the amount of free space on the BTRFS file system.

Return to the root filesystem and take a snapshot of A: sudo btrfs su sn A B

You now have two subvolumes: "A" and "B". Observe the amount of free space on the BTRFS file system is unchanged. Observe that B contains the same folders and files as A.

Now, open subvolume A and delete folder "2". Observe the file system still has the same amount of free space.

Open subvolume B and delete folder "1". Note the file system space is unchanged.

You still have a copy of all the files you started with, but folder 1 is only in A and folder 2 is only in B.

Exercise: By percentage of the original space used by subvolume A and it's two folders "1" and "2"; How much free space is recovered if you now delete subvolume A?

[oshunluvr]

Answer: 50%.

The steps to the answer:
Subvol A with folders 1 and 2 occupy 100% of the space.
Subvol B with folders 1 and 2 occupy 100% of the space - the same 100% occupied by subvol A. Therefore no additional space is required.
When folder 2 is deleted from subvol A, no space is freed because folder 2 is still in subvol B. 100% of the space is still occupied.
When folder 1 is deleted from subvol B, no space is freed because folder 1 is still in subvol A. 100% of the space is still occupied.
After the above actions, Folder 1 exists only in subvolume A and folder 2 exists only in subvolume B.
When you delete subvolume A, folder 1 is also totally deleted. Only subvolume B and folder 2 remain. Therefore, 50% of the space is freed - the portion formerly occupied by folder 1.

[oshunluvr]

How much space do snapshots take, for those who have tiny SSDs?

Actually Clay, I forget to mention here another advantage to BTRFS - unified free space.

In the old EXT4 way, to have a separate home or a second linux install you must partition. Partitioning requires that you guess how much space you'll need for each partition and it makes it incredibly difficult to amend.

EXT4 example:
128GB SSD partitions:
1 - swap - 4GB
2 - kubuntu - 16GB
3 - other Linux - 16GB
4 - home - 92GB

Kubuntu ends up using 12GB - 4 wasted GB, "other" linux uses 8GB - 8 wasted GB. Or wait - what if other linux needs 20GB? Time to re-partition and hope gparted doesn't let you down.

BTRFS Example, same drive:
1 - swap 4GB
2 - everything else - 124GB.

Kubuntu, other linux, and home are in separate subvolumes but on the same filesystem. Free space is available to all the subvolumes. If other linux only uses 8 or grows to 20GB no matter - it expands and contracts as needed and no free space is languishing.

[GreyGeek]

To be brutally honest, only a bit. I'm a tough nut to crack!

Okay, let's add to the scenario I outlined. After I immediately delete everything except the snapshot, I then (again, a mental exercise) zero out the space occupied by the contents I deleted (don't worry about how I did that; again, an exercise), but again, leave the snapshot intact. What happens to the 'contents' of the snapshot now? Is it still able to restore what I nuked, or is it now also, and empty container?

I have these three cups ... and a pea. Follow closely!

[mr_raider]

Actually Clay, I forget to mention here another advantage to BTRFS - unified free space.

In the old EXT4 way, to have a separate home or a second linux install you must partition. Partitioning requires that you guess how much space you'll need for each partition and it makes it incredibly difficult to amend.

EXT4 example:
128GB SSD partitions:
1 - swap - 4GB
2 - kubuntu - 16GB
3 - other Linux - 16GB
4 - home - 92GB

Kubuntu ends up using 12GB - 4 wasted GB, "other" linux uses 8GB - 8 wasted GB. Or wait - what if other linux needs 20GB? Time to re-partition and hope gparted doesn't let you down.

BTRFS Example, same drive:
1 - swap 4GB
2 - everything else - 124GB.

Kubuntu, other linux, and home are in separate subvolumes but on the same filesystem. Free space is available to all the subvolumes. If other linux only uses 8 or grows to 20GB no matter - it expands and contracts as needed and no free space is languishing.

This is a big reason I use BTRFS on all my installs, no more playing Tetris with SSD partitions.

I have one ext4 install left, it's an ubuntu server where / and /home actually reside on separate physical drives. All my other PCs are BTRFS.

[oldgeek]

Well, I decided to give btrfs a try and installed Kubuntu 18.04 in a virtual machine (QEMU/KVN) with btrfs. When I checked it out, I saw no difference on Dolphin from ext4. I guess I did something wrong. When it came to the installing part, I clicked on Manual and got the menu. I couldn't figure out what to do next, so I clicked on the device name and a new partition came up, with the same value as the original (20 gigas). I clicked on btrfs and let it continue. The result? No discernable difference. The btrfs install is a sub-partition of the original device partition--is that OK? I'm really interested in trying out btrfs before I commit myself to a new install, but I want to try it out first. What did I do wrong?

What should I see when I click on Dolphin is a btrfs system? Thank you in advance.

[oshunluvr]

Well, I decided to give btrfs a try and installed Kubuntu 18.04 in a virtual machine (QEMU/KVN) with btrfs. When I checked it out, I saw no difference on Dolphin from ext4. I guess I did something wrong. When it came to the installing part, I clicked on Manual and got the menu. I couldn't figure out what to do next, so I clicked on the device name and a new partition came up, with the same value as the original (20 gigas). I clicked on btrfs and let it continue. The result? No discernable difference. The btrfs install is a sub-partition of the original device partition--is that OK? I'm really interested in trying out btrfs before I commit myself to a new install, but I want to try it out first. What did I do wrong?

What should I see when I click on Dolphin is a btrfs system? Thank you in advance.

You won't see any difference at the user level at all. BTRFS allows mounting if the subvolumes as though they were partitions. Open a terminal and type "mount" and enter. You should see both "/" and "/home" are on the same partition. If you look at /etc/fstab, you will see the lines to mount "/" and "/home" are identical except there will be an option subvol=@ on the root line and subvol=@home for the home mount.

To view the subvolumes you need to mount the root volume with no "subvol=" option.

I'll assume your partition is sda1, if not substitute the correct partition device name. Open a terminal and;

Create a mount point:
sudo mkdir /subvol

Mount the root file system:
sudo mount /dev/sda1 /subvol

Look at the listing:
ll /subvol

You should see

@
@home

in the listing.

Take a snapshot:
sudo btrfs subvolume snapshot /subvol/@ /subvol/@_snap1

List the subvolumes again:
ll /subvol

You should see:

@
@home
@_snap1

[oldgeek]

OK, I tried the three steps and they worked, so I guess I did get it right. However, when I list the subvolume, I not only get @ and @home, but also two others: './' and '../'. What are those?