BTRFS in openSUSE Tumbleweed

[claydoh]

All this is just some evidence that btrfs is possibly not always useful for us mere mortal Desktop users

[kubicle]

All this is just some evidence that btrfs is possibly not always useful for us mere mortal Desktop users

As has been demonstrated, you don't really need snapper to enjoy the benefits of modern filesystems

[oshunluvr]

The point about the root folder is it's generally so small, why not include just include it in the root subvolume backup? Sure you have some stuff in there, do most people? I think not.

Well, my system automatically takes a snapshot every morning at 5am and makes a backup every Sunday morning as that seemed sufficient for my needs, so that doesn't apply to me. If you can't do a basic script and set a cronjob, there are plenty of automated snapshot tools, and AFAIK none of them require the hodgepodge approach used by openSuSE. If I wanted to, I could snapshot hourly and backup 4 times a day or whatever, but that seems overkill for me. In an Enterprise environment? Sure. more often would be better. Regardless, you can't protect people from themselves. Their setup or an Ubuntu one, you still have to make your snapshots and backups or you're SOL whatever subvolume scheme you have.

Recovering an accidentally deleted file is opening Dolphin, browsing to the identical location that I deleted it from - except in the snapshot subvol - and copying it back. Not exactly what I would describe as "scattered". BTW an ironic word to use considering the topic - "scattered" might be used to descrbe the btrfs setup openSuSE is using, LOL.

As far as a rollback because of a damaged update or whatever, taking a snapshot just before running update would be smart and automating it even smarter. I have had to recover from damage from a failed update not long ago. The manual sequence of events;

Take snapshot.
Do sudo apt full-upgrade.
Reboot.
Reboot fails - system won't boot.

Recovery:
1. Boot to live USB or in my case, a second install - ANY alternate install (I usually have 5-6 various distros or versions available).
2. Navigate to the BTRFS root file system and rename the damaged OS subvolume.
3. Re-snapshot the latest snapshot to the original subvol name
4. Reboot
Done

Wouldn't your example require a tool? Can't EFI get bashed by an update (honestly asking - I avoid it) along with the OS? How do you know which parts need to be restored? You don't and you have to rely on snapper or whatever? Sorry, I don't like that idea at all.

The re-snapshot part is due to my keeping my snapshots in a subfolder on the root FS rather than along side the bootable and home subvols, just for neatness. Also re-snapshotting a snapshot keep the original snapshot alive to reuse again if needed. Boot logs or whatever are still there - the "bad" subvolume doesn't have to be erased right away.

So here's the real difference in my mind: I could talk someone through my procedure in a few minutes. But with their setup? You'd BETTER have a vendor utility or an average user is struggling or just plain out of luck. That to me means a utility is required. Not in my case.

Also, "Snapper" was a disaster when it was introduced. I can't help but think they must have improved it, but anyway, no thanks. My Dolphin BTRFS service menu and my anacron snapshot script is enough for my needs. I'm sure there's some use-case that all that mess applies to, but none that I know anything about. Honestly I can't see one single thing about their config that helps a single machine regular desktop user.

Your installation must be way more complicated than mine, which I would expect if one were a developer.

[kubicle]

The point about the root folder is it's generally so small, why not include just include it in the root subvolume backup?

Because I don't need or want to backup the whole root ("/") subvolume. When you have a dozen hosts pushing daily, weekly and monthly backups on a local server, and that server pushing encrypted backups on a remote server, it gets kind of important that you back up only the things you need to, and not unnecessary data...for efficiency reasons (even if you have only a few hosts, a couple dozen extra gigs backed up off-site over the internet tends to add up in both space and bandwidth...and it can take a while, too). We all know on-disk snapshots are not backups.

Well, my system automatically takes a snapshot every morning at 5am and makes a backup every Sunday morning as that seemed sufficient for my needs, so that doesn't apply to me. If you can't do a basic script and set a cronjob, there are plenty of automated snapshot tools, and AFAIK none of them require the ho dgepodge approach used by openSuSE.

Sure, but "more than 2 subvolumes" still doesn't count as "hodgepodge" in my book. People have been using separate partitions for various directories even on desktops for ages for various reasons, this is much more manageable with no real downsides. Is it required or always useful, no. Does it have benefits in some use cases, yes. Is there obvious harm in any use case, no.

Recovering an accidentally deleted file is opening Dolphin, browsing to the identical location that I deleted it from - except in the snapshot subvol - and copying it back. Not exactly what I would describe as "scattered". BTW an ironic word to use considering the topic - "scattered" might be used to descrbe the btrfs setup openSuSE is using, LOL.

I was not talking about a single file, I meant scalability...let's say you removed 27.137 files matching a specific regex across several filesystems accidentally. Can you recover them manually, or with a script or another tool, sure. But I don't see how their way is worse? (But I can see how drag-and-drop in Dolphin is worse.)

As far as a rollback because of a damaged update or whatever, taking a snapshot just before running update would be smart and automating it even smarter. I have had to recover from damage from a failed update not long ago. The manual sequence of events;

It obviously has to be automatic, or it doesn't exist. And this makes it automatic. Can do use your own solution? yes, but the tasks (for making all your own solutions) are sort of starting to build up here (and I still don't see how their way is worse? Especially for a "regular" user uncomfortable with things like scripting).

Take snapshot.
Do sudo apt full-upgrade.
Reboot.
Reboot fails - system won't boot.

Recovery:
1. Boot to live USB or in my case, a second install - ANY alternate install (I usually have 5-6 various distros or versions available).
2. Navigate to the BTRFS root file system and rename the damaged OS subvolume.
3. Re-snapshot the latest snapshot to the original subvol name
4. Reboot
Done

You can do that manually, too, still don't see how their way is worse. And you can still do just that if you use their system, just restore / (you don't need to restore the rest of the subvolumes)

Wouldn't your example require a tool?

Yes, snapper. (or you can do it manually if you wish)..you can boot your snapshots from grub. You can just make the creation of boot entries automatic with snapper (which makes them quite a bit more accessible to the "regular" user).

Can't EFI get bashed by an update (honestly asking - I avoid it) along with the OS?

Everything is possible, but you still have the manual restoration option if all else fails.

How do you know which parts need to be restored? You don't and you have to rely on snapper or whatever?

You usually do have a fairly good idea what is NOT causing the issue, like that icon theme or a piece of software you have worked on in /usr/local for hours in the morning and don't wish to lose. And you always can restore all, it's just an option to pick and choose if you wish (and have a hunch where the problem is or not).

Also re-snapshotting a snapshot keep the original snapshot alive to reuse again if needed. Boot logs or whatever are still there - the "bad" subvolume doesn't have to be erased right away

But they could be inaccessible or deleted , depending on what went wrong. (this could obviously also happen on /var/log subvolume, but is generally less likely because of the compartmentalization).

So here's the real difference in my mind: I could talk someone through my procedure in a few minutes.

Someone, yes. Everyone, no. That's why we have vendor utilities.

My Dolphin BTRFS service menu and my anacron snapshot script is enough for my needs. I'm sure there's some use-case that all that mess applies to, but none that I know anything about. Honestly I can't see one single thing about their config that helps a single machine regular desktop user.

Like I said, everyone should use what fits their needs best, or even what they want to use (in the case it does not fit their needs the best), but that doesn't really mean other solutions are a mess (especially since the "mess" is essentially still "there are more than two subvolumes which I'm used to"). Single machine desktop users don't really have to worry about the configuration or the technical details, they can use the vendor utilities to roll back/backup or whatever. Nothing stops from creating their own snapshotting schemes, timed and automatic before upgrades, and scribbling down their own recovery procedures etc. if they wish, but that's not going to happen without effort either.

To me it seems regular desktop users could use their system fairly effortlessly (I haven't actually used it so it may be that you're right and it's a mess after all, but the idea is sound). Am I going to switch to a system like that in the foreseeable future, no. I kind of like the way I have my rigs set up, and it has served me well for years. But if I had to start from scratch, I'd give it a go (although I'd probably spent similar hours configuring it as I would rewriting my current set up...I'm rarely fully satisfied with someone else's solution, even if it's clearly better than mine).

[GreyGeek]

....
Also, "Snapper" was a disaster when it was introduced. I can't help but think they must have improved it, but anyway, no thanks. ....

I tried snapper when I first adopted BTRFS almost 5 years ago. Playing around with the Snapper in Tumbleweed I can see some definite improvements in speed and functionality. I also have experience with snapper snapping and leaving a mess. I learned to clean it up which, unlike possible messes by TimeShift if you delete it without first deleting the snapshots you made while using it, which wasn't too hard.

So, for grins and giggles, I am going to install snapper and its gui from the repository and play with it again, to see how it has changed and/or improved. One thing a manual snapshot under <ROOT_FS> has going for it as long as you can boot <ROOT_FS> you can rollback @ or @home from a LiveUSB that has BTRFS installed.

In snapper, the subdirectories into which subvolumes are stored are under /.snapshots and /home/.snapshots. Those are inside @ and @home. IF either of them break your odds of recovering them are slim to none. How do you roll back to a previous snapper snapshot number if you can't access snapper or /etc/snapper/snapper.conf? Anyway, I'm going to play around and see what improvements, if any, have been made.

Thanks for the contributions to the thread everyone!

[jlittle]

My 2c...

I use snapper, and the separate snapper-gui. The automatic 2 hourly snapshots have proven their worth for me; perhaps I'm prone to computer clumsiness. A script in APT does automatic snapshots before full-upgrade; I'm not sure if snapper set that up, but it happens even when I've stopped snapper. do-release-upgrade created pre and post snapshots (unnecessarily, because I ran it on a snapshot anyway, but it matters little.)

My first reaction reading this thread about the multiplicity of Suse subvolumes was similar to other posters, but on reflection I've come around a bit. The rationale about rollbacks seems very arbitrary. In the 4 years since I installed btrfs I've been splitting up the @ subvolume a little, as different parts of it are better with different backup strategies, so the Suse approach makes some sense. But I think their splits will only suit a fraction of users; I suspect that some Suse people evolved splits to suit themselves, and sent them out to the world. I suppose that it's good as an example. The separate /root subvolume has me shaking my head, but I run a Ubuntu derivative where logging on as the root user doesn't happen.

[kubicle]

In snapper, the subdirectories into which subvolumes are stored are under /.snapshots and /home/.snapshots. Those are inside @ and @home. IF either of them break your odds of recovering them are slim to none. How do you roll back to a previous snapper snapshot number if you can't access snapper or /etc/snapper/snapper.conf? Anyway, I'm going to play around and see what improvements, if any, have been made.

Whether the snapshot hierachy is nested or flat, it really shouldn't matter. If you lose the "original" due to filesystem corruption or disk failures, the snapshots aren't going to do you much good (they aren't backups) whether they're nested or flat. if the filesystem is ok (mounts ok), you can use the snapshots. Please report back on your experiences, hands on notes are worth more than my theoretical ramblings.

The separate /root subvolume has me shaking my head, but I run a Ubuntu derivative where logging on as the root user doesn't happen.

In case it wasn't clear, I don't log in as root (except from a terminal), but root still has it's settings, ssh-keys, git-configuration, zsh-configuration, vpn-certificate infrastructure, byobu (tmux) settings, mostrc, vimrc, aliases just to name a few. It isn't much in kilobytes, mostly just text configs...but it still stuff that I want to keep and backup regularly, so having a subvolume for it makes sense (I don't currently have one, but it's still an essential part of my back up regime). Some of it (the host-agnostic and not sensitive) is also on my git server for easy push/pulls across hosts, but it's always good to have more than one back up strategies.

[GreyGeek]

My 2c...

I use snapper, and the separate snapper-gui. The automatic 2 hourly snapshots have proven their worth for me; perhaps I'm prone to computer clumsiness. A script in APT does automatic snapshots before full-upgrade; I'm not sure if snapper set that up, but it happens even when I've stopped snapper. do-release-upgrade created pre and post snapshots (unnecessarily, because I ran it on a snapshot anyway, but it matters little.)

I always do a "sudo apt update, sudo apt full-upgrade". After I installed snapper I now get pre-post entries in the "snapper -h root list"

$ [B]sudo snapper -c root list[/B]
 # | Type   | Pre # | Date                            | User  | Cleanup | Description        | Userdata
---+--------+-------+---------------------------------+-------+---------+--------------------+---------
0  | single |       |                                 | root  |         | current            |         
1  | single |       | Thu 30 Apr 2020 10:02:29 PM CDT | jerry |         | single following 0 |         
2  | pre    |       | Thu 30 Apr 2020 10:28:31 PM CDT | root  | number  | apt                |         
3  | post   |     2 | Thu 30 Apr 2020 10:28:32 PM CDT | root  | number  | apt                |         
4  | single |       | Fri 01 May 2020 10:40:52 AM CDT | root  | number  | boot               |

The #4 single was entered because of the boot up this morning. My home listing shows no such entry.

[B]$ sudo snapper -c home list[/B]
 # | Type   | Pre # | Date                            | User  | Cleanup | Description        | Userdata
---+--------+-------+---------------------------------+-------+---------+--------------------+---------
0  | single |       |                                 | root  |         | current            |         
1  | single |       | Thu 30 Apr 2020 10:02:54 PM CDT | jerry |         | single following 0 |

I have to specifically make a manual entry for my home account after an full-upgrade. Here is the result of installing kreversi using muon:

[B]$ sudo snapper -c root list[/B]
 # | Type   | Pre # | Date                            | User  | Cleanup | Description        | Userdata
---+--------+-------+---------------------------------+-------+---------+--------------------+---------
0  | single |       |                                 | root  |         | current            |         
1  | single |       | Thu 30 Apr 2020 10:02:29 PM CDT | jerry |         | single following 0 |         
2  | pre    |       | Thu 30 Apr 2020 10:28:31 PM CDT | root  | number  | apt                |         
3  | post   |     2 | Thu 30 Apr 2020 10:28:32 PM CDT | root  | number  | apt                |         
4  | single |       | Fri 01 May 2020 10:40:52 AM CDT | root  | number  | boot               |         
[COLOR=#ff0000]5 [/COLOR] | pre    |       | Fri 01 May 2020 12:23:49 PM CDT | root  | number  | apt                |         
[COLOR=#ff0000]6[/COLOR]  | post   |     5 | Fri 01 May 2020 12:23:52 PM CDT | root  | number  | apt                |

The home listing showed no changes.

However, there are tons of applications that store files in both @ and @home, and not coordinating snapshots of @ and @home is, IMO, a serious shortcoming if you want to restore your total system to what it was before any particular event. Because of that, five years ago, I created a bash script to do my own pre-post snapshots using snapper. It appears to be necessary today if one wants to keep coordinated snapshots of @ and @home.

#!/bin/bash
# created by Jerry L Kreps on July 20, 2015 and released under the GPL 2.0.
# This script merely creates a snapshot in both root and home with the designation 
# of PRE or POST as the type, which indicates that the user created it before an 
# action or afterwards.
# This script is run with PRE as the TYPE before an action which you may want to reverse
# AND this script run again with POST as the TYPE after that action has been completed.  Both snapshots are singletons because
# no timeline is used.  Only the "PRE" or"POST" in the description, along with a timestamp, 
# links the pre to the post snapshot.
# To reverse the action run the following two snapper commands:
#
# snapper -c home udochange n..m   
# sudo snapper -c root undochange o..p
# 
# where n or o is the number of the PRE and m or p is # the number of the POST snapshot
#
# or 
#
# sudo snapper -c home rollback n
# sudo snapper -c root rollback o
# 
# and rebooting after either method.
#
# After running those two commands both of the empty snapshots folders can be deleted using
#
# snapper -c home delete n-m
# sudo snapper -c root delete o-p
#
#
# actual program starts here
NOW=$(date +%Y%m%d%H%M)
echo Enter snapshot type PRE or POST:
read TYPE
echo Enter description
read DESC
STR=$TYPE" "$DESC" "$NOW
echo $STR
HCMD='snapper -c home create -d "'${STR}'"'
RCMD='sudo snapper -c root create -d "'${STR}'"'
eval "$HCMD"
eval "$RCMD"

My first reaction reading this thread about the multiplicity of Suse subvolumes was similar to other posters, but on reflection I've come around a bit. The rationale about rollbacks seems very arbitrary. In the 4 years since I installed btrfs I've been splitting up the @ subvolume a little, as different parts of it are better with different backup strategies, so the Suse approach makes some sense. But I think their splits will only suit a fraction of users; I suspect that some Suse people evolved splits to suit themselves, and sent them out to the world. I suppose that it's good as an example. The separate /root subvolume has me shaking my head, but I run a Ubuntu derivative where logging on as the root user doesn't happen.

IF ALL applications installed by the user, only for the user's use, were stored in the user's account, including libraries, gui's, datafiles, configuration files, etc., sort of like an AppImage, then one could keep their @home snapshots independently of their @root snapshots, and dividing @root into several smaller subvolumes wouldn't matter. Rollback to a previous @home snapshot wouldn't bother anything in the @ subvolume(s). But, that is currently not the case.

Also, it is still, IMO, a serious problem to store the root .snapshot directory under root, in /.snapshots and the home snapshot directory under home, in /home/.snapshots. If, for example, the active root subvoume were corrupted and one could not mount @, then there would be no way to access previous snapshots stored in /.snapshots, and recovery would be impossible. Ditto for previous snapshots of @home stored in /home/.snapshots. However, storing @ and @home snapshots in an /mnt/snapshots subdirectory, where /mnt is mounted using

mount /dev/disk/by-uuid/someuuid /mnt

are accessible IF <ROOT_FS> is mountable, i.e., the HD. Only if /mnt/snapshots is corrupted would access to the @yyyymmdd and @homeyyyymmdd snapshots not be possible. If the HD is not mountable then all is lost and reformatting the HD, if possible, would allow one to re-use it for what ever purposes. One could install the OS using BTRFS as the root file system, and use btrfs send & receive to call back @yyyymmdd and @homeyyyymmdd snapshots from an external storage device to reconstruct @ and @home by making snapshots of them without the -r parameter;

btrfs subvol snapshot /mnt/snapshots/@yyyymmdd /mnt/@
btrfs subvol snapshot /mnt/snapshots/@homeyyyymmdd /mnt/@home
reboot

would restore the system to what it was when those two snapshots were sent off to external storage.

Over all, snapper's capabilities and limitations have not appeared to change over the last five years since I last tried it. Ergo, I am purging it from my system. It is just so easy to do BTRFS snapshots manually, and to archive them, and you don't need a nested file system, or a gui, to do it.

[kubicle]

Also, it is still, IMO, a serious problem to store the root .snapshot directory under root, in /.snapshots and the home snapshot directory under home, in /home/.snapshots. If, for example, the active root subvoume were corrupted and one could not mount @, then there would be no way to access previous snapshots stored in /.snapshots, and recovery would be impossible.

I don't understand your problem. If your @ is corrupted, all your on-filesystem snapshots of it are worthless, regardless of where they reside, top-level or nested. On-filesystem snapshots are not full backups, they are just deltas (changes from the original, that's why they are initially very small, but tend to grow over time as more changes are made), you can't restore anything from the delta alone (in this case, if @ is corrupted). It would be like trying to restore a play by Shakepeare by knowing only that word number 2367 was previously "You" (this is not a completely accurate analogy technically, but you get the idea).

If you'd still prefer to have your snapshots in top-level (for whatever reason), you can have your .snapshots to be a separate top-level subvolume, there are instructions for this in the Snapper section of archwiki, for example: https://wiki.archlinux.org/index.php...esystem_layout

However, there are tons of applications that store files in both @ and @home

Applications do store things in @home, but no upgrade ever touches @home, so snapshotting it for upgrades doesn't make that much sense (especially to do it automatically every single time you upgrade anything)

[GreyGeek]

I don't understand your problem. If your @ is corrupted, all your on-filesystem snapshots of it are worthless, regardless of where they reside, top-level or nested. On-filesystem snapshots are not full backups, they are just deltas (changes from the original), not full back ups, you can't restore anything from the delta alone (in this case, if @ is corrupted).

If you'd still prefer to have your snapshots in top-level (for whatever reason), you can have your .snapshots to be a separate top-level subvolume, there are instructions for this in the Snapper section of archwiki, for example: https://wiki.archlinux.org/index.php...esystem_layout

I understand what you are saying, but I do a lot of messing around and on occasions I break things. On several occasions, after my experimenting, @ has been unable to load and I recovered by booting from a persistent liveusb with btrfs installed and merely restoring @ from /mnt/snapshots/@yyyyddmm and rebooting. Apparently, sometimes, when the part of @ that boots gets corrupted the bootable part of @yyyymmdd doesn't ? I've also restored @ from @yyyymmdd after retrieving it from my 3rd HD. But, all of that had happened on 18.04 and 16.04. One time, on 16.04, I had to reformat and reinstall. So far, I've never had a problem with @ or @home on 20.04 caused by my testing things out, or changing things, like removing the snapd loop device mount points, experimenting, etc...

[kubicle]

@ has been unable to load

Sure, but just because it doesn't load doesn't mean it's corrupted (which would make all your on-filesystem snapshots unusable). It's a different problem that could just as well be resolved by restoring a nested snapshot...or skip the livecd altogether and just boot the snapshot from grub and restore @ from there. There is nothing wrong with using a live-cd either, but you shouldn't think that you can use a livecd to restore a top-level on-filesystem snapshot in case a nested snapshot becomes unusable because of filesystem corruption or hardware failure, it doesn't work that way. You need off-filesystem back ups in addition to your on-filesystem snapshots (having the snapshot top-level makes no difference in this regard).

[jlittle]

It is just so easy to do BTRFS snapshots manually

Great, but ... maybe I'll forget. I'd like it to happen automatically. Ok, a simple cron job. But then, the snapshots pile up, so let's have another cron job to judiciously prune those snapshots, according to some reasonable schedule. I'd like a snapshot before something that might screw up too, I can do it manually, but ... maybe I'll forget. Ok, just use hooks (apt, systemd...) to run a snapshot beforehand. I'd have to extend the pruning job to cope with these.

Well, I've just implemented most of snapper. If that's what I want, why not use snapper?

There's some extra functions I don't use, but it's good that someone makes it track changes in APT and the like (f.ex. do-release-upgrade) and the boot process.

[GreyGeek]

Great, but ... maybe I'll forget. I'd like it to happen automatically. Ok, a simple cron job. But then, the snapshots pile up, so let's have another cron job to judiciously prune those snapshots, according to some reasonable schedule. I'd like a snapshot before something that might screw up too, I can do it manually, but ... maybe I'll forget. Ok, just use hooks (apt, systemd...) to run a snapshot beforehand. I'd have to extend the pruning job to cope with these.

Well, I've just implemented most of snapper. If that's what I want, why not use snapper?

There's some extra functions I don't use, but it's good that someone makes it track changes in APT and the like (f.ex. do-release-upgrade) and the boot process.

When you do apt upgrades and snapper puts in a pre & post root snapshot, but not home's, what do you do for snapshotting your home account?

BTW, I noticed apt-btrfs-snapshot in the repository and installed it. If fails with a namespace error when I attempted to test it.

[kubicle]

When you do apt upgrades and snapper puts in a pre & post root snapshot, but not home's, what do you do for snapshotting your home account?

As I mentioned before, apt (upgrades, installs or purges) never touch anything in @home, so snapshotting it automatically for every apt operation is not really necessary (@home never changes).

The use case for automatic apt snapshots is that you can easily recover from broken upgrades/installs (or to revert a purge that took away a number of packages because of dependencies), in which case you never need to restore @home. The automatic apt snapshots are not a good option for things like "I want to go back to back to before I installed gnome a few months back and have been using since", as you can't rely on automatic snapshots still being available as these are (and should be) removed in a schedule. For things like that you should create manual snapshots, which you can also do with snapper (then you could include a snapshot of @home as *running* gnome will change it)...or stick to timelined snapshots, if enabled (which are also removed on schedule, but older ones are kept longer).

But manual snapshots are usually your best option for specific "rollback points" (of course you should also take care that the restore of @home doesn't take away other changes you've made since then or your data files if they are on @home and not in a separate subvolume (or filesystem/disk)...so often it's actually easier/better not to restore @home at all, but you can if you wish to.