Btrfs, Snapper and apt-btrfs-snapshot

[GreyGeek]

I've been playing with two CLI utilities for btrfs, snapper and apt-btrfs-snapshot.

After playing extensively with both I decided to delete apt-btrfs-snapshot. The primary reason is that I didn't want system wide PRE & POST snapsnots eating up HD space everytime I ran apt-get or the muon updater ran. Secondarily, the apt-btrfs-snapshots were created by mounting my partition and saving them as @apt-snapshot-yyyymmddhh:mm:ss, which are not visible without mounting my active partition again.

My Kubuntu partition is 336GB. With only Kubuntu 14.04 installed I was eating about 15% of the partition space. Over a period of five days, with the standard snapper configurations for both / (@) and /home (@home), and all the apt-get snapshots, about 60% of my partition was eaten up. Here is the root config file with my present settings:

# subvolume to snapshot
SUBVOLUME="/"

# filesystem type
FSTYPE="btrfs"

# users and groups allowed to work with config
ALLOW_USERS="jerry"
ALLOW_GROUPS=""

# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"


# run daily number cleanup
NUMBER_CLEANUP="yes"

# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="5"                [B]  <---- initially was 50[/B]
NUMBER_LIMIT_IMPORTANT="3"


# create hourly snapshots
TIMELINE_CREATE="yes"

# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"

# limits for timeline cleanup
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="0"  [B]<------ Initially was 10[/B]
TIMELINE_LIMIT_DAILY="1"      
TIMELINE_LIMIT_MONTHLY="3"
TIMELINE_LIMIT_YEARLY="1"

# cleanup empty pre-post-pairs
EMPTY_PRE_POST_CLEANUP="yes"

# limits for empty pre-post-pair cleanup
EMPTY_PRE_POST_MIN_AGE="1800" [B]  <----- I still don't know what the 1800 means ???[/B]

My changes took 60 snapshots off of my btrfs installation and gave me 66% free space. / and /home now have 4 snapshots each, instead of 30+, and those 8 snapshots are taking 20% of my partition.

I was curious to see if the /.snapshots directory snapshots contained and data in their "home" subdirectory. They did not. Likewise, my /home/.snapshots directory contained only my ~/jerry directory. So, there is a need when installing snapper to create the second /home account. When one uses snapper and does not signifiy which account to use then the action defaults to the root account. So, if one wants to create a snapshot in ones home account using snapper here is the syntax:
sudo snapper -chome create --description "my snapshot in home"
Without the "-chome" section your snapshot would be a snapshot of / and would be stored in /.snapshots.

Snapper also gives you the ability to do a "pre" and "post" snapshot and include a command in between. Snapper referes to snapshots by the number it creates for them, not by the btrfs ID number. Using the "status" or "diff" switches on snapper you can use the numbers of two snapshots to compare them and see what's changed. If you want to roll back the changes between snapshot #x and snapshot #y then use this command:
sudo -croot undochange x..y
It is important to specifiy either root or home because they both could have the same snapshot numbers (but not the same data) and if you meant to roll back the home account to what it was before "x" you might end up rolling back root instead.

Obviously, something like an apt-get may change both the / and /home/youracct subvolumes. So before you run apt-get sequences you'd run snapper to create a snapshot for both root and home specifying the "--pre-number". After the update you run snapper again with the same "pre" number, which snapper sees as a "post" number.

...
--pre-number number
For post snapshots the number of the pre snapshot must be provided.

And, the part that allows you to do a command in between "pre" and "post" snapshots:

--command command
Create a pre and post snapshot and run command in between.

Very powerful stuff!

[oshunluvr]

Nice report, GG. I see many uses for snapshots, but I think the snapper tool might need some work before prime-time.

I am surprised how much of your space was being used. Did you compare "df" to "btrfs fi df /" to verify free space was being reported correctly? Sometimes the standard df gets fooled, but maybe that's only with multi-device filesystems.

What about auto-removal of older snaps? I would think an incremental delete would make snapshots a must-have feature. Maybe a daily cronjob snap and a snap every time certain programs were run like apt-get install, then automatic removal of the last five snaps.

[GreyGeek]

When I deleted most of the snapshots I ran the btrfs defrager on both @ and @home, and I defraged the directory meta data, too.

sudo btrfs filesystem defragment -v -r -f / (and /home)
sudo find / (and /home) -xdev -type d -print -exec btrfs filesystem defragment '{}' \;

The used space was down to 44% when there were 8 snapshots. Now, there are 10 more and the usage has climbed to 54% (Conky reports 66% free)


Here's the current df utility and btrfs's df:
:~$ sudo df --total
Filesystem Type 1024-blocks Used Available Capacity Mounted on
/dev/sda4 btrfs 353010688 101708436 234676364 31% /
....
/dev/sda4 btrfs 353010688 101708436 234676364 31% /home
total 712657668 203428968 475979804 30% -

:~$ sudo btrfs fi df /
Data, single: total=178.00GiB, used=95.79GiB
...


31% vs 54%

As far as auto-removal of older snapshots there is a mechanism in snapper to do that:

  Cleanup Algorithms
       Snapper provides several algorithms to clean up old snapshots. The algorithms are executed in a daily cron-job. This can be
       configured in the corresponding configurations files along with parameters for every algorithm.

       number
           Deletes old snapshots when a certain number of snapshots is reached.

       timeline
           Deletes old snapshots but keeps a number of hourly, daily, monthly and yearly snapshots.

       empty-pre-post
           Deletes pre/post snapshot pairs with empty diffs.

Here is snapper's root config:

:~$ sudo snapper -croot get-config
Key                    | Value
-----------------------+------
ALLOW_GROUPS           |      
ALLOW_USERS            | jerry
BACKGROUND_COMPARISON  | yes  
EMPTY_PRE_POST_CLEANUP | yes  
EMPTY_PRE_POST_MIN_AGE | 1800 
FSTYPE                 | btrfs
NUMBER_CLEANUP         | yes  
NUMBER_LIMIT           | 5    
NUMBER_LIMIT_IMPORTANT | 3    
NUMBER_MIN_AGE         | 1800 
SUBVOLUME              | /    
TIMELINE_CLEANUP       | yes  
TIMELINE_CREATE        | yes  
TIMELINE_LIMIT_DAILY   | 1    
TIMELINE_LIMIT_HOURLY  | 0    
TIMELINE_LIMIT_MONTHLY | 3    
TIMELINE_LIMIT_YEARLY  | 1    
TIMELINE_MIN_AGE       | 1800

Snapper does do not explain the meaning of the config perameters very well. I haven't figured out what the "1800"means, but the other numbers refer to the number of snapshots. I "think" the total number of snapshots existing at any moment with the settings shown amounts to 5+3 = 8. And, with a TIMELINE_LIMIT_DAILY of 1, I see these snapshots in /.

:~$ sudo snapper -croot list
Type   | #   | Pre # | Date                            | User | Cleanup  | Description | Userdata
-------+-----+-------+---------------------------------+------+----------+-------------+---------
single | 0   |       |                                 | root |          | current     |         
single | 1   |       | Sun 26 Jan 2014 09:17:01 AM CST | root | timeline | timeline    |         
single | 88  |       | Thu 30 Jan 2014 12:17:01 AM CST | root | timeline | timeline    |         
single | 104 |       | Thu 30 Jan 2014 04:17:01 PM CST | root | timeline | timeline    |         
single | 105 |       | Thu 30 Jan 2014 05:17:01 PM CST | root | timeline | timeline    |         
single | 106 |       | Thu 30 Jan 2014 06:17:01 PM CST | root | timeline | timeline    |         
single | 107 |       | Thu 30 Jan 2014 07:17:01 PM CST | root | timeline | timeline    |         
single | 108 |       | Thu 30 Jan 2014 08:17:01 PM CST | root | timeline | timeline    |         
single | 109 |       | Thu 30 Jan 2014 09:17:01 PM CST | root | timeline | timeline    |

When I jump ahead of the cron task and do a manual clean up here are the results for root:

:~$ [FONT=courier new]sudo snapper -croot cleanup timeline[/FONT]
jerry@jerry-Aspire-V3-771:~$ sudo snapper -croot list
Type   | #  | Pre # | Date                            | User | Cleanup  | Description | Userdata
-------+----+-------+---------------------------------+------+----------+-------------+---------
single | 0  |       |                                 | root |          | current     |         
single | 1  |       | Sun 26 Jan 2014 09:17:01 AM CST | root | timeline | timeline    |         
single | 88 |       | Thu 30 Jan 2014 12:17:01 AM CST | root | timeline | timeline    |         
jerry@jerry-Aspire-V3-771:~$

which seems to agree with "NUMBER_LIMIT=3"

EDIT: I found the meanings of the config terms:

http://doc.opensuse.org/products/dra...snapper.limits

[oshunluvr]

I've never run defrag, but balance on multi-disk filesystems helps determine actual usage. From what I've read regular system df is almost useless on a btrfs raid. I'm not sure how that reflects on a single disk system.

[GreyGeek]

Well, when I limit the ability of snapper to auto create snapshots I'm left with the number 0 snapshot (created when snapper was first installed) and two timeline snapshots taken during the most recent two hours. Combined with /home's snapshots the six take up 20% of my HD space. I'm going to purge snapper and its hidden subdirectories and use btrfs's snapshot command by creating a script that snapshots / and /home before any planned updates or changes. If the updates or changes work out OK then I'll delete the snapshots. If not, then I'll use them to rollback and then delete them. Like you said, Oshunlover, snapper is not ready for prime time.

[oshunluvr]

I bet either of us could easily write a script and cronjob that would do it better than snapper.

You'd need a rollback script, maybe with some triggers like when you activate apt-get or how about in a service menu? Right-click on the desktop, choose "Make a snapshot", done. Hmmm, an easy GUI way to handle snaps would be very cool.

Maybe when I retire again, I'll have time <rolls eyes>...

[GreyGeek]

I was thinking about using kdialog with a bash script to do that.

Snapper is nasty. I made a crucial mistake removing it. Before I purged snapper I should have used it to delete all of its snapshots. However, I ASSuMEd that a purge of snapper would remove all of its files, dependencies and constructs. It does not. To make matters worse, in the /home/.snapshots/ directory were the timeline snapshots. Snapper saved them as readonly. Even as root using mc I cannot delete them so they must have the immutable setting on them. I mounted my partition as /mnt and even from there I could not remove them. Need to do some googling and finid out how to remove immutable ro files.

[oshunluvr]

Could be CoW causing your issue. Try mounting the root volume with nodatacow option or to change individual files/folders chattr +C <target> should stop CoW on them.

The nodatacow option will disable copy-on-write on the entire filesystem. You might want to go that route, delete all the snaps, then remount normally or reboot.

[GreyGeek]

I'll try that. I confirmed that /mnt was mounted as rw. And, I checked several of the files that I couldn't delete becaue they were ro, but I couldn't see an "i" immuntable flag.

[GreyGeek]

mounting it nodatacow did not work. When I use mc to view the file permissions that I can't remove they are shown as rw, but when I try to delete them using mc it complains that they are ro.
The lsattr does not show ro on any file, nor does it show the immutible attribute set on any of the files that I can't delete. Strange.

[GreyGeek]

I got it sorted out. I reinstalled snapper and using it I could delete all the / snapshots except "0". It wouldn't let me delete the /home snapshots because my messing around trying to delete them probably deleted the info.xml file.

So, I tried booting into a LiveCD and mounting my /dev/sda4 and using mc again, but the results were the same. While I was usiing the LiveCD I umounted /dev/sda4 and ran the btrfs check against it. No errors were reported. I googled for "snapper can't delete snapshot" and found a post by the guy who wrote snapper. He said that only the parts of the path named "snapshot" are subvolumes. The ".snapshots" and the numbered parts regular directories. So, I tried
sudo btrfs subvolume delete /home/.snapshots/114/snapshot
and it worked! I quickly deleted all the remaining snapshots on both / and /home. I used rmdir to remove /home/.snapshots/114 and the rest of the leftovers. Then I defraged / and /home and the meta data and then ran scrub.

I am back up to 70% free according to df and I've USED 73% (89GB out of 121GB on a 336GB partition) according to btrfs fi df / the remaining "unused" portion being reserved for the potential creation of a snapshot of the entire system.

LESSONS LEARNED:
1) The btrfs filesystem is very stable. And, it does not need swap partitions, so don't waste your disk space on swap if you want to use btrfs. Considering all the violence I did to it, it while learning how to use it, it never misbehaved and the btrfs check /dev/sda4 on the unmounted volume showed no errors. The documentation, however, is sparse. Btrfs is still is in such rapid development that writing extensive documentation at the current time would be mostly a wasted effort. Fortunately, if you use Google Search's time limitations to restrict your search to recent documents, and to Ubuntu or debian based documentation, information is not hard to find.

2) Btrfs is, IMO, not meant for single HDs or a single partition unless that HD or partition is TWICE OR 3X as big as the maximum size of your anticipated storage needs. My 336GB partition barely qualifies, IMO. After Kubuntu and all of my favorite apps are installed, and with NO snapshots, the size of @ and @home combined uses 90GB out of 121GB. IF I read "btrfs fi /" right, I have only 31GB of unused space left on which to install more apps or save more data before my btrfs filesystem becomes full. By comparison, a 336GB EXT4 partition would show about 30GB taken with Kubuntu and about 300GB remaining for additional applications and data, which is 10X as much as btrfs makes available with the same installs. However, the ability to easily add or remove other storage devices with relative ease, and the ability to make snapshots of your system and to roll back if adversity hits, makes it a worthwhile install, IMO. Certainly a lot easier than making tar backups of critical files and/or copyng them to other devices or locations and then doing the OS reinstall and copy routine. I have an active 7 port USB device plugged into my Acer. I could plug in 7 64GB USB sticks (448GB, of which about 120GB woudl be usable) and add them to my storage pool. Or, I could plug in my 350GB USB HD and get the same usable space. With development proceeding things can only get better. One caution: if you do something like that, and come close to filling up that additional space, you will have to add even more devices to give you working room while you trim down your stored files. Also, If you boot off of an EXT4 /boot partition be advised that snapshots do not apply, so rollbacks of Muon updates will not revert to the previous kernel, but the current kernel's headers in /usr/src will be gone.

3) Snapper , as Oshunlover said, is NOT ready for primetime. And, I doubt it will ever be because of design flaws, one of which is too many snapshots automatically created. True, one can change the snapper config settings, but then it becomes a matter of which ones are critical and how do you specifically tell snapper to do only those? Snapper isn't that fine grained, which is why it appears to me that custom shell scripts will do better. Snapshots can quickly eat up your disk space. It would be a LOT easier and more compatible to use btrfs commands in a shell script to do activities similar to that which snapper does, and have those scipts tailered exactly to your needs, and run ONLY when you need them. Snapper allows one to manually create "pre" and "post" snapshots and execute a command in between, which is something that one might want to do when using apt-get or when the Muon updater runs. The pre and post numbers which snapper uses to identify its snapshots can be used to remove changes made between then before they are deleted. Thus, "snapper -croot delete nn..mm" and "snapper delete -chome nn..mm" is supposed to remove any changes to the system caused by any commands run between those two snapshots. Enter apt-btrfs-snapshots.

4) The package apt-btrfs-snapshots is supposed to automatically make pre and post snapshots when ever apt-get is run, either in a console or by use of a wrapping gui. That's when I thought about snapper and apt-btrfs-snapshot busily churing away, breeding children automatically. The number of automatic snapshots being created on my system was eating my free space alive! Limiting snapper's snapshots to 4 each for @ and @home, but not being able to limit apt-btrfs-snapshots it became clear that after a few Muon updates and some apt-get update & apt-get upgrades of my own I realized that I had less than a week's worth of disk space remaining. I quickly purged apt-btrfs-snapshots but, to my dismay, it did NOT delete the snapshots it made! Fortunately, all of them were created as unmounted subvolumes, i.e., "@apt-snapshot-yyyymmddhh:mm:ss" on the partition my system was running on. It was easy to mount /dev/sda4 to /mnt and use "btrfs subvolume delete @apt-snapshot-yyyymmddhh:mm:ss", then umount.

5) Btrfs by itself is adequate. Be sparse with your snapshots. To give you and example of the power of btrfs on Kubuntu read the following article about making backups of your home account using "btrfs sub create": http://arstechnica.com/information-t...filesystems/3/

[Snowhog]

I hope/anticipate that you will write up a clear and concise report on your observations and provide it to the btrfs development team. It's quality 'testers' like yourself that development teams need to hear from.

[oshunluvr]

The only point GG made above I would differ slightly with is the utility of btrfs on a single disk.

IMO, btrfs has many benefits for smaller single-drive users but snapshots is not one of them. For example, if you have a small drive but want to multi-boot, you are stuck splitting your drive up into multiple partitions and thus waste a tremendous amount of space. With btrfs in the same scenario, you would need only a single file system and all your free space would be available to all your installs and your home all the time without managing partition sizes.

Regardless of filesystem size, snapshots require paying close attention to their maintenance. Snapshots are not a set-it-and-forget-it function as GG learned with his experiments. Under certain conditions; one-off upgrades like me or risky things like compiling from an little-known source, the ability is awesome. But you must return to the snapshot and remove it as soon as it is no longer needed. In theory, over time a snapshot could grow to the same size as the subvolume it was made from. Take 4 snapshots and you have the potential to use a lot of filesystem space! With a larger amount of free space and a heavier work load, a scheduled series of snapshots - with included removal of same as they age - is a very large benefit.

A snapshot is not a backup and should not be used to replace one. Rather, it's a "photograph" of your system that can be used to "remember" it previous state. Once that remembrance is no longer needed, chuck that snapshot like you would pictures of your ex-girlfriend.

[GreyGeek]

I recreated the /.snapshots subvolume and the /home/.snapshots subvolume. Then I created snapshots in both of them using the same name:
snap201402011310 (Feb 1, 2014 at 13:10PM).
Later, I ran sudo btrfs sub show / and sudo btrfs sub show /home and noticed the following, in bold print:

:~$ [FONT=courier new]sudo btrfs sub show /[/FONT]
/
        Name:                   @
        uuid:                   70593792-0f70-7c42-8e26-f1a7cae7d28e
        Parent uuid:            -
       [B] Creation time:          2014-01-21 17:09:59[/B]
        Object ID:              257
        Generation (Gen):       21115
        Gen at creation:        6
        Parent:                 5
        Top Level:              5
        Flags:                  -
        Snapshot(s):
                              [B]  .snapshots/snap201402011310[/B]

:~$ [FONT=courier new]sudo btrfs sub show /home[/FONT]
/home
        Name:                   @home
        uuid:                   c4055db3-93f1-cc4a-833d-d8d8e516ac0b
        Parent uuid:            -
        [B]Creation time:          2014-01-21 17:10:00[/B]
        Object ID:              258
        Generation (Gen):       21114
        Gen at creation:        9
        Parent:                 5
        Top Level:              5
        Flags:                  -
        Snapshot(s):
                                [B].snapshots/snap201402011310[/B]

I first thought that btrfs resurrected the old snapshots subvolumes that I created with snapper, but my Muon history shows that I didn't install snapper until 1/23/14, two days after the creation date given for those two subvolums, both of which I created earlier this afternoon! Muon's history shows that my first update on this system was 1/20/14. So, I have no clue where btrfs got its 1/21/14 creation date.

BTW, usage rose from 89.96GB to 90.45GB. The two snapshots were created in milliseconds.

[dibl]

Nice work, GG -- and nice writeup -- thanks for that. My impressions of BTRFS, after 3 years of use, pretty much agrees with yours. I don't see it as a meaningful improvement over ext4 for the single-user desktop or laptop system for everyday use, and I don't see anything indicating that the BTRFS developers had that target market in mind in the first place. I take oshunluvr's point that for the habitual upgrader, it does have a nice built in answer to the perpetual linux user's question "how do I go back?". But beyond that, there's nothing special for the OS filesystem.

Where BTRFS really shines and outperforms ext4 in numerous ways is its multi-device filesystem implementation for large data storage. I only use 2 drives, 1 TB each, for my data. But I can imagine a small or mid-size enterprise that happens to be in a line of business that generates lots of data, like the engineering firm that I retired from in 2010, really benefitting from a Linux-based server implementation with multi-drive BTRFS for main data storage and backup. Never mind RAID controllers and proprietary drivers -- you could simply start buying consumer-grade server motherboards and SATA hard drives and don't stop until you have enough terabytes, and some spares, to support your data storage requirements. Very, very economical and yet very secure and easy to support.