Btrfs snapshots using btrbk

[GreyGeek]

I've looked into other snapshot tools for Btrfs and the last one I have yet to look at is BtrBk. Here is the Readme in the app's package. btrbk has a couple of caveats:
1) btrbk does NOT do restores. You will have to do that manually. And when you do restore
2) Don't mv a ro snapshot to @ or @home and then use the property set to make it rw. The reason is that this low-level command leaves “Received UUID” in a false state which causes btrbk to fail on subsequent incremental backups. Instead, use a btrfs subvolume snapshot created without the -r flag.

So, like snapper, btrbk creates snapshots automatically based on hourly, daily, weekly, monthly and yearly schedules. It will also create incremental snapshots, but that ability comes with a warning related to the use of "RAW" output of a btrbk send & receive command:

For incremental backups ("incremental yes"), please note that:

• As soon as a single incremental backup file is lost or corrupted, all later incremental backups become invalid, as there is no common parent for the subsequent incremental images anymore. This might be a good compromise for a vacation backup plan, but for the long term make sure that a non-incremental backup is triggered from time to time.
  
  
• There is currently no support for rotation of incremental backups: if incremental is set, a full backup must be triggered manually from time to time in order to be able to delete old backups.
  

The "RAW" parameter is NOT part of the Btrfs command structure and is produced by using the "-f" parameter, which sends the snapshot as an ASCII data stream, which btrbk divides into to text files: the main data file containing the btrfs send stream, and a sidecar file ".info" containing metadata:
<snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|.xz][.gpg]
<snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|.xz][.gpg].infoIn summary, btrbk is just the first part of snapper, except that the snapshots are saved at the <ROOT_FS> level, which is superior to snapper's technique. Snapper, as you may recall, stores its snapshots UNDER "/" and "/home", inside the @ and @home subvolumes. As long as grub boots without problems you'll be OK using Snapper because you can access your snapshots, otherwise you cannot. Btrbk requires you to rollback manually, meaning that as long as you can mount your system HD you can access @, @home and the <ROOT_FS>/snapshots subdirectory.

Taking snapshots is drop-dead easy, regardless of how often you do it. Rolling back is equally as easy. Since btrbk only does half of those tasks its utility is questionable. IF one is not doing software development, or some other activity where important changes are made minute by minute or hour by hour, then taking snapshots hourly or daily is not necessary. Taking them before significant changes in your system is all that is required, and nothing has to be running in cron to do that. If I do "sudo apt update" and see that 197 apps are coming down the pipe my first action will be to open a Konsole, "sudo -i" to root, mount my system drive to /mnt and take two dated snapshots, one of @ and one of @home. Then I umount /mnt and exit root. Then I run "sudo apt full-upgrade" and start the process. IF all ends well and my system runs OK and boots without problems I won't be rolling back, but those dated snapshots are there just incase something happens later. To keep them safe I send & receive them to my archival HDs. When I accumulate 6-10 pair of snapshots on my system HD I cull them back to 4 pair by using "btrfs subvol delete -C /mnt/@YYYYMMDD", etc... About 20 snapshot pairs reside on my archival HDs.

Here is btrbk's abridged README.txt.

Introduction
btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs specific capabilities to create atomic snapshots and transfer them incrementally to your backup locations.

The source and target locations are specified in a config file, which allows to easily configure simple scenarios like “laptop with locally attached backup disks”, as well as more complex ones, e.g. “server receiving backups from several hosts via ssh, with different retention policy”.

Key Features:

• Atomic snapshots
• Incremental backups
• Configurable retention policy
• Backups to multiple destinations
• Transfer via ssh
• Resume of backups (if backup target was not reachable for a while)
• Encrypted backups to non-btrfs destinations
• Wildcard subvolumes (useful for docker and lxc containers)
• Transaction log
• Comprehensive list and statistics output
• Resolve and trace btrfs parent-child and received-from relationships
• Display file changes between two backups

btrbk is designed to run as a cron job for triggering periodic snapshots and backups, as well as from the command line (e.g. for instantly creating additional snapshots).


Synopsis
Please consult the btrbk(1) man-page provided with this package for a full description of the command line options.

Configuration File
Before running btrbk, you will need to create a configuration file. You might want to take a look at btrbk.conf.example provided with this package. For a detailed description, please consult the btrbk.conf(5) man-page.

When playing around with config-files, it is highly recommended to check the output using the dryrun command before executing the backups:

btrbk -c /path/to/myconfig -v dryrun

This will read all btrfs information on the source/target filesystems and show what actions would be performed (without writing anything to the disks).

Example: laptop with usb-disk for backups
In this example, we assume you have a laptop with:

• a disk having a btrfs root subvolume (subvolid=5) mounted on /mnt/btr_pool, containing a subvolume rootfs for the root filesystem (i.e. mounted on /) and a subvolume home for the user data,
• a directory or subvolume /mnt/btr_pool/btrbk_snapshots which will hold the btrbk snapshots,
• a backup disk having a btrfs volume mounted as /mnt/btr_backup, containing a subvolume or directory mylaptop for the incremental backups.

Retention policy:

• keep all snapshots for 2 days, no matter how frequently you (or your cron-job) run btrbk
• keep daily snapshots for 14 days (very handy if you are on the road and the backup disk is not attached)
• keep monthly backups forever
• keep weekly backups for 10 weeks
• keep daily backups for 20 days

/etc/btrbk/btrbk-mylaptop.conf:

snapshot_preserve_min 2d
snapshot_preserve 14d


target_preserve_min no
target_preserve
 20d 10w *m
snapshot_dir btrbk_snapshots


volume /mnt/btr_pool
subvolume rootfs
target send-receive
 /mnt/btr_backup/mylaptop

subvolume home
target send-receive /mnt/btr_backup/mylaptop

/etc/cron.daily/btrbk:

#!/bin/sh
exec /usr/sbin/btrbk -q -c /etc/btrbk/btrbk-mylaptop.conf run


This will create snapshots on a daily basis:

• /mnt/btr_pool/btrbk_snapshots/rootfs.YYYYMMDD
• /mnt/btr_pool/btrbk_snapshots/home.YYYYMMDD

And create incremental backups in:

• /mnt/btr_backup/mylaptop/rootfs.YYYYMMDD
• /mnt/btr_backup/mylaptop/home.YYYYMMDD

If you want the snapshots to be created only if the backup disk is attached, simply add the following line to the config:

snapshot_create ondemand

Example: host-initiated backup on fileserver
Let’s say you have a fileserver at “myserver.mydomain.com” where you want to create backups of your laptop disk, the config would look like this:

ssh_identity
 /etc/btrbk/ssh/id_rsa

volume /mnt/btr_pool
subvolume rootfs
target send-receive /mnt/btr_backup/mylaptop
target send-receive ssh://myserver.mydomain.com/mnt/btr_backup/mylaptop

In addition to the backups on your local usb-disk mounted at /mnt/btr_backup/mylaptop, incremental backups would also be pushed to myserver.mydomain.com.

Example: fileserver-initiated backups from several hosts
If you’re a sysadmin and want to trigger backups directly from your fileserver, the config would be something like:

ssh_identity /etc/btrbk/ssh/id_rsa

volume ssh://alpha.mydomain.com/mnt/btr_pool
subvolume rootfs
target send-receive /mnt/btr_backup/alpha

 subvolume home
target send-receive /mnt/btr_backup/alpha

volume ssh://beta.mydomain.com/mnt/btr_pool
subvolume rootfs
target send-receive /mnt/btr_backup/beta

 subvolume dbdata
target send-receive /mnt/btr_backup/beta

This will pull backups from alpha/beta.mydomain.com and locally create:

• /mnt/btr_backup/alpha/rootfs.YYYYMMDD
• /mnt/btr_backup/alpha/home.YYYYMMDD
• /mnt/btr_backup/beta/rootfs.YYYYMMDD
• /mnt/btr_backup/beta/dbdata.YYYYMMDD

Example: local time-machine (hourly snapshots)
If all you want is to create snapshots of your home directory on a regular basis:

/etc/btrbk/btrbk.conf:

timestamp_format long
snapshot_preserve_min 18h
snapshot_preserve 48h 20d 6m

volume /mnt/btr_pool
snapshot_dir btrbk_snapshots
subvolume home

/etc/cron.hourly/btrbk:

#!/bin/sh
exec /usr/sbin/btrbk -q run

Note that you can run btrbk more than once an hour, e.g. by calling sudo btrbk run from the command line. With this setup, all those extra snapshots will be kept for 18 hours.

Example: multiple btrbk instances

Let’s say we have a host (at 192.168.0.42) running btrbk with the setup of the time-machine example above, and we need a backup server to only fetch the snapshots.

/etc/btrbk/btrbk.conf (on backup server):

target_preserve_min no
target_preserve 0d 10w *m


volume ssh://192.168.0.42/mnt/btr_pool
subvolume home
snapshot_dir btrbk_snapshots
snapshot_preserve_min all
snapshot_create no


 target send-receive /mnt/btr_backup/my-laptop.com

If the server runs btrbk with this config, 10 weeklies and all monthlies are received from 192.168.0.42. The source filesystem is never altered because of snapshot_preserve_min all.

Example: backup from non-btrfs source
First create a btrfs subvolume on the backup server:

# btrfs subvolume create /mnt/btr_backup/myhost_sync

In your daily cron script, prior to running btrbk, sync your source to myhost_sync, something like:

rsync -a --inplace --delete -e ssh myhost.mydomain.com:/data/ /mnt/btr_backup/myhost_sync/

Then run btrbk, with myhost_sync configured without any targets as follows:

volume /mnt/btr_backup
subvolume myhost_sync
snapshot_name myhost

 snapshot_preserve_min latest

snapshot_preserve 14d 20w *m

This will produce daily snapshots /mnt/btr_backup/myhost.20150101, with retention as defined with the snapshot_preserve option.
Note that the provided script: “contrib/cron/btrbk-mail” has support for this!

Example: encrypted backup to non-btrfs target
If your backup server does not support btrfs, you can send your subvolumes to a raw file. This is an experimental feature: btrbk supports “raw” targets, meaning that similar to the “send-receive” target the btrfs subvolume is being sent using btrfs send (mirroring filesystem level data), but instead of instantly being received (btrfs receive) by the target filesystem, it is being redirected to a file, optionally compressed and piped through GnuPG.

/etc/btrbk/btrbk.conf:

raw_target_compress xz
raw_target_encrypt gpg
gpg_keyring /etc/btrbk/gpg/pubring.gpg
gpg_recipient btrbk@mydomain.com


volume /mnt/btr_pool
subvolume home
target raw ssh://cloud.example.com/backup
ssh_user btrbk
# incremental no

This will create a GnuPG encrypted, compressed files on the target host. For each backup, two files are created:

• /backup/home.YYYYMMDD.btrfs.xz.gpg: main data file containing the btrfs send-stream,
• /backup/home.YYYYMMDD.btrfs.xz.gpg.info: sidecar file containing metadata used by btrbk.

I you are using raw incremental backups, please make sure you understand the implications (see btrbk.conf(5), TARGET TYPES).

Setting up SSH
Since btrbk needs root access on the remote side, it is very advisable to take all the security precautions you can. Usually backups are generated periodically without user interaction, so it is not possible to protect your ssh key with a password. The steps below will give you hints on how to secure your ssh server for a backup scenario. Note that the btrbk executable is not needed on the remote side, but
you will need “/sbin/btrfs” from the btrfs-progs package.

btrbk comes with a shell script “ssh_filter_btrbk.sh”, which restricts ssh access to sane calls to the /sbin/btrfs command needed for snapshot creation and send/receive operations (see ssh_filter_btrbk(1)). Here is an example on how it can be used with ssh:

Step 1 (client): Create a ssh key dedicated to btrbk, without password protection:

ssh-keygen -t rsa -b 2048 -f /etc/btrbk/ssh/id_rsa -C btrbk@mydomain.com -N ""

Step 2 (server): Copy the “ssh_filter_btrbk.sh” from the btrbk project to “/backup/scripts/”.

Step 3 (server): Add contents of the public key (/etc/btrbk/ssh/id_rsa.pub) to
“/roo/.ssh/authorized_keys”, and configure “ssh_filter_btrbk.sh” to be executed whenever this key is used for authentication. Example lines:

# example backup source (also allowing deletion of old snapshots)
command="/backup/scripts/ssh_filter_btrbk.sh -l --source --delete" <pubkey>...


# example backup target (also allowing deletion of old snapshots)
command="/backup/scripts/ssh_filter_btrbk.sh -l --target --delete" <pubkey>...


# example fetch-only backup source (snapshot_preserve_min=all, snapshot_create=no),
# restricted to subvolumes within /home or /data
command="/backup/scripts/ssh_filter_btrbk.sh -l --send -p /home -p /data" <pubkey>...

You might also want to restrict ssh access to a static IP address within your network:
from="192.168.0.42",command="/backup/scripts/ssh_filter_btrbk.sh [...]" <pubkey>...

Please refer to ssh_filter_btrbk(1) for a description of the “ssh_filter_btrbk.sh” options, as well as sshd(8) for a description of the “authorized_keys” file format.

Also consider setting up ssh access for a user dedicated to btrbk and choose either:

• backend btrfs-progs-btrbk to completely get rid of ssh_filter_btrbk.sh, in conjunction with btrfs-progs-btrbk,
• backend btrfs-progs-sudo, configure /etc/sudoers, and consider using “ssh_filter_btrbk.sh –sudo” option.

For even more security, set up a chroot environment in /etc/ssh/sshd_config (see sshd_config(5)).

Restoring Backups
btrbk does not provide any mechanism to restore your backups, this has to be done manually. In the examples below, we assume that you have a btrfs volume mounted at /mnt/btr_pool, and the subvolume you want to have restored is at /mnt/btr_pool/data.

Important: don’t use btrfs property set to make a subvolume read-write after restoring. This is a low-level command, and leaves “Received UUID” in a false state which causes btrbk to fail on subsequent incremental backups. Instead, use btrfs subvolume snapshot (without -r flag) as described below.

Example: Restore a Snapshot
First, pick a snapshot to be restored:

btrbk list snapshots

From the list, pick the snapshot you want to restore. Let’s say it’s /mnt/btr_pool/_btrbk_snap/data.20150101.
If the broken subvolume is still present, move it away:

mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN

Now restore the snapshot:

btrfs subvolume snapshot /mnt/btr_pool/_btrbk_snap/data.20150101 /mnt/btr_pool/data

That’s it; your data subvolume is restored. If everything went fine, it’s time to nuke the broken subvolume:

btrfs subvolume delete /mnt/btr_pool/data.BROKEN

Example: Restore a Backup

First, pick a backup to be restored:

btrbk list backups

From the list, pick the backup you want to restore. Let’s say it’s /mnt/btr_backup/data.20150101.
If the broken subvolume is still present, move it away:

mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN

Now restore the backup:
btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/
btrfs subvolume snapshot /mnt/btr_pool/data.20150101 /mnt/btr_pool/data
btrfs subvolume delete /mnt/btr_pool/data.20150101

Alternatively, if you’re restoring data on a remote host, do something like this:

btrfs send /mnt/btr_backup/data.20150101 | ssh root@my-remote-host.com btrfs receive /mnt/btr_pool/

If everything went fine, nuke the broken subvolume:

btrfs subvolume delete /mnt/btr_pool/data.BROKEN

FAQ
Make sure to also read the btrbk FAQ page. Help improve it by asking!

[GreyGeek]

To summarize my exploration of btrfs backup tools:
1) I would NEVER use TimeShift. Reason: it's storage location is locked in concrete and links @ & @home to subdirectories that can be easily deleted, trashing your system. Also, before you remove it you MUST FIRST delete all snapshots made by it or you will trash your system trying to delete them manually.
2) I wouldn't use btrbk because it adds little automation to the snapshot process and isn't any better than doing manual snapshots. Also, it doesn't do restores, you have to do those manually. Snapper does a much better job of snapshotting and rolling back, IF you can live with the snapshots being inside @ and @home and not at the <ROOT_FS> level. One can easily remove Snapper's snapshots and file structure manually without harming your system.

Despite Snapper's shortcomings, it works well if one limits the number and frequency of snapshots by eliminating the timeline and making all snapshots singletons. I wrote a script that does that and is run before ("PRE") and after ("POST") any system change that one might want to be able to reverse. I call it

snap_pre_post.sh

#!/bin/bash
# created by Jerry L Kreps on July 20, 2015 and released under the GPL 2.0.
# This script merely creates a SINGLETON snapshot in both root and home with the designation of PRE or POST as the type, 
# which indicates that the user created it before an action or afterwards.
# This script is run with PRE as the TYPE before an action which you may want to reverse
# AND this script run again with POST as the TYPE after that action has been completed.  Both snapshots are singletons because
# no timeline is used.  Only the "PRE" or"POST" in the description, along with a timestamp, links the pre to the post snapshot.
#
# To reverse the action run the following two snapper commands:
#
# snapper -c home udochange n..m   where n or o is the number of the PRE and m or p is the number of the POST snapshot
# sudo snapper -c root undochange o..p
#
# After running those two commands both of the empty snapshots folders can be deleted using
#
# snapper -c home delete n-m
# sudo snapper -c root delete o-p
#
#
#
NOW=$(date +%Y%m%d%H%M)
echo Enter snapshot type PRE or POST:
read TYPE
echo Enter description
read DESC
STR=$TYPE" "$DESC" "$NOW
echo $STR
HCMD='snapper -c home create -d "'${STR}'"'
RCMD='sudo snapper -c root create -d "'${STR}'"'
eval "$HCMD"
eval "$RCMD"

Oshunluver also wrote a script that snapshots his system.
oshunluver_bkup_script.sh

#!/bin/bash

# Log the script activity
exec 1> >(logger -s -t $(basename $0)) 2>&1


## Set the variables
## Edit this section as needed
distro_sub="@KDEneon"       # Distro subvolume to snapshot
home_sub="@KDEneon_home"    # Home subvolume to snapshot
source_mount="/subvol/"     # Root file system mount location
addname="_daily_"           # This is added to the subvolume name of the snapshots
oldsnapnum="7"              # Snapshot sequence number for removal


## Assemble the above variables into single variables and a few additional needed variables
today=`date +%Y-%m-%d`                                  # Today's date in a format that match the 'stat' command output below
distro_source="$source_mount""$distro_sub"              # The root mount and distro subvolume name combined
home_source="$source_mount""$home_sub"                  # The root mount and home subvolume name combined
newdistrosnap="$distro_source""$addname""1"             # New snapshot names are the source name with 
newhomesnap="$home_source""$addname""1"                 # the contents of addname and 1 appended to the end 
olddistrosnap="$distro_source""$addname""$oldsnapnum"   # Distro subvolume to remove
oldhomesnap="$home_source""$addname""$oldsnapnum"       # Home subvolume to remove


## Begin process
# Has a snapshot already occurred today? Quit if yes
if [ `stat -c "%y %s %n" $newdistrosnap | cut -b 1-10` == $today ]
then 
    exit 0
fi


# Delete the oldest snapshot if it exists
if [ -d "$olddistrosnap" ]
    then 
    btrfs su de -c "$olddistrosnap"
fi


# Roll the current snapshot names by adding 1 to each trailing number
for i in {6..1} 
    do
      mv "$distro_source""$addname"$i "$distro_source""$addname"$(($i+1))
      mv "$home_source""$addname"$i "$home_source""$addname"$(($i+1))
    done
    
# Take new read-write snapshots;
    btrfs su sn "$distro_source" "$newdistrosnap"
    btrfs su sn "$home_source" "$newhomesnap"


# Touch these new snapshots to set the file date as today;
    touch "$newdistrosnap"
    touch "$newhomesnap"


# Script complete
exit 0