Btrfs-sync, a backup script for subvolumes
#BTRFS
The script and its manual are found here:
https://ownyourbits.com/2018/03/09/e...th-btrfs-sync/
There is an installation procedure using wget, or you can copy the code from the site and save it into a file titled btrfs-sync, or any other name you want, in your path and with execute permissions for root, or any sudo enabled owner you want. The author suggests creating a user called "btrfs", and shows how to create it and how to make a cron task.
I'm copying the comparison of this code with an rsync procedure because it is informative, and it could act as the starting template for creating your own tailored code:
Note that this code does NOT create snapshots, it only syncronizes existing (src) snapshots, or folders containing snapshots, with snapshots at the destination (dst). The user would have to manually (or with snapper) create snapshots that this program would send & receive from the source to the destination. A cron task would make this process automatic. It syncs incrementally by comparing UUID's of the dst with UUID's of the src. This program would probably fit well with sending snapper's snapshots to a remote server.
[CODE]
#!/bin/bash
#
# Simple script that synchronizes BTRFS snapshots locally or through SSH.
# Features compression, retention policy and automatic incremental sync
#
# Usage:
# btrfs-sync [options] [...] [[user@]host:]
#
# -k|--keep NUM keep only last sync'ed snapshots
# -d|--delete delete snapshots in that don't exist in
# -z|--xz use xz compression. Saves bandwidth, but uses one CPU
# -Z|--pbzip2 use pbzip2 compression. Saves bandwidth, but uses all CPUs
# -q|--quiet don't display progress
# -v|--verbose display more information
# -h|--help show usage
#
# can either be a single snapshot, or a folder containing snapshots
# requires privileged permissions at for the 'btrfs' command
#
# Cron example: daily synchronization over the internet, keep only last 50
#
# cat > /etc/cron.daily/btrfs-sync /dev/null )
[[ $? -ne 0 ]] && { echo "error parsing arguments"; exit 1; }
eval set -- "$OPTS"
while true; do
case "$1" in
-h|--help ) print_usage; exit 0 ;;
-q|--quiet ) QUIET=1 ; shift 1 ;;
-d|--delete ) DELETE=1 ; shift 1 ;;
-k|--keep ) KEEP=$2 ; shift 2 ;;
-z|--xz ) ZIP=xz PIZ=( xz -d ); shift 1 ;;
-Z|--pbzip2 ) ZIP=pbzip2 PIZ=( pbzip2 -d ); shift 1 ;;
-v|--verbose) SILENT="" VERBOSE=1 ; shift 1 ;;
--) shift; break ;;
esac
done
SRC=( "${@:1:$#-1}" )
DST="${@: -1}"
# detect remote dst argument
[[ "$DST" =~ : ]] && {
NET="$( sed 's|:.*||'
The script and its manual are found here:
https://ownyourbits.com/2018/03/09/e...th-btrfs-sync/
There is an installation procedure using wget, or you can copy the code from the site and save it into a file titled btrfs-sync, or any other name you want, in your path and with execute permissions for root, or any sudo enabled owner you want. The author suggests creating a user called "btrfs", and shows how to create it and how to make a cron task.
I'm copying the comparison of this code with an rsync procedure because it is informative, and it could act as the starting template for creating your own tailored code:
The main difference between these two methods is that BTRFS works at the block level, whereas rsync works at the file level.
Because rsync works with files, it will not detect things such as renaming or moving a file, so it can only send it again. Also, it needs to analyze the existing files at the destination to see if they have been updated or not.
In order to achieve this, it can either analyze modification dates and sizes, which is relatively fast, or compare checksum of file chunks at both ends which is more robust but slower. In any case, there will be a significant processing overhead when you are synchronizating a whole partition with many thousands of files. (IOW, rsync is a LOT slower than sending a snapshot - GG)
A plus of this approach is that you are able to exclude certain files or folders, where BTRFS works by subvolumes in an all or nothing fashion.
BTRFS on the other hand understands blocks, and because it is a COW filesystem, it already knows what bytes have changed between a snapshot and the next. If we renamed the file, only some tiny metadata has changed, and BTRFS knows that we don’t need to transfer the whole file again, only those few bytes.
The same happens when a big file changes internally, such as an image file for a virtual machine where we have been working.
This is the same reason why snapshots in COW filesystems are so space efficient, allowing us to create instant safety copies of huge volumes that only takes extra space as we change the files in them.
Obviously a drawback is that you need a BTRFS filesystem at both ends, but why would we stick to an old generation filesystem where we now have more modern and featureful ones?
Because rsync works with files, it will not detect things such as renaming or moving a file, so it can only send it again. Also, it needs to analyze the existing files at the destination to see if they have been updated or not.
In order to achieve this, it can either analyze modification dates and sizes, which is relatively fast, or compare checksum of file chunks at both ends which is more robust but slower. In any case, there will be a significant processing overhead when you are synchronizating a whole partition with many thousands of files. (IOW, rsync is a LOT slower than sending a snapshot - GG)
A plus of this approach is that you are able to exclude certain files or folders, where BTRFS works by subvolumes in an all or nothing fashion.
BTRFS on the other hand understands blocks, and because it is a COW filesystem, it already knows what bytes have changed between a snapshot and the next. If we renamed the file, only some tiny metadata has changed, and BTRFS knows that we don’t need to transfer the whole file again, only those few bytes.
The same happens when a big file changes internally, such as an image file for a virtual machine where we have been working.
This is the same reason why snapshots in COW filesystems are so space efficient, allowing us to create instant safety copies of huge volumes that only takes extra space as we change the files in them.
Obviously a drawback is that you need a BTRFS filesystem at both ends, but why would we stick to an old generation filesystem where we now have more modern and featureful ones?
Note that this code does NOT create snapshots, it only syncronizes existing (src) snapshots, or folders containing snapshots, with snapshots at the destination (dst). The user would have to manually (or with snapper) create snapshots that this program would send & receive from the source to the destination. A cron task would make this process automatic. It syncs incrementally by comparing UUID's of the dst with UUID's of the src. This program would probably fit well with sending snapper's snapshots to a remote server.
[CODE]
#!/bin/bash
#
# Simple script that synchronizes BTRFS snapshots locally or through SSH.
# Features compression, retention policy and automatic incremental sync
#
# Usage:
# btrfs-sync [options] [...] [[user@]host:]
#
# -k|--keep NUM keep only last sync'ed snapshots
# -d|--delete delete snapshots in that don't exist in
# -z|--xz use xz compression. Saves bandwidth, but uses one CPU
# -Z|--pbzip2 use pbzip2 compression. Saves bandwidth, but uses all CPUs
# -q|--quiet don't display progress
# -v|--verbose display more information
# -h|--help show usage
#
# can either be a single snapshot, or a folder containing snapshots
# requires privileged permissions at for the 'btrfs' command
#
# Cron example: daily synchronization over the internet, keep only last 50
#
# cat > /etc/cron.daily/btrfs-sync /dev/null )
[[ $? -ne 0 ]] && { echo "error parsing arguments"; exit 1; }
eval set -- "$OPTS"
while true; do
case "$1" in
-h|--help ) print_usage; exit 0 ;;
-q|--quiet ) QUIET=1 ; shift 1 ;;
-d|--delete ) DELETE=1 ; shift 1 ;;
-k|--keep ) KEEP=$2 ; shift 2 ;;
-z|--xz ) ZIP=xz PIZ=( xz -d ); shift 1 ;;
-Z|--pbzip2 ) ZIP=pbzip2 PIZ=( pbzip2 -d ); shift 1 ;;
-v|--verbose) SILENT="" VERBOSE=1 ; shift 1 ;;
--) shift; break ;;
esac
done
SRC=( "${@:1:$#-1}" )
DST="${@: -1}"
# detect remote dst argument
[[ "$DST" =~ : ]] && {
NET="$( sed 's|:.*||'
Comment