Automatic Backup Strategy - thoughts?

[oshunluvr]

By way of explanation for the discussion:
For quite a while (years) I have been leaving my desktop PC on 24/7. Recently, I've changed my mind on that and have been shutting it down when I know I'm not going to be using it soon - like within a couple hours. This change is mostly due to the fact that with SSDs, grub timing, sddm, and KDEneon/Kubuntu I can be power-on-to-desktop in like 12 seconds.

For the last six months or so I have been doing automated snapshots and backups of my desktop install and home. The system takes a daily snapshot and makes 2 full backups to 2 separate drives once a week for both / and /home. It keeps the latest backup plus the most previous, and keeps three daily snapshots. This was adequate in my view and has in fact "saved" me twice.

However, now that I am not running the PC all-day-every-day, the automation has failed (two scripts running via cron). It has failed because:

1. The script uses file date to determine auto-removal and now that it's not on every day, it leaves older unwanted snapshots and backups behind.
2. The backups are done on Sundays so if the machine is not on that day, no backup is made at all.

I want to transition to a different plan so I'm looking for suggestions. I want it to be completely hands-off.

My current thoughts are to re-write the scripts to use the count of backups/snapshots instead of dates, keep 7 snapshots instead of just 3 (drive space will support this), and then send every 7th snapshot as a backup. This will result in a "weekly" backup based on seven days of use rather than seven calendar days and I'd increase my snapshot footprint. I'm not sure if 7 snapshots is necessary, but it seems easy to do and I have the space.

I thought I would create a snapshot at boot time - but only if one had not already been done that day.

The backup procedure would be harder. I could launch it right away when I boot up, but if I needed to reboot or wanted to leave I'd have to leave it running or manually re-do the backup after reboot - both of which would not be optimal. I could detect the existence of a backup, but interrupted backups would consume space while also being totally useless so again - not optimal. Note that backups using btrfs occur totally in the background so running the backup would not hamper my normal use. Previously, I ran backups at 3 am on Sunday knowing I would not be using the machine and thus would not accidentally interrupt the procedure - it does take quite some time to finish all 4 backups.

Anyone have any suggestions or better ideas on how to accomplish this?

[oshunluvr]

I've decided the snapshots will be simple - instead of a cron script I will run a script at boot up in /etc/rc5.d that will:

• Determine if a snapshot is needed (no existing snapshot made on the current day).
• Delete the oldest snapshot if more than 7 exist.

Still wondering about how to do the backups. I think I need a script that runs at startup that:

• Determines if a backup is needed (existing backup > 7 days old).
• Starts the backup process.
• When complete, removes the oldest backup if more than 2 exist.
• If not complete, prevents shutdown/reboot.

• Determines if a backup is needed (existing backup > 7 days old).
• Notifies the user that a backup is needed and asks if this is a good time to do it.
• Starts the backup process when activated.
• When complete, removes the oldest backup if more than 2 exist.
• If not complete, prevents shutdown/reboot.
• Notifies the user when complete.

I've also decided that I will use conky to do the notifications as I already have a "Backup Status" indicator in my conky script. This keeps the notification "alive" until I activate the backup process. This isn't totally automated but seems like a reasonable compromise. I lined out the first task list above because if the backup run takes 4 hours, I wouldn't want it to activate itself 10 minutes before I left town for a week.

So it seems I need four scripts/files:

• Daily snapshot
• Backup check-and-notify
• Do the backup
• Prevent shutdone/reboot until backup complete

and to modify conky. I will want a .desktop file to run the "Do the backup" so I can access it from the menu or from a desktop icon. All the scripts would run automatically except "Do the backup". Time to get out some script-foo.

Still open to comments/suggestions (or scripting help ).

[oshunluvr]

Here's the conky output test for the backups:

[dequire]

Anyone have any suggestions or better ideas on how to accomplish this?

Great work! I look forward to seeing the finalized solution. By the way, this seems like a great question for Ask Ubuntu. Lots of knowledgeable peeps over there.

[Snowhog]

My only 'advice' would be to flow chart the processes you are contemplating and at each decision point ask "What would happen if...".

[oshunluvr]

Half the work is already done for this from my previous scripts - just some re-writes are needed.

At this point, I'm looking into the best way to determine the oldest subvolume for deletion. The root volumes theses snapshots and backups exist on have multiple other subvolumes, so simply determining the oldest isn't enough - it also has to be of the correct name. As long as I use some sort of stable naming convention, it shouldn't be too hard. With my previous efforts, leaving the system off for 24 hours caused the skipping of removing old snapshots and eventually lead to over-use of drive space. I need functionality that works regardless of gaps in time.
Snowhog, you're absolutely right about that. I've already seen some unintended results (as in a wiped install - but I had a backup ) from one of my earliest attempts.

Snapshots flow chart:

Does one exist from earlier today>

Yes> Do not make a snapshot, end.
No> Make a snapshot >

Is this snapshot the eighth one?>

Yes> Delete the oldest one.
No> End.

Assuming something doesn't cause extra snapshot(s) that leave a total greater than 8, this would be all I need. Note that I often make manual snapshots when updating or doing something scary. Those I manually clean up and I use a different naming structure so it's clear which snap was for what.

Backups are a bit more work because the criteria is different. I want to keep only two backups and I want them seven days apart. Under this new idea, that's seven "operational" days (system up) verses seven calendar days like before. I also do some fancy editing of the snapshots to make them bootable in place (from the back up drive) which also requires a re-snapshoting effort - because snapshots used to send|receive must be Read-Only to transmit (make a backup). All this I have a good handle on - just working out the what-to-keep and what-to-delete parts.

Backups flow chart:

Is a backup needed (seven operational days since last backup)>

No> End.
Yes>

Make a R/O snapshot
Send the R/O snapshot to both backup drives
Delete the source R/O snapshot
Re-snapshot the target snapshots as R/W
Delete the target R/O snapshots
Edit the R/W snapshot with my sed one-liners
Are there 3 backups?>

No> End.

Yes> Delete oldest backup, end.

I may just use a number at the end of the subvolume name, then roll down before the next snapshot - like 6 becomes 7, 5 becomes 6, etc., and maybe the same game for the backups. One interesting behavior of the btrfs snapshot feature is the snapshot retains the creation date of the source - I don't know if that's due to the mounting options I'm using or if it's the default behavior. I "correct" this by a simple "touch" command after each snapshot so I can tell at a glance when the snapshot was made.

As far as the "Seven operational days" I should be able to compare the date of the oldest snapshot to the backup or something like that. Still mulling that over...

[Snowhog]

Backups flow chart:

Is a backup needed (seven operational days since last backup)>

No> End.
Yes>

Make a R/O snapshot
Send the R/O snapshot to both backup drives
Delete the source R/O snapshot
Re-snapshot the target snapshots as R/W
Delete the target R/O snapshots
Edit the R/W snapshot with my sed one-liners
Are there 3 backups?>

No> End.

Yes> Delete oldest backup, end.

How are you defining "operational days" ("Is a backup needed (seven operational days since last backup)>"); what constitutes an 'operational day'?

[oshunluvr]

A day the computer is actually on. For example, I travel for work. So if I shut down Sunday night and don't power up again until Friday night, that's 2 operational days. So if I traveled three weeks in a row:
Sunday: day 1
Friday: day 2
Saturday: day 3
Sunday: day 4
Friday: day 5
Saturday: day 6
Sunday: day 7 - Backup day

In this case the backup would be three weeks apart, but that's OK because I it was only on for seven days. On the weeks I'm home a backup would occur every week.

[Snowhog]

Something you should also take in to consideration, is what to do when a previous step in the procedure 'fails'. You do NOT want your script to continue, but instead, create/send a status notification of the failure and then exit. Given that you might not be around/available, some kind of flag should also be created that your scripts can check for. If that 'flag' exists, do not run the scripts. This prevents what could otherwise end up being a catastrophic event.

[oshunluvr]

Something you should also take in to consideration, is what to do when a previous step in the procedure 'fails'. You do NOT want your script to continue, but instead, create/send a status notification of the failure and then exit. Given that you might not be around/available, some kind of flag should also be created that your scripts can check for. If that 'flag' exists, do not run the scripts. This prevents what could otherwise end up being a catastrophic event.

Good point as well. I have a backup status displayed in my conky (as noted above) but not a snapshot status as of yet.

For now, this appears to rotate the snapshots correctly:

#!/bin/bash

# Log the script activity
exec 1> >(logger -s -t $(basename $0)) 2>&1


## Set the variables
## Edit this section as needed
distro_sub="@KDEneon"       # Distro subvolume to snapshot
home_sub="@KDEneon_home"    # Home subvolume to snapshot
source_mount="/subvol/"     # Root file system mount location
addname="_daily_"           # This is added to the subvolume name of the snapshots
oldsnapnum="7"              # Snapshot sequence number for removal


## Assemble the above variables into single variables and a few additional needed variables
today=`date +%Y-%m-%d`                                  # Today's date in a format that match the 'stat' command output below
distro_source="$source_mount""$distro_sub"              # The root mount and distro subvolume name combined
home_source="$source_mount""$home_sub"                  # The root mount and home subvolume name combined
newdistrosnap="$distro_source""$addname""1"             # New snapshot names are the source name with 
newhomesnap="$home_source""$addname""1"                 # the contents of addname and 1 appended to the end 
olddistrosnap="$distro_source""$addname""$oldsnapnum"   # Distro subvolume to remove
oldhomesnap="$home_source""$addname""$oldsnapnum"       # Home subvolume to remove


## Begin process
# Has a snapshot already occurred today? Quit if yes
if [ `stat -c "%y %s %n" $newdistrosnap | cut -b 1-10` == $today ]
then 
    exit 0
fi


# Delete the oldest snapshot if it exists
if [ -d "$olddistrosnap" ]
    then 
    btrfs su de -c "$olddistrosnap"
fi


# Roll the current snapshot names by adding 1 to each trailing number
for i in {6..1} 
    do
      mv "$distro_source""$addname"$i "$distro_source""$addname"$(($i+1))
      mv "$home_source""$addname"$i "$home_source""$addname"$(($i+1))
    done
    
# Take new read-write snapshots;
    btrfs su sn "$distro_source" "$newdistrosnap"
    btrfs su sn "$home_source" "$newhomesnap"


# Touch these new snapshots to set the file date as today;
    touch "$newdistrosnap"
    touch "$newhomesnap"


# Script complete
exit 0

Now, just where/when to activate it. It has to run with root privilege.

I used a ton of variables so I could modify it easily for other installations.

[jlittle]

I noticed that this has problems:

if [ `stat -c "%y %s %n" $newdistrosnap | cut -b 1-10` == $today ]
then 
exit 0
fi

If the file in $newdistrosnap doesn't exist, messy errors occur. As well, I prefer:

• [[...]] over [...], much saner syntax and better errors^*
• $( ... ) over backticks
• not having redundant options; %s %n are not used here
• to use awk fields over cut with columns
• explicit use of ISO dates, over formats that are not explicitly so
• to think about symbolic links
• "; then"
• always using double quotes around shell variables, as a habit

So, I offer:

if [[ -e "$newdistrosnap" ]]; then
 if [[ $(/bin/ls -L --full-time "$newdistrosnap" | awk '{print $6}') == "$today" ]]; then
     exit 0
 fi
fi

(/bin/ls to avoid aliases for ls) or maybe for the condition

$(printf "%(%F)T" $(stat -Lc %Y "$newdistrosnap") )

because strftime(3) date formatting codes (%F here) are used in a lot of contexts, so it's worth knowing them, though printf is very bash.

Shell scripting has become an art, with as many opinions as script writers, I'm just presenting ideas.

^* I know I'm in a minority with this. I came to bash from ksh, and ksh didn't much like [ ... ], though it did work. Coming from when bash was considered slow and bloated, the term "bashism" is used with a sneer, even for features that are actually POSIX. I think that if you put #!/bin/bash at the top of the script, avoiding bash features for "compatibility" is silly, particularly if it results in (IMO) kludge idioms, no less kludgy for being hallowed by ancient usage. Look at a configure script produced by autotools if you want a kludge-fest example.

[oshunluvr]

I noticed that this has problems:

if [ `stat -c "%y %s %n" $newdistrosnap | cut -b 1-10` == $today ]
then 
exit 0
fi

If the file in $newdistrosnap doesn't exist, messy errors occur.

I see what you mean, but that's rather the point. If it doesn't exist, then it needs to be created. Terminating the script here if the snapshot doesn't exist is a failure of the intended script function. So this:

if [[ -e "$newdistrosnap" ]]; then

causes the script to fail. I do not want that. I suppose to make it cleaner, a case statement might be better, but an error message is not going to cause any real problems whereas failing to make a snapshot when it's needed will cause a problem.

To these suggestions/comments:

[[...]] over [...], much saner syntax and better errors

I've seen this before but never had it explained why it's better. What is "saner" in this context and what qualifies one error as better than another?

$( ... ) over backticks

I have no idea what you mean here.

not having redundant options; %s %n are not used here

Agreed and fixed.

to use awk fields over cut with columns

Why?

explicit use of ISO dates, over formats that are not explicitly so

I don't know what this means either, but I assume you're referring to using "stat -c %y" rather than your suggested "stat -Lc %Y". I don't understand your point here either. I absolutely do not want to know the exact hour/minute/second the snapshot was created, just if it was created today. My "stat" command returns 2018-04-17 - the exact format of "date +%Y-%m-%d" - and thus an easy comparison. Whereas "stat -Lc %Y" gives 1523960785. To me, that forces additional work to be done for zero benefit. Please explain what you meant here.

to think about symbolic links

Again - no clue what you meant here. There are no symbolic links used in my script.

"; then"

A difference in semantics. I might use this if I had nested if statements, but otherwise I prefer to have "then" on a separate line. I find it visually easier to follow especially if there are also "else" actions included in the statement.

always using double quotes around shell variables, as a habit

In principal I agree (and I left out it's use once where I should have) but what is the purpose of double quotes around a variable that's an integer? There cannot be a space or a symbol in a decimal integer so why add extra text? Especially when the integer variable is formed within the script, not as a command-line switch.

So, I offer:

if [[ -e "$newdistrosnap" ]]; then
 if [[ $(/bin/ls -L --full-time "$newdistrosnap" | awk '{print $6}') == "$today" ]]; then
     exit 0
 fi
fi

(/bin/ls to avoid aliases for ls) or maybe for the condition

In my view, shorter is better - easier to track and trouble shoot - but obviously, function trumps all. Within the context of this script, why is "/bin/ls" better than "stat" and why is "awk" better than "cut"?

$(printf "%(%F)T" $(stat -Lc %Y "$newdistrosnap") )

This I don't get either. Please explain "%(%F)T"

because strftime(3) date formatting codes (%F here) are used in a lot of contexts, so it's worth knowing them, though printf is very bash.

No clue here what this means.

Shell scripting has become an art, with as many opinions as script writers, I'm just presenting ideas.

Thank you and I appreciate your comments. I am looking forward to your responses to my questions if you have time.

One thing not addressed in this script is the mounts. Previously, I had the mounts not mounting at boot time for boot-time reduction and had my previous script do the mounting. I set the three mounts as "noauto, user" so I didn't need root permission to mount them. Currently, I changed the mounts to "auto". This allows the script to run immediately after log in. I think I still need to do a sanity check on the mounts when I write the backup script to be sure I'm not sending a backup to a folder rather than the intended, but this script would fail immediately if the root volume was unmounted. I suppose for the sake of correctness, I still need to verify the needed volume is mounted.

[Snowhog]

I suppose for the sake of correctness, I still need to verify the needed volume is mounted.

Absolutely! And, that it's mounted rw and not ro.

[jlittle]

I see what you mean, but that's rather the point. If it doesn't exist, then it needs to be created. Terminating the script here if the snapshot doesn't exist is a failure of the intended script function. So this
causes the script to fail. I do not want that.

You've got the sense wrong. The test checks for the existence of the file, and if it's not there nothing happens and the script continues without error. If it is there, then the script checks its date against today.

I've seen this before but never had it explained why it's better. What is "saner" in this context and what qualifies one error as better than another?

With [ ... ], ordinary command line processing occurs, with file globbing, word splitting, quote removal, and so on, with many opportunities for unintended consequences. F. ex. if you want to check that $var begins with an "m":
[[ $var = m* ]] does what you mean, but the result of [ $var = m* ] depends on what happens to be in the current directory, and it may be a bash: [: too many arguments error. Again, I stress that many eschew [[ ... ]] as a bashism.

$( ... ) over backticks

I have no idea what you mean here.

`command` can be written as $(command). The $(...) form is preferred even in a POSIX shell. $( ... ) easily nests, and the interaction between backticks and the other quote marks and word splitting can get very messy.

to use awk fields over cut with columns

Why?

If the output of the command being piped to cut changes slightly, the cut column numbers can be wrong. A common case is when there's an extra long file name pushing things out.

Also, awk can do a lot more, and if you start with it it's quick to keep going. It does grep more simply than grep; say you want lines with Peter and Paul in them, awk '/Peter/ && /Paul/' is plain, but you'd have to chain greps to do that. I suppose if you've got a hammer, the world looks like a nail, and awk is a very big and well-used hammer in my toolbox.

I don't know what this means either, but I assume you're referring to using "stat -c %y" rather than your suggested "stat -Lc %Y". I don't understand your point here either. I absolutely do not want to know the exact hour/minute/second the snapshot was created, just if it was created today. My "stat" command returns 2018-04-17 - the exact format of "date +%Y-%m-%d" - and thus an easy comparison. Whereas "stat -Lc %Y" gives 1523960785. To me, that forces additional work to be done for zero benefit. Please explain what you meant here.

There's been a lot of pain with dates and date formatting, especially if one is not from North America. Output can change depending on the locale, and Qt went painfully backwards with Qt 5, and thus also KDE with plasma. (The idea that date formats are fixed by the region setting is hopeless.) The output of stat changed at some time in the past to use ISO format, I think. I prefer to always explicitly use an ISO date (yyyy-mm-dd).

Again - no clue what you meant here. There are no symbolic links used in my script.

Simple to cope with them existing in your backup directory by adding the -L flag. A good practice IMO.

A difference in semantics. I might use this if I had nested if statements, but otherwise I prefer to have "then" on a separate line. I find it visually easier to follow especially if there are also "else" actions included in the statement.

A matter of preference, akin to the brace style in C. I don't watch TV but I'm told there's a Big Bang Theory episode where one of the characters drops his girlfriend because he didn't like her brace style.

That's all I've got time for now, I'll respond to the rest later.

[oshunluvr]

Jlittle, thanks for your time.

I figured out what you meant by "backticks" - I've not heard that word before and should have been more specific in my questions of that comment. You meant the "grave" key. You see, I speak "US English" (or in this case Xorg lingo ) while you speak "English English" (or in some cases Aussie English).

I think I understand most of what you said above. I see that in the case of "awk" and the different "stat" commands in the larger picture - if the script was to be distributed. However, I still don't think that's an issue here. I am not betting that the linux world will up-end the entire date structure.

I'll post my newest version later today.