Would like some assistance

[Snowhog]

Today was 'day 1' running my script. It worked flawlessly. Full Backups of @ and @home were created on my external USB HDD. Tomorrow will be 'day 2', creating Incremental Snapshots. Personally, I think they should referred to as Incremental Backups, but, however one pronounces 'salmon', it's still a fish. If tomorrows results are what I expect them to be; no errors; I'll post my script.

I've learned/relearned A LOT since I started this Topic 13 days ago.

[Snowhog]

Geez! 61 preceding posts in this Topic. I'm trending!

It took me 16 days to get my script to work for my use case and in the way I wanted without any errors! That occurred about an hour ago. I'm so happy.

The script isn't actually a finished/polished product yet. Now that it works as designed, I can, at my leisure, re-examen it for the purpose of simplifying and improving the code. So maybe later this evening or sometime tomorrow I'll present it here for all to heap their accolades upon or to say WTF! I'm good either way.

But until then, I'm taking some time to mentally unwind.

[Snowhog]

And here it is.

#!/bin/bash
clear
STARTCLOCK=$(date +%s)
# snapshot.sh
# Created on 2022-03-16
#
# Based on the script by GreyGeek
# and the grateful assistance
# of GreyGeek, claydoh and others
# without who's help I might not have
# gotten this done.
#
# Currently this script only does the following:
#
# The first time this script is run, no snapshots yet exist.
# Snapshots are made of the /@ and /@home sub-volumes on the
# internal HDD and sent to the external USB HDD. This is called
# initial bootstraping, and corresponds to a full backup.
# The internal HDD now has one /@ and /@home snapshot.
# The external USB HDD now has one FULL BACKUP of /@ and /@home.
# The script is finished at that point.
#
# On subsequent runs a new snapshot is made of the /@ and /@home
# on the internal HDD.
# The internal HDD now has two /@ and /@home snapshots.
# Incremental snapshots of the /@ and /@home sub-volumes on the
# internal HDD are created on the external USB HDD.
# The internal HDD now has two /@ and /@home snapshots.
# The external USB HDD now has one FULL BACKUP of /@ and /@home
# and one INCREMENTAL BACKUP of /@ and /@home.
# The previous /@ and /@home snapshot on the internal HDD are
# deleted.
# The internal HDD now has one /@ and /@home snapshot.
#
# Change Log:
# Syntax to use:
# YYYY-MM-DD
# What was done
#

# To be run as root from /
#
# Before executing this script:
# Plug in external HDD and ignore Disk & Devices pop-up.
# External drive will be powered and identified by system but not yet mounted.

# Check backup drive connection status

USBDRIVE=($(ls /dev/disk/by-label/|grep 'USBHDD'))
TEST="USBHDD"
if [[ $USBDRIVE == $TEST ]];
then
echo "USB HDD is connected and will be mounted"
read -p "Press any key to continue, Ctrl+C to Quit"
else
echo "USB HDD needs to be connected"
read -p "When connected, press any key to continue, Ctrl+C to Quit"
sleep 2
fi
clear

LAPTOP="mount /dev/disk/by-label/LAPTOP /mnt"
USBHDD="mount /dev/disk/by-label/USBHDD /backup"

# Mount Drives
## Internal Drive

echo "Mounting Internal HDD"
echo "Command is"
echo ""
echo " "$LAPTOP
echo ""
eval $LAPTOP
sleep 2

# When this script is run for the first time,
# and no snapshots exist,
# the two array variables following will be NULL
# otherwise they will contain at least one element each

SNAPLIST1=($(ls /mnt/snapshots/|grep 'ROOT'))
SNAPLIST2=($(ls /mnt/snapshots/|grep 'HOME'))

echo ""
echo "Current content of /mnt"
echo ""
vdir /mnt
echo ""
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""

## External Drive

echo "Mounting USB HDD"
echo "Command is"
echo ""
echo " "$USBHDD
echo ""
eval $USBHDD
sleep 2
echo ""
echo "Current content of /backup"
echo ""
vdir /backup
echo ""
read -p "Press any key to continue, Ctrl+C to Quit"

# Snapshot Routine

echo
if [[ ${#SNAPLIST1[@]} -le 0 ]];
then
clear
echo "Day 1 snapshot process"
echo ""
read -p "If this is not Day 1 press Ctrl+C NOW!"
clear

# Create Day 1 snapshots of @ and @home on /mnt/snapshots and send to /backup
# Variables

# Base Names

ROOTNAME=$(date +%F_%T)'ROOT'
HOMENAME=$(date +%F_%T)'HOME'

# Make Snapshots

MKSNAP1="btrfs subvolume snapshot -r /mnt/@ /mnt/snapshots/$ROOTNAME"
MKSNAP2="btrfs subvolume snapshot -r /mnt/@home /mnt/snapshots/$HOMENAME"

# Send Snapshots

SNDSNAP1="btrfs send /mnt/snapshots/$ROOTNAME | btrfs receive /backup"
SNDSNAP2="btrfs send /mnt/snapshots/$HOMENAME | btrfs receive /backup"

echo "This two step process will create a Full Backup of /@ and /@home sub-volumes"
echo ""
echo "Step 1 creates READ-ONLY snapshots and saves them in /mnt/snapshots"
echo "Step 2 creates and sends the Full Backup to /backup"
echo ""
read -p "Press any key to continue"
clear
echo "Step 1. Creating READ-ONLY snapshots"
echo ""
echo "Commands are"
echo ""
echo " "$MKSNAP1
echo " "$MKSNAP2
echo ""
echo "Creating snapshots"
echo ""
eval $MKSNAP1
eval $MKSNAP2
echo ""
echo "Snapshots created"
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""
read -p "Press any key to continue"
clear

# Initial bootstrapping

echo "Step 2. Creating Full Backups"
echo ""
echo "Sending $ROOTNAME to /backup"
echo "Sending $HOMENAME to /backup"
echo ""
echo "Commands are"
echo ""
echo " "$SNDSNAP1
echo " "$SNDSNAP2
echo ""
echo "This will take a while"
echo ""
eval $SNDSNAP1
eval $SNDSNAP2
sync;sync;sync
echo ""
echo "Creation of Full Backups complete"
echo "Current content of /backup"
echo ""
vdir /backup
echo ""
echo "Initial snapshot process completed"
echo ""

# Array Variables now contain one element each

else

# This is run two or higher, so one snapshot of /@ and /@home
# from previous run exist and will be identified as
# PREVIOUSSNAP1
# PREVIOUSSNAP2

# Array Variables

SNAPLIST1=($(ls /mnt/snapshots/|grep 'ROOT'))
SNAPLIST2=($(ls /mnt/snapshots/|grep 'HOME'))
#
# Each Array Variable now contain one or more elements

# Variables

ROOTNAME=$(date +%F_%T)'ROOT-Inc'
HOMENAME=$(date +%F_%T)'HOME-Inc'

# ONLY used if two or more of each Snapshot is being saved

OLDESTSNAP1=${SNAPLIST1[0]}
OLDESTSNAP2=${SNAPLIST2[0]}

PREVIOUSSNAP1=${SNAPLIST1[-1]}
PREVIOUSSNAP2=${SNAPLIST2[-1]}

CURRENTSNAP1=$ROOTNAME
CURRENTSNAP2=$HOMENAME

# Take Snapshots

MKSNAP1="btrfs subvolume snapshot -r /mnt/@ /mnt/snapshots/$ROOTNAME"
MKSNAP2="btrfs subvolume snapshot -r /mnt/@home /mnt/snapshots/$HOMENAME"

# Make Incrementals
## @

MKINC1='btrfs send -p /mnt/snapshots/'$PREVIOUSSNAP1
MKINC1=$MKINC1' /mnt/snapshots/'$CURRENTSNAP1
MKINC1=$MKINC1' | btrfs receive /backup'

## @home

MKINC2='btrfs send -p /mnt/snapshots/'$PREVIOUSSNAP2
MKINC2=$MKINC2' /mnt/snapshots/'$CURRENTSNAP2
MKINC2=$MKINC2' | btrfs receive /backup'

# Make Snapshots

clear
echo "Running Day 2+ process"
echo ""
read -p "If this is not Day 2+ press Ctrl+C now!"
echo "Otherwise press any key to continue"
clear

echo "This two step process will create an Incremental Backup of /@ and /@home sub-volumes"
echo ""
echo "Step 1 creates READ-ONLY snapshots and saves them in /mnt/snapshots"
echo "Step 2 creates and sends the Incremental Backup to /backup"
echo ""
read -p "Press any key to continue"
clear
echo "Step 1. Taking READ-ONLY snapshots"
echo ""
echo "Commands are"
echo ""
echo " "$MKSNAP1
echo " "$MKSNAP2
echo ""
eval $MKSNAP1
eval $MKSNAP2
echo ""
echo "Snapshots created"
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""
read -p "Press any key to continue"
clear

# Make Incremental Snapshots

echo "Step 2. Creating Incremental snapshots"
echo ""
echo "Commands are"
echo ""
echo " "$MKINC1
echo " "$MKINC2
echo ""
eval $MKINC1
eval $MKINC2
eval 'sync;sync;sync'
echo ""
echo "Incremental snapshots received to /backup"
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""
echo "Current content of /backup"
echo ""
vdir /backup
echo ""
read -p "Press any key to continue"
clear
fi

# Delete Oldest Snapshot

DELSNAP1='btrfs subvolume delete -C /mnt/snapshots/'$OLDESTSNAP1
DELSNAP2='btrfs subvolume delete -C /mnt/snapshots/'$OLDESTSNAP2

# Delete Oldest Snapshots

echo "There are now two snapshots of /@ and /@home on /mnt/snapshots;"
echo "the ones created during this run, and the ones created during the previous one."
echo "The previous snapshots will now be deleted from /mnt/snapshots."
echo ""
echo "Commands are"
echo ""
echo " "$DELSNAP1
echo " "$DELSNAP2
echo ""
echo "Deleting"
echo ""
echo " "$OLDESTSNAP1
echo " "$OLDESTSNAP2
echo ""
eval $DELSNAP1
eval $DELSNAP2
eval 'sync;sync;sync'
echo ""
echo "Snapshots have been deleted"
echo ""
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""
read -p "Press any key to continue"
clear

# Calculate and display how much time creating and sending snapshots took

STOPCLOCK=$(date +%s)
SECS=$((STOPCLOCK - STARTCLOCK))
DURATION=$(date --date "0 $SECS sec" +%Hh:%Mm:%Ss)

echo "Finished. Total process took: $DURATION"
echo ""
echo ""
echo "Drives remain mounted. Remember to umount drives and power-off USB HDD"

[GreyGeek]

Interesting to see how your mind works!

So, how long did your day 2 snapshots and increments take?

[Snowhog]

GreyGeek Thank you. My mind operates differently than I believe most peoples do.

As my laptop really doesn't have much more the the installed OS; very little has been added by me (/home/paul); processing the root and home sub-volumes took just little over 14-minutes (the initial 'full backups'). Verifying the script, I waited for some time to elapse then ran it again (Incremental Snapshots). Those only took under 1-minute, as you'd expect.

[GreyGeek]

GreyGeek Thank you. My mind operates differently than I believe most peoples do.

After readings your posts for the last 13 years I believe you, and I say that as a compliment! Can you believe it's been 13 years!

[USER="9548"]
As my laptop really doesn't have much more the the installed OS; very little has been added by me (/home/paul); processing the root and home sub-volumes took just little over 14-minutes (the initial 'full backups'). Verifying the script, I waited for some time to elapse then ran it again (Incremental Snapshots). Those only took under 1-minute, as you'd expect.

I probably have several GB more than you in my system, but my long times are around 16 minutes and my inc times are about 30 seconds, based on that piece of code I stole from you!

[Snowhog]

After readings your posts for the last 13 years I believe you, and I say that as a compliment!

Thank you, and I do (take it as a compliment).

My father once told me that when person A is communicating anything to person B, if person B isn't comprehending, or misunderstands A, the fault isn't with B; A is always responsible for 'effective communication'. I believe that, and do my level best to put it into daily practice.

based on that piece of code I stole from you!

I won't tell!

Can you believe it's been 13 years!

In six days, I will have been a member of KFN for 15 years!

[GreyGeek]

Thank you, and I do (take it as a compliment).

My father once told me that when person A is communicating anything to person B, if person B isn't comprehending, or misunderstands A, the fault isn't with B; A is always responsible for 'effective communication'. I believe that, and do my level best to put it into daily practice.

The only advice I got from my father was "If you get thrown in jail don't call me". I was around 14 or 15 at the time. That was the sum total of his lessons in life. But, I regret not going to his funeral. His journey though life was full of hard knocks not all of his own making.

In six days, I will have been a member of KFN for 15 years!

I'm 13 days beyond my 13th year! (Feb 4, 2009)

[Snowhog]

SNAPLIST1=($(ls /mnt/snapshots/|grep 'ROOT'))
SNAPLIST2=($(ls /mnt/snapshots/|grep 'HOME'))

These lines are now erroring on 23.10. Well, the first one; line 89; which throws me out of the script; the second line isn't executed, but it too, would error, as it's written with the same syntax.

./snapshot.sh: 89: Syntax error: "(" unexpected

I haven't attempted to run this script since installing Kubuntu 23.10 on the laptop. The script hasn't been modified since I finished and verified that it worked; flawlessly. The only thing different is 23.10 vs the prior version that was on the laptop.

Why is this syntax not liked by bash now?

[Snowhog]

So there were changes made in bash version 5.2. At the time I wrote/used my script, bash version was 5.1.

New Features
============

This is an update to the fifth major release of bash.

Read the file NEWS in the bash-5.2 distribution for a complete description
of the new features. A copy of the relevant portions is included below.

This release fixes several outstanding bugs in bash-5.1 and introduces
a number of new features.

There are a number of bug fixes, including several bugs that caused the
shell to crash. Complete details are available in the CHANGES file.

The most notable new feature is the rewritten command substitution parsing
code, which calls the bison parser recursively. This replaces the ad-hoc
parsing used in previous versions, and allows better syntax checking and
catches syntax errors much earlier. The shell attempts to do a much better
job of parsing and expanding array subscripts only once; this has visible
effects in the `unset' builtin, word expansions, conditional commands, and
other builtins that can assign variable values as a side effect. The `unset'
builtin allows a subscript of `@' or `*' to unset a key with that value for
associative arrays instead of unsetting the entire array (which you can still
do with `unset arrayname'). There is a new shell option, `patsub_replacement'.
When enabled, a `&' in the replacement string of the pattern substitution
expansion is replaced by the portion of the string that matched the pattern.
Backslash will escape the `&' and insert a literal `&'. This option is enabled
by default. Bash suppresses forking in several additional cases, including
most uses of $(<file).

All the new features are described below.

Readline has new features as well. There is a new option:
`enable-active-region'. This separates control of the active region and
bracketed-paste. It has the same default value as bracketed-paste, and
enabling bracketed paste enables the active region. Users can now turn off
the active region while leaving bracketed paste enabled. Two new bindable
string variables are available; their values are terminal escape sequences
that set the color used to display the active region and turn it off,
respectively. If set, these are used in place of terminal standout mode.
Finally, Readline now checks for changes to locale settings (LC_ALL/LC_CTYPE/
LANG) each time it is called, and modifies the appropriate locale-specific
display and key binding variables when the locale changes.

There are a few incompatible changes between bash-5.1 and bash-5.2. Here-
documents and here-strings use temporary files if the shell compatibility
level is 50 or lower. The `unset' builtin in bash-5.2 treats array subscripts
`@' and `*' differently than previous versions, and differently depending on
whether the array is indexed or associative. Bash-5.2 attempts to prevent
double-expansion of array subscripts under certain circumstances, especially
arithmetic evaluation, by acting as if the `assoc_expand_once' shell option
were set. Set the compatibility level appropriately to revert to previous
behavior; details are in the file COMPAT.

Bash can be linked against an already-installed Readline library rather
than the private version in lib/readline if desired. Only readline-8.1 and
later versions are able to provide all of the symbols that bash-5.2 requires;
earlier versions of the Readline library will not work correctly.

A complete list of changes between bash-5.1 and bash-5.2 is available in
the file CHANGES; the complete list is too large to include in this
message.

Change in the command substitution parsing code. Is this what's causing the two lines to fail?

[jlittle]

I recall an error like that being due to a problem many lines previously. There's nothing wrong with SNAPLIST1=($(ls /mnt/snapshots/|grep 'ROOT')) as far as I can see.

[Snowhog]

The error may have been due to my not having created the snapshot directory in /mnt. /mnt/snapshot has to exist before running the script. It’s there now. I’ll test later.

[Snowhog]

There's nothing wrong with SNAPLIST1=($(ls /mnt/snapshots/|grep 'ROOT')) as far as I can see.

First, I created /mnt/snapshots and reran the script this morning. Bash still complains about the first of these two lines with the same ./snapshot.sh: 89: Syntax error: "(" unexpected

I tested this line manually and it works; bash doesn't complain. I'm at a loss at this point (but still searching the 'Net) as to why it isn't liked when executed within the script.

[jlittle]

Is the script in post #63 still the current version? I expect not, because the "DELSNAP" lines have unclosed quotes.

[Snowhog]

jlittle Yes, the script I'm trying to use is the same as what is posted in post #63. I got the script to run to completion just a bit ago, with no errors. As root (sudo -i), I ran: bash snapshot.sh No errors. Took a bit over 18-minutes to create and send to the backup drive, my @ and @home directories.

When I was receiving the error on the variable definition line, I was launching the script with sh ./snapshot.sh So a difference in how sh and bash handle things. The script file is set executable, so I should be able to run it directly: snapshot.sh I'll try that next time.

You have to keep your eyes open and be aware of nuances! They're pesky little buggers!