My newest BTRFS backup script - I found a new bash function - arrays

[oshunluvr]

I rearranged my server which means rearranging the backup automation scripts. I now have 2 backup drives that are 6TB and the media storage drive is 10TB, so I want to divide the 15 subvolumes across the 2 backup drives. Basically, I'm left with source, backup1, and backup2 and I want subvolumes 1-15 on different backups drives.

My previous script didn't work this way so I began anew. The first riddle was "Can I have one script that successfully divides the subvolumes between the two backup devices?" I'm not so concerned with current sizes of the subvolume because they grow and shrink as needed but I wanted to use all the capacity I have available.

Here's what I came up with. I removed everything but the parts that pertain to this post. I would like comments if there's a better way to do it:

declare -a set=(@Audio @Backups @Documents @Downloads @Home_Movies @Incoming @Movies @Music @Music_Videos @Pictures @Projects @TV_Shows @Videos @WWW)
declare -a set1=(@Audio @Backups @Documents @Downloads @Home_Movies @Incoming @Music @Music_Videos @Pictures @Projects @Videos @WWW)

for subvol in ${set[@]} ; do  
if [[ ${set1[@]} = *"$subvol"* ]]; then backup_mount=backup1_mount ; else backup_mount=backup2_mount ; fi

"set" = all the subvolume names, "set1" = those I want to go to backup1, any missing from set1 are sent to backup2.

It's my first time using an array variable this way. Works like I want and amazingly short. I wanted to avoid repetitive code or dual scripts. If I decide later to re-arrange the backups or add more subvolumes I only have to add or subtract from the "set" and "set1" arrays.

It occurred to me that I could have the "set" array built from a simple "ls" command. Then any newly appearing subvolumes would be automatically added to the process and sent to backup2.

Or I could even adjust the target backup location by using a trailing number in the subvolume name - like a 1 or 2 at the end of the name to set the backup location or read the list from a text file instead of in the script itself, but I doubt I'll be changing this around much.

[dibl]

Wow, you do good work, Stuart! Very impressive.

[GreyGeek]

Indeed!
I always used Python when I wanted to use arrays. I wasn't aware that bash offered them. So, I went searching for them:
https://www.thegeekstuff.com/2010/06...The+Geek+Stuff
"If the index number is @ or *, all members of an array are referenced. "

[oshunluvr]

I can see a lot of functionality by using arrays. I have a couple other scripts that need re-writes too.

[jlittle]

Your array use has a potential problem. It is not seen in your example, but if you removed @Music from set1, your test will still find it in set1 because it matches @Music_Videos.

There isn't a simple way to check if something is in a bash array that I like, there's only what seems to me tricky, or involved:

• mark the end of each item with something sure to be not found in the names, f.ex.
  ${set1[@]/%/ZZZ} = *${subvol}ZZZ*
• use regex matching with spaces or something around the names:
  " ${set1[@]} " =~ " $subvol "
• define a function, then use that in the test

  is_in () {local x item=$1;shift;for x;do [[ "$x" = "$item" ]] && return 0;done; return 1; }
  if is_in $subvol "${set1[@]}"; then ...

[oshunluvr]

Your array use has a potential problem. It is not seen in your example, but if you removed @Music from set1, your test will still find it in set1 because it matches @Music_Videos.

There isn't a simple way to check if something is in a bash array that I like, there's only what seems to me tricky, or involved:

• mark the end of each item with something sure to be not found in the names, f.ex.
  ${set1[@]/%/ZZZ} = *${subvol}ZZZ*
• use regex matching with spaces or something around the names:
  " ${set1[@]} " =~ " $subvol "
• define a function, then use that in the test

  is_in () {local x item=$1;shift;for x;do [[ "$x" = "$item" ]] && return 0;done; return 1; }
  if is_in $subvol "${set1[@]}"; then ...

I expected you'd have some sage advice for me

Nice catch, I didn't think of that - "Music" exists in two names. Also "Videos" and "Movies" does too.

So I ran a test. Before the test I had already decided to swap the "sets" because set2 (the group going to backup2) was much smaller - only @Movies and @TV_Shows. For the test , I added @Music to set2. Here's the test script:

#!/bin/bash

declare -a set=(@Alt_Movies @Audio @Backups @Documents @Downloads @Home_Movies @Incoming @Movies @Music @Music_Videos @Pictures @Projects @TV_Shows @Videos @WWW)
declare -a set2=(@Movies @TV_Shows @Music)

for subvol in${set[@]};do
if[[${set2[@]}= *"$subvol"* ]];thenecho"set2"$subvol;elseecho"set1"$subvol;fi

done

exit 0

and the output:

[FONT=monospace][COLOR=#000000]root@server:~# ./autotest    [/COLOR]
set1 @Alt_Movies
set1 @Audio
set1 @Backups
set1 @Documents
set1 @Downloads
set1 @Home_Movies
set1 @Incoming
set2 @Movies
set2 @Music
set1 @Music_Videos
set1 @Pictures
set1 @Projects
set2 @TV_Shows
set1 @Videos
set1 @WWW

[/FONT]

So clearly, "@Music" is in set 2 and "@Music_Videos" is in set 1.

[oshunluvr]

Thinking maybe the order would matter, I added "@Music_test" near the top of the list and "@Music_test2" near the bottom, but the results look accurate:

set1 @Alt_Movies
set1 @Audio
set1 @Music_test
set1 @Backups
set1 @Documents
set1 @Downloads
set1 @Home_Movies
set1 @Incoming
set2 @Movies
set2 @Music
set1 @Music_Videos
set1 @Pictures
set1 @Projects
set1 @Music_test2
set2 @TV_Shows
set1 @Videos
set1 @WWW

[oshunluvr]

Finally, I added "@Music_test2 to the front of "set2" list and:

root@server:~# ./autotest    
set1 @Alt_Movies
set1 @Audio
set2 @Music_test
set1 @Backups
set1 @Documents
set1 @Downloads
set1 @Home_Movies
set1 @Incoming
set2 @Movies
set2 @Music
set1 @Music_Videos
set1 @Pictures
set1 @Projects
set2 @Music_test2
set2 @TV_Shows
set1 @Videos
set1 @WWW

it FAILED. Both @Music_test2 and @Music_test1 appeared in the set2 list.

[oshunluvr]

Clearly something is wacky. Strange that it seems to work fine until I get to the @Music_test vs @Music_test2. Also, if I change "@Music_test" to "@Music_test1" it doesn't fail.

Finally, I changed "@Music" in set 2 to "@Music_Videos" and they both listed as set 2. Seems the order of the sets controls the outcome.

Anyway: Adding the spaces as JLittle suggests, changing this:

if [[ ${set2[@]} = *"$subvol"* ]];

to this

if [[ " ${set2[@]} " = *" $subvol "* ]];

Seems to work fine.