Rsync to backup home and cron job questions.

[oshunluvr]

So I've finally begun cleaning up my digital mess over here and decided it was high time to automate backups. Since I have a server, it seemed and obvious choice for storing a backup of my home directory. Additionally, my server houses a few (thousand) files I'd also not like to lose - mostly family photos and I have plenty of space on my desktop to backup those.

Easy right? So I've decided a daily backup would be sufficient for now. So I made a little rsync script (one line) and put it in /etc/cron.daily:

rsync -avz --delete-after /home/stuart /backup/home

This copies my home directory to an NFS mounted share. Likely, I'll add lines to it for my install and for the photos and anything else I can think of, but before I do I have a question: It seems like a dangerous possibility that one or more of the drives could be off-line or fail and this could be catastrophic.

For example, if for some reason my /home partition became un-mounted or unreadable, wouldn't then the --delete-after option delete my backup? Obviously, this would be bad! Or in the other direction, what if the NFS share was off-line? Then the backup would be re-created entirely on the / partition under the target mount directory thus filling my root partition until it was full and this is again b-a-d!

So I think I need a way to detect the existence of one or more mounts before executing each backup command and a notification waiting for me if one of them fails.

Digging for ideas but any suggestions would be helpful. Right now I'm at:

libsmount="/backup/home";

mountpoint $libsmount
if [ $? -eq 0 ] ; then
echo "mounted"
else
echo "not mounted"
fi

This tests for a mount point and works correctly. Now I guess I nest IFs until I reach the desired state...

[oshunluvr]

Re: Rsync to backup home and cron job questions.

This seems to work well enough - although not very terse:

target1="/backup/home"; source1="/home"; 
mountpoint $target1 > /dev/null
if [ $? -eq 0 ] ; then
 mountpoint $source1 > /dev/null
 if [ $? -eq 0 ] ; then
 echo "mounted"
 else
 echo "not mounted"
 fi
fi

Now how to message me when it fails...

[tnorris]

Re: Rsync to backup home and cron job questions.

This is what I use to detect a mount point in my backup script.

# Check if BACKUP_DIR is mounted
if [ `$MOUNT | $GREP "$SNAPSHOT_DIR" | $WC -l` -eq 0 ]; then
 log_error "Exiting. Snapshot Directory is not mounted."            
 exit                                     
fi

You might also want to consider using an rsync type connection rather than over NFS. Rsync is supposed to be more efficient when using an rsync daemon. Also, it avoids the need to have the the NFS mount point at all which avoids a point of failure. You can check rsync's man page to see how to run it as a daemon and how to connect to said daemon.

[wizard10000]

Re: Rsync to backup home and cron job questions.

Tellya what I do - I don't let rsync delete files.

Once a month or so I'll go into the backup location and blow away all the dotfiles and just let rsync recreate them - my documents and multimedia don't need to be cleaned up. If I need to remove a document I'll remove it manually from both source and target but I don't think I've ever thrown away pictures or music.

It's just me, but I also back up /etc and /root - there are only a couple of files I need in /etc - some grub configuration, the system crontab, NFS exports, rc.local and a couple of apt sources but that way I've got it all just in case I need something else.

I grab /root so that I can restore settings there as there are some KDE and GTK settings I'd like to not have to recreate every time I reinstall - and root's just an itty bitty directory

As tnorris said if you're worried about rsync filling up your root partition if NFS fails, back up directly to the remote machine instead of to a local mount point like this-

rsync -avz --delete-after /home/stuart stuart@target.machine.ip.address:/destination/directory

or use rsync over ssh.

Or better yet, have the backup server pull the files rather than pushing files to it

[oshunluvr]

Re: Rsync to backup home and cron job questions.

Both good Ideas - didn't think about the server side pulling the files, but I'd still have to do checks for some mounts. The server has 4 drives mounted every which way. Plus I want to eventually have five or six different sets of backups both onto and off of the server.

My thought process is - I have a ton of hard drive space right now so I have no excuse not to backup as much as possible, but I want it totally automatic and fail-safe (as possible). The delete-after function works for my desires because keeping things "clean" should also be automatic.

How about I do a daily backup without delete-after but a monthly with it. That way I have a way to recover something accidentally discarded say, last week but still end up with a trash-less backup. I'd only have to remember which day of the month my deletes were going to occur. I think that might be the best of both worlds.

I didn't realize I could directly access the other computer without using a mount but I'm not sure that solves much. If the target isn't mounted then that directory will fill-up - same issue different location. Or is there a logical check in rsync that I am unaware of? What's the functional difference between using the rsync daemon vs. cron? Efficiency as far as time goes isn't really an issue as these computers run 24/7 and this will occur around 2am.

@tnorris: What do I need to add to your script to make it work? I'm getting

./test7: line 3: : command not found
./test7: line 3: -l: command not found
./test7: line 3: [: -eq: unary operator expected

If i add SNAPSHOT_DIR="/backup/home", I get

./test7: line 3: -l: command not found
./test7: line 3: /backup/home: Is a directory
./test7: line 3: [: -eq: unary operator expected

I'm a total bash-idiot so talk slowly...

@wizard10000: Sidebar - just curious as to which files you edit in /root and for what purposes? I always copy my edited files in /etc because I do a lot of fine-tuning after every install.

[wizard10000]

Re: Rsync to backup home and cron job questions.

@wizard10000: Sidebar - just curious as to which files you edit in /root and for what purposes? I always copy my edited files in /etc because I do a lot of fine-tuning after every install.

Fonts, themes, dolphin options and so on. I also change the background color of root's theme just a little bit so I can tell at a glance whether the window in which I'm working has root privileges. If I restore root's settings I don't have to recreate them

Plus, if you apply a GTK+ theme and want it to run in say, synaptic it's root's configuration that needs to change, not a user's.

[wizard10000]

Re: Rsync to backup home and cron job questions.

I didn't realize I could directly access the other computer without using a mount but I'm not sure that solves much. If the target isn't mounted then that directory will fill-up - same issue different location. Or is there a logical check in rsync that I am unaware of?

Another reason to do a server pull rather than client push - that way if the source files aren't available rsync has nothing to do

On another note - it's just my preference but I never automate file deletion. It wouldn't be too difficult for you to just rm -rf your backup directory (or parts of it) once a month and just let the nightly rsync job run without delete

[tnorris]

Re: Rsync to backup home and cron job questions.

@tnorris: What do I need to add to your script to make it work? I'm getting

./test7: line 3: : command not found
./test7: line 3: -l: command not found
./test7: line 3: [: -eq: unary operator expected

If i add SNAPSHOT_DIR="/backup/home", I get

./test7: line 3: -l: command not found
./test7: line 3: /backup/home: Is a directory
./test7: line 3: [: -eq: unary operator expected

I'm a total bash-idiot so talk slowly...

Sorry, I left out that beginning of the script. The assignments are there so that someone can't override the program name and make it do something else. For example, someone could place a program called "mount" in ~/bin and then the script would pick that up instead of the "mount" in /usr/bin.

Here's the full script. The basic idea is it uses Hard Links and rsync to create incremental rolling backups that take up a minimum amount of space by having files that are the same in multiple snapshots share the same hard link. My /home directory is about 20GB and it only takes about 15 minutes to do the backup each night over a wireless connection to my NAS. The majority of that time is spent deleting the oldest backup. The moves are fast because it's just an I-NODE update.

#!/bin/bash
# Derived from [url]http://www.mikerubel.org/computers/rsync_snapshots/[/url]

unset PATH

# Command Locations
ECHO=/bin/echo
RM=/bin/rm
MV=/bin/mv
CP=/bin/cp
TOUCH=/bin/touch
MOUNT=/bin/mount
GREP=/bin/grep
WC=/usr/bin/wc
RSYNC=/usr/bin/rsync
DATE=/bin/date

# File Locations
SNAPSHOT_DIR=/media/backup
SNAPSHOT_MAX=7
SNAPSHOT_FILE=snapshot
BACKUP_DIR=/home/tnorris
RSYNC_DEST=tnorris@192.168.1.6::tnorrisbackup
EXCLUDES=/home/tnorris/.exclude
PASSWORD_FILE=/home/tnorris/.rsync_pw

log_info () {
 $ECHO "`$DATE` - $1"
}

log_error () {
 $ECHO "`$DATE` - $1" >&2
}

log_info "Starting snapshot"

# Check if BACKUP_DIR is mounted
if [ `$MOUNT | $GREP "$SNAPSHOT_DIR" | $WC -l` -eq 0 ]; then
 log_error "Exiting. Snapshot Directory is not mounted."
 exit
fi

# Delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_DIR/$SNAPSHOT_FILE.$(($SNAPSHOT_MAX-1)) ]; then
 $RM -rf $SNAPSHOT_DIR/$SNAPSHOT_FILE.$(($SNAPSHOT_MAX-1))
 status=$?
 if [ $status -eq 0 ]
 then
  log_info "Removed $SNAPSHOT_DIR/$SNAPSHOT_FILE.$(($SNAPSHOT_MAX-1))"
 else
  log_error "Removal of $SNAPSHOT_DIR/$SNAPSHOT_FILE.$(($SNAPSHOT_MAX-1)) failed."
  exit $status
 fi
fi

# Shift each snapshot back one
for ((i=($SNAPSHOT_MAX-2); i>=0; i--))
do
 FROM=$i
 TO=$(($i+1))
 if [ -d $SNAPSHOT_DIR/$SNAPSHOT_FILE.$FROM ]; then
  $MV $SNAPSHOT_DIR/$SNAPSHOT_FILE.$FROM $SNAPSHOT_DIR/$SNAPSHOT_FILE.$TO
  status=$?
  if [ $status -eq 0 ]
  then
   log_info "Moved $SNAPSHOT_DIR/$SNAPSHOT_FILE.$FROM to $SNAPSHOT_DIR/$SNAPSHOT_FILE.$TO"
  else
   log_error "Move of $SNAPSHOT_DIR/$SNAPSHOT_FILE.$FROM to $SNAPSHOT_DIR/$SNAPSHOT_FILE.$TO failed."
   exit $status
  fi
 fi
done

# rsync from the $BACKUP_DIR into the most recent snapshot
$RSYNC -a --stats --delete --delete-excluded --exclude-from="$EXCLUDES" --link-dest=../$SNAPSHOT_FILE.1 --password-file="$PASSWORD_FILE" "$BACKUP_DIR/" "$RSYNC_DEST"/$SNAPSHOT_FILE.0
status=$?
if [ $status -eq 0 ]
then
 log_info "Rsynced $BACKUP_DIR/ to $RSYNC_DEST/$SNAPSHOT_FILE.0 linked to ../$SNAPSHOT_FILE.1"
else
 log_error "Rsync of $BACKUP_DIR/ to $RSYNC_DEST/$SNAPSHOT_FILE.0 linked to ../$SNAPSHOT_FILE.1 failed."
 exit $status
fi


# Update the mtime of $SNAPSHOT_FILE.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_DIR/$SNAPSHOT_FILE.0
status=$?
if [ $status -eq 0 ]
then
 log_info "Touched $SNAPSHOT_DIR/$SNAPSHOT_FILE.0"
else
 log_error "Touch of $SNAPSHOT_DIR/$SNAPSHOT_FILE.0 failed."
 exit $status
fi

log_info "Snapshot Complete"

This is what the email I get looks like for a successful run. I have several machines in my house running this nightly.

Thu Jun 16 04:00:01 EDT 2011 - Starting snapshot
Thu Jun 16 04:10:10 EDT 2011 - Removed /media/backup/snapshot.6
Thu Jun 16 04:10:10 EDT 2011 - Moved /media/backup/snapshot.5 to /media/backup/snapshot.6
Thu Jun 16 04:10:10 EDT 2011 - Moved /media/backup/snapshot.4 to /media/backup/snapshot.5
Thu Jun 16 04:10:10 EDT 2011 - Moved /media/backup/snapshot.3 to /media/backup/snapshot.4
Thu Jun 16 04:10:10 EDT 2011 - Moved /media/backup/snapshot.2 to /media/backup/snapshot.3
Thu Jun 16 04:10:10 EDT 2011 - Moved /media/backup/snapshot.1 to /media/backup/snapshot.2
Thu Jun 16 04:10:11 EDT 2011 - Moved /media/backup/snapshot.0 to /media/backup/snapshot.1

Number of files: 47282
Number of files transferred: 1396
Total file size: 16898534894 bytes
Total transferred file size: 4126596402 bytes
Literal data: 1561455959 bytes
Matched data: 2565140443 bytes
File list size: 1303538
File list generation time: 0.176 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1563537523
Total bytes received: 1121231

sent 1563537523 bytes received 1121231 bytes 972744.02 bytes/sec
total size is 16898534894 speedup is 10.80
Thu Jun 16 04:36:59 EDT 2011 - Rsynced /home/tnorris/ to tnorris@192.168.1.6::tnorrisbackup/snapshot.0 linked to ../snapshot.1
Thu Jun 16 04:37:00 EDT 2011 - Touched /media/backup/snapshot.0
Thu Jun 16 04:37:00 EDT 2011 - Snapshot Complete

[leetuckert]

Re: Rsync to backup home and cron job questions.

Greetings:

I have scanned over these posts and don't really have anything to add except that you might want to consider Unison. I have a server in my office down stairs running Centos 5. I have two notebooks and a desktop computer all running Kubuntu. Everybody in the family as a Unison icon located on the tool bar. You edit a configuration file that uses a specific format which tells unison which directories or, even specific files to keep in sync. When you run Unison on the client; i.e., the notebooks or the desktop computer, it syncs everything in to the server. One post mentioned that he never lets rsync delete files. There is some wisdom in this; however, since I keep both my notebooks, my personal and my work notebooks in sync, I usually don't worry about inadvertently deleting something because I figure I'll catch it before it get propagated to all three. Also, you can configure Unison so that it doesn't automatically delete files. Unison may not work in your situation; however, it has worked very well for me.

What! Crap! I just deleted two weeks worth of work!

[oshunluvr]

Re: Rsync to backup home and cron job questions.

Unison looks promising, but the GUI is too crippled for my setup. It might be a nice cli tool though. I'll try it out along with rsync.

As far as the "Delete, or Not Delete" mini-debate: What I want is a snapshot type backup which by definition includes deleting files from the backup at some point. That, along with the desire of total automation, dictates allowing the script to do the dirty work.

I think my idea of blending non-delete daily scripts, monthly "clean-up" delete scripts, and sufficient protection from odd-ball occurrences (network down, mounts not mounted, drive failure) is really what I'm after.

Thanks everyone for all the input. I will attempt to wrench this out Sunday. Outcome to be determined!

[Linkmaster]

Re: Rsync to backup home and cron job questions.

@oshunluvr:
You should perhaps make a backup of your system on a different media the day before it does the delete feature. Just to be on the safe side

[oshunluvr]

Re: Rsync to backup home and cron job questions.

Ok, here's what I've come up with. It's very wordy but I haven't coded in 25 years. I can't yet get my mind around case vs. goto so I've got a lot of if's, but it seems to do what I want.

Checks to verify all mounts are available.
Allows backups of those mounts that are OK and skips those not available.
Reports all errors and successes.
Backs up daily and deletes absent source files from the targets monthly.

I'm sure there's a dozen better ways to do this so please enlighten me.

#!/bin/bash
# Backup script
# Delete actions taken on the day of the month in DELETEDAY
#

unset PATH

# Source Locations
SOURCE1=server://mnt/shared
SOURCE2="server://mnt/videos/Family Videos"
SOURCE3=server://mnt/videos/Projects
SOURCE4=server://
SOURCE5=office://home
SOURCE6=office://

# Target Locations
TARGET1=office://mnt/sda6 #1
TARGET2=office://mnt/sdb6 #234
TARGET3=server://home_backup #56

# Mountpoint checks
#  server side
MOUNT1=/mnt/shared
MOUNT2=/mnt/videos
MOUNT3=/home_backup
#  office side
MOUNT4=/mnt/sda6
MOUNT5=/mnt/sdb6
MOUNT6=/home

# Command Locations
ECHO=/bin/echo
RM=/bin/rm
MV=/bin/mv
CP=/bin/cp
TOUCH=/bin/touch
CHECKSERVER="/usr/bin/rsh -p 2222 daddy@server /bin/mountpoint"
CHECKLOCAL=/bin/mountpoint
GREP=/bin/grep
WC=/usr/bin/wc
RSYNC="/usr/bin/rsync -az "
DATE=/bin/date

# Varibles
MOUNTERROR=0
#  Day of the month to clean out deleted files
DELETEDAY=28


# Begin events

log_info () {
 $ECHO "`$DATE` - $1"
}

log_error () {
 $ECHO "`$DATE` - $1" >&2
}

log_info "Starting backup - Checking mount points"

# Check if each mount is available
ISUP1=$CHECKSERVER $MOUNT1 | $WC -l
ISUP2=$CHECKSERVER $MOUNT2 | $WC -l
ISUP3=$CHECKSERVER $MOUNT3 | $WC -l
ISUP4=$CHECKLOCAL $MOUNT4 | $WC -l
ISUP5=$CHECKLOCAL $MOUNT5 | $WC -l
ISUP6=$CHECKLOCAL $MOUNT6 | $WC -l

if [ $ISUP1 -eq 0 ]; then
 log_error $MOUNT1" is not mounted."
 MOUNTERROR=1
fi

if [ $ISUP2 -eq 0 ]; then
 log_error $MOUNT2" is not mounted."
 MOUNTERROR=1
fi

if [ $ISUP3 -eq 0 ]; then
 log_error $MOUNT3" is not mounted."
 MOUNTERROR=1
fi

if [ $ISUP4 -eq 0 ]; then
 log_error $MOUNT4" is not mounted."
 MOUNTERROR=1
fi

if [ $ISUP5 -eq 0 ]; then
 log_error $MOUNT5" is not mounted."
 MOUNTERROR=1
fi

if [ $ISUP6 -eq 0 ]; then
 log_error $MOUNT6" is not mounted."
 MOUNTERROR=1
fi

if [ $MOUNTERROR -eq 1 ]; then
 log_error "Not all mountpoints available!"
else
 log_info "All mount points available."
fi

# Is today delete day?
if [ `$DATE +%d` -eq $DELETEDAY ]; then
 DELETE=" --delete-after"
fi

if [ $ISUP1 -eq 1 -a $ISUP4 -eq 1 ]; then
 $RSYNC $DELETE $SOURCE1 $TARGET1
 log_info $SOURCE1": Backup complete."
else
 log_error $SOURCE1": No backup made."
fi

if [ $ISUP2 -eq 1 -a $ISUP5 -eq 1 ]; then
 $RSYNC $DELETE $SOURCE2 $TARGET2
 $RSYNC $DELETE $SOURCE3 $TARGET2
 log_info $SOURCE2" and "$SOURCE3": Backup complete."
else
 log_error $SOURCE2" and "$SOURCE3": No backup made."
fi

if [ $ISUP5 -eq 1 ]; then
 $RSYNC $DELETE $SOURCE4 $TARGET2
 log_info $SOURCE4": Backup complete."
else
 log_error $SOURCE4": No backup made."
fi

if [ $ISUP3 -eq 1 -a $ISUP6 -eq 1 ]; then
 $RSYNC $DELETE $SOURCE5 $TARGET3
 log_info $SOURCE5": Backup complete."
else
 log_error $SOURCE5": No backup made."
fi

if [ $ISUP3 -eq 1 ]; then
 $RSYNC $DELETE $SOURCE6 $TARGET3
 log_info $SOURCE6": Backup complete."
else
 log_error $SOURCE6": No backup made."
fi

if [ $MOUNTERROR -eq 1 ]; then
 log_error "Some mounts not available"
else
 log_info "All backups complete"
fi

[oshunluvr]

Re: Rsync to backup home and cron job questions.

I fixed a few things that I missed the first go-round but there's still problems

Biggest issue is my setup uses a non-standard port for ssh and a shared key security for the admin user on my server (not the same username as my desktop). Thus, if I run the script as root, it fails because the root user on my desktop doesn't have a shared key. If I run it as my regular user, it partially fails because I can't backup the other users on my system or any files I don't have read permission for.

So is it safe security wise to allow root shared key access to my server? I guess so... back to the keyboard.