Need help learning how to backup/restore

[TwoFistedJustice]

I still consider myself new to linux even though I've been using it for a couple years now. I need help coming up with a BASIC system backup and restore strategy. I would appreciate some help learning what programs/commands to use to backup/restore and what files/directories to save.

My current OS is Kubuntu 20.04. I don't have any web servers or anything like that running. I just use my computer for web browsing, some javascript development... pretty mundane stuff.

I have an external USB SSD which I can use as a backup destination.

I keep my 'voluntary' documents, pictures, etc on a separate partition on the same physical drive as my OS. I don't keep any important 'voluntary' files in my /home
directory.

My goal is to be able to bring my system back to a point similar to a fresh OS install but with important system settings and my most used applications intact.

Things I know I want to back up:

git settings and configurations
public/private keys for github

custom symbolic links for launching applications

wifi networks/passwords

The 'voluntary' files that I keep all on their own partition.


I have no idea what system files to save and need some extra guidance in this area.

[jglen490]

I use, and recommend, the CLI program rsync. There are some GUI front ends for rsync, but I don't use them, I just use a text file to keep a few rsync commands handy.

I backup everything in /home, which includes the system configs that pertain to my usage of Kubuntu, created and saved data files, music, movies, etc.

There are lots of ways to use rsync. You can include and exclude specific directories. You can specify a source of files to be backed up and a target for where the files will be backed up.

I use a rotating set of four external USB hard drives, and my needs are fairly simple. Each hard drive has "label", such as Backup1, Backup2, etc, and that label shows up when I attach the drive.

So a typical backup command looks like this:

sudo rsync -auv /home /media/john/Backup1

but there are lots of ways to use rsync, but this pattern tells rsync to copy everything in my /home to the Backup drive mounted to /media/john, and to only copy those files that have a newer timestamp than may exist already on the Backup drive.

From time to time, there have been issues with this approach, and since I have a fairly static file load in /home, I have a command to delete everything on a specific Backup drive, such as:

sudo rm -rf /media/john/Backup1/*

This is useful particularly if I have reorganized or moved directories or files around within /home, as it prevents duplicates that rsync wouldn't see from being "updated" to the target Backup drive.

There are a hundred ways to do this, as will be pointed out by others, I'm sure

[claydoh]

Things I know I want to back up:

git settings and configurations
public/private keys for github

custom symbolic links for launching applications

wifi networks/passwords

All the above are stored in your $HOME, mainly across different hidden files and dirs, Desktop setup, application options, browser data, etc. So this is always a good place to start.
How you do it, and what you use is harder to choose as there are soooooo many options from simple and to complex and highly configurable. I haven't made up my mind yet, and I have been trying things for years now

[TwoFistedJustice]

Okay so I need to backup /home because even if I thought I didn't use it, I actually used it a lot.

On a related note, my current system is having some issues which seem to be increasing over the last few days and I wonder if porting my home directory to a new install may port these problems as well?

The problems are: (not looking for discrete solutions to these at this point)

• I can't update my linux kernel. It's stuck on one from over a year ago. (can install but system won't boot with a later one)
• I can't update Python, it claims to be broken in Muon.
• I can no longer install anything, it says I have no authorization.
• Muon, Firefox, and associated symlinks all disappeared from my system without my intervention.

[claydoh]

The first two items are not related to anything in your user settings or user data at all

The last probably are not, but still could be.


If your do go fresh do look at using btrfs
Snapshots and Timeshift would be useful in this sort of situation.

Sent from my LM-V600 using Tapatalk

[Fred47]

This is the way I have been doing for many years. If things go really wrong, the reinstalling Linux is about an hour process, depending on download speed, how custom your install is, etc. To that I just copy all my files by dragging the home folder to a USB drive. I do this about every few weeks, but I could search for all files created today and copy that over every day as well. I do all this with Dolphin, which means I can restore using any OS if I decide to change.

[TwoFistedJustice]

The first two items are not related to anything in your user settings or user data at all

The last probably are not, but still could be.

I believe Firefox was corrupted by a network error during an update. I have really terrible rural internet. I found the failed download in Chromium and it wasn't one I initiated.

Muon I think has to do with Python being broken. Not sure why it seems to have been completely erased from my system. But I think it's to do with python since I can't even install it, and the errors all seem to revolve around python dependencies.

[claydoh]

Again zero of this is happening on files in your /home dir. Only the user data used or created by an application. Your individual app settings, theming bits, desktop settings, browser caches and bookmarks, all that is in $HOME. Installed program executables, libraries, system level stuff that needs a password to edit or delete, none of that is stored in your user's homer dirs.

I found the failed download in Chromium and it wasn't one I initiated.

Not sure what you mean by this.

[GreyGeek]

In a Konsol you can issue

sudo dpkg --configure -a
sudo apt install -f

several times until you get no error msgs or the error msgs remain dispite doing the sequence several times.
Other possible fixes are repairing the package cache.

sudo apt autoremovve  (or sudo apt-get autoremove)
sudo apt autoclean   (or sudo apt-get autoclean)

For serious surgery on the package cache:

sudo rm -r /var/lib/apt/lists/*
sudo mkdir /var/lib/apt/lists/partial
sudo apt-get clean
sudo apt-get update

Since you seem to be having a LOT of trouble with your Kubuntu 20.04 installation perhaps re-installing and using BTRFS as the root file system would be in order. I've been running 20.04 + BRTFS since it was in alpha and it has been faultless for me.

About backing up...
I'm a strong believer in using BTRFS as the root file system. Using it, backing up the root (@) and home (@home) accounts usually take less than a minute manually. Doing a differential backup to you external SDD could take 1 -3 minutes depending on how many file changes you've made on your installation. Of the GUI's available for backing up I'd suggest TimeShift (in the repository), although I don't use it (and don't like it, but a lot of people do). There is a BTRFS forum on KFN with tutorials, advice, suggestions, etc ....

That said, if you decide to stay with EXT4 then TimeShift is probably still your best backup solution. EXT4 does NOT use a copy-on-write paradigm so backing it up involves actual file transfers. With something like TimeShift doing it in the background, and assuming you have a fast system (8 GB RAM, 8 core processor, SSD drive), running TimeShift in the background shouldn't interfere with your normal operations, unless you are playing intensive games.

Here is a good tutorial for TimeShift (not on Kubuntu but that doesn't matter).

[jglen490]

And once you get a good Linux install, assuming you reinstall, then rsync can be used to backup what you have in /.

I truly believe that having / on one partition and /home on another partition results in the most recoverable Linux for most users. You can backup and restore one without impacting the other to the least extent, and gives you a lot of flexibility for backing up.

It is true that there are a lot of backup solutions, and there are a lot of ways to setup a Linux. I do not use tools like Timeshift, nor do I use btrfs. If tools like that are not configured correctly, you can end up with a full file system at exactly the wrong moment. Out of the box, Timeshift is a good point in time rollback tool, as it creates small files when changes are made, such as updating an application file. If that updated application fails or doesn't act properly, you can roll it back to the previous version and press on. But if those small files are not managed properly and cleared out when not needed, they just take up file space until there is no more space.

I'm not describing a problem that the OP has right now, I'm just talking about good system administration - even on a not so complex Linux.

As for btrfs, in my opinion, it is a good filesystem for mission critical Linux systems and complex enterprise solutions, precisely because good system administration tools are available. For the average user, a good install on a mature filesystem, such as ext4, and a proper backup and recovery plan is more than sufficient.

[TwoFistedJustice]

Using rsync I have now copied my /home directory and all of my voluntary files to removable media.

Once I have a fresh OS installed, what is the best/safest way to copy my backup back into /home?

[GreyGeek]

IF you decide to go with rsync then a good GUI for it is BackInTime (backintime-qt in the repository -- DON'T use backintime-qt4 because 20.04 is based on qt5)

The first time you use rsync it will make a complete copy of your system to your SSD. How long that will take depends on how much data you have. Typically, for 150Gb of /root + /home it would probably take in the order of 15 to 30 minutes to create the first backup. After that, rsync sends "deltas", the difference between the file system on the SSD and your live system. This reduces the backup time to a minute or less in most cases, unless you are prone to moving lots of 100Mb file around.

Restoration, assuming you can boot, is copying the entire system back from your SSD to your sda1 partition. This can take as long as the initial first copy that rsync made of your system.

While lots of companies, including Facebook and others use BTRFS in their enterprise environment, the BTRFS command structure is VERY simple because it is auto-tuning. When I take a snapshot of root (@):

sudo btrfs su snapshot -r /mnt/@ /mnt/snapshots/@yyyymmdd

it literally takes only an instant. It is also empty except for some meta-data. When you change a file, copy of the original file is moved to the snapshot. The next day I do the same commandd (via my script) and create @yyyymmdd+1. Rinse and repeat. There is a limit to the number of snapshots you can keep on your system, but what really matters is how long you keep them. Eventually, as changes are made, a snapshot will become as big as the system it is copying. BTRFS devs say that the number of snapshots should be limited to around a dozen or so per subvolume.

When I experiment, or see that a big update of 150 or so packages are coming I create a standby snapshot:

sudo btrfs su snapshot -r /mnt/@ /mnt/snapshots/@beforebigupdate

If my system reboots and behaves normally I'll delete the beforebigupdate snapshot. Otherwise, I'll boot into my Kubuntu Live USB stick, mount my sda1 drive to /mnt and do a rollback:

mv /mnt/@ /mnt/@old  
btrfs subvol snapshot /mnt/snapshots/@beforebigupdate /mnt/@

Note: I only backup and restore @ because I've merged @home into @ deliberately. I was in the habit of backing up both @ and @home at the same time, using the same timestamp, and when I restored one I always restored its matched pair. Some apps split their files between root and home. Restoring a root snapshot of one date with a home snapshot of another date could create a mismatch. Your home account has the menu links to start a program, and the config files to control it, but the binaries are in /opt or some other place. Mixing @'s and @home's you might end up with missing pieces and unable to run an app.

No matter which backup method you choose, if you stick with it you will become proficient in using it after a little while and using it will become second nature.

[GreyGeek]

Using rsync I have now copied my /home directory and all of my voluntary files to removable media.

Once I have a fresh OS installed, what is the best/safest way to copy my backup back into /home?

Why would you copy your /home backup from your corrupt old system onto a pristine new /home ?
Rsync makes exact copies of your "voluntary" files onto your SSD. You can install mc (midnight commander) and run it as root.
Set your SSD home in the left panel and your system home in the right panel. Right click on your voluntary files and folders to highlight them, then left click on one highlighted file and drag it (with the rest) into your right panel and into the same location. You could use Dolphin but it is a lot slower. MC is a Konsole app.

Or, if your Linux-fu is up to it, you can use the mv command manually to move what you want from where it is on the SSD to where you want it in your /home/yourname folder.

[TwoFistedJustice]

rsync is not working as expected.

sudo rsync -auv [target] [destination]

started copying and overwriting every file that it had previously copied, though the vast majority hadn't even been opened.

sudo rsync -auv --list-only [SOURCE] [DESTINATION]

listed every file

I was under the impression that the -u parameter prevented that behavior?

How do I make it only copy new and changed files?

[jglen490]

Is the full path for your "destination" different from the full path for your "source"? The destination should be an external, USB drive of some sort. The directory structure of the backup on the "destination" will be identical to the directory structure that is in your "source". That's what rsync does.

You NEVER want a backup to be on the same drive as the files you are backing up. A backup will provide the ability to restore files after rebuilding from a hardware failure.

Please don't confuse a "backup" with a "rollback" (or "recovery") - two very different processes, two different purposes. A "rollback" (or "recovery") provides the ability to recover from a failed update or corrupted file or set of files. The "rollback" (or "recovery") data should be on the same drive as your normal filesystem.