[SOLVED] Problems Making Backups

[oldgeek]

I have been trying to make a backup of my system to an external HD using btrfs but so far have been unable to do so. I took snapshots of my system using Greygeek's steps in the BTRFS Guide in this thread. Everything seemed to work out right, until the end after I had copied my ltest @home to the HD; it ended with an error message which I did not copy down. Nonetheless, the folder appears in my HD and I don't know if it is valid or not. Next I tried to send the latest @home to the HD, but it stopped partially through saying it couold not read one file. I checked the folder on my HD and indeed it is incomplete. I would like to start all over again but I don't know how to delete the suspect folders on my HD, as I get an error message if I try to copy to them, saying that the folder already exists. Really, I have not had much luck with btrfs so far with respect to backing up and would like to know how to delete the suspect folders so I can try again.

[oshunluvr]

Ok, when working with BRTFS and backups, you're working with subvolumes. I wouldn't trust a partially completed backup. So...

...go to the backup file system and delete the backup subvolume
sudo btrfs subvolume delete -c <your_backup_name>
sync

Then retry the backup send|receive command. If it fails again, either write down the error or take a screen shot or cell phone pic of it so you can paste it here.

You don't want to "copy" the actual folders/files because that sort of defeats the purpose of using BTRFS to make backups. Backup your @home with a read-only snapshot that you send to the external hard drive.

[oldgeek]

This time I followed your instructions correctly and I was able to delete the suspect files (subvolumes?) without a hitch. I then sent my latest snapshot of @ to the ext hd and it 'copied' correctly. I'm still waiting for the @home snapshot to finish, but so far it is sending correctly. So thanks. I suspect I had made a typing error somewhere, and Konsole doesn't forgive typing errors. The process I used with the computer on was to first mount the internal hd to /mnt and then the external hd to /backup/. Then I deleted the offending files (subvolumes?) and then proceeded to send the snapshots to the ext hd. I did not use sync after deleting--was that necessary?--but did use it after sending @ and @home (just finished, successfully). Did I do everything correctly? Did I need to mount the internal hd to /mnt?

What is the purpose of sync?
When I used the ext4 file system, I backed up my files using Aptik, and was able to use it twice to restore my files. It seemed to take the same amount of time. What is the difference between using Aptik as a backup system and btrfs? I was unable to restore my @ and @home subvolumes awhile ago when my system crashed, but I think it was due to errors in the process. But the question remains--how is btrfs better in restoring files and configurations? Note: I use the term copy inaccurately when I mean send. Ext4 habits.
Thanks again for all your help.

[GreyGeek]

This time I followed your instructions correctly and I was able to delete the suspect files (subvolumes?) without a hitch. I then sent my latest snapshot of @ to the ext hd and it 'copied' correctly. I'm still waiting for the @home snapshot to finish, but so far it is sending correctly. So thanks. I suspect I had made a typing error somewhere, and Konsole doesn't forgive typing errors. The process I used with the computer on was to first mount the internal hd to /mnt and then the external hd to /backup/. Then I deleted the offending files (subvolumes?) and then proceeded to send the snapshots to the ext hd. I did not use sync after deleting--was that necessary?--but did use it after sending @ and @home (just finished, successfully). Did I do everything correctly? Did I need to mount the internal hd to /mnt?

Yes, and Yes.

The purpose of using /mnt, or any other directory (/subvol) is to mount the Btrfs <ROOT_FS> system, which is above the level of @ and @home. Since both /mnt and /subvol were created as root they are owned by root and are under root (/).

IF you have only one HD in your PC then the "Internal hd" is that HD, and is usually represented by /dev/sda1, the partition on which you installed / with Btrfs as the root file system. As long as you do not have other storage devices plugged into your system it is safe to use /dev/sda1 as
mount /dev/sda1 /mnt (or subvol if you are oshunluver! ) However, I am assuming that the external drive you mounted as /backup is a USB HD. RIght?

I have personally experienced the switching of sdX values when I plugged in my third HD. (My Acer has two bays and I removed the CDROM and replaced it with an HD Caddy) I was running two HDs as RAID1 (sda and sdb). When I plugged in the 3rd HD it became sdb and what was sdb became sdc. My RAID1 continued to function normally because it was mounted with the uuid in fstab, but I was continually getting errors that sdc has the right uuid but the wrong sdX designation. I eventually re-balanced my RAID1 back to RAOD0, to double my disk space.

IF, however, you have three internal HD like I have, then care must be taken that the system doesn't switch sdX device designations on you during the mount process. That is why I use
mount /dev/disk/by-uuid/theuuidofmysda1 /mnt

I haven't seen USB sticks switching sdX designations but ... Btrfs expects UUID to truly be unique, which is why you DO NOT want to use block level clones on Btrfs subvolumes and you should use UUID's to mount your <ROOT_FS> to /mnt.

What is the purpose of sync?

sync writes any data buffered in memory out to disk. This can include (but is not limited
to) modified superblocks, modified inodes, and delayed reads and writes. This must be
implemented by the kernel; The sync program does nothing but exercise the sync(2) system
call.

The kernel keeps data in memory to avoid doing (relatively slow) disk reads and writes.
This improves performance, but if the computer crashes, data may be lost or the filesystem
corrupted as a result. sync ensures that everything in memory is written to disk.

When I used the ext4 file system, I backed up my files using Aptik, and was able to use it twice to restore my files. It seemed to take the same amount of time. What is the difference between using Aptik as a backup system and btrfs? I was unable to restore my @ and @home subvolumes awhile ago when my system crashed, but I think it was due to errors in the process. But the question remains--how is btrfs better in restoring files and configurations? Note: I use the term copy inaccurately when I mean send. Ext4 habits.
Thanks again for all your help.

I've never used Aptik but from what I've read it seems to be a piece meal technique unless you click the "backup everything" button. When you mount the <ROOT_FS> to /mnt, for example, you are working outside of @ and @home, which includes everything in the / file system, including the file system, and everything under /home, including your home account. You are working on @ and @home while it is live and running. I don't know about Aptik, but many backup or cloning programs cannot copy a program that is running. Running programs will be skipped. Btrfs copies everything, running or not.

You use "btrfs su snapshot -r /mnt/@ /mnt//snapshots/@YYYYMMDD" to create an exact image of @ as a read only snapshot, identical in every way at that instant. Ditto for @home. The act of "sending it": "btrfs send /mnt/snapshots/@YYYYMMDD | btrfs receive /backup" causes the snapshot to be fully populated, so the time it takes to send it is comparable to the time it would take any other process to copy the same data. IF something messes up in / or /home/youraccount and you can't boot up or log in and need to recover from a snapshot you don't have to re-install your distro and then import your copied data. The act of replacing the old @ and @home with the most recent working snapshot recovers your ability to boot and log in.

[oshunluvr]

BTRFS and the use of logical containers (subvolumes) and it's handling of multiple devices offers several advantages over other file systems, not the least of which is "one-stop shopping" for features you would otherwise have to obtain by adding on some other piece of software.

In the case of backups - since I can't comment on or compare to Aptik as I've never used it - I can state that the backup and restore capability is now an integral part of your file system. Rather than relying on some outside tool, you now can backup and restore using the file system itself. I wouldn't expect it to be faster - data is data after all and you're still backing up all your data - but it's simpler, operates in the background, and backups are immediately accessible without other outside tools.

Same goes for file compression, RAID, snapshots, migrating devices, and more - all handled natively.

I have taken snapshots, rolled back to a snapshot, made and restored from backups, migrated data from one device to another (changed hard drives), created RAID layers, changed RAID levels, removed RAID altogether, compressed, uncompressed and changed file compression types, all without additional software and all - with the exception of a snapshot rollback - without even rebooting. No other file system does this so easily, simply, and without outside software.

[oldgeek]

Thanks to both of you, very informative. I understand then that while I am sending a subvolume to my external HD (which is not a thumb drive, by the way) I can continue to use the computer to surf the net, use programs, and whatnot. Do I understand correctly? I used Aptik to restore my whole HD before I had btrfs and afterwards, when I had trouble restoring my saved subvolumes. It works OK, but I would rather just use btrfs--hence all the questions (and mounting mistakes).

[oshunluvr]

Yes, btrfs will do it's work in the background and you will hardly notice it. Part of the reason you have to have a read-only snapshot to send|receive is that prevents you from corrupting the data in the snapshot during the copy operation.