BTRFS usage- Adding devices, moving data, converting to RAID1, all without rebooting

[jlittle]

diff -r @Pictures/ /mnt/pool/@Pictures/ | grep @Pictures/ | awk '{print $4}' > pictures.txt

Using diff -r is handy and convenient, but for terabytes of pictures that will read them all, it might take a while. If you just want to check for existence, diffing a listing would be far faster, say

diff <(cd @Pictures/; find . | sort) <(cd /mnt/pool/@Pictures/; find . | sort)

or for the error prone like me

cd @Pictures
find . | sort  > /tmp/a
cd /mnt/pool/@Pictures
find . | sort > /tmp/b
diff /tmo/a /tmp/b

[oshunluvr]

Thanks for that, jlittle. Always looking for better ways to to things. In this case, it was more the number of files rather size that was at issue. 30,000+ files in 80GB. It did take quite awhile. It's not something I normally do but it was more of a checking of the procedure so that when the inevitable question of veracity came up, I could claim validation. Also, I had deleted my backups to re-configure to RAID1 so I was being cautious.

I'll add your code above to my cheat sheet.

[NoWorries]

I decided that I would try to install the zesty beta 2 on my experimental desktop and use btrfs. As this would require formatting my root and home partitions, I attached a 256 GiB SSD via a usb port and installed the system on this SSD using btrfs for both the root and home. I then copied all home files from the experimental desktop onto this new installation on the usb port. This system work quite well, so I did the next step below.

I then installed beta 2 onto the desktop with btrfs for both the root and home partitions. I then copied back all the home files (including hidden files) from the usb SSD onto the desktop home.

I did not add any RAID devices to this system. I do find that /dev/sda1 is / and /dev/sda6 is /home and both are btrfs. When I issue the device scan command I get

:~$ btrfs device scan
Scanning for Btrfs filesystems
ERROR: cannot open /dev/sda1: Permission denied
ERROR: cannot open /dev/sda6: Permission denied

Have I done anything wrong up to now or should I have taken a different approach?

I did find an interesting and positive report on btrfs at http://www.virtualtothecore.com/en/2...xt-filesystem/.

[oshunluvr]

ERROR: cannot open /dev/sda1: Permission denied
ERROR: cannot open /dev/sda6: Permission denied

You forgot sudo. Device level access always requires sudo.

Your explanation does not make it clear as to how you got to having / and /home on separate partitions. This is not the normal nor is it optimal. Nor do you state if you are using subvolumes for / and /home. If not, you are not getting any benefit from using btrfs.

Unless they changed the installer for Zesty, the default and preferred installation is to install to a single file system (whether it exists on a single partition, whole device, or multiple devices). The installer then automatically separates / and /home into subvolumes: @ for / and @home for /home.

Reasons:

Using a single file system allows all the subvolumes to share the free space (benefit #1).
Having everything in subvolumes allows snapshots. Snapshots are required to be able to:

Rollback
Backup
Send/Receive

(benefits #2,3, and 4)
If you needed to increase your storage capacity, you could add another drive or partition to your filesystem while still using the file system (benefit #5).

If somehow your / and /home are not in subvolumes, I recommend you start over. If they are, in fact, in subvolumes you can either re-install or just move the @home subvolume to the same filesystem as @.

Let me know how you plan to proceed and I'll help you with the steps.

[NoWorries]

Thanks for your willingness to help with me with getting btrfs to work as it should.

My initial attempt was to install Zesty using the SSD connected to my usb port and I chose the manual disk approach and selected a single btrfs / partition. I found it interesting that separate root and home partitions were created and the files from the desktop SSD home partition were copied onto the usb SSD home partition. I found that there was the @home and @root which were the same size as those on the desktop.

I then shutdown, disconnected the usb SSD and followed the same approach to install Zesty on the desktop with a single btrfs partition. When I did this there was only a single / partition on reboot. Both the root and home did not have the @home or @root designation that was on the usb SSD. I will be very interested to know how to get this right. I do like to see separate root and home partitions

[oshunluvr]

Terminology question:

...I chose the manual disk approach and selected a single btrfs / partition. I found it interesting that separate root and home partitions were created...

Did you mean "separate root and home subvolumes" ??

Subvolumes are not partitions, although they can be mounted as if they were. The difference is huge. I would be surprised if the Zesty install acts differently than previous versions. I'll test it later for my own eyes.

What you should see if the install was done correctly: "df" will show you both / and /home mounted with the same device name and sizes. If this is the case, then you have a single btrfs file system with 2 subvolumes. To view the subvolumes at the higher level, create a mount point and mount the primary device (partition). For example if your partition containing the btrfs file system is /dev/sda2, then mount /dev/sda2 to somewhere. Example:

sudo mkdir /mnt/myinstall
sudo mount /dev/sda2 /mnt/myinstall

Now df will show three mounts all have the same device name and sizes. A listing (ls) of /mnt/myinstall would look like:

@    @home

Try this and post back...

[GreyGeek]

Oshunluver,
I re-read your OP to see if I could glean additional Btrfs info from it when I noticed something that I missed the first time ... a home computer system using EIGHT TERABYTES of data!

Around 1981 or 1982 a client of mine, Centaur Parts in G.I., had been using a two disk drive Apple ][+ to do their inventory using software I had written. With the new system he could handle a larger inventory of nuts, bolts, etc... than he had before, doing everything manually. With the larger inventory two 5.25" floppies wasn't going to cut it. So, he bought a Corvus 5MB HD drive for the Apple. The instructions said to turn on the drive and let it warm up for 15 minutes to stablize the temperature before you turned on the Apple attached to it. It ran well once warmed up, and for the time it was very fast. It took most of a night to back up the data on that drive to VHS cassette tape, which was also part of the package. When he first saw it in action after we put the entire contents of the two floppies (226 Kb) on it his first reaction was "this thing is a barn! We'll never fill it up." He continued to increase his enventory, which increases his customer base, which increased his inventory, etc.... Within a year he had me transfer my software to a new IBM PC with a 10MB HD. A year later he needed a 20Mb HD, and his capacity needs kept climbing.

8,000 GB ... That would hold the contents of 1,600,000 Corvus drives!

[NoWorries]

I will start with what the KDE Partition Manager shows. I have the usb connected that I did my first install on. The first is the desktop SSD


The next is the SSD attached to the usb


I will now show what Dolphin gives in the /mnt/ listing


To me the above shows that the usb SSD installation created two partitions with the correct @ while the SSD on the Desktop had the correct @ designation for both root and home. The df command shows

:~$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
udev             3863704         0   3863704   0% /dev
tmpfs             777312      9520    767792   2% /run
/dev/sda5      243024896  20882712 221299576   9% /
tmpfs            3886560         0   3886560   0% /dev/shm
tmpfs               5120         4      5116   1% /run/lock
tmpfs            3886560         0   3886560   0% /sys/fs/cgroup
/dev/sda5      243024896  20882712 221299576   9% /home
tmpfs             777312         0    777312   0% /run/user/119
tmpfs             777312        12    777300   1% /run/user/1000
/dev/sda5      243024896  20882712 221299576   9% /mnt/myinstall
/dev/sdb1       11697996   7509360   2885456  73% /mnt/myinstallusb
/dev/sdb2      238318024 143393584  93423232  61% /mnt/myinstallusb2

This shows that all is well. My main concern is the consequence of a new installation and the possibility of the home files (including hidden files) being lost. Is this what will happen and, if so, what is the best way to stop this happening?

[oshunluvr]

I agree the "myinstall" drive looks OK. Only thing is why was a logical partition used, but this is of no consequence, just that it wasn't necessary.

The USB drive concerns my somewhat because of the two partition structure. In install on that drive has the very real possibility of filling up with only 11GB available. Granted, unless you use it full time it will not likely be an issue. However, once again, an unnecessary complication. Additionally, those snapshots, while not an issue now, could each grow to the size of the initial install. If you're going to keep the USB drive as it is, you'll want to regularly delete those snapshots.

My main concern is the consequence of a new installation and the possibility of the home files (including hidden files) being lost. Is this what will happen and, if so, what is the best way to stop this happening?

This, of course, has nothing to do with btrfs or your install. It's a matter of backups. btrfs is like any other file system. The good news is it's much simpler to do backups with btrfs and since you have a USB drive with btrfs on it, it's a natural target to send your backups to.

Here's my thoughts with what I see: You haven't explained your "use case" for the USB drive. As long as you're not planning on actually booting the USB drive frequently, you can leave it as-is. However, I see a real possibility for a great way to use the USB drive.

Here's what I'd do with your set up:

I'd wipe the USB drive partitions leaving a single partition (unless you want a swap partition).
Make the USB drive bootable (install GRUB to it).
Rather than maintaining two separate installs, I'd use the fixed drive install and then send it to the USB drive as a backup - and then "elevate it" as the bootable install. Once initially set up, you could do incremental backups to save time.

Here's how this would work:

Initially, you plug in the USB drive, take read only snapshots of @ and @home and then do a full send|receive of both @ and @home from the desktop to the USB drive, adding the work "_backup" to each of them when you send them, like this:

$ sudo -i
# btrfs subvolume snapshot -r @ @_backup
# btrfs subvolume snapshot -r @home @home_backup
# btrfs subvolume send @_backup | btrfs receive /mnt/myinstallusb2/
# btrfs subvolume send @home_backup | btrfs receive /mnt/myinstallusb2/

Now you have a full duplicate of everything.
Your USB drive now contains @_backup and @home_backup, both read-only copies of your @ and @home from the fixed drive.

Now you want to prepare the USB drive to be bootable, so we need read-write copies of the subvolumes:

**NOTE** From here on out I will use the shorter version of the commands. Most btrfs commands can be shortened to their first to letters, so "subvolume" becomes "su" and "snapshot" becomes "sn" etc.

# btrfs su sn /mnt/myinstallusb2/@_backup /mnt/myusbinstall2/@
# btrfs su sn /mnt/myinstallusb2/@home_backup /mnt/myusbinstall2/@home

Now your USB drive contains:

# ls -l /mnt/myinstallusb2
total 0
drwxr-xr-x 1 root root 236 Mar 24 10:52 @
drwxr-xr-x 1 root root 236 Mar 24 10:52 @_backup
drwxr-xr-x 1 root root 32 Mar 24 10:52 @home
drwxr-xr-x 1 root root 32 Mar 24 10:52 @home_backup

The final hurdles to make this USB drive bootable are GRUB and fstab. I think the easiest solution is to use partition labels or device names instead of UUIDs. Before sending the backup to the USB drive, simply edit fstab to use labels or device names. To fix GRUB, uncomment (or add) this line in /etc/default/grub:

GRUB_DISABLE_LINUX_UUID=true

Now update-grub will use the device name or label to select the boot drive and your USB drive should be bootable. Do this on both drives.

However often you backup, you plug in the USB drive and send an incremental backup to the USB drive. This requires a new snapshot on the fixed drive while still keeping the previous version, then you send|receive the differences to the same target on the USB drive. Those commands look like:

# btrfs su sn -r @ @_backup_new
# btrfs send -p @_backup @backup_new | btrfs receive /mnt/myinstallusb2/@_backup

Finally, on the USB drive you delete the previous @ and @home and again make new r/w copies of @_backup and @home_backup and you're good to go.
You would also need to remove the first backups on the fixed drive and rename @backup_new to @backup so you were ready for the next incremental send|receive

Hopefully I didn't leave anything out.

The coolest thing would be to script and automate this so that when you plugged in your USB drive, the incremental backup would begin automagically. I might try this myself!

[oshunluvr]

Oshunluver,
I re-read your OP to see if I could glean additional Btrfs info from it when I noticed something that I missed the first time ... a home computer system using EIGHT TERABYTES of data!...8,000 GB ... That would hold the contents of 1,600,000 Corvus drives!

I know, a LOT right? Total space is actually 16TB (2x6TB + 2x2TB), but most of that is RAID1 via btrfs so after drive and filesystem overhead it's 7.3TB for storage with the remainder for images of the boot disk - a 60GB SSD. Currently, it's about 50% used so lots of room to grow.

It holds all our movies, TV shows, family videos, music, shared documents, family photos (back to 1957 - before I was born!). All that, plus nextcloud for phone backups and my family to use for file storage. I also use it to store my backups from my desktop. The two large drives hold the movies and tv shows. All else is on the two smaller drives.

[GreyGeek]

...

The final hurdles to make this USB drive bootable are GRUB and fstab. I think the easiest solution is to use partition labels or device names instead of UUIDs. Before sending the backup to the USB drive, simply edit fstab to use labels or device names. To fix GRUB, uncomment (or add) this line in /etc/default/grub:

GRUB_DISABLE_LINUX_UUID=true
Now update-grub will use the device name or label to select the boot drive and your USB drive should be bootable. Do this on both drives.
...

You lost me on this. I can make a USB stick bootable by setting the boot flag when I format it, and saving grub to it, but without installing an OS on it what is it going to do? And by booting the USB stick aren't I "unbooting" (disconnecting) my HD?

Are you saying that if I have booted to and am running on my HD and that if I make the changes to the grub and fstab ON MY HD, my USB stick will boot when I plug it in, even though I am running off of my HD? Or, the changes are on the USB grub & fstab (OS installed?) and it will "boot" without disconnecting my HD?

Why isn't just automounting sufficient?

[oshunluvr]

You lost me on this. I can make a USB stick bootable by setting the boot flag when I format it, and saving grub to it, but without installing an OS on it what is it going to do? And by booting the USB stick aren't I "unbooting" (disconnecting) my HD?

Are you saying that if I have booted to and am running on my HD and that if I make the changes to the grub and fstab ON MY HD, my USB stick will boot when I plug it in, even though I am running off of my HD? Or, the changes are on the USB grub & fstab (OS installed?) and it will "boot" without disconnecting my HD?

Why isn't just automounting sufficient?

Well, I was thinking about the case where you would send a btrfs install to any other drive and then have it bootable right away, on the same computer, without any manual intervention. If UUIDs are in use by GRUB or in fstab, you would have to amend the UUIDs in at least those two locations prior to booting. If you avoid the use of UUIDs altogether, you could - in theory - boot to a received subvolume without doing anything special except making it RW vs. RO. You cannot, at least normally, mount two drives with the same UUID. I'm supposing you might use a USB device as a backup boot device and want to boot to it for recovery purposes.

Point being, as long as the USB drive had GRUB on it, and GRUB was looking for "@" to boot to, you wouldn't actually have to install to it at all. You could send a bootable OS to it using btrfs.

[GreyGeek]

Ah, OK. That's in essence what I've done with my bootable USB stick with Neon installed using Btrfs. It's a bootable and runable stick, not the classic LiveUSB ISO used for installing. If my HD bites it I can boot into the USB stick and have a Btrfs system that can see my internal HD and my external USB 320GB WD Passport HD, from which I can send/receive from the Passport to my internal HD, with all the work being done within the booted USB stick with Btrfs on it.

I'm going to shamelessly steal your idea of using a diff snapshot to send/receive to my @_backup and @home_backup snapshots on the 320GB WD passport. After, or course, I verify that changes made since the last diff/s/r are good.

ALL without having to bring down your system. You gotta love Btrfs!

[oshunluvr]

ALL without having to bring down your system. You gotta love Btrfs!

+1

[NoWorries]

I agree the "myinstall" drive looks OK. Only thing is why was a logical partition used, but this is of no consequence, just that it wasn't necessary.

The USB drive concerns my somewhat because of the two partition structure. In install on that drive has the very real possibility of filling up with only 11GB available. Granted, unless you use it full time it will not likely be an issue. However, once again, an unnecessary complication. Additionally, those snapshots, while not an issue now, could each grow to the size of the initial install. If you're going to keep the USB drive as it is, you'll want to regularly delete those snapshots.

Thanks very for you comprehensive response.

When I saw the snapshots, I decided to find out how big they were. The root partition was 11.16 GiB with 9.02 GiB used. The properties for the @ showed 5.9 GiB, for @apt-snapshot-2017-03-20_17... showed 5.0 GiB and for @apt--snapshot-2017-03-20_19... showed 5.2GiB. The total is then 16GiB which is greater than the partition size of 11.16GiB. To me this indicates that the snapshots contain pointers to the files in @.

On thing about the two partition organization on the usb is that a new install does not required backing up the @home. Ever since I have been using Kubuntu, I have never backed up before doing a new install, which makes a new install very fast. So I must confess that I am still preferring the two partition approach and keep an eye on the snapshot files.

I will give your suggestions a try and use the backup home files from my Laptop EXT4 system.