BTRFS usage- Adding devices, moving data, converting to RAID1, all without rebooting

[oshunluvr]

I decided to move away from using backups on my server to using RAID1 with btrfs instead - less CPU overhead and almost instant recovery.

BTW, I realize RAID1 isn't the same as a regular backup because if you delete a file accidentally, a backup saves you, RAID1 doesn't. However, this makes sense to me for use on the server. Anything I want to keep badly enough, I'll make an additional backup on my desktop. Mostly the server holds music and videos and my family uses it for their backups from their computers and phones.

**Notes**

I did actually have to reboot once during the whole process because I changed boot devices and I wanted to be sure I could boot to the new install location before wiping the old one. Otherwise, no rebooting necessary.
My server supports hot swapping drives so I never powered down. All that's needed is sudo partprobe after a drive swap.
If you're new to BTRFS, I reference "subvolumes" below. On a btrfs, a filesystem can be divided into subvolumes. This keeps data separate while still existing on a single filesystem. This has many benefits including all free space on the filesystem is available to any subvolume to use when needed. Subvolumes are mountable just like a partition so no partitioning is necessary. I currently have 3 different installs on a single btrfs filesystem, each in their own subvolume.

I have been a long time btrfs user (since 0.19) and am very comfortable with using it. I recommend it to anyone who will listen. Here's one example why:

My server had 1 6TB drive for data and 3x2TB drives used for backups. All 4 btrfs file systems in single mode. The new configuration is adding a small 60GB SSD as a boot device, a 2nd 6TB drive for RAID1 with the first drive, and removing one of the 2TB drives. I will use the 6TB drives in RAID1 for my DVD collection and the 2TB drives in RAID1 for everything else (pics, docs, music, etc.). This will increase my total data storage to 8TB. A small part of the 2TB drives is reserved (partitioned) for a backup for the installation (Ubuntu server 14.04) so if the SSD fails I can boot up to one of the 2TB drives and be running again in short order - BIOS fallback in boot order is automatic.

Tasks:

1. Remove the extra 2TB drive and install the SSD and the new 6TB drive
2. Copy Ubuntu to the SSD and set up the SSD as the boot device
3. Repartition the 2 remaining 2TB drives and create a new RAID1 filesystem on the 2TB drives
4. Add the new 6TB drive to the current filesystem on the older 6TB drive
5. Convert the 12TB filesystem to a 6TB RAID1 filesystem
6. Move target data from the 6TB RAID1 to the 2TB RAID1 (docs, etc)
7. Make backup copies of Ubuntu on the 2TB drives

1. Since 2TB drive I was removing was only used for backups, I just pulled it. I inserted the 6TB drive in it's place and connected the SSD to an unused sata port. I ran partprobe and all the drives were up.

2. One of BTRFS best features? send/receive. I created a new btrfs filesystem on the SSD, mounted it, and then sent a full copy of my currently running install to it. That's right, I didn't have to boot to another install or a USB drive. The steps to do this are: Make a read-only snapshot of the install subvolume, issue the send|receive command, then wait for it to complete. Once it's received, since it's read-only, simply make a new read-write snapshot of the subvolume, delete the read-only version, and you're good-to-go. I went into the install and manually edited fstab and grub.cfg to reflect the UUID of the new filesystem and rebooted. I set BIOS to boot to the new SSD and it booted first try.

3. My new scheme required some partitioning. I made a 60GB partition on both 2TB drives that allowed me to duplicate the entire SSD as a backup. The remaining space was used for a new storage filesystem in RAID1. One interesting hiccup - I deleted the old partition tables, re-partitioned, and created the BTRFS RAID1 filesystem. The first attempt showed a large portion of it in use even though no files were on it. I went back and used wipefs to remove the original BTRFS superblocks, re-created the RAID1 filesystem and all was well. Creating a RAID filesystem with btrfs is a single command: mkfs.btrfs -m raid1 -d raid1 /dev/sdc2 /dev/sdd2 That's it. Took about a second to complete. The two switches, -m and -d, refer to metadata and data. Making them both RAID1 puts a copy of everything on both drives. If you've ever tried to do this with mdadm you can appreciate just how easy btrfs really is.

4. BTRFS handles multiple devices easily as well. A single command: btrfs device add /dev/sdb /mnt/pool and it's all done. This added the new 6TB drive to the already existing 6TB filesystem mounted at /mnt/pool. I now had a 12TB filesystem with all the data on the first drive.

5. With another single command: btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/pool and we were off to the races. Unlike the previous commands, this one did not happen instantly, I went to bed. I imagine it took several hours to copy 5.3TB of files. When complete, I had a full 6TB RAID1 filesystem with all my data duplicated on both drives.

6. Now I needed to move all the data I wanted to relocate from the 6TB filesystem to the 2TB filesystem. Just like in step 2, send|receive did the work. Again though, this takes time. I moved about 1.5TB from one filesystem to the other.

7. TBD: As of this morning I haven't decided the best way to make bootable backups of the install. With changing UUIDs and such, I have a couple issues to work out. For now, I made send|receive backup copies of my install, but the ultimate goal would be auto-fallback to a current copy of the install in the event of a failure. My BIOS will handle drive selection, but booting to a degraded RAID might be problem. BTRFS will do a incremental send|receive but I assume this will over-write any edits to fstab and grub.cfg which would be need to boot correctly.

These operations did not even require unmounting any of the subvolumes except the ones deleted when re-partitioning the 2TB drives. The entire time my server was up and running and all files accessible. I can't talk enough about how awesome btrfs is.

[GreyGeek]

+1

[#]btrfs[/#]

[dibl]

@oshunluvr, you are the MAN!

[GreyGeek]

Indeed!

[oshunluvr]

Lol

[oshunluvr]

Update:

Once all my data folders were sent from the main btrfs pool to the new one, I wanted to be sure everything transferred. So I ran this command on all the subvolumes that I transferred:

diff -r @Pictures/ /mnt/pool/@Pictures/ | grep @Pictures/ | awk '{print $4}' > pictures.txt

This should compare the two subvolumes and list any missing files. None showed up as missing.

Finally, I unmounted the original subvolumes and mounted the new subvolumes in place for a final check of functionality and completeness before removing the original subvolumes.

Looks good so far!

[NoWorries]

Very impressive with what you have done with Btrfs. I am not such a heavy data user, but I do require to unplug my Laptop and relocate every couple of weeks. Also when I travel, I take my Laptop with me as I can load and edit pictures that I have taken. This saves a lot of time when I get home to finish this process.

Now with this type of activity, does Btrfs make what I do easier as I currently use Ext4?

[GreyGeek]

BTRFS usage- Adding devices, moving data, converting to RAID1, all without rebooting

My experience with btrfs shows that as far as application use goes the only caveat is that if you use VirtualBox you MUST create your virtual drives with no copy on write, nocow, to avoid corruption of the virtual HD. Other than that Btrfs blows Ext4 away, especially in reversing unwanted changes or restoring to previous conditions. Also, with the default automatic snapshot settings one can eat up HD space rather quickly. On my 625Gb HD using Ext4 with my usual installs I'd have around 525Gb of unused space. With Btrfs using Snapper the "unused" space was 160-200Gb. After neutering Snapper's config settings I used my own script with Snapper and got about 400Gb. Snapshots are virtually empty at first but get bigger as changes are made to the system as old versions of changed files are moved to the snapshot. Rollbacks are a breeze. Oshunluver has several excellent posts documenting the use of Btrfs.


Sent from my iPhone using Tapatalk

[NoWorries]

I cannot remember the last time I needed to do a rollback. Probably because I install a new system every 6 months.

My main concern with Btrfs was related to when you have mounted disks that store all the rollback files and other systems. I have found problems closing down one computer mounted to another which are Ethernet connected. In the case of a system with Btrfs connections to other storage devices, what happens when you remove one computer to use elsewhere and then return? Is the removed computer useable and do you have to unmount and mount to re-establish Btrfs?

[oshunluvr]

I'm unclear what exactly you're doing or what you mean, but as far as I know, btrfs doesn't connect over Ethernet. It's a file system not a network protocol. The connection of a mounted drive is over NFS or SAMBA or?

"Rollbacks" aren't "files" per se, they are snapshots of subvolumes. If you were taking a snapshot for potential rollback use you certainly wouldn't want to transmit it to another computer. It wouldn't be of any use to you there. If you were sending a snapshot for backup then why would not having that computer "attached" during normal use be a threat? It would be no different than using a thumb drive for backup. You'd only need to have it attached to restore from a backup.

[oshunluvr]

Very impressive with what you have done with Btrfs. I am not such a heavy data user, but I do require to unplug my Laptop and relocate every couple of weeks. Also when I travel, I take my Laptop with me as I can load and edit pictures that I have taken. This saves a lot of time when I get home to finish this process.

Now with this type of activity, does Btrfs make what I do easier as I currently use Ext4?

whether or not btrfs would benefit would depend on your setup and your needs. For example, if your laptop has a separate home partition you could combine all your data -install root and home - into a single btrfs filesystem. This would allow all your free space to be available to both the install and home.

Backups are as simple as snapshot, send, delete snapshot. Restore is just as easy. You can have a simple script do it automatically with cron. Rollbacks are just snapshots. Before you do something potentially dangerous, like an upgrade, simply snapshot first. If all is well, just delete the snapshot. If not, replace your install damaged by the upgrade with the snapshot and delete the non-working install.

Many of the really awesome features are when your using multiple devices like I talk about above but there are many other benefits as well.

[GreyGeek]

Computer A running Btrfs works normally with itself and Computer B running EXT4 works normally by itself. Having them connected to the same subnet doesn't change anything. Either one could be used as a network server. But, if you are talking about mutiple storage devices attached to a single computer then here is the latest info.

[NoWorries]

Sorry for the confusion. When I use an Ethernet connection, I always use NFS. And if I mount another computer, I find that I have to make sure that it is unmounted before closing down.

I now have a 256GB SSD attached to my Laptop's USB 3 port, so if I changed to Btrfs and used this to store the rollback files, I understand that I would not loose any free space on my Laptop Home directory. If I then disconnect my Laptop to say go and do a presentation, this arrangement would not cause any problems from disconnecting, making changes to the Laptop stored memory and then reconnecting the SSD to my USB port.

[oshunluvr]

NoWorries: I'm not sure, but I think you might have a couple misconceptions. If it seems I'm talking down to you, please forgive me, but I want to explain a few basic things;

Networked File System (NFS): A file system, or format, resides only on the hard drive it's installed on and does not transmit itself over a network connection. Point being; the file system is irrelevant to the NFS issues you're experiencing. BTRFS will neither add to or reduce your issues in this area. Likely, if you find that removing the NFS shared computer without unmounting causes problems, it is how you have it mounted that is the source of the issue. The most obvious error you might see is "NFS stale file handle." Explore the mounting options soft/hard, bg, and intr to help with this issue. You could also explore a method of auto-mounting and unmounting the share when it is detected or missing.

BTRFS and SNAPSHOTS: BTRFS Snapshots are full copies of the current state of a BTRFS subvolume and may be read/write or read-only. They initially take no space because, rather than actually copying each file, their file "handles" (location and storage information) are stored. When you begin making changes to the source subvolume (the one you took a snapshot of) then the snapshot begins to grow because any file changes that occur to the source are not saved in the snapshot. So for example, you take a snapshot of your home and delete 20 files from it. You would not see any additional available free space because the files were not actually deleted. Their file handles are removed from the source subvolume but not from the snapshot. The snapshot continues to grow as the source subvolume ages (assuming it's in use and changes are being made to it) until it is deleted.

The only place that using a networked device with BTRFS would matter is if you wanted to send (transmit) a BTRFS SUBVOLUME SNAPSHOT over the network. In order to receive the transmitted snapshot, you would either have to have a BTRFS filesystem on the receiving drive or send the snapshot to a file rather than to a subvolume. The disadvantage to sending a snapshot to a file is that you won't be able to enter (view or copy files) from the snapshot until it's restored to a BTRFS file system as a subvolume.

Snapshots and Subvolumes have many fundamental benefits, two of which are: backups and rollbacks. Which is most beneficial depends on your needs.

BACKUPS: Making a backup of your home to a USB disk is a classic use for btrfs snapshot and send/receive. The steps and explanations of them are:

1) Mount the root filesystem that contains the @home subvolume.

The default way Ubuntu and variants like Kubuntu install to btrfs is to create two subvolumes at install time named @ and @home. These subvolumes are then mounted; @ at / and @home at /home. During normal usage you would never actually notice the difference or "see" the subvolumes. In order to make a snapshot of your home subvolume, you would need to mount the primary file system. You could then see and snapshot @ or @home.

2) Make a read-only snapshot of the @home subvolume.

Before a snapshot can be transmitted using send/receive it cannot be in use in any way, otherwise data can be corrupted. The way this is guaranteed is by making the snapshot read-only.

3) Mount the target (USB or whatever) drive.
4) Use btrfs to send/receive the read-only snapshot to the target drive.

This makes a full copy of every file in the snapshot. As you might expect, this may take quite some time depending on the size of the subvolume.

5) Delete the read-only snapshot.

If you don't delete it, it will grow until it reaches the size of the source subvolume.

Backups need not be "sent" anywhere. If you have the drive space simply leaving the snapshot in place will serve as a backup. Sending the backup to another device and then deleting the source snapshot conserves space on the source drive (if needed) and protects you against device failure - just like any other backup method.

One thing to note is the received snapshot is also read-only because it is a copy of the source snapshot. This is usually fine for a backup, but if you wanted to make it read/write for some reason, you simply make a new snapshot of the received subvolume as read-write and delete the read-only one.

ROLLBACKS: A "Rollback" is really just a description of how you use a snapshot to protect yourself from unintended results from an action like an upgrade. The steps and explanations of them are:

1) Mount the root filesystem that contains the @ subvolume.

2) Make a read/write snapshot of the @ subvolume.

It must be read-write so it can be used to replace the source subvolume if needed.

3) Do the action you're protecting yourself from (upgrade, etc.)
If your install is OK after the action:
4) Delete the snapshot.
If your install is NOT OK after the action:
4) Delete the failed subvolume and rename the snapshot to match the deleted subvolume and reboot.

Obviously, if you're working from your install you would have to boot to a LiveUSB install or another install before you could delete and rename the subvolume. Another method to boot to the snapshot be to edit the necessary grub.cfg and fstab to allow booting to the snapshot without renaming, but this is a little complicated so I'm not going into it here. One of the small but mighty benefits to using subvolumes is they can reside on the same filesystem. This means all the UUIDs are identical and booting to a different subvolume can be done without changing UUIDs. The mounting of a subvolume is just a mount option in fstab.

I now have a 256GB SSD attached to my Laptop's USB 3 port, so if I changed to Btrfs and used this to store the rollback files, I understand that I would not loose any free space on my Laptop Home directory. If I then disconnect my Laptop to say go and do a presentation, this arrangement would not cause any problems from disconnecting, making changes to the Laptop stored memory and then reconnecting the SSD to my USB port.

NoWorries, in direct reply to your comments; using the USB drive to store backups is a great way to use BTRFS and if you follow the steps I outlined it wouldn't take any extra space on your laptop drive. However, storing subvolume snapshots on a different drive does not qualify as a "rollback." Once you move the snapshot off the root drive it isn't available to use as a rollback. You would have to go through a full restoration of the snapshot-ed subvolume - just like with any other backup method. The point of making a snapshot for rollback usage is it allows you to quickly and easily restore your system to it source state at the time of the snapshot. Remember a snapshot takes zero space until changes are made to the source subvolume. If you want rollback capability, you'll need to snapshot when you feel it necessary and delete those snapshots when they are no longer needed.

Regarding drive space; if you used BTRFS on your single-drive laptop you wouldn't need to have multiple partitions for Linux. You could change your home, root install, and any other current Linux partitions into subvolumes of a single file system on a single partition. Thus, all the free space is combined and is available to any of the subvolumes. But because the data is segregated by subvolumes, the data remains separate and is accessed just like it is today - only mounted using an option indicating the subvolume name. In effect you would have more usable free space because it isn't artificially divided. This is another amazing benefit to btrfs and it's use of subvolumes.

[GreyGeek]

BTRFS usage- Adding devices, moving data, converting to RAID1, all without rebooting

This is such an excellent post it, and all the other's oshunluver has posted about BTRFS should be grouped into its own forum and pinned.


Sent from my iPhone using Tapatalk