HOWTO: Using btrfs subvolumes for home data files

[oshunluvr]

I am in the process of changing over to 12.04 from 11.04. I usually preserve my previous working install and do a full new install along side so I don't leave myself computer-less if something goes wrong. Since I have multiple installs on the computer, I tend to keep /home within each install so settings and the like are kept unique. This has the potiential to leave things like documents and downloads scattered all over the place. To prevent this scattering and to reduce the required space for each install (having six separate /homes can use a lot of hard drive space!) I have a large single partition and keep Documents, Pictures, Music, Videos, and Downloads folders on it. Then after each new install, I mount the data partition, remove the default cooresponding folders from my /home directory, and create symlinks to the folders on the data partition.

This time I decided to try something different. Since I am using btrfs anyway I decided to get more into the filesystem's abilities and use subvolumes for these files. The primary functional difference is rather than a symlink, I use mount points defined in fstab for each of the four folders.

Why do it this way? Several reasons;

Easier and more efficent backups: btrfs has subvolume "snapshot" capabilities that you can't do with just subdirectries. Also, by having the file types in different subvolumes I can prioritize backups.

To learn how to do it: This usually is enough reason for me.

Visually consistant: The folders appear as "normal" folders in my home rather than links.

Flexability of the filesystem: btrfs allows on-the-fly (while mounted) expansion and contraction. If you run out of room - simply add in another partition or drive.

HOWTO:
I will skip how to create the initial btrfs filesystem. The link above explains that in enough detail.

Example terminal commands assuming your btrfs partition is /dev/sda5:

Mount the root directory (not a subvolume) somewhere and navigate to it;

sudo mkdir /mnt/data
sudo mount /dev/sda5 /mnt/data
cd /mnt/data

Create a subvolume and take ownership of it;

sudo btrfs subvolume create @mydocuments
sudo chown myuser:myuser @mydocuments

Move your existing files to the new subvolume;

mv ~/Documents/* /mnt/data/@mydocuments

Edit /etc/fstab using your preferred method and add the mount points:

/dev/sda5 /home/myuser/Documents  btrfs  space_cache,compress,subvol=@mydocuments 0       0

You can use the rootvolume UUID instead of the device name if you prefer.

Mount the new subvolume in the correct place and you're done:

sudo mount /home/myuser/Documents

One interesting (and not benefitial) difference is when moving files across subdirectories on a single partition, the file "pointers" are moved but not the actual data. When using subvolumes, the file is actually moved from one subvolume to the other, even if they exist on the same partition. Obviously, this takes quite a bit longer.

Another small downside is you can't share directories this way. A symlink is still the best way to go for that.

I'll likely have more to add as I play with this new setup...

[SecretCode]

Interesting - the flexibility (dynamic resizing, etc) of btrfs seems like a major advantage here.

The primary functional difference is rather than a symlink, I use mount points defined in fstab for each of the four folders.

...

Visually consistant: The folders appear as "normal" folders in my home rather than links.

Technically I think you could do this with multiple ext4 partitions as well - but of course it gets to be quite a hassle having lots of logical partitions at the disk level.

One interesting (and not benefitial) difference is when moving files across subdirectories on a single partition, the file "pointers" are moved but not the actual data. When using subvolumes, the file is actually moved from one subvolume to the other, even if they exist on the same partition. Obviously, this takes quite a bit longer.

That's the same as it would be with multple ext4 partitions. From Linux's pov, moving files within a filesystem is just a case of making a new folder entry pointing to the same inode; but inodes are specific to a filesystem and to move to a different one, it requires a physical copy. Although it looks like btrfs subvolumes don't use different extents within the partition (from a v brief reading) - so conceivably a file could be moved to another subvolume without a physical copy/move of the file data - it might require a smarter Linux driver to support this.

Another small downside is you can't share directories this way. A symlink is still the best way to go for that.

Do you mean sharing directories between different installed OSes? Or between users?

[oshunluvr]

Do you mean sharing directories between different installed OSes? Or between users?

I was thinking between users. You could have different mount points for each user to the same directory, but a link to a single mounted directory makes more sense to me. For example, the default skel user has a "Public" folder intended for Samba file-sharing. In my case, I have 4 people that sometimes use my computer. I can replace that folder with a link to @Public subvolume and give it world access. A somewhat safer course would be to have group ownership of the public areas and add that group to all your users. In actual practice, I have a home server so that acts as the "Public" storage/file exchange.

I decided to do this post to introduce btrfs to more people. Dibl and I have been using it for a couple of years now and we've both found it to be quite stable. I suspect once the file system check tool is fully functional, it will be the default fs for many distros. They are also going to change the way subvolumes are handled likely sometime this year, so they'll be a format change then (which will be fun!) but that should solve a lot of larger issues and make the fs easier to use.

In a desktop/server environment, btrfs really shines. Without taking the filesystem off-line you can add and remove drives and partitions at will and RAID happens almost automatically and is configurable to your needs. In my previous server setup, mdadm RAID was used, but when a drive needed removal it was a real chore. With btrfs, it's a breeze.

Re. Inodes: From a post by Ian Kent, a btrfs dev.:

1) You cannot hardlink between subvolumes. This is because subvolumes have their own inode numbers and such, think of them as separate mounts in this case, you cannot hardlink between two mounts because the link needs to point to the same on disk inode, which is impossible between two different filesystems. The same is true for subvolumes, they have their own trees with their own inodes and inode numbers, so it's impossible to hardlink between them.

Another way to use subvolumes: rather than mounting each of them, you can "CD" into them like any other partition. In the case of a Public subvolume, this might be a better course than mounting.

[dibl]

My btrfs filesystem on a pair of WD1002FAEX 6GB/s drives is about 16 months old now, and I see no issues or signs of problems. I chose the drives to take advantage of the Marvell 9128 controller on my Asus P6X58D-E motherboard. I used the defaults for a multi-drive btrfs installation, resulting in mirrored metadata and striped data. My pile of data amounts to about 768GB, out of a total filesystem capacity of 1.8TB, today:

root@imerabox:/home/don# btrfs fi df /mnt/DATA
Data, RAID0: total=820.00GB, used=768.36GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=72.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=12.62GB, used=1.55GB
Metadata: total=8.00MB, used=0.00

I'm sold on btrfs as a "easy" way to set up large multi-drive filesystems -- and cost-efficient, when compared to buying a serious hardware RAID controller.

[SteveRiley]

Oshun and Dibl... can you guys comment on any improvements to the speed of Btrfs? Last time I looked at it, about a year ago, it took almost an hour for a 300 MB apt-get upgrade to churn -- and on an SSD, no less. Same thing on EXT4 finished in 4 minutes.

[oshunluvr]

I can't speak for the apt-get portion of your time. I created a roughly 400MB file (dd bs=4096 count=102400 all zeros) and copied it between two ext4 partions and two btrfs partitions:

ext4-to-ext4
real 0m3.297s
user 0m0.008s
sys 0m0.596s

btrfs-to-btrfs
real 0m0.504s
user 0m0.004s
sys 0m0.364s

ext4-to-btrfs
real 0m0.452s
user 0m0.000s
sys 0m0.404s

btrfs-to-ext4
real 0m3.139s
user 0m0.012s
sys 0m0.624s

This is way less than a comprehensive test. Small file performance and really large file performance would likely vary. Also, these are non-RAID ex4 partitions and a three disk btrfs array using a few "advanced" mounting options. btrfs: space_cache,compress ext4: nobh,data=writeback,noatime.

I posted some more extensive testing way back a like a year ago or so and compared the performance to ext4 in a 4-disk mdadm RAID to a 4-disk btrfs RAID array. The ext4 RAID performed better in some areas and the btrfs array better in others,. However, that was before space_cache and compress were available. At the time I felt the ease of setup and use of the btrfs made it the clear winner, not to mention the other features (like snapshots and subvolumes).

Remember though, I'm using it in a server and performance desktop both with multiple drives and was using mdadm RAID everywhere until I switched to btrfs. There are also some other considerations to btrfs that haven't been worked out yet. For example, in a RAID array, btrfs doesn't handle large differences in partition size well - it tends to treat the larger partition as though it were the same size as the smaller one. Simple work around is to partition to nearly equal sizes and expand the number of partitions in an array. For example, my server has 2x2TB and 2x1TB drives. I had reserved 200GB on the 1TB drives before I added the 2TB drives to the mix. Since I had a functioning btrfs array of 2x800GB, I created 2 800GB partitions on each 2TB drive and added all 4 of them to the existing array giving me 6x800GB (4.8TB) on that array and used the remaining 2x500GB on the 2TB drives for a second array. Obviously, having partitions on the same physical device in an array doesn't add any performance, but it doesn't appear to hurt any either. Next year when I re-do the server I'll make the main array 4x1TB (1 per drive) and the secondary 2x750GB (unless I acquire a couple more larger drives )

[SteveRiley]

Those Btrfs writes in your test are much faster than the EXT4 writes -- but since your Btrfs mounts include caching and compression, it makes sense that a file of all zeros would show such difference in performance. How would the numbers look if you created a 400 MB file from /dev/urandom instead?

[oshunluvr]

For my buddy, Steve...

ext4-to-ext4
real 0m3.627s
user 0m0.004s
sys 0m0.564s

ext4-to-btrfs
real 0m6.955s
user 0m0.008s
sys 0m0.404s

btrfs-to-btrfs
real 0m0.524s
user 0m0.004s
sys 0m0.452s

brtfs-to-ext4
real 0m3.178s
user 0m0.004s
sys 0m0.548s

As I said before - the 3-disk btrfs array will clobber the ext4 single partition every time. A better test would be a 3-disk mdadm RAID0 with ext4. Unfortunatly, I just installed 12.04 beta 1 so I'm busy migrating my setup.

For the record the ext4 partitions are on different disks. The 2 btrfs arrays are on the same set of disks. The times were retrived using a simple time cp file target/file command.

[dibl]

Oshun and Dibl... can you guys comment on any improvements to the speed of Btrfs? Last time I looked at it, about a year ago, it took almost an hour for a 300 MB apt-get upgrade to churn -- and on an SSD, no less. Same thing on EXT4 finished in 4 minutes.

There's no reason btrfs shouldn't work fine for the OS partition, but I really wouldn't recommend it, as I don't think it would offer any advantage over ext4, especially on a SSD. But I don't know why an update or a dist-upgrade should take any longer on btrfs.

Btrfs is really targeted to large, and possibly dynamic, data storage requirements. While I haven't had any reason to use the dynamic storage features that it offers, it's obvious to me why it should be very attractive to the IT guys at the server farm.

One caveat -- btrfs absolutely sucks for hosting a Virtual Machine (which is really a single very large file). It's a known issue and I would suppose that situation will be improved as the development continues.

[SecretCode]

One caveat -- btrfs absolutely sucks for hosting a Virtual Machine (which is really a single very large file).

Why is that? I mean why should it be any worse than ext3/4? (I can see why advantages might be lost.)

And would it have any implications for things like database tools that keep masses of data in one file with lots of random seeks and writes? (Like Microsoft Access. I'm not actually sure if there are real-world cases of this in common Linux apps.)

[dibl]

Why is that? I mean why should it be any worse than ext3/4? (I can see why advantages might be lost.)

And would it have any implications for things like database tools that keep masses of data in one file with lots of random seeks and writes? (Like Microsoft Access. I'm not actually sure if there are real-world cases of this in common Linux apps.)

I really have not studied it much -- I wasn't aware of the issue, and tried doing something on a backup VM that I had parked on the btrfs partition, and was surprised to see how slow it ran. A quick google search turned up this and so that's how I learned about the issue, and that's about all I know about it.

But is your MS Access toolbox 10GB or 20GB or more in size? That's the scale of a VM hard drive.

[SecretCode]

I have no Ms Access files at all! What kind of person do you take me for?

I'm just remembering stuff that people used to talk about.

I think huge databases are the only things that compare with VMs in size (even video files aren't), but such large dbs would often run on dedicated partitions and use multiple files (and NOT use Ms Access!) - and some of the database engines have their own raw file systems do they not?

eta: I like the language in that bug report ... "I will now spend some time getting this to suck less" etc

[dibl]

I have no Ms Access files at all! What kind of person do you take me for?

Whew! Well, I did work a bit with MS Access many years ago, but I could not imagine a 20GB Access file.

I would be surprised if a conventional RDBMS had a big problem running its data tables on a btrfs filesystem -- they surely had that kind of application in mind when they designed it in the first place. But my stuff is just videos, music, images, docs, etc., so I don't really know. I think the problem must be related to the unique way a VM opens the virtual hard disk and uses it.

eta: I like the language in that bug report ... "I will now spend some time getting this to suck less" etc

Yeah, they seem to have a good attitude about it.

[oshunluvr]

I'm using a btrfs array for my os (11.04 and 12.04) and btrfs array for my home that includes my VirtualBox stuff and I haven't noticed any issues, speed or otherwise.

[SteveRiley]

Thanks, Oshun, for running the second test. One thing that caught my eye was the difference between the ext4-to-btrfs runs. Here are your numbers:

File of zeros:

ext4-to-btrfs
real 0m0.452s
user 0m0.000s
sys 0m0.404s

File of random bytes:

ext4-to-btrfs
real 0m6.955s
user 0m0.008s
sys 0m0.404s

Less than half a second to almost seven seconds...if I'm interpreting your numbers correctly, it would appear that Btrfs with data that doesn't lend itself to compression is still performing not too well...?

But I don't know why an update or a dist-upgrade should take any longer on btrfs.

It was over a year ago, using Arch. I could have had a bad build or something. The symptom was incessant disk thrashing, very similar to what's described in that Redhat bug post you found.