Announcement

Collapse
No announcement yet.

BTRFS Testing and use

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    BTRFS Testing and use

    dibl and I have been trying out the btrfs filesystem over the last few months. Since it's been included in the Alternate install CD for (K)Ubuntu and has a ton of potential for the future, I decided we should have our conversation about this publicly.

    dibl has been testing by transferring files to and from an ext4 partition and a btfrs partition. I'll let him fill in anything he sees fit to add.

    I have been using a btrfs partition on my desktop computer as a backup device. Last week I added 2-2tb drives to my server and rather than use RAID or LVM, I opted to use btrfs. One immediately positive result is the size (4tb) of this device is easily handled by btrfs.

    I am transferring the contents of my backup btrfs device to the new brtfs drives and when the 1.2tb is moved, I plan on creating several new devices to compare speeds and such. Since I am still using 10.04 as my primary install, I will also be installing 10.10 to take advantage of newer btrfs features not available to the 2.6.32 or below kernels. I currently use several RAID0 devices and reisferfs filesystems. I will compare those along with EXT4 to the BTRFS single and multiple devices and report speeds.

    Finally, I also have a currently unused SSD of average speed that I will also test using BTRFS.

    I am a busy guy so it may be a while to get all this done but I wanted to begin the public discourse asap.

    What follows is the contents of dibls emails regarding this topic and any we'll post any new information we develop as we go along...


    Please Read Me

    #2
    Re: BTRFS Testing and use

    @dibl:

    Today I created a directory with 24,850 files in 520 sub-folders, total size 3.8GiB. I used it to re-run the "Small Test" case. (The "large file" test is what it is -- no need to re-do that one). Here are the results:

    BTRFS Copy Speed Test

    Without space_cache and compress options

    Large File From To Time
    1 ext4 btrfs 1:33
    2 btrfs ext4 3:54

    Small Files
    1 ext4 btrfs 1:40
    2 btrfs ext4 2:43


    With space_cache and compress options

    Large File From To Time
    1 ext4 btrfs 1:52
    2 btrfs ext4 3:48

    Small Files
    1 ext4 btrfs 1:26
    2 btrfs ext4 2:27

    Now it is possible to discern a significant performance difference. I re-ran the "small files" test twice -- I think this is real.

    Please Read Me

    Comment


      #3
      Re: BTRFS Testing and use

      Thanks oshunluvr.

      The "large file" for this test is a single 16GiB video, .mpg format. As stated above, I created a directory of 24,850 files in 520 subfolders, total size of 3.8GiB, for the "small files" test. You can do the math -- these are all very small html and .png files. The drives involved are very fast already -- the ext4 filesystem is on a PCI-e bus SSD, and the btrfs fileystem is on a pair of WD1002FAEX SATA 6GiB/s drives on a Marvell 88SE9123 SATA-3 controller.

      My tentative conclusion is that btrfs file operations won't be noticeably faster on transfers of very large files, when the space_cache and compress options are enabled. However, transfers of many small files appear to be roughly 10% faster on a btrfs filesystem with those two options enabled (2.6.37 or later kernel required, and the mount options don't take effect until you reboot the system).

      I don't have the ability to run transfers between two separate btrfs filesystems -- I only have the single 2-disk btrfs filesystem. Btrfs is still in development, and we know from experience that "tweaking" the code for performance is normally the last developmental activity prior to release for general use, so it probably is not as fast today as it ultimately will be.

      Truly geek news .... 8)

      Comment


        #4
        Re: BTRFS Testing and use

        Ok - I finally got my testing done. Please remember this is very unscientific.

        For my testbed I used my daily desktop Kubuntu 10.04 with 2.6.32-30-generic kernel. This is important to note because there are significant upgrades to BTRFS with the .38 kernel and the utilities changed - more options available that speed things up. SInce I'm using .32 I don't get those upgrades.

        For partitions I used 2 sets of 4x6gb partitions on 4 exactly the same hard drives. For reiserfs and for ext4 I first create RAID0 devices using mdadm. For btrfs I used the btrfs-raid option.

        For all partitions I used only the default options - no special tweaks during creation or mounting. This was mostly a function of my time required to research all the choices.

        I used a single 3.0gb large avi file for the large file test and for the many files test I used a folder containing 13.2GB 7740 files 93 subfolders - mostly jpg's. I ran a timed copy command in a terminal, didn't use the computer for anything else during the tests, deleted the target after each copy (so overwrite lag isn't a factor) and ran each test six times - throwing out the largest of each set and then averaging the remaining five.

        All timings came out with real, user, and sys times

        Large Many
        file test files test
        reiserfs 13.415 real 186.944
        0.057 user 0.247
        10.657 sys 75.300

        ext4 11.254 real 291.344
        0.030 user 0.287
        6.443 sys 45.160

        btrfs 14.891 real 160.022
        0.037 user 0.247
        4.703 sys 36.330

        So clearly ext4 is better on the large file but remember no special settings were used and ext4 has plenty of options. One thing I noticed was the CPU time of reiserfs was way higher than the others so even though it was a bit faster in partical use, it slower other tasks down considerably. btrfs won out handily on the many files test.

        I am preparing to make the change to 11.04 and along with that I am moving to btrfs. Even if the results were different reiserfs is practically dead and ext4 is a temporary fs - btrfs is the future...





        Please Read Me

        Comment


          #5
          Re: BTRFS Testing and use

          Good job -- thanks for posting it!

          Comment


            #6
            Re: BTRFS Testing and use

            Interesting discussion. The only thing that keeps me away from btrfs is the lack of an fsck tool. Aren't you worried that there may be a problem?

            Having said that, from what I read by those using btrfs already it is a blinder.
            Once your problem is solved please mark the topic of the first post as SOLVED so others know and can benefit from your experience! / FAQ

            Comment


              #7
              Re: BTRFS Testing and use

              As it says in the wiki "btrfs is stable on a stable machine" -- my experience indicates that is true. My installation (pair of WD1000s) is on a big tower that is running on a UPS, and I shut it down when I'm gone from the house all day. As long as you don't suffer a hard crash, and keep all data backed up, I don't think anything bad is likely to happen. I wouldn' t advise btrfs on a laptop, for example.

              Comment


                #8
                Re: BTRFS Testing and use

                Supposedly the fsck tool is "almost ready." I the meantime having a backup will suffice. With the cost of hard drives, there is no reason to not have space for a backup.

                I agree with dibl - my system is similar to his although I leave my computer running 24/7.

                I really can't remember the last time I needed to run a filesystem check that wasn't related to my own actions.

                Please Read Me

                Comment


                  #9
                  Re: BTRFS Testing and use

                  Originally posted by oshunluvr
                  Supposedly the fsck tool is "almost ready."
                  Ah, I didn't know (checked some time last week). I'll keep an eye on it.
                  Once your problem is solved please mark the topic of the first post as SOLVED so others know and can benefit from your experience! / FAQ

                  Comment


                    #10
                    Re: BTRFS Testing and use

                    I've been trying to install the Natty Daily Current to a btrfs raid0 4 partition with no luck.

                    It occurs to me that although btrfs has the RAID0,1,10 options that the downside of RAID0 might outweigh any upside.

                    From what I can find no one has reported anything re. using RAID0 vs. regular btrfs setup.

                    I think that things like snapshots and adding/subtraction devices I might be better off not selecting RAID0 and sticking to LZO compression.

                    It appears this will give me better performance than the 4 device RAID0 resierfs I'm replacing anyway.

                    Opinions?

                    Please Read Me

                    Comment


                      #11
                      Re: BTRFS Testing and use

                      I installed it with default values, based on this from the "multiple devices" section of the btrfs wiki:

                      By default, metadata will be mirrored across two devices and data will be striped across all of the devices present.
                      That sounded like a reasonable configuration for my main data storage, which amounts to some 450GB. So, addressing my pair of WD1002FAEX drives, I did something like

                      Code:
                      sudo mkfs.btrfs /dev/sdd /dev/sde
                      You have to remember that btrfs is really a different concept than what we are use to thinking about, when we say "RAID": https://btrfs.wiki.kernel.org/index.php/SysadminGuide

                      So bearing in mind how it actually works, when I check my btrfs filesystem RAID status, it shows the "data" and "metadata" just as per the wiki:

                      Code:
                      root@aptosidbox:/home/don# btrfs filesystem df /mnt/DATA
                      Data, RAID0: total=484.00GB, used=445.07GB
                      Data: total=8.00MB, used=52.00KB
                      System, RAID1: total=8.00MB, used=40.00KB
                      System: total=4.00MB, used=0.00
                      Metadata, RAID1: total=1.62GB, used=634.77MB
                      Metadata: total=8.00MB, used=0.00
                      I dunno whether that sheds any light on your question ....

                      Also, AFAIK, Grub still won't boot a btrfs filesystem, so /boot will have to be elsewhere on ext2/3/4.

                      Comment


                        #12
                        Re: BTRFS Testing and use

                        There really isn't much commentary regarding the RAID options for btrfs. It appears the only difference is how the metadata is treated, which again may not be worth the bother.

                        Roger on the /boot. Since I've been using RAID for a couple of years I almost do that by reflex anyway... https://launchpad.net/btrfs-grub2

                        Please Read Me

                        Comment


                          #13
                          Re: BTRFS Testing and use New Install: 11.04 using btrfs

                          I did two installs for testing.

                          First I installed to a single partition using the ALT installer disk. This worked fine although it was incredibly slow. I really don't know how long it took because I left, but I know it was more than two hours.

                          I then attempted to use the "device add" function of btrfs - no joy. I got a general ERROR and no other info. I attempted this both while using the partition and when not using it (from the second install). At this point I would say this function doesn't work. An interesting thing that happened suiring this install was since I did not specify a separate /home partition, the installer automatically created a subvolume for / and one for /home with a separate mount line in fstab. This allows you to easily backup and/or make a "snapshot" of your /home subvolume! What a great function!

                          I did the second install from the LiveCD version to a brtfs four partition array using the "-m raid0" switch which spreads metadata (rather than duplicating) across all disks. This time I also used a separate /home partition created with the metadata on all devices also across four partitions.

                          This install also created subvolumes for root and home although they are on their own partitions. Again, genius - since you can later add more subvolumes if you desire.

                          I haven't tweaked mount options yet or done any speed testing but I believe the btrfs project is on the right path and will be THE disk format of choice for all distros in very short order.

                          Please Read Me

                          Comment


                            #14
                            Re: BTRFS Testing and use

                            Time for a small update: I've been using btrfs as my primary filesystem for five "devices" since June. So far (knocking on wood...) no issues at all.

                            All data backed up, of course.

                            This week I decided to remove one of the four hard drives on my desktop computer to use it in a DVR. So I have had to delete three partitions from three btrfs devices in order to clear this drive of data.

                            I did the first delete from a rarely used install partition - no issues.

                            As I type I am deleting a partition from my /home btrfs device. This device is 240gb total in size 60gb X 4 partitions. I am doing this while logged in and running from my home which is on this device! Amazing. I can tell the system is busy because of an occasional lag, but it's brief and not often.

                            Next will be removing the last partition from my install which I will also do while running from it. As a long-time hard drive abuser I am totally in love with this filesystem. I can't wait until it's the default.

                            Please Read Me

                            Comment


                              #15
                              Re: BTRFS Testing and use

                              Thanks for the update. Here, my btrfs filesystem has been 100% free of any cause for concern -- nada. Earlier this summer I started recording my collection of vinyl record albums, so my filesystem usage now looks like this:

                              Code:
                              root@aptosidbox:/home/don# btrfs filesystem df /mnt/DATA
                              Data, RAID0: total=762.00GB, used=638.96GB
                              Data: total=8.00MB, used=7.73MB
                              System, RAID1: total=8.00MB, used=64.00KB
                              System: total=4.00MB, used=0.00
                              Metadata, RAID1: total=11.88GB, used=1.16GB
                              Metadata: total=8.00MB, used=0.00
                              I haven't looked lately to see if there is any news about the fsck tool -- have you seen anything about that?

                              BTW, with the increase in amount of data, backup has become a little more challenging. I decided to try a 7z compressed tar backup, which worked perfectly after I figured out how to do it, but it took 36 hours due to the heavy compression options that I used. With a lot of music and video content, which does not compress much, I think maybe I'll just have to spring for more/bigger backup drives and stick with "cp" as my backup tool.

                              Comment

                              Working...
                              X