Btrfs

[jglen490]

So I've been following the various BTRFS discussions that seem to be everywhere and a part of a lot of different threads. I'm not convinced that I need BTRFS, but I probably could be convinced to want BTRFS. I need to understand what hardware and software that is needed to make this move.

Right now John-Desktop has two mechanical hard drives - 80 GB and 250 GB - each using ext4. These are setup as follows:

John@John-Desktop:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            4.0G     0  4.0G   0% /dev
tmpfs           804M  9.4M  795M   2% /run
/dev/sda1        12G  6.8G  4.5G  61% /
tmpfs           4.0G  876K  4.0G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           4.0G     0  4.0G   0% /sys/fs/cgroup
/dev/sda2        59G   24G   33G  43% /home
/dev/sdb1       230G   26G  192G  12% /home/multimedia
tmpfs           804M     0  804M   0% /run/user/118
tmpfs           804M   28K  804M   1% /run/user/1000

John-Desktop also uses two external mechanical hard drives, in enclosures, connected via USB on a rotating basis to take backups using rsync.

The 250 GB drive has enough room to accommodate the data on both internal drives. So I'm guessing that if I were to put everything on the existing 250 GB using BTRFS, I would need at least one more 250 GB drive. But that's just a guess.

I'm guessing that under BTRFS, I could do away with the external backups and somehow configure BTRFS to accommodate recovery requirements. But that too is just a guess.

Help me to understand this process. Thank you!

[vinnywright]

Well I may not be the best one to try to explain it all ,,,,,,,@oshunluvr is our resident expert on btrfs.

but I do use it ,,,,, you do not NEED any special hardware and "btrfs-tools" would be the only package you may not have that you would need ... other than that the file system support is built in.

the btrfs file system can be made in a partition or you can make it on the bare disk with no partitioning necessary. in btrfs land you can have sub volumes on the file system that can contain OS installs (or backups) , so on one btrfs file system you could have several OS's and their is no need to set a set size for the sub volumes as all will expand to use all available space as needed

@hear I am setup like this

vinny@vinny-Bonobo-Extreme:~$ sudo parted -l
[sudo] password for vinny: 
Model: ATA HGST HTS725050A7 (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags: 

Number  Start   End    Size    Type      File system     Flags
1      8225kB  323GB  323GB   primary   btrfs           boot
3      323GB   379GB  56.3GB  primary   ext4
4      379GB   496GB  117GB   extended
6      379GB   436GB  57.0GB  logical   ext4
5      436GB   496GB  59.8GB  logical   ext4
2      496GB   500GB  4295MB  primary   linux-swap(v1)

Model: ATA HGST HTS721010A9 (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
1      1049kB  1000GB  1000GB  btrfs        primary

sda1 is btrfs and has 2 OS installs in it (Neon-LTS and Kubuntu-17.04)

sda3 ext4 Kubuntu-14.04

sda5 ext4 Debian-8-KDE

sda6 ext4 Kubuntu-16.04 with Neon/dev/stable added to the sources.list

sdb1 btrfs is my data drive and is linked to all my installs

sdb1 was an ext4 file system and had my data on it when I did an ,in place, conversion to btrfs ,,,,,,,the data was intact .

the sda1 looks like this

vinny@vinny-Bonobo-Extreme:/media/vinny/ff5d66d4-35b6-4c9c-a64e-8dfbe2aa1e31$ ls
@  @17.04  @17.04snap  @home  @home17.04  @home17.04snap  ubiquity-apt-clone  var

(I'm in the 16.04 , and mounted the partition with dolphin so thats why the /media/vinny/XXXXXXX)

the @ and @home are the Neon-LTS sub volumes ,,,,the @17.04snap and @home17.04snap are "snapshots" (backups) of the 17.04 install I took just before doing something that may have broken the system ,,,,,,,had things gone south all I would have had to do was delete the subvolumes and rename the snaps (remove the "snap" ) and all would have been back to where I started .

btrfs allso has it's own capabilities ,,,,,, but I know little about that ,,,, @oshunluvr dose however , and will chime in on this one soon I'm sure ...

O ,,,is their any reason your using a 32bit install ?

VINNY

[GreyGeek]

I am running a 5 year old 17" Acer Aspire V3-771G with 6GB RAM and an Intel HDGraphics as the primary GPU and an NVidia GT650M as the secondary GPU, which cannot be set in the BIOS as the primary GPU. However, the nvidia-781 nvidia-378 driver configures the GT650M as if it were the primary and my plasma desktop and everything it runs uses the nvidia GT650M. Running Minecraft 1.13 with Forge and Optifine I get up to 500 fps. Universe Sandbox^2 runs all simulations smoothly.

I am using a Hurricane IPv6 tunnel with Google's IPv6 DNS to give me a 20/20 at ipv6-test.com.

I'm running KDE Neon User Edition, fully updated, with btrfs on two 750Gb HD's configured in RAID1, with a third 750Gb HD using btrfs as a backup drive.

[FONT=monospace][COLOR=#000000]$ [B]sudo btrfs filesystem df /[/B][/COLOR]
Data, RAID1: total=102.00GiB, used=98.86GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=2.00GiB, used=1.09GiB
GlobalReserve, single: total=181.88MiB, used=0.00B

[COLOR=#000000]$ [B]sudo btrfs filesystem show /[/B][/COLOR]
Label: none  uuid: 12980ae8-4117-4cc5-bbb8-8065e82af93d
        Total devices 2 FS bytes used 99.95GiB
        devid    1 size 691.19GiB used 104.03GiB path /dev/sda1
        devid    2 size 698.64GiB used 104.03GiB path /dev/sdc

[COLOR=#000000]$ [B]sudo btrfs filesystem show /dev/sdb1[/B][/COLOR]
Label: none  uuid: 17f4fe91-5cbc-46f6-9577-10aa173ac5f6
        Total devices 1 FS bytes used 265.65GiB
        devid    1 size 698.63GiB used 270.02GiB path /dev/sdb1

[/FONT]

I mount the sdb1 HD only as storage, using btrfs send & receive dated ro snapshots of @ and @home. Then I umount it.

I've been using Btrfs for over 2 years and haven't had a single problem with it, nor have i had to do a roll back. However, I have browsed into snapshots on occasion to fetch config files or previous versions of libraries or programs.

As a retired programmer, among other things, my system holds copies of every major program I've ever written for myself or clients since 1978, plus copies of all my email since around 2000. I'm using 99Gb of HD space.

The only caveats about using Btrfs are:
1: It doesn't need or use swap, so don't waste your space, that also means no suspend or hibernate
2: It uses Copy On Write (COW) so a dynamic virtual HD in VirtualBox is out of the question. Make your virtual HDs static or the directory in which you will save them as NOCOW. I chose static virtual HDs and never had any problem.

I've found that it is easier to create snapshots manually, and takes less space, than to use a tool like Snapper. There are excellent posts describing how to use Btrfs in this forum.

If you do decide to use Btrfs I suggest that your do a fresh install using Btrfs and give your entire drive, /dev/sda1 to it, making sda your boot flag. Also, don't bother with the root & boot partitions or dividing /home into several partitions. With Btrfs that is unnecessary. Btrfs on Ubuntu will create two major subvolumes: @ and @home. (/ and /home) You can create others if you want but that just adds to the complexity. SUSE, for example, makes a subvolume out of just about every major directory in the Linux file system hierarchy.

You can format your 80Gb drive independently with Btrfs without adding it to the pool. Then you can mount it and use it to store backup snapshots. You can also create ASCII copies of snapshots and use it as the destination for those copies. Plus, much, much more .... I haven't even scratched the surface of the power and flexibility of Btrfs

[#]BTRFS[/#]

[oshunluvr]

jglen490, I'm busy working this week but later this evening I should have some time to post a couple comments and advice. Check back around 6pm.

[kubicle]

1: It doesn't need or use swap, so don't waste your space, that also means no suspend or hibernate

Technically, no file system "needs or uses" swap.

Swap (either swap files or swap partitions) are something the kernel uses as a sort of (slow) extension of ram memory (that resides on disk), not really related to file systems.

Last I checked, btrfs didn't support swap files (swap storage inside a file system), and you can't use swap partitions on a partitionless btrfs disk (using subvolumes)...but you can still use swap partitions with separate btrfs partitions if you wish (if you want hibernation, for example).

[GreyGeek]

I have 6GB of RAM and when I was running Kubuntu with Btrfs I created a 12 GB swap partition specifically for hibernation, in case I wanted to use it (but never have). I noticed on systemguard that the swap was never used, so I deliberately loaded many programs and started as many processes (DL, searches, etc) to use all of my eight cores so I could see what it would take to force the system to use swap. I never could get swap into action. When I went to Neon I didn't install a swap partition and no matter what program or process I use I never experience a slowdown or stall, with one exception. Kdeconnectd would often create a minor delay but I never use it so I disabled it and those delay's ceased.

[kubicle]

I have 6GB of RAM and when I was running Kubuntu with Btrfs I created a 12 GB swap partition specifically for hibernation, in case I wanted to use it (but never have). I noticed on systemguard that the swap was never used, so I deliberately loaded many programs and started as many processes (DL, searches, etc) to use all of my eight cores so I could see what it would take to force the system to use swap. I never could get swap into action. When I went to Neon I didn't install a swap partition and no matter what program or process I use I never experience a slowdown or stall, with one exception. Kdeconnectd would often create a minor delay but I never use it so I disabled it and those delay's ceased.

Most people will probably not need swap on modern machines with plenty of ram (regardless of the file system they use), except for hibernation, and are unlikely to ran out of ram unless there is a bad memory leak or they do some really memory intensive tasks.

It's not uncommon to have no swap used (in fact, if your system uses swap a lot you probably need to install more ram...swap is to be considered "emergency" memory rather than "extended" memory, just that you don't lose unsaved work if you ran out of ram)...although this depends on the system's "eagerness" to swap (the configured vm.swappiness value [sysctl vm.swappiness]).

I'm not taking a stand on whether one should have swap space or not, just wanted to mention that you could even if you use btrfs.

[GreyGeek]

.... swap is to be considered "emergency" memory rather than "extended" memory, just that you don't lose unsaved work if you ran out of ram)...although this depends on the system's "eagerness" to swap (the configured vm.swappiness value [sysctl vm.swappiness]).....

That was how Bill Ball explained it to me in his now antiquated book Learn Linux in 24 Hours back in May of 1998. (Amazon has the date wrong by a year) It had RH 5.0 on a CD in back of that paperback.

IIRC (a BIG if), fallocate and mkswap were used to create a swap file (not partition) and swapon activated it. When I was running RH5 on my old Sony VAIO desktop, a PCV-90 with a 166MHz processor, 16MB RAM and a 2.1GB HD, I created a 32 Mb swap file (twice RAM because that's what Ball said was needed). When the graphical installers on LiveCDs came into common use they all had swap partition options instead, and that's when swap became a partition for me. I remember the swap file being used on RH5 and SuSE 5.3 but since the advent of GB RAM in the laptops I bought I can't recall a single instance of the swap partition being used. I don't use hibernate or suspend so sometime around Kubuntu 12 I stopped using swap partitions.

[vinnywright]

btrfs allso has it's own capabilities ,,,,,, but I know little about that ,,,, @oshunluvr dose however , and will chime in on this one soon I'm sure ...

VINNY

OOPS this was a typo it was supposed to read

btrfs allso has it's own RAID capabilities ,,,,,, but I know little about that ,,,, @oshunluvr dose however , and will chime in on this one soon I'm sure ...

and GG has touched on it some as well ...

VINNY

[oshunluvr]

So, to the OP, Why you might want to use btrfs:

Subvolumes:

No partitioning: You no longer need to partition or you can use a single partition in place of many. I currently have 4 installs and respective /homes all on a single btrfs file system contained within subvolumes. The immediate benefit to this is all free space is available to any of the subvolumes since they all reside on the same file system.
Backups: Subvolumes can be sent from one file system to another in their entirety.

Scale-ability using multiple devices:

Drives or partitions can be added or subtracted from a file system to expand or retract space as needed.
RAID or Not-RAID (JBOD) can be configured or re-configured on multiple devices simply and easily.

The order of importance of these factors will depend on your preferences and needs.

A real-world use case might be if you had a single small hard drive but wanted to multi-boot several Linux installs. Using btrfs, you would not have to guess how much space to partition for each install and would never have to re-size a partition. Just install them all to a single file system using subvolumes and share the space.

Your current configuration,

/dev/sda1        12G     6.8G  4.5G    61% /
/dev/sda2        59G     24G   33G     43% /home
/dev/sdb1        230G    26G   192G    12% /home/multimedia

you could combine into a single file system using both drives. Easiest case, you can create a single file system with your current partitions as is without re-partitioning. You would have a single file system of 301G with 230G available. By using subvolumes, you would still have segregation of / and /home and /home/multimedia if you choose to or you could combine /home and /home/multimedia - my assumption is /home/multimedia exists because there's not enough room on sda to hold your videos and music, etc. All the 230G available space could be used by any of the subvolumes.

Of course, you could re-partition, combining sda1 and sda2 into a single partition of 71G and then match it to a new 71G partition on sdb and use RAID0 (142G) for speed or RAID1 (71G) for redundancy and use the remaining space for backup storage.

Here's the amazing part: With the sole exception of actually re-partitioning your hard drives, you can do everything I listed above while still using your computer - not even a reboot needed.

[Qqmike]

Trying to think through this ... So, if using UEFI/GPT, I wonder how this works with btrfs. Trying to think through this, where UEFI will look for an ESP (EFI System Partition), marked as such, and containing the bootloaders, and so on.

[Qqmike]

UEFI and btrfs?

After 10 minutes of intense searching, I get the sense that this could take some time. Not finding anything (yet) at rodsbooks.com (Rod Smith), but it may be there (UEFI and btrfs). Nor at Arch (but it may be there, too). This hints at it:

https://bbs.archlinux.org/viewtopic.php?id=151589

(and concludes you'd need a separate ESP partition)

[Qqmike]

One source, so far:

Cannot use [URL="https://wiki.archlinux.org/index.php/UEFI"]UEFI[/URL] to boot.

https://wiki.archlinux.org/index.php/Btrfs

[oshunluvr]

My understanding (I do not use UEFI) is you need a separate EFI partition and then you can boot using UEFI.

The reference Qqmike links to is the section regarding "Partitionless Btrfs disk" which means the entire disk is used without benefit of a partition table. AFAIK, you can't boot to this configuration with GRUB either. My server's data drives are all partitionless, but they are not boot devices.

It's no different than booting to a GPT disk using GRUB - you need a EF02 (grub boot partition) for GRUB to install to. The current practice for this configuration (I do use this) is when partitioning using GPT, set the parameters using gdisk to "expert", unset the parameter that forces all partitions to a 2048 bite boundary alignment, partition the beginning of the disk (up to but not including byte 2048) as an EF02 type partition, then partition as normal. Then you can install GRUB to the GPT disk.

Qqmike is the expert on EFI booting, but I think it's not a whole lot different than the above.

[jglen490]

Great info! I appreciate the input and the discussions.

While this discussion started I did some research on BTRFS and especially Copy On Write (COW). Were I to be engineering an enterprise level or a business system, BTRFS would be a definite consideration, among other things. COW is not the only consideration for BTRFS, but it seems to be important; Redirect On Write (ROW) is another. From what I've read, COW adds some processing intensity in that for any write there are three I/O activities, and that changes recorded when writing to an existing file are done so in block operations - not file. So the changed data is written in a nearby block and has a reference/pointer to the actual file. That block write can provide the basis of a sort of incremental backup, but is entirely dependent on the existence of the related file. Which brings up the subject of implementing some RAID level. To be effective there needs to be redundancy of storage hardware. The blocks of changes are no good if a hardware problem results in a single redundant drive saying adios - based on some low level of RAID. Higher levels of RAID will make use of additional storage hardware.

Recovery still requires some backup strategy other than incremental.

With that, converting to BTRFS makes no sense at this point with my small system, which does not include multiple users or multiple OS implementations, and which is already being backed up to off-system drives using a rotating media strategy. Maybe the next LTS upgrade, but for now BTRFS is nice but not necessary.

Thank you for the input!