btrfs replace and disk sizes

[Philip]

Hello,Lets assume, a BTRFS disk fails and want to replace it. I have two possibilities (in theory):

1. I do a "btrfs replace" or
2. I do a "btrfs delete" and then a "btrfs add"

Is there any difference concerning the exploitable disk space. Example: I have a RAID5 with 3 1TB disks and now one breaks. I want to replace it with a 2TB disk. In which case, (1) or (2), I get more exploitable disk space? The question is, does a "btrfs replace" exactly reuse 1TB of the new 2TB hard drive or more?Thanks for your help

[kubicle]

The question is, does a "btrfs replace" exactly reuse 1TB of the new 2TB hard drive or more?Thanks for your help

According to the documentation, replace will initially use (after the replace) only the 1TB, but after the replace you can use resize (btrfs filesystem resize <devid>:max <path>) to expand the space to the full 2TB

https://btrfs.wiki.kernel.org/index....failed_devices
https://btrfs.wiki.kernel.org/index..../btrfs-replace

[GreyGeek]

What kubicle said, and the official status page for btrfs (including raid56) is:
https://btrfs.wiki.kernel.org/index.php/Status#RAID56

I checked that because I noticed you mentioned RAID5, which wasn't ready for general use the last time I checked. It appears that except for the write hole it is "mostly OK":

RAID56

Some fixes went to 4.12, namely scrub and auto-repair fixes. Feature marked as mostly OK for now.
Further fixes to raid56 related code are applied each release. The write hole is the last missing part, preliminary patches have been posted but needed to be reworked. The parity not checksummed note has been removed.


Device replace

Device replace and device delete insist on being able to read or reconstruct all data. If any read fails due to an IO error, the delete/replace operation is aborted and the administrator must remove or replace the damaged data before trying again.

[oshunluvr]

I've not used RAID5 with BTRFS, but my understanding using raid 5 you have 2 drives of capacity on a 3 drive array. RAID 5 will always be limited use the capacity of the smallest disk so you won't see any storage increase in your example. This is backed up accoring to this:

https://www.carfax.org.uk/btrfs-usage/

Given your example of changing from 3x1TB to 2x1TB+1x2TB one could actually move away from RAID 5 to RAID 1. RAID 5 still has issues. BTRFS will use the three drives to spread the data RAID1 style so all space is used. Not really a gain in capacity, but less possible problems using RAID1 than 5.

BTW - for anyone who cares, there's new tools available for BTRFS. THey're not in the 18.04 package, but in the 20.04 package of "python3-btrfs" are four new commands:

/usr/bin/btrfs-balance-least-used
/usr/bin/btrfs-orphan-cleaner-progress
/usr/bin/btrfs-space-calculator
/usr/bin/btrfs-usage-report

The third one there will answer the "How much space will I have" question.

[oshunluvr]

Example of above tool:

stuart@200vm:~$ sudo btrfs-space-calculator -m raid5 -d raid5 2T 1T 1T
Target metadata profile: RAID5
Target data profile: RAID5
Mixed block groups: False
Total raw filesystem size: 3.64TiB
Device sizes:
Device 1: 1.82TiB
Device 2: 931.32GiB
Device 3: 931.32GiB
Metadata to data ratio: 1:200
Estimated virtual space to use for metadata: 10.00GiB
Estimated virtual space to use for data: 1.81TiB
Total unallocatable raw amount: 931.32GiB
Unallocatable raw bytes per device:
Device 1: 931.32GiB
Device 2: 0.00B
Device 3: 0.00B
stuart@200vm:~$

RAID1:

stuart@200vm:~$ sudo btrfs-space-calculator -m raid1 -d raid1 2T 1T 1T
Target metadata profile: RAID1
Target data profile: RAID1
Mixed block groups: False
Total raw filesystem size: 3.64TiB
Device sizes:
Device 1: 1.82TiB
Device 2: 931.32GiB
Device 3: 931.32GiB
Metadata to data ratio: 1:200
Estimated virtual space to use for metadata: 10.00GiB
Estimated virtual space to use for data: 1.81TiB
Total unallocatable raw amount: 0.00B
Unallocatable raw bytes per device:
Device 1: 0.00B
Device 2: 0.00B
Device 3: 0.00B
stuart@200vm:~$

[kubicle]

Good points by both of you, I understood (maybe incorrectly) that the example was hypothetical, and the question was just that "can you get the same same space with replace as with delete/add" (for which the answer was yes).
Possible problems and limitations with RAID5/btrfs should obviously be taken into account in real world situations (for example, you should not get wasted space on RAID5 if you have 2x1TB + 2x2TB).

[GreyGeek]

python3-btrfs has the following comment in the change list:

python-btrfs (11-2) unstable; urgency=medium

and that was marked on Fri, 22 Mar 2019 01:12:12 +0100.

I did some investigating and found that v10 is the stable version, v11 is the unstable version, and there is a version called the "latest". The author's web page is here:
https://pypi.org/project/btrfs/
He has a release history and a set of files to download.

I got curious and searched for "btrfs" on that site and found these:
https://pypi.org/search/?q=btrfs

Which list 31 python projects for btrfs. There are some VERY interesting offerings.

• btrfs 11
  
  Python module to inspect btrfs filesystems
  
• btrfs-backup 0.3.1
  
  Intelligent, feature-rich backups for btrfs
  
• btrfs-sxbackup 0.6.11
  
  Incremental btrfs snapshot backups with push/pull support via SSH
  
• buttersink 0.6.9
  
  Buttersink is like rsync for btrfs snapshots
  
• btrfs-simple-snapshots 0.1.4
  
  Snapshot you btrfs subvolumes and apply a retention policy
  
• shadow 0.39
  
  btrfs snapshot utility
  
• snapshotbackup 0.0.16
  
  backups with `rsync` and `btrfs` snapshots
  
• bedup 0.10.1
  
  Deduplication for Btrfs filesystems
  
• butter-hopper 0.2.0
  
  
• btrfsmaint 0.5.2
  
  BTRFS Filesystem Maintenace Scripts.
  
• buttervolume 3.7
  
  Docker plugin to manage Docker Volumes as BTRFS subvolumes
  
• butterknife 0.2.0
  
  Butterknife makes bare-metal Linux deployment dead-simple using the Linux Containers (LXC) and Btrfs filesystem.
  
• btb 0.1-dev
  
  Backup system using rsync and btrfs snapshots
  
• ebt 2.0.92
  
  Flexible backup framework
  
• snapperS 1.1.8
  
  A set of supplementary commands to be used with snapper
  
• snapbtr 0.1
  
  Simple btrfs snapshots management
  
• reflink 0.2.1
  
  Python reflink wraps around platform specific reflink implementations
  
• bytterfs 0.2
  
  Backup script for btrfs send/receive.
• snapper-systemd-boot 0.1.0.dev2
  
  Generate systemd-boot entries from snapper btrfs snapshots.