Migrating From EXT4 to BTRFS?

[TWPonKubuntu]

oshunluvr;
Thanks for your reply to gnomek. It did, kind of, answer my own question... backup, backup, backup is the proper way to handle a system upgrade.

Does btrfs offer a way to protect SOME data during a system upgrade? Yes, I get that I could restore from backup to get my data back intact, but that seems like a needless extra step.

[GreyGeek]

What Oshunluver says.
The best approach is to KISS it!

You can still keep your data in another subvolume on your system HD or on a second HD.

For example: after you install, say, 18.04 using Btrfs in the normal way (to /dev/sda1 giving it Btrfs as "/") you'll end up with two partitions, @ and @home, which fstab binds to / and /home. You can use the partitionmanager to format a second HD using Btrfs, giving it the label "data". Then, create a subdirectory on your main HD as /data. Then use "mount /dev/path-partuuid/uuid-of-data /data" to mount the drive and change ownership to yourname:yourname. Add an entry in /etc/fstab the mounts the drive at boot usiing its uuid. Save all your data to /data.

root@jerry-Aspire-V3-771:~# [B]mount /dev/disk/by-uuid/b3131abd-c58d-4f32-9270-41815b72b203 /backup[/B]
root@jerry-Aspire-V3-771:~#[B] mount /dev/disk/by-uuid/47a4794f-b168-4c61-9d0c-cc22f4880116 /mnt[/B]
root@jerry-Aspire-V3-771:~# [B]vdir /mnt[/B]
total 0
drwxr-xr-x 1 root root 262 Jun 11 14:26 @
drwxr-xr-x 1 root root  10 May  3 23:06 @home
drwxr-xr-x 1 root root 288 Jun 13 11:13 snapshots

root@jerry-Aspire-V3-771:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           581M  8.7M  572M   2% /run
/dev/sda1       699G  126G  573G  18% /
tmpfs           2.9G   40M  2.8G   2% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup
/dev/sda1       699G  126G  573G  18% /home
tmpfs           581M   32K  581M   1% /run/user/1000
[B]/dev/sdc1       699G  256G  441G  37% /backup[/B]
/dev/sda1       699G  126G  573G  18% /mnt

[FONT=courier new][B]root@jerry-Aspire-V3-771:~# btrfs su snapshot -r /backup /backup/@snapshot[/B][/FONT]
Create a readonly snapshot of '/backup' in '/backup/@snapshot

root@jerry-Aspire-V3-771:~# vdir /backup
total 9872
drwxr-xr-x 1 root  root      262 May 26 18:46 @_20180525
drwxr-xr-x 1 root  root      262 May 31 11:38 @_20180531
...
drwxr-xr-x 1 root  root       10 May 26 19:16 @home_20180525
drwxr-xr-x 1 root  root       10 May 31 12:10 @home_20180531
.....
drwxrwxr-x 1 jerry jerry     252 Apr 11  2017 SimpleScreenRecorder
[B]drwxr-xr-x 1 root  root      848 May 31 12:22 @snapshot[/B]
drwxrwxr-x 1 jerry jerry     606 Jan  8 21:53 WebPlotDigitizer-linux-x64
-rw-r----- 1 jerry jerry 9369618 Sep 29  2010 wireless_kismet.pdf

Now, I can mount an external, removable storage and use btrfs send & receive to send the /backup/@snapshot to that storage for safe keeping at another location. If I accidently delete or destroy a something on /backup I can browse @snapshot and copy it back out into /backup. When I am not using /backup I can umount it.

PS- Since /backup is not a <ROOT_FS> system it does not have a @ and @home subvolume and mounting the HD to /backup doesn't give access to a higher ID 5 level.

PS:PS -- My sdc1 HD is for holding archived snapshots of @ and @home from my system drive. As such I just send & receive the -r snapshots to that hd mounted as /backup. When I snapshotted /backup as /backup/@snapshot I violated my basic principle of never nesting backup snapshots like snapper does. IF sdc1 gave me mounting problems I could never access @snapshot to restore files on /backup.

Because I formated sdc1 as as a btrfs subvolume without a path designation when mounted as /backup that directory becomes ID=5. All my @YYYYMMDD and @homeYYYYMMDD snapshots are thus at ID=5 level. To give my files the same level of backup I created a subvolume @backup. I moved all the files (but none of the snapshots) into @backup. I created a /snapshots subdirectory (not subvolume) for storing snapshots of @backup. With that I can do either mount /dev/uuid /backup or /dev/uuid/@backup /backup

[TWPonKubuntu]

GG;

That answered my question, thanks. Looks like an external HD is in order for my systems in the future.

[gnomek]

Thank you for great explanation.

No. I haven't tried this but I suspect you would get two btrfs file systems with zero subvolumes. Thus removing another great benefit to using btrfs in the first place - subvolumes and their snapshots and backups. I'm going to test this and report back.

I would be grateful. Maybe there is a command for this to do after install? Something that would mount /home as @home in fstab or something.

Personally, I don't use EFI nor do I see any advantage to doing so. I don't boot Windows to bare metal (no one should IMO) and why should I add an extra layer of complication when GRUB works just fine? I'm currently booting 8 distros from a single BTRFS file system without using anything other than grub.

Are you saying that if I don't use windows I don't need EFI partition even on new type of motherboards? I still use an old motherboard with BIOS and Linux only.

What if I have, as you advise, external USB for backups. But it is 1TB and I don't want to allot entire space to backups but only a part of it. The rest must be NTFS to be easily accessed elsewhere. What would be reasonable space for brtfs partition for backups, considering I will backup standard desktop Kubuntu. I read that it is not easy to tell, but from time to time I could delete all snapshots, I mean wipe this partition and do a full system backup. For example every year I can do full backup on entire system and than snapshots with changes.

Now on ext4 I do a full system backup every half a year with Clonezilla and it takes about 5GB. Ext4 with Clonezilla vs. brtfs snapshots seems to be like a time (consuming solution) vs. disk space solution.

Did you know that Linux Mint introduced a tool for system snapshots:
https://linuxmint-installation-guide...timeshift.html

[GreyGeek]

GG;

That answered my question, thanks. Looks like an external HD is in order for my systems in the future.

Keep in mind that I could have created a /home/yourname/data as the data directory and mounted the @data to it instead of to /data.

That said, since /home is actually @home the question arises that if I snapshot @home don't I also snapshot /home/yourname/data?
The answer is no.

In the <ROOT_FS> resides @ and @home. In /etc/fstab @ is bound to "/" and @home is bound to /home, which appears to be a directory under / (root). Taking a snapshot of @ and browsing it with Dolphin when one looks into @/home it is empty! That is because a subvolume B under subvolume A is not snapshotted when subvolume A is snapshotted.

Therefore, it makes more sense to make subvolumes like @data at the <ROOT_FS> level when ever possible.
btrfs subvolume create /mnt @data
Then, in /etc/fstab, mount @data as /data or /home/yourname/data, but always take snapshots at the <ROOT_FS> level (ID=5) the same way snapshots of @ and @home are taken, but not necessarily at the same time because the data supposedly stored in @data has no relationship to @ or @home.

[TWPonKubuntu]

I'll admit to some confusion on this...

I'm sure there is a rational description of WHY btrfs is configured as it is... Not your explanation, GG, but what were the developers wanting to achieve in their design.

I do see advantages to btrfs, compared to others such as EXT4. It will take some time to wrap my mind around btrfs as well as I have done with EXT4. At least it is not a Redmond product...

[GreyGeek]

Migrating From EXT4 to BTRFS?

https://btrfs.wiki.kernel.org/index.php/Btrfs_design

[oshunluvr]

oshunluvr...

...Does btrfs offer a way to protect SOME data during a system upgrade? Yes, I get that I could restore from backup to get my data back intact, but that seems like a needless extra step.

No file system that I know of offers that sort of protection, a backup is your protection. Having said that, btrfs does not "re-format" the drive in the way that almost all other file systems do. It does not wipe the existing data, it only creates the locations for the metadata. You'll notice the difference if you create an EXT4 (or god forbid an NTFS) file system and time how long it takes. Then create a btrfs file system on the same partition. It will happen almost instantly. What this means is your data is intact until btrfs allocates that space for a file. One could if need be, totally recover from accidentally formatting the wrong partition to btrfs if no data was written to it before the attempt at recovery.

I'm not really sure how this would help anyone.

[oshunluvr]

I would be grateful. Maybe there is a command for this to do after install? Something that would mount /home as @home in fstab or something.

I'm running the test now, but I still don't understand why anyone would want two btrfs file systems on separate partitions on the same drive. It literally makes no sense at all. Frankly, if you insist on partitioning your drive so you can put /home on a separate partition I would not recommend using btrfs at all. There would be no point to it.

TEST COMPLETE: I installed to two btrfs file systems, setting one as / and the other as /home. The Kubuntu 18.04 installer created the same default subvolumes, putting @ on / and @home on /home. This does not change my above opinion.

Are you saying that if I don't use windows I don't need EFI partition even on new type of motherboards? I still use an old motherboard with BIOS and Linux only.

Yes, that is precisely what I am saying. One of my computers is configured with EFI because it's a ChromeBox that I hacked and EFI was baked in. None of my other computers use EFI unless they have Windows on them.

What if I have, as you advise, external USB for backups. But it is 1TB and I don't want to allot entire space to backups but only a part of it. The rest must be NTFS to be easily accessed elsewhere. What would be reasonable space for brtfs partition for backups, considering I will backup standard desktop Kubuntu. I read that it is not easy to tell, but from time to time I could delete all snapshots, I mean wipe this partition and do a full system backup. For example every year I can do full backup on entire system and than snapshots with changes.

Seems like you've assumed btrfs will require the entire drive. BTRFS is a file system like any other. It will use the space you allot to it so partition your drive accordingly. If you prefer, you can make subvolume backups to a file that can be transferred to any other file system. So you could leave your USB drive NTFS and just copy the backup file to it. The only disadvantage to doing this is you can't access the files inside the backup until you restore the file back to a btrfs file system.

Your backup size will equal your subvolume size. If you want only a single backup of each subvolume then you need the same amount of space as they require. If you want two backups, then double the space, etc. As far as the rest of what you're proposing, I don't understand the purpose or benefit of backing up a series snapshots unless you want to be able to roll back to a year ago. Seems unnecessary to me, but have at it. You could do the same thing by simply doing INCREMENTAL backups of your subvolumes and retaining them for as long as you like. I've never heard of anyone wiping their drive annually or on any other schedule. Sounds like a lot of work.

Look, no offence intended but if you're not going to take advantage of the ease of use and benefits that BTRFS has to offer then don't switch. I can understand why you would want to do things the way you always have and that's fine. But if you are going to continue to do all that, don't bother reformatting to btrfs. There's no reason to.

Now on ext4 I do a full system backup every half a year with Clonezilla and it takes about 5GB. Ext4 with Clonezilla vs. brtfs snapshots seems to be like a time (consuming solution) vs. disk space solution.

I'm not sure what you mean here. If you mean booting to Clonezilla and waiting all the time it takes to make a backup by copying the entire partition and all the unneeded space it uses to do so is a horrible waste of both space and time, then yes I agree. BTRFS snapshots take less than a second to make and a single command sends a snapshot to another btrfs file system in the background. Since you can continue to use your computer while making the backup, it effectively takes zero time. And since it copies only data and not file system overhead space, it's more efficient than many backup options. And you can access your backup at the file level if you wish without "restoring" anything. BTRFS is the clear winner when it comes to backups. Anything else is just doing it the hard way.

Did you know that Linux Mint introduced a tool for system snapshots:
https://linuxmint-installation-guide...timeshift.html

I am aware, but I prefer to make my own scripts an have my system do them automatically. The time required to simply manually take a snapshot is trivial so I do that whenever I need to. I don't see any benefit to the added workload of maintaining a mess (IMO only) created by someone else's program or spending the extra time to launch a GUI program to do something I can do myself in 2 seconds without a GUI. I'm sure it's a fine tool for those that want one.

[oshunluvr]

BTW for all of you considering using a USB drive to store BTRFS backups using the send|receive functionality; The btrfs developers I have chatted with don't recommend doing that. The concern is that USB interfaces are notoriously unreliable and a disconnect in the middle of a receive operation would corrupt the backup. They prefer using the send to a file operation, transferring the file to the USB drive, then restoring the file to a subvolume there. I believe the difference is that during a normal file copy operation, a disconnect of the target USB drive would generate a system notification and you would know to retry the file copy. Whereas a disconnect for a send|receive operation may not send any notification and you could be left unaware that your backup had failed. One could just use the send-to-a-file-and-copy-it method and not bother to restore it to a subvolume until it was needed and thus saving a step that might not be needed anyway. Alternately, I suppose you could use some other method to validate the send|receive operation had completed correctly like a simple "data space used" validation check.

I'm not trying to throw a wrench into anyone's plans but I wanted to make you all aware of this slight but potential pothole.

[oshunluvr]

oshunluvr;
Thanks for your reply to gnomek. It did, kind of, answer my own question... backup, backup, backup is the proper way to handle a system upgrade.

Does btrfs offer a way to protect SOME data during a system upgrade? Yes, I get that I could restore from backup to get my data back intact, but that seems like a needless extra step.

I re-read your post here and think I may have misunderstood your intent with my first reply.

If by "system upgrade" you mean upgrading packages on your system, the answer is a resounding YES. That is precisely what snapshots are. GG and I do this every time we upgrade our systems. "Snapshot before upgrade" is the mantra of BTRFS . A simple renaming of the subvolumes and a reboot "rolls" you back to your previous state. The reboot takes longer than the roll-back.

[TWPonKubuntu]

GreyGeek; Thank you for the link the design documents on btrfs. I've bookmarked it and am digging through it.

oshunluvr: Again, you offer excellent advice and it is appreciated.

[TWPonKubuntu]

I re-read your post here and think I may have misunderstood your intent with my first reply.

If by "system upgrade" you mean upgrading packages on your system, the answer is a resounding YES. That is precisely what snapshots are. GG and I do this every time we upgrade our systems. "Snapshot before upgrade" is the mantra of BTRFS . A simple renaming of the subvolumes and a reboot "rolls" you back to your previous state. The reboot takes longer than the roll-back.

I'm referring to a full kernel upgrade. As you know, with EXT4, I'm able to keep the data in my /home partition by not formatting it. I like this ability and am looking for something from btrfs which accomplishes the same result.

More simple "package" upgrades, as a general rule, don't try to modify my EXT4 /home partition so are not a problem.

Reading your previous posts, I think this (the desired result; keep my data intact) is available with btrfs. The steps in the process with btrfs ARE different, I get that too.

Thank you for your detailed replies, they DO help.

[oshunluvr]

Any "system upgrade" including kernel upgrades or even full release (do-release-upgrade) upgrades to a new version do not format or re-format anything. The only time reformatting would occur is during a new installation, not an upgrade.

However, with btrfs it is possible to do a new install without formatting at all. Just rename the subvolumes of the existing installation before doing the new installation, select your btrfs file system as the installation target, and do not format it. I do this every time I do a new install. I just looked and I have 9 distros installed to one btrfs filesystem.

[GreyGeek]

Migrating From EXT4 to BTRFS?

mmm... I've had more failures with USB HD's (CDW's Passport drives) than with any USB stick I've used. In fact, since I began using USB sticks for storage I've never had one fail on me during the middle of a write. That said, I just bought a 256 GB USB 2.0 stick for $25 from Amazon. I plan to format it with Btrfs using compression to make it appear to be about 500GB as an experiment. The stick may be Chinese junk Either way I'll soon know.

Oshunluver, your analogy of rafts in a swimming pool is perfect! A @ raft and a @home raft, each one capable of expanding to support more passengers until their combined size filled the pool. One doesn't have to know in advance how much space to assign to the @ raft or the @home raft to maximize the use of the pool's surface.

Now, assume, as gnomek wants to do, having TWO swimming pools, with a @ raft floating in one pool and a @home raft floating in the other pool. Unless both filled their respective pools surface completely one or both are wasting surface space and neither can benefit from the other's unused space. Also, unless one has a 3rd internal HD (like I do) they are forced to save their snapshots to the drive their @ or @home is also on. The only place to send&receive a snapshot would be to an external storage device. And, like you say, a snapshot is NOT a backup unless it is saved on an external storage device.

The best course of action would be to combine the pools (HD's) by first installing Btrfs on one as the root file system, creating both @ and @home subvolumes. Then, after booting into the system, opening a Konsole and using the btrfs device command to add the second HD to the first, without creating a RAID1. IOW, just doubling the pool size which both @ and @home can expand into as needed. That's the best way to use two HDs, but only if you need their combined size to act as one pool.

When I first added a second HD to my system I made them into a RAID1. After a while I realized that making snapshots and storing them externally gave me all the protection I needed, so I converted them into RAID0 combination that gave me 1.2Tb of usable space. Realizing that my @ and @home snapshots combined never exceeded 130GB I also realized that I was wasting HD space. So, I degraded sdb1 and removed it from my pool. Then I converted it to a standard Btrfs subvolume (no @ or @home subvolumes). Now, it and sdc1, formatted with Btrfs, are my two storage HDs.

I mount my sda1 to /mnt to make @ & @home snapshots, and I mount sdb1 to /backup and sdc1 to /media as receivers of sent snapshots. Just a reminder for gnomek and the others: I cannot can create a snapshot of @ on /mnt with a destination of /backup/snapshots/@YYYYMMDD because but once I am done sending and receiving previously made snapshots residing under /mnt/snapshots to /backup/snapshots I will be umounting /backup and /media. IOW, Btrfs snapshots cannot should not cross pool boundaries. Btrfs would not be able to make changes to /backup/snapshots/@YYYYMMDD if it isn't mounted and active, so no changes could trickle down to the snapshot, which would no longer reflect changes in my system. The Btrfs send & receive commands are for allowing the transfer of snapshots from one pool to another pool.