REVISITING - More BTRFS fun Multibooting to subvolumes - FIXING GRUB

[oshunluvr]

If you are adventurous enough to have followed my previous post here on how to do multiple installs to a single BTRFS file system, you might be running into trouble booting to all your installs. I did mention there to edit 40_custom in the original post, but several people have missed that part, so here's a re-visit to just that piece of the process in greater detail:

The issue:
os-prober - the tool that GRUB uses to detect installs - does not look inside subvolumes to detect other installs. I doubt it ever will. My untested belief is, if you mounted each install first, then run os-prober, it might find them. Even if it did, it would be useless to you because they grub.cfg would have the mounts in it's menu - which wouldn't exist on the next reboot.

This write-up assumes the following:
You have installed several version of *buntus or a few select other distros that install to btrfs using subvolumes, Manjaro for example.
You have followed my previous post's instructions to rename the subvolumes to unique names.
You have installed grub with each install.
When you reboot after each install, you only see the latest distro in your grub menu.

Rest assured, as long as you don't select "Format this partition" during installation AND you have previously renamed the subvolumes from @ and @home to something else, they will still be there after the second (and beyond) installs to the same btrfs volume.

So here's the Catch 22: If you allow grub to be installed with each new installation, you will only be able to boot to the new install each time. If you don't allow grub to be install each time, you will only be able to boot to the installation that was last installed with grub. Is that clear?

Example:
First you install Kubuntu LTS.
You rename @ to @Kubuntu and @home to @Kubuntu_home and do the necessary edits to grub.cfg and fstab.
You boot to Kubuntu and run update-grub - and all is good.

Second install WITH grub installation:
You install Lubuntu 17.04 to the same btrfs file system and you install grub to the same boot device.
Upon reboot - you can only select Lubuntu

Second install WITHOUT grub installation (launch the installer with the -b switch to install without boot loader):
You install Lubuntu the same btrfs file system and grub is not installed.
Upon reboot - you can only select Kubuntu and Lubuntu doesn't have grub within it at all.

Either way, you're stuck, but there are several solutions - This is Linux after all

Solution 1:
Install grub every time to the same boot device (i.e. you're on a single disk system):

Install your primary OS (I'm assuming Kubuntu for this example), then install all the others you want - all of them with grub.

This ensures all the installations have all the necessary grub files to boot.

Follow the same renaming routine after each install - renaming @ and @home to something unique and editing fstab and grub.cfg as required (see the linked older post if you don't understand this).

After each install, reboot to the new install and edit /etc/grub.d/40_custom and add this to it:

menuentry 'Kubuntu' {
    insmod part_gpt 
    insmod btrfs
    search --no-floppy --fs-uuid --set=root <YOUR BTRFS UUID HERE>
    configfile /@Kubuntu/boot/grub/grub.cfg
}

Now make 40_custom executable and run update-grub.

This allows grub to switch another menu. Now when you reboot, Kubuntu is also available in the menu.
When you re-boot to Kubuntu, edit 40_custom in Kubuntu again, but now add each new installation to it.

For example, you have Kubuntu, Lubuntu and Manjaro all installed to the same btrfs filesystem using unique subvolumes, but you prefer to boot to Kubuntu on a daily basis.

Your 40_custom file under Kubuntu might look like:

menuentry 'Lubuntu 17.04' {
   insmod part_gpt
   insmod btrfs
   search --no-floppy --fs-uuid --set=root <YOUR BTRFS UUID HERE>
   configfile /@Lubuntu/boot/grub/grub.cfg
}
menuentry 'Manjaro' {
   insmod part_gpt
   insmod btrfs
   search --no-floppy --fs-uuid --set=root <YOUR BTRFS UUID HERE>
   configfile /@Manjaro/boot/grub/grub.cfg
}

As you install other distros, simply edit 40_custom under it and add your Kubuntu install, reboot to Kubuntu and add the new distro to 40_custom there as described above and you'll be able to boot to all of your installs.

Solution 2:
Install grub every time to a different boot device (you have more than one boot-able disk):

This is basically the same technique as above except when installing your primary boot distro, you install grub to your primary boot drive (usually sda).

When you install all the other distros, install grub to your second drive (usually sdb). This ensures all the necessary grub files are there with each distro but they're not "taking over" as the boot distro.

With this method, you need only edit 40_custom within your bootable installation (Kubuntu in the above example) and you needn't bother with 40_custom in any of the other installs.

GRUB NOTE: When you use grub this way - multiple menus - here are a couple tips:

• In each installation, make the grub menu visible (edit /etc/default/grub or /etc/grub.d/00_header depending on the distro).
• You can return to the previous menu from a secondary one with the ESC key.

CAVEAT: In both above solutions you are relying on a single distro to boot all the others. This is fine until you decide you no longer want the distro you're booting to. Don't forget, if you delete the boot-able install, you delete the grub files needed to boot along with it.

It's a simple matter to make any other of the distros the main boot-able install. Just copy your full 40_custom file over the the installation of choice, boot to it, run update-grub and then run grub-install. Verify you boot to the chosen distro by default, then delete the distro you no longer need.

Solution 3:
Make a bootable grub partition (another multi-disk solution):

This is my method but it's not the easiest to initially set up. I have more than one drive on my system, so I created and mounted a 128MB EXT2 partition on one of them (actually two of them so I have a boot-able backup drive). I copied /boot/grub from one install to this partition. Then ran grub-install with the --root-directory option pointing at the grub partition. Whenever I install a distro, I tell it to boot from sdc, but my boot drive is sda (sdb as backup) so my grub is never messed with.

Now regardless of deleting installs, I can always boot. I manually edit the grub.cfg on this bootable partition and add and subtract installs as I do them - using the same menuentry as you would in 40_custom. Here's the tail (the part after the header info) of my grub.cfg from that partition:

[FONT=monospace][COLOR=#000000]### BEGIN Custom menu ### [/COLOR]
menuentry 'KDE Neon' { 
   insmod part_gpt 
   insmod btrfs 
   search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8 
   configfile /@KDEneon/boot/grub/grub.cfg 
} 
menuentry 'KDE Neon - new' { 
   insmod part_gpt 
   insmod btrfs 
   search --no-floppy --fs-uuid --set=root fcc94981-05e4-4846-92e7-87bf9885178d 
   configfile /@KDEneon_new/boot/grub/grub.cfg 
} 
menuentry 'Kubuntu 16.04' { 
   insmod part_gpt 
   insmod btrfs 
   search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8 
   configfile /@Kubuntu_16_04/boot/grub/grub.cfg 
} 
menuentry 'Ubuntu 16.04' { 
   insmod part_gpt 
   insmod btrfs 
   search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8 
   configfile /@Ubuntu_16_04/boot/grub/grub.cfg 
} 
menuentry 'Manjaro free' { 
   insmod part_gpt 
   insmod btrfs 
   search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8 
   configfile /@Manjaro_free/boot/grub/grub.cfg 
} 
menuentry 'Manjaro nonfree' { 
   insmod part_gpt 
   insmod btrfs 
   search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8 
   configfile /@Manjaro_nonfree/boot/grub/grub.cfg 
} 
menuentry 'Memory test (memtest86+)' { 
   insmod part_gpt 
   insmod ext2 
   search --no-floppy --fs-uuid --set=root 4bd4447b-48df-43d8-9781-d444d68ce462 
   knetbsd /boot/memtest86+.elf 
} 
menuentry 'Memory test (memtest86+, serial console 115200)' { 
   insmod part_gpt 
   insmod ext2 
   search --no-floppy --fs-uuid --set=root 4bd4447b-48df-43d8-9781-d444d68ce462 
   linux16 /boot/memtest86+.bin console=ttyS0,115200n8 
} 
menuentry 'KDEneon ISO' { 
   set isofile="/neon-useredition-20170608-2351-amd64.iso" 
   loopback loop (hd0,2)$isofile 
   linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$isofile noprompt noeject 
   initrd (loop)/casper/initrd.lz 
} 
### END Custom menu###

[/FONT]

You can see by the UUID that all my installs reside on the same UUID. I also moved MEMTEST to the grub partition so it's not installed 10 times and the last stanza allowed me to boot from an ISO without burning it to a USB or DVD first.

[#]btrfs[/#] [#]grub[/#] [#]multiboot[/#]

[steve7233]

I can't boot Kubuntu. It gose in to rescue mode. I created a new subvolume for @ and @ home. Fstab.cgfg and grub.cfg are edited. I checked with Kubuntu live DVD. I deleted @ and @home. Before I rebooted I listed the subvo!umes @ and @home were deleted and in their place was the word deleted.

[steve7233]

It won't boot now! Goes to emergency mode. I did a lot of google searches. Nothing worked. I read Ubuntu stuff on BTRFS multiboot, reloading and reinstalling GRUB2 to a BTRFS system. I tried boot-repair, but it had an install problem. I have a snapshots foldure but if I use those it might not boot if it is the same problem. I will goto the store and when I get back I will install Kubuntu 17.04. I will check this thread first incase some one thinks they can help fix Grub2. I tried purgeing and reinstalling Grub2 using chroot but get something like 'failed to get conical path to device.'.

[vinnywright]

I can't boot Kubuntu. It gose in to rescue mode. I created a new subvolume for @ and @ home.

what exactly dose this mean?

befor you could not boot ? ,,,,,, after you could not boot ? ,,,,, is this what you did to "rename the subvolumes" ? ,,,,,,why did you create new subvolumes ?

Fstab.cgfg and grub.cfg are edited. I checked with Kubuntu live DVD. I deleted @ and @home. Before I rebooted I listed the subvo!umes @ and @home were deleted and in their place was the word deleted.

I hope you did not think creating new subvolumes was the same as renaming the subvolumes the got installed as @ and @home ,,,,,,IF you deleted those 2 ,,,,,you are hosed .

+ you do not just delete a subvolume ,,,,you use the btrfs command "btrfs subvolume delete -c <path to the subvolume>"

VINNY

[steve7233]

what exactly dose this mean?

befor you could not boot ? ,,,,,, after you could not boot ? ,,,,, is this what you did to "rename the subvolumes" ? ,,,,,,why did you create new subvolumes ?

I hope you did not think creating new subvolumes was the same as renaming the subvolumes the got installed as @ and @home ,,,,,,IF you deleted those 2 ,,,,,you are hosed .

+ you do not just delete a subvolume ,,,,you use the btrfs command "btrfs subvolume delete -c <path to the subvolume>"

VINNY

Yes, btrfs subvolume delete. Was I supposed to run some other subvolume command after the btrfs delete? Maybe something to move meta data or something?

[vinnywright]

Yes, btrfs subvolume delete. Was I supposed to run some other subvolume command after the btrfs delete? Maybe something to move meta data or something?

you did not answer the rest of the questions ,,,,that are more important ATM

VINNY

[steve7233]

what exactly dose this mean?

befor you could not boot ? ,,,,,, after you could not boot ? ,,,,, is this what you did to "rename the subvolumes" ? ,,,,,,why did you create new subvolumes ?

I hope you did not think creating new subvolumes was the same as renaming the subvolumes the got installed as @ and @home ,,,,,,IF you deleted those 2 ,,,,,you are hosed .

+ you do not just delete a subvolume ,,,,you use the btrfs command "btrfs subvolume delete -c <path to the subvolume>"

VINNY

I read oshunluvr's several threads on mulltibooting and have been trying to figure all that out. Basicly he says to rename @ and @home then edit fstab and grub.cfg. I think I goofed in my method of renaming the subvolumes. I thought If I kept it simple I would avoid hosing my system. Unfortunatly it looks like I made a hosing mistake anyway. I thought I could clone @ and @home. I created @Kubuntu1404 and @homeKubuntu1404 then btrfs deleted the @ and @home. I rebooted after the file edits. All seemed fine. listing the sub volumes showed the orginal volumes were now marked as deleted and checking the mounts showed that @Kubuntu1404 and @homeKubuntu1404 were mounted and @ and @home were not.

Based on that, I thought I understood and then deleted the deleted ones. I rebooted and ... emergency mode I don't mind -to much-, having to reinstall as I have learned that occasionaly all Linux true Linux users hose themselves every now and then. . I just want to know what I did wrong and how to do it the correct way. Now I will go run the 16.04 install so I can try again hopfully with out borking my system this time. I look at it this way, So I made a silly mistake. Now I can help others to avoid the mistake and look like I know what I am doing to a Linux newbie. I just hope I have internet after the install.

I am learning how to fix a no internet machine with all these reinstalls!

[GreyGeek]

It's called "paying your dues"!
Keep up the good work ... that's really how one learns.

[vinnywright]

LOL yup just what I thought .

he meant ,,,to literally just rename them ,,,,,rite click in dolphin>rename>and type in a new name for @ and @home like hear

vinny@vinny-Bonobo-Extreme:/media/vinny/ff5d66d4-35b6-4c9c-a64e-8dfbe2aa1e31$ ls
@  @17.04  @17.04snap  @home  @home17.04  @home17.04snap  ubiquity-apt-clone  var

the @17.04 and @home17.04 where named @ and @home ,when installed ,,,then I just renamed them as you see ,,,the CURENT @ and @home are a neon-LTS install ,,,,,,the @home17.04snap is a snapshot of @home17.04

VINNY

[steve7233]

Can VB handle BTRFS? If so then I can learn it via a VM then delete the VM and do it inm the host. Now why didn't I think of this sooner? Now that I think of it. Didn't some one suggest that in another thread?

[GreyGeek]

Can VB handle BTRFS? If so then I can learn it via a VM then delete the VM and do it inm the host. Now why didn't I think of this sooner? Now that I think of it. Didn't some one suggest that in another thread?

COW will give VB fits unless you turn it off for the folder in which your plan on placing your virtual HDs. Or, my approach, was to create static virtual HDs preset to a fixed size, usually 40-60Gb, depending on what I planned to do with the virtual OS. That way the dynamic change of HD size is avoided and BTRFS is happy.

[steve7233]

COW will give VB fits unless you turn it off for the folder in which your plan on placing your virtual HDs. Or, my approach, was to create static virtual HDs preset to a fixed size, usually 40-60Gb, depending on what I planned to do with the virtual OS. That way the dynamic change of HD size is avoided and BTRFS is happy.

I always use fixed disks anyway since they work better in VB, plus you don't risk running out of host disk space.

[steve7233]

I have 17.04 installed now. Before I can try multi booting I first need to figure out what I did wrong and what I should have done.

[oshunluvr]

Yeah, I guess I need to add a note there or something. Rename simply means "RENAME", as in the Linux command "mv <oldname> <newname>". I didn't anticipate this causing confusion to anyone and you're not the first who thought is was something special other than just renaming. I should also add a note about reading the btrfs command list - there's no rename command specially for btrfs. Here's my subvolumes on my main btrfs file system:

drwxr-xr-x 1 root root 286 Aug 23 11:32 @KDEneon
drwxr-xr-x 1 root root 286 Aug 23 11:32 @KDEneon_170925-145522
drwxr-xr-x 1 root root 288 Jun  7 09:36 @KDEneon_backup1
drwxr-xr-x 1 root root 288 Jun 23 15:29 @KDEneon_backup2
drwxr-xr-x 1 root root 302 Jun 30 07:58 @KDEneon_backup3
drwxr-xr-x 1 root root  48 Sep  6 16:46 @KDEneon_home
drwxr-xr-x 1 root root  48 Sep  6 16:46 @KDEneon_home_170925-145537
drwxr-xr-x 1 root root  38 Apr  1 09:30 @KDEneon_home_backup1
drwxr-xr-x 1 root root  38 Apr  1 09:30 @KDEneon_home_backup2
drwxr-xr-x 1 root root  38 Apr  1 09:30 @KDEneon_home_backup3
drwxr-xr-x 1 root root  48 Sep  6 16:46 @KDEneon_home_save1
drwxr-xr-x 1 root root 196 Jun 13 16:49 @KDEneon_new
drwxr-xr-x 1 root root  12 Jun 13 16:41 @KDEneon_new_home
drwxr-xr-x 1 root root 286 Aug 23 11:32 @KDEneon_save1
drwxr-xr-x 1 root root 286 Sep 22 13:49 @KDEneon_save2
drwxr-xr-x 1 root root 262 Dec 13  2016 @Kubuntu_16_04
drwxr-xr-x 1 root root  32 Dec 22  2016 @Kubuntu_16_04_home
drwxr-xr-x 1 root root 282 Jun 27 09:31 @Manjaro_free
drwxr-xr-x 1 root root  12 Jun 11 08:11 @Manjaro_free_home
drwxr-xr-x 1 root root 216 Jun 10 11:55 @Manjaro_nonfree
drwxr-xr-x 1 root root  12 Jun 10 11:49 @Manjaro_nonfree_home
drwxr-xr-x 1 root root 234 Jun 13 17:21 @Ubuntu_16_04
drwxr-xr-x 1 root root  12 Jun 13 16:14 @Ubuntu_16_04_home

Six of these are subvolumes I can boot to, the rest are homes or snapshots.

As far as "cloning" a subvolume, you're mixing tasks. There's really no need to ever "clone" (as in make-a-full-copy-of) a subvolume because taking a snapshot does it better and faster and without error. To expand a bit:

You can take a "snapshot" of a subvolume which is effectively a full copy of a subvolume. The only rules are the snapshot must be on the same btrfs file system. So if your subvolume @ is on a file system on sda2, the the snapshot of it must also be on sda2. You can copy a subvolume snapshot from one file system to another (a different partition or drive) IF the snapshot is read-only - Then you can use the btrfs send|receive commands to transmit a read-only snapshot from one btrfs file system to another. In my list above, the subvolumes ending in ".backup." are read-only backups that have been transmitted to another file system (my backup btrfs drive) without yet deleting them.

So, steve7233, back to your issue: Assuming you now have a working copy of 17.04 installed to a btrfs file system using the default @ and @home subvolumes. For the list below, I will use /dev/sda2 as the partition that your btrfs file system resides on and assume you will use @Kubuntu1704 as your new subvolume name. Obviously, use the correct partition device name and whatever new subvolume names you want when you actually do this:

1) Mount the parent btrfs file system somewhere.

sudo mkdir /mnt/parent
sudo mount /dev/sda2 /mnt/parent
cd /mnt/parent

You are now in the parent file system. To verify all is well, a simple directory listing will work:

ls

the result should be

@ @home

2) Snapshot (regular, not read-only) both subvolumes using your desired new subvolume names.

sudo btrfs subvolume snapshot @ @Kubuntu1704
sudo btrfs subvolume snapshot @home @Kubuntu1704_home

Now a directory listing (ls command) results in:

@ @home @Kubuntu1704 @Kubuntu1704_home

You now have two copies of your install and your home. At this point, note a few things: Your snapshots have not taken any additional space on your drive. The snapshot will not change as you make changes to the source subvolume and the source subvolume will not change if you change something in the snapshot. A snapshot is an exact subvolume copy frozen at the time it was made. So if you delete a file in @home it will still be in @Kubuntu1704_home and vice-versa. As you make changes to either the source or the snapshot subvolume, the snapshot subvolume will begin to consume space on the drive.

3) Enter the new root subvolume and edit /etc/fstab and change the mount options from @ and @home to your new names.

kate @Kubuntu1704/etc/fstab

Find the two lines that are similar to:

UUID=8f0c1661-4e84-4512-b875-23bcfd5be1d8 / btrfs rw...subvol=@ 0 1
UUID=8f0c1661-4e84-4512-b875-23bcfd5be1d8 /home btrfs rw...subvol=@home 0 2


Edit the subvol=@ to subvol=@Kubuntu1704 and @home to @Kubuntu1704_home and save.

MAKE ABSOLUTELY SURE there are no typos in these edits and have a bootable device handy as a backup.

4) Enter /boot/grub/grub.cfg and manually edit the boot menuentry to reflect the new root subvolume name.

sudo chmod u+w /boot/grub/grub.cfg
kate /boot/grub/grub.cfg

In this file, find this line (usually about line 133) :

### BEGIN /etc/grub.d/10_linux ###

Then look about 25 lines lower for a line beginning with:

linux /@/boot/vmlinuz...

and ending with:

....rootflags=subvol=@

Change@ in both places to @Kubuntu1704 and also one line lower you'll see /@/boot/initrd... Change that to @Kubuntu1704 also, and save the file.

If you've done everything correctly, at this point you should be able to reboot and you'll be using the snapshot subvolume instead of the previous @ and @home. To verify this, simply list your mounts and look at what mount options are in place:

mount |grep btrfs

The output should be similar to:

/dev/sda2 on / type btrfs (rw...subvol=/@Kubuntu1704)
/dev/sda2 on /home type btrfs (rw...subvol=/@Kubuntu1704_home

I shortened the output to just show the last bit where the subvolume is indicated.

Assuming all is as expected, run update-grub so all your grub menuentries are pointing to the new subvolumes:

sudo update-grub

At this point, you can keep @ and @home or install over them or delete them and @Kubuntu1704 and @Kubuntu1704_home will not be touched.

[steve7233]

3) Enter the new root subvolume and edit /etc/fstab and change the mount options from @ and @home to your new names.

[/FONT]

I would have had it if I had realized I was editing the wrong sub volume! Then I deleted it, when I deleted @. ... OK you can stop laughing now. I used to have a card that said "to error is human, but to really screw up is to use a computer." You would think with all my years of using computers I would notice that. It's always the simple things that we forget and then have to laugh about later when we realize. Thanks for helping me clear it up.