Never saw this before

[GreyGeek]

A month ago I installed KDE Neon User Edition LTS onto a Dell Vostro 1500. It installed without problems, replacing an old XP installation, and worked without problems. The owner, a father, gave the laptop to his daughter.


She had created a second user account with KUser, and then deleted it when it couldn't do root maintenance. She then attempted to rename the primary account using KUser and it failed, so she renamed it back to the original name. (She's tech support for a corporation that uses Windows and Window-think created problems for her.) Things appeared to work normally until she rebooted, which gave the black screen at login time. That stumped her.

I rebooted into the recovery screen and logged in as root and remounted the file system: "mount -o remount, rw /" in order to use usermod, chmod and chown to attempt to repair things. It failed to mount and I was kicked back into a non-functioning recovery terminal. I rebooted. That was the first time I tried the remount option on a btrfs system. Perhaps it doesn't work on Btrfs file systems?


At the black screen again I hit Ctrl+Alt+F2 and logged into a terminal using the original name and password. Then I "sudo -i" to get root and mounted /dev/sda1 to /mnt

I mv'd @ to @old and @home to @homeold. Then I used
"btrfs subvol snapshot /mnt/snapshots/@20171205 /mnt/@" followed by "sync"
and
"btrfs subvol snapshot /mnt/snapshots/@home20171205 /mnt/@home" followed by "sync"

Then I deleted @old
"btrfs subvol delete -c /mnt/@old"
and then
"btrfs subvol delete -c /mnt/@homeold"

The second delete command presented "file not found". A repeat with the up arrow gave the same.

"vdir /mnt" gave the same msg.
So did "vdir /"
So did "cd /"

Only commands that were part of bash itself would work. Commands residing on the file system would not respond. The file system had disappeared. I have never seen this kind of behavior before in the 19 years I've used Linux. Nothing was shown in the logs. Systemd's last attempt to write to its log failed with a "file not found" error.

There was nothing to do but hit the power button. The roll back worked. The system, as it was on the day I first installed it (12/5/17), came back to life and worked normally. I deleted @homeold and proceeded to test the system. Everything worked as it should. KPartitionManager's Smart Report gave the HD a clean bill of health. The system had passed a 12 hour RAM check when I first installed Neon, so I did not repeat it. About 400+ apps came down the pipe and all upgraded normally. Dmesg after a reboot showed nothing in the log out of the norm.

The user had installed the Chromium browser so I reinstalled it. Then I made snapshot backups of @ and @home in case things went south again. The system has been up and running for three hours without any problems.

I still need to change the dad's name to the daughter's name.

[kubicle]

"btrfs subvol delete -c /mnt/@old"

Were you still using that subvolume as your root file system when you deleted it (you didn't chroot away from it...or reboot)?

That could explain why external commands on the file system could not be found (bash would still work as it was loaded in ram).

[oshunluvr]

That is indeed odd. Seems like there were two failures, the user renaming and the difficulty with the mounting.

Perhaps it doesn't work on Btrfs file systems?

As a test, I booted a KDEneon vbox machine into recovery mode and had no problem remounting the btrfs root folder read-write using the exact same command in your OP.

Curiously, I was unable to test renaming the user using kuser because - at least here - that functionality does not exist in kuser or I couldn't find it. I can rename the "full name" but not the username. Any chance you're not getting the whole story?

To change a username, AFAIK you have to use the usermod and groupmod commands:

sudo usermod -l <NEW_USER_NAME> -m -d /home/<NEW_USER_NAME> <OLD_USER_NAME> && sudo groupmod -n <NEW_USER_NAME> <OLD_USER_NAME>

I tested this also and it worked on an added second user. However, since only the primary user has the ability to use sudo, the command fails because the user is in use at the time of the command. I definitely think you didn't get the whole story about what she tried to do. I suppose one could do the above from the recovery prompt after remounting read-write, but it seems unlikely she knew to do that.

At least the roll-back worked. BTRFS saved the day once again!

[oshunluvr]

I suspect deleting the subvolume in use at the time was the source of the problems with the empty file system. I haven't tested that process you went through, but I suspect if you changed the order of events, you would have been OK. Like restore snapshot, rename subvolumes, reboot, THEN delete the old subvolumes.

[GreyGeek]

I normally mount /dev/sda1 to / mnt but don’t cd to it. I use the full path in the commands. But, in the case of the failure I DID cd to /mnt and used relative paths. Since I was on /mnt I didn’t put /mnt/@home, I used @home. IIRC

[kubicle]

I normally mount /dev/sda1 to / mnt but don’t cd to it. I use the full path in the commands. But, in the case of the failure I DID cd to /mnt and used relative paths. Since I was on /mnt I didn’t put /mnt/@home, I used @home. IIRC

I meant that the commands you ran seemed to delete the filesystem you were using as the root filesystem (at boot one filesystem is set as the root.)

When you moved (renamed) @ to @old, that subvolume (now just renamed @old) was still your root file system (/). And if you subsequently deleted that, then nothing under / existed anymore (because the filesystem mounted there was removed), therefore the system can't find commands that are commonly found in $PATH. If you wish to change the root filesystem on a running machine, you need to use the 'chroot' command ('cd' doesn't do it, it only changes the current working directory)...of course, a reboot should have changed the root file system as well, if you had done it before deleting @old...since you already restored a previous snapshot as the new @.

[GreyGeek]

Renaming @ to @old and @homeold doesn’t kill ol the system. That’s why @old and @homeold are deleted after creating a rw snapshot of @ from @YYYYMMDD and a rw snapshot of @home from @homeYYYYMMDD before deleting the @old and @homeold.

What I may have done was inadvertently deleted @ instead of @old Regardless, the beauty of Btrfs is that from a LiveUSB I was able to recreate @ and delete @homeold and reboot my system without losing anything , and it took less than five minutes. In my use case of Btrfs I haven’t found any down sides.

I’ll mark the incident off as an age related brain fart. Btrfs is an fsfof.

[kubicle]

Renaming @ to @old and @homeold doesn’t kill ol the system.

Yes it does, it you delete the renamed filesystem you are still using as the root filesystem (without rebooting or chrooting from it).

You can try it again if you don't believe me

And yes, I'm only deducing this based on the commands you posted in your first post. If you performed steps that were not in your first post or made some other mistakes (like deleting a wrong subvolume), this might not be an accurate description of what went wrong.

This should be fine:
1. move @ to @old
2. cp @somesnapshot to @
3. reboot (or chroot to the new @)
4. delete @old

This should fail:
1. move @ to @old
2. cp @somesnapshot to @
3. delete @old
4. Now system won't function as the file system used as the root file system (@old) is gone and there is nothing in / (or /usr/bin etc.), bash (and builtins) will stiil work as long as it is loaded in memory.
5. After a reboot, the new @ (the copied @somesnapshot) is used (mounted) as the root file system, so the system will work again.

[GreyGeek]

Here is the command sequence I've been using for two years:

Rolling back:

1) Open a Konsole and issue
"sudo -i"

2) Mount sda1 on /mnt
"mount /dev/sda1 /mnt"

3) Move @ to @old
"mv /mnt/@ /mnt/@old", followed by "sync"

4) Move @home to @homeold
"mv /mnt/@home /mnt/@homeold" followed by "sync"

5) Copy /mnt/snapshot/@YYYYMMDD to /mnt/@
"btrfs subvol snapshot /mnt/snapshots/@YYYYMMDD /mnt/@" followed by "sync"

6)Copy /mnt/snapshots/@homeYYYYMMDD to /mnt/@home
"btrfs subvol snapshot /mnt/snapshots/@homeYYYYMMDD /mnt/@home"

7) Delete @old and @homeold
"btrfs subvol delete -c /mnt/@old"
"btrfs subvol delete -c /mnt/@homeold"

8) Exit root and Konsole
"exit"
"exit

9) Reboot

The problem occurred at step #7 when I deleted @old. When I attempted to delete @homeold the btrfs command was not found. Everything: @, @home, / and /home were missing. That's when I used a LiveUSB to do steps 5-9 over.

I've used this procedure several times since the problem occurred without experiencing any repeat of that problem.

[kubicle]

You can use this command to check the currently mounted root filesystem:

mount | grep '/ '

Before you start it should report something like:
/dev/sda1 on / type btrfs (rw,relatime,ssd,space_cache,subvolid=XXX,subvol=/@)
(This will show that subvolume @ is mounted as the root filesystem "/")

After you move @ to @old, it should report something like:
/dev/sda1 on / type btrfs (rw,relatime,ssd,space_cache,subvolid=XXX,subvol=/@old)
(The same filesystem is still mounted as the root filesystem, now only renamed to "@old", so if you delete it, you are destroying your currently mounted root filesystem "/"...and that pretty much guarantees the running system becomes inoperable.)

Of course the part that I haven't tested is whether restoring the @snapshot to @ will actually remount the new @ as the root file system, but I'd be really surprised if it did, doing that on it's known would probably be causing more problems than it would ever solve, but perhaps you can check that the next time you go about something like that?

Note that this is not a problem if you run the process from a liveusb (or a separate installation), since in that case the mounted root filesystem resides on the usb drive (or a different subvolume in case of a separate installation) and not on the file system(s) you are working on (/dev/sda1 subvolume @).

[GreyGeek]

The “before” is just what is in fstab, which uses UUID’s.
Your “after” proposition is interesting. I’m going to test it.
Both @ and @old have the same UUID and fstab isn’t changed.
/ isn’t umounted when /dev/sda1 is mounted to /mnt, and no “busy” or similar msg interrupts the mounting process.

[kubicle]

The “before” is just what is in fstab, which uses UUID’s.

Yes, /etc/fstab is the configuration file that tells the system what should be mounted where, so the mount command output will report the same options listed in fstab (if you haven't made any mount or file system changes since boot). Note that fstab doesn't tell what is currently mounted, it just defines what the system will mount at boot (any runtime changes to mounts are not reflected in fstab...the 'mount' command without options will list all active mounts)

Your “after” proposition is interesting. I’m going to test it.

Feel free to test it, but I did test the subvolume move myself before I posted, and that is what happens: subvolume move does not change active root file system mount. In other words, the same (moved) filesystem is still mounted as the root filesystem.

Both @ and @old have the same UUID and fstab isn’t changed.

See above, fstab doesn't change as it does not list active mounts, only what the system should mount at boot.

/ isn’t umounted when /dev/sda1 is mounted to /mnt, and no “busy” or similar msg interrupts the mounting process.

Yes, you effectively have the same filesystem (/dev/sda1 subvolume @) mounted at two mount points, "7" and "/mnt/@" and any changes you do on /mnt/@ will directly affect /@ as well as the target is the same subvolume, like moving (renaming) @ to @old or deleting @old. Again this doesn't happen if run from liveUSB, for example, because /dev/sda1 subvolume @ is not mounted at / in that scenario.

[GreyGeek]

You nailed it, Kubuicle!

root@jerry-Aspire-V3-771:~# mount | grep '/ '
/dev/sda1 on / type btrfs (rw,relatime,space_cache,subvolid=1083,subvol=/@)
root@jerry-Aspire-V3-771:~# mount /dev/sda1 /mnt
root@jerry-Aspire-V3-771:~# mv /mnt/@ /mnt/@old
root@jerry-Aspire-V3-771:~# mount | grep '/ '
/dev/sda1 on / type btrfs (rw,relatime,space_cache,subvolid=1083,subvol=/@old)
root@jerry-Aspire-V3-771:~# mv /mnt/@old /mnt/@
root@jerry-Aspire-V3-771:~# mount | grep '/ '
/dev/sda1 on / type btrfs (rw,relatime,space_cache,subvolid=1083,subvol=/@)
root@jerry-Aspire-V3-771:~#

Doing a mv @ to @old does make @old the current OS.
When I create a rw @ from /mnt/snapshot/@YYYYMMDD the @old is still the current OS.

However, I can still do step #7, in that order, and not lose my current OS when I delete @old, because I can still use btrfs subvol delete to delete @homeold.


I must have mistyped the cmd and didn't realize it. Thankfully, btrfs allows for easy recovery for klutzes like me.