[SOLVED!!] - Katastrophic Koala upgrade - system completely borked...

[bag]

Right then folks, I need some help here... had a problem during the upgrade and now my system seems completely borked...

Need to explain a little detail.

I had a fully working 9.04 Kununtu install. Several days ago (a week or so?) I updated it to a 4.3 system using backports - everything was OK, although I lost all my settings etc... Anyway, I finally got it straight, fully updated and stable. No problems. And it looked very nice too - was really looking forward to getting into Karmic with the same eye-candy.

Then I tried to update it to 9.10.

The downloads went fine and the system started to install. I know from the earlier warning at the start of the whole process, that it could be canceled during the downloads but not once the install proper actually started.

Anyway, the install started and then at some point kdesudo crashed on me and threw up an error message. I don't have access to the error message because although I managed to save it as a text file, I couldn't get it off my system. I couldn't mount a usb stick to copy it to (all USB mounts had gone down), I couldn't print it out (CUPS was not responding), and I couldn't send it out by e-mail because I couldn't get either Thunderbird or Firefox to start up. KDE was hanging on and off, plasma was not responding although I could still do a desktop cube spin!!!

Anyway, I couldn't open up a terminal to try and change anything - so I took a massive gamble. Given that my system was up but essentially useless, and I couldn't open a terminal, I rebooted. Plasma unhung long enough for me to press the relevant buttons for a reboot.

The plan was to get it to some sort of command line, and then try again. Gotta be worth a try, right?

Upon rebooting it hung - didn't get to a proper command line. So I'm a little bit stuck.

At the moment, it's currently showing error messages. I'll type in longhand the error that's currently on the screen - please see what you can make of it to give me some guidance.

One or more of the mounts listed in /etc/fstab cannot yet be mounted:
(ESC for recovery shell)
/: waiting for /dev/disk/bu-uuid/blad-di-blah-di-blah1
/tmp: waiting for /dev/disk/bu-uuid/blad-di-blah-di-blah2
/boot: waiting for /dev/disk/bu-uuid/blad-di-blah-di-blah3
/distros: waiting for /dev/disk/bu-uuid/blad-di-blah-di-blah4
/home: waiting for /dev/disk/bu-uuid/blad-di-blah-di-blah5
/restore: waiting for /dev/disk/bu-uuid/blad-di-blah-di-blah6
swap: waiting for /dev/disk/bu-uuid/blad-di-blah-di-blah7
/common: waiting for /dev/md1

mountall: Cancelled
init: mountall main process (879) terminated with status 1
General error mounting filesystems.
A maintenance shell will now be started.
CONTROL-D will terminate this shell and re-try.
root@(none):~#

Pretty bad, huh!! Although, I'm not panicking just yet... I know that all my data is still there but still, it's starting to get a little worrying.

I have not yet tried to CONTROL-D. I am not sure what is the best option here. Should I Ctrl^D out and see if it mounts this time? Is there anything that I should know before I do?

I am pretty sure that if I could get the mounts up and running, then get to a command line etc, then I could rescue the system. I really don't want to go through the whole reinstall - I just wanted to do a simple upgrade this time and reinstall on the next release (LTS?). As a fallback plan, I'm currently downloading a copy of Kubuntu 9.10 CD on the laptop so that I could install from scratch again if I had to... I have a backup of my /home folder anyway, so as a last resort, it can be done - but I'd rather not if it can be rescued...

I need some help here - from someone who knows a lot more about this than I do. How can I get it to a safe command line so that I can try again, and when I do, what are the command line actions I need to take to try it all again.

Many thanks in advance for any help you can give me.

Cheers,

Bag.

[jglen490]

Re: Katastrophic Koala upgrade - system completely borked...

It appears that you are at some sort of a prompt. Have you tried any commands?

[casperboy77]

Re: Katastrophic Koala upgrade - system completely borked...

I'm having same problem. How do I fix this?

[bag]

Re: Katastrophic Koala upgrade - system completely borked...

jglen490,

thanks for the post. I've not tried any commands yet. It does drop to a command line but it's not registering my computer - note that it shows that I am "root@(none):" - which, unfortunately, is not the name of my PC...

Will try to run a uname -a later when I get back home, and other simple commands like that. I've experienced a maintenance shell before on a couple of occasions (what a wonderful world, Linux is!!) and I've always found it quite limiting - many commands not known etc...

Anyway, the point is, I'm really not sure how to proceed from here. Especially if it tells me that it can't mount my drives/partitions...

Would anyone know if there is a way to at least get /boot mounted so that I can see what kernels are available to be used? I'm wondering if I can find a way to boot into a command line on a kernel that does have access to all the partitions, so that I can then kick off the upgrade again by hand on the CLI.

Or am I just shooting in the dark here? Can anyone tell me if this is even the correct way to go?

Finally - let's assume that I do get there and that I'm able to kick off the upgrade again from the CLI... (I am nothing if not an optimist!!). I've now already downloaded all of the required files for the upgrade. How do I make sure that the system knows about these - and doesn't try to download them all again...? We're on a slow connection, and the 690MB download already took me 18 hours, so really don't want to go there if I can. How can I make sure that the update manager (when I get there) can see the files already downloaded in the /temp folder?

Many thanks for all your help.

Cheers.

Bag.

[mudlark]

Re: Katastrophic Koala upgrade - system completely borked...

That's where my update failed.

I am not suggesting you do this but what I did was to download a fresh version of 9.10, boot into the live cd, copied my personal folder and rebuilt everything. I also kept a copy of my fstab file so I can get easy access to my network folders. backing up the home folder doesn't save many of the hidden settings files so i lost lots of passwords and firefox bookmarks. A right old pain, but the new system is fully settled in Ext4 and the latest grub. (In facrt, my view is that grub has buggered things up damaging the mbr)

Good luck and i will watch with interest.

M.

[bag]

Re: Katastrophic Koala upgrade - system completely borked...

Mudlark,

Well, that's the fallback position, if I have to... although I'd rather not, if I can possibly avoid it. We'll see if anyone with more experience can come up with an elegant way for me to get to a CLI and try the upgrade again...

I went home at lunch and did a CTRL^D. It dropped straight out and tried to mount the drives again... then got stuck whilst it waited for a response from the system... Should have tried some simple commands first - d'oh! Ho hum, if it fails and then gets to a CLI again later then I'll try some simple commands and have a poke around.

In the meantime, had a thought. Maybe someone can validate the idea and let me know how to go about it...

If I was to:

1. Boot into a 9.10 disk as a live CD
2. And then chroot

does this give me access to the installed system, from which I can try to upgrade again?

I presume that I could use any kernel that's in there, could I not? Pick a kernel, any kernel, chroot into it and then run the upgrade path?

I seem to remember doing something similar on a borked Gentoo system a few years back. It was quite impressive to be able to boot another CD and then effectively get into a 'dead' system on the disk to repair it from the inside.

Would this work? Is this methodology pretty much as I remember it? And if so, can anyone point me in the right direction for instructions on how to do this?

It was a bit of a lightbulb moment when I thought of it - but am I going along the right path here, or am I wandering off in the wrong direction.

As always, anyone who knows more about this is very much encouraged to pipe up...!!! I am quite officially lost, lost, lost on this one...! I've not given up hope yet - I know the data's there - but still, I don't want to be without a system for weeks while I try to recover it...

Many thanks,

Bag.

[bag]

Re: Katastrophic Koala upgrade - system completely borked...

Right then folks, I've had a bit of a surf, and after a few false starts, I've found this...

http://ubuntuforums.org/showthread.php?t=157250

It talks about how to use Knoppix to chroot into a dead system and update it.

I have an old Knoppix disk lying around somewhere, so will try it out later. See how I get on.

I presume that I'll also need to mount other filesystems/partitions that will be needed too (and I'm thinking /boot and /temp here etc etc).

So then, going back to an earlier question I had about what I'd do if I got to a command prompt and could kick it all off again...

Finally - let's assume that I do get there and that I'm able to kick off the upgrade again from the CLI... (I am nothing if not an optimist!!). I've now already downloaded all of the required files for the upgrade. How do I make sure that the system knows about these - and doesn't try to download them all again...? We're on a slow connection, and the 690MB download already took me 18 hours, so really don't want to go there if I can. How can I make sure that the update manager (when I get there) can see the files already downloaded in the /temp folder?

Any ideas?

Many thanks in advance.

Bag.

[flarson]

Re: Katastrophic Koala upgrade - system completely borked...

Long before you got dropped to a maintenance shell you should have seen a grub menu that would allow yo to select the kernel that would boot before. Why not chose that one and then complete the upgrade? I would strongly recommend that you complete the upgrade from a text terminal after having exited X.

[bag]

Re: Katastrophic Koala upgrade - system completely borked...

mando_hacker

OK, oh so simple...! Hadn't tried rebooting though 'cos I didn't know what other damage I could do to the machine...

Will look at the grub menu as you say - presume it would/should be fine unless it was one of the files or programs that was already replaced/screwed up before the kdesudo crash... I presume that those programs that no longer work when kdesudo crashed were those that had already been replaced as part of the installation routine.

Will give that a go first.

Just to be clear - what is the best way to drop to a text terminal and exit X when/if I reach a login prompt?

And do I just do:

sudo apt-get dist-upgrade

??

Would that be it?

How do I force it to use the already downloaded packages, is that possible? I really would prefer not to wait another 18 hours for the download of the files when they're already there on the disk...

Many thanks for your help.

Cheers.

Bag.

[flarson]

Re: Katastrophic Koala upgrade - system completely borked...

The fact that you get to the maintenance shell indicates that grub is OK. You seem to be booting into a kernel that has a bad initrd.img file and lacks support for your filesystems. The grub menu should be OK. It is possible you will find the initrd.img for the old kernel has been screwed up also but the only way to know is to try it.

CTL-ALT-F1 should take you to virtual terminal 1, F2 to VT2 ... Then run

sudo service kdm stop

or

sudo invoke-rc.d kdm stop

or

sudo /etc/init.d/kdm stop

to stop KDM.

Either aptitude or apt-get will use the already downloaded packages. Aptitude is smarter than apt-get and I recommend it. You may find that you need to run

sudo dpkg --configure -a

before you can complete the upgrade depending on what state your system was left in.

I suggest that once you get logged in you run

sudo dpkg --configure -a

then

sudo aptitude -f full-upgrade

Keep doing some combination of those until the upgrade completes without error.

[bag]

Re: Katastrophic Koala upgrade - system completely borked...

Hmmm.... well I'm not getting much joy from this trying to "boot into an alternative kernel" lark...

I've got nine different options available and I've now tried them all. I get the same problem each time. It hangs when trying to complete the mounts. Same error messages that showed up earlier.

So now I'm wanting to ask a different question.

If all of my kernels boot but all of them fail at the same time because they can't mount anything, doesn't that suggest that it's actually a problem with the filesystems/drives/data on the partitions etc.? That, in itself, is a more scary question...

So, can anyone tell me what the error messages might mean? What is actually likely to be causing the errors in the first place? Up until now, I've just been trying to concentrate on getting past the errors and fixing it again from inside. I've been working on the assumption that the data is there, and that it's usable but that I just can't access it. However, now I need to know - once I get in and am able to see the mounts - is it actually gonna be worth fixing?

If anyone can give me an idea of what it realistically means when it says that that the mounts cannot yet be mounted then that would be great...

This will give me a better idea about whether I should try the chroot option or not.

Many thanks,

Bag

[Snowhog]

Re: Katastrophic Koala upgrade - system completely borked...

Have you tried simply running off of the LiveCD? Do you have one? If you can boot a LiveCD, then you can mount the HD and take a look around. Assuming you do that, locate and copy/paste the /etc/fstab file so we can see how you have it set up.

[flarson]

Re: Katastrophic Koala upgrade - system completely borked...

The LiveCD suggestion is a good one. I expect you will find that the drives and partitions are fine. When the kernel boots it first mounts an initrd.img file as the root filesystem. This file is created after the installation of a new kernel and after a new version of initramfs-tools they might all be rebuilt. That file must contain the kernel modules that are needed to mount your real filesystems. If they do not you will see exactly what you are seeing. Probably the best solution is to boot from the LiveCD and chroot into the installation and try something like

sudo aptitude reinstall linux-image-2.6.31-14-generic

or whatever the latest version installed is. This should result in the generation of an initrd.img file in /boot for at least that kernel. While you are in there it would be nice to use the dpkg--configure -a and aptitude commands to try to complete the upgrade, because it is possible that the files that tell initramfs-tools which modules to include in the inittrd.img file are damaged.

[bag]

Re: Katastrophic Koala upgrade - system completely borked...

Hey Snowhog,

Yeah - I have a couple of different live CD's around. I've booted it into Knoppix and now I'm officially worried...

Knoppix doesn't register any hard disks at all. Nothing. Nada. Rien de tout...

This is unusual because (although it's been a long time since I used Knoppix) I seem to remember that all of the partitions were set up on the desktop with links to them all. You could then mount any one of them by just clicking on it...

I'd normally expect to see a bunch of different partitions up there for review. But since there's nothing there I got very curious. Running QtParted brings up only Knoppix stuff and a /dev/hda - all related to knoppix and none of the partitions that I have on my PC.

So I've rebooted and looked at booting into 9.10 Kubuntu live disk to see if that picks up anything else. I'm surprised to see that I can see all the devices in Kubuntu though! Hurrah!.

Will finish this post now and then connect again from the live cd environment so that I can copy and paste in the /etc/fstab file...

Back soon.

Bag

[bag]

Re: Katastrophic Koala upgrade - system completely borked...

Mando_hacker, great stuff. Many thanks.

Right then, for the record, here is the fstab file - which to me (in my limited experience) doesn't seem to be a real problem (unless the UUID's have changed somehow).

# /etc/fstab: static file system information.
#
# Use 'vol_id --uuid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>  <type> <options>    <dump> <pass>
proc      /proc      proc  defaults    0    0
# / was on /dev/sdc3 during installation
UUID=faaa3c7c-93e3-4b24-a03f-8f95a1f616f2 /        ext3  relatime,errors=remount-ro 0    1
# /boot was on /dev/sdc1 during installation
UUID=23fadc37-9946-43b0-87ac-de7f91470690 /boot      ext3  relatime    0    2
# /distros was on /dev/sdc7 during installation
UUID=be7f9061-fe34-49e0-8947-65036bf95edc /distros    ext3  relatime    0    2
# /home was on /dev/sdc8 during installation
UUID=0f73537f-eda3-452f-abb9-681c0dc1d0c3 /home      ext3  relatime    0    2
# /restore was on /dev/sdc6 during installation
UUID=3e9f2bca-fb6a-4162-a2de-d8b49c65626e /restore    ext3  relatime    0    2
# /tmp was on /dev/sdc5 during installation
UUID=1fe46969-367b-4b6e-b09f-1dd263374a99 /tmp      ext3  relatime    0    2
# swap was on /dev/sdc2 during installation
UUID=54eb59f3-1e7d-4268-94c8-e80f39ea6464 none      swap  sw       0    0
/dev/scd0    /media/cdrom0  udf,iso9660 user,noauto,exec,utf8 0    0
/dev/scd1    /media/cdrom1  udf,iso9660 user,noauto,exec,utf8 0    0

# /common created during installation, manually attached to /dev/md1 raid device later
/dev/md1				/common 	ext3  	defaults    0    0

Right then, so this is the gameplan...

1. Stop KDE on the LiveCD.
2. Chroot from the LiveCD to the system on my disks.
3. Try to reinstall the latest kernel there.
4. Then run an upgrade path on the command line and see if it all comes clean in the wash.

Does this sound about right?

Let me know quick if I'm about to make a big mistake...!!!

Thanks,

Bag.