update-initramfs stuck at 96% FATAL TO SYSTEM

[oshunluvr]

After yesterdays update, apt gets stuck when trying to create/update initramfs for the new 5.15.0-76 kernel. I left it overnight and it's STUCK.

I re-ran update-initramfs manually for the new kernel and got this:

sudo update-initramfs -c -v -k 5.15.0-76-generic
~~~
​Calling hook mdadm
Adding binary /sbin/mdadm
Adding binary /sbin/mdmon
Adding module /lib/modules/5.15.0-76-generic/kernel/drivers/md/linear.ko
Adding module /lib/modules/5.15.0-76-generic/kernel/drivers/md/multipath.ko
Adding module /lib/modules/5.15.0-76-generic/kernel/drivers/md/raid0.ko
Adding module /lib/modules/5.15.0-76-generic/kernel/drivers/md/raid1.ko
Adding module /lib/modules/5.15.0-76-generic/kernel/drivers/md/raid456.ko
Adding module /lib/modules/5.15.0-76-generic/kernel/drivers/md/raid10.ko

Progress: [ 96%] [################################################## ################################################## ################################################## ####.......]
​

Basically the same results. I even change the modules setting from "MOST" to "DEP" to see if that would help, but exact same results.
Also same results for 5.19.0-46-generic

I thought about removing mdadm since I don't use it anyway (and assuming it's actually the issue) but I can't because dpkg won't let me do anything until it's finished.

I'm boxed in. I'd rather not roll-back if I can learn a way to fix this.

[Snowhog]

Can you interrupt the process with CTRL+C

[oshunluvr]

Yeah, but that still leaves me unable to update initramfs or install or remove any packages.

[Snowhog]

but that still leaves me unable to update initramfs or install or remove any packages

IF you interrupt, you can't then run sudo dpkg --configure -a

[oshunluvr]

So I went ahead and rolled back to try the update again. But when I did it took several minutes to reboot. It hung for 2-3 minutes trying to shut down 7-8 mdadm instances, which is super odd because I'm not using it.

The only odd -ball thing I am aware of is I had an unreadable CD-ROM in the drive. When it finally rebooted it also took several minutes for the attempt to read the CD-ROM to fail.

I'm going to start over the the CD out and see what happens...

[Snowhog]

IF it's mdadm that's the culprit, and as you stated you don't use it anyway, then https://askubuntu.com/questions/1255...nitramfs-stuck and the first answer.

[oshunluvr]

Yup, that looks like the answer. BUT...

I did a roll-back and took out the CD and it worked. So now I'm off to try it again with the CD back in...

[Snowhog]

The only odd -ball thing I am aware of is I had an unreadable CD-ROM in the drive.

Why would you have such ‘mounted’?

[oshunluvr]

It's not mounted. You can't mount a bad disc. It was just left in the drive. It's an external USB BluRay drive.

The question that needs answering is WHY is update-initramfs hanging because a USB CDROM has a disc in it at all. What part of that process is attempting to scan the contents of a CDROM via USB no less?

[oshunluvr]

So re-rolling back and attempting the update with the bad disk back in the drive still fails. Clearly, the cause revolves around mdadm and/or the bad disc.

Now attempting to make the mdadm config change as detailed in the link.

[oshunluvr]

It is definitely having the CDROM in the drive causing the hang. My system is total screwed in this state. mdadm is hung in 5 processes and they cannot be killed. Basically, encountering this means your install is toast. No way to recover except to restore from backup.

This is the most egregious bug I have encountered in many many years.

[oshunluvr]

To clarify, here the timeline and status of this:

Left unreadable CDROM in external USB BluRay drive.
Attempted "apt full-upgrade" which included a kernel update.
"update-initramfs" hangs when processing mdadm modules (96% complete) due to mdadm scanning the unreadable CDROM.

This results in broken upgrade because:
The hung mdadm scan processes CANNOT be killed with kill -9 or -SIGKILL.
Reboot is also hung by mdadm thus a manual reset is required.

I was able to boot to a previous kernel with the bad CDROM removed, then dpkg --configure -a finished and it's usable again.

If you had no other bootable kernel you would be left with an unusable installation. What a total and unnecessary PITA.

[claydoh]

Kind of a dumb question, but do you have the preinstalled-pool repo enabled/present in /etc/apt/sources.list.d ?

This sort of reminds me of really old *buntu bugs with it scanning an optical drive for the pre-installed pool (the install media, which emulates an optical disk) when updating. Dunno how that would be involved with updating the initramfs though.

Just a random thought.

[oshunluvr]

Kind of a dumb question, but do you have the preinstalled-pool repo enabled/present in /etc/apt/sources.list.d ?

This sort of reminds me of really old *buntu bugs with it scanning an optical drive for the pre-installed pool (the install media, which emulates an optical disk) when updating. Dunno how that would be involved with updating the initramfs though.

Just a random thought.

No. I think they fixed that issue a while ago. I haven't seen the ISO left in sources for several years, but still I checked and no.

The actual cause of the problem is that "mdadm --scan ..." scans ALL devices for mdadm RAID indiscriminately. Clearly, one would not have 2 CDROMs running in RAID. That's probably not even possible. For whatever reason, the way it's called causes the process to hang in a way that cannot be killed, at least not in any way I was able to discover. I eventually killed all the processes related to it, but could not ever kill the mdadm scan process.

In my case, removing the CDROM from the drive, booting to an available kernel - since the one that was getting installed never got initramfs built - and then running dpkg reconfigure allow the initramfs to build.

Anyway - seems to me in some cases, a less experienced user or someone who ended up without any bootable kernels would be in a situation where re-installing from scratch would be their only option. The cause isn't obvious as there's no indication that the CDROM was the problem and it took quite awhile before I realized the CD was even in the drive. The "dead" CD does also cause reboot to hang for some time, but you get a console message indicating that. Since I couldn't soft reboot due to the mdadm hang, pushing the reset button was my only option.

I suspect this combination of events - mdadm installed, bad CDROM in the drive, initramfs rebuild - is esoteric enough that it's never really come to the attention of the developers. Seems like there should be a programatic way to skip "srX" devices during the scan.

I'd report it as a bug if I could find where to do it Seems I never can find the correct "official" webpage to report something like this.

[GreyGeek]

So even "kill -9 <somepid>" wouldn't work?