mount: / is busy and system hangs without shutting down

[Frank616]

oshunluvr:

OK, here are the screenshots of what went flashing by. lsof | more did the trick. It doesn't look like ps ran.

Sorry they are out of order. I thought I uploaded them in order.

To my untrained eye, it looks like everything shut off except for a few at the bottom that are system services, and I don't see network manager in there anywhere.

Let me know where to go from here.

Frank.

[Frank616]

Actually, it looks more like lsof did not run. I just checked outputs of both of those commands on my running machine, and the pictures above look more like ps ajx.

Frank.

[oshunluvr]

Yeah, nothing useful there that I can see. Ok, lets try this; We're going to see what processes are running before the umountroot script is called.

I assume this happens both on reboot and shutdown?

Do ls /etc/rc6.d (reboot scripts, rc0.d is shutdown scripts) and find the umountroot link. It will be named S60umountroot or something similar. We need to create a files that run just before umountroot - they execute in alphabetical order, so use a name that comes before S60umountroot but just before. Then put this in it:

#! /bin/sh

ps -Af >> process.log

and make it executable. Then reboot. Once you're back up, post the contents of /process.log and we'll have a look. Also, go ahead and undo the edits you did to the umountroot script.

[Frank616]

oshunluvr:

OK, did as you said. Had to rename a file S59xxx to S58xxx to make room for it, but it ran. Output below:

UID        PID  PPID  C STIME TTY          TIME CMDroot         1     0  0 15:40 ?        00:00:01 /sbin/init
root         2     0  0 15:40 ?        00:00:00 [kthreadd]
root         3     2  0 15:40 ?        00:00:00 [ksoftirqd/0]
root         5     2  0 15:40 ?        00:00:00 [kworker/0:0H]
root         6     2  0 15:40 ?        00:00:00 [kworker/u4:0]
root         7     2  0 15:40 ?        00:00:00 [rcu_sched]
root         8     2  0 15:40 ?        00:00:00 [rcuos/0]
root         9     2  0 15:40 ?        00:00:00 [rcuos/1]
root        10     2  0 15:40 ?        00:00:00 [rcu_bh]
root        11     2  0 15:40 ?        00:00:00 [rcuob/0]
root        12     2  0 15:40 ?        00:00:00 [rcuob/1]
root        13     2  0 15:40 ?        00:00:00 [migration/0]
root        14     2  0 15:40 ?        00:00:00 [watchdog/0]
root        15     2  0 15:40 ?        00:00:00 [watchdog/1]
root        16     2  0 15:40 ?        00:00:00 [migration/1]
root        17     2  0 15:40 ?        00:00:00 [ksoftirqd/1]
root        18     2  0 15:40 ?        00:00:00 [kworker/1:0]
root        19     2  0 15:40 ?        00:00:00 [kworker/1:0H]
root        20     2  0 15:40 ?        00:00:00 [khelper]
root        21     2  0 15:40 ?        00:00:00 [kdevtmpfs]
root        22     2  0 15:40 ?        00:00:00 [netns]
root        23     2  0 15:40 ?        00:00:00 [writeback]
root        24     2  0 15:40 ?        00:00:00 [kintegrityd]
root        25     2  0 15:40 ?        00:00:00 [bioset]
root        26     2  0 15:40 ?        00:00:00 [kworker/u5:0]
root        27     2  0 15:40 ?        00:00:00 [kblockd]
root        28     2  0 15:40 ?        00:00:00 [ata_sff]
root        29     2  0 15:40 ?        00:00:00 [khubd]
root        30     2  0 15:40 ?        00:00:00 [md]
root        31     2  0 15:40 ?        00:00:00 [devfreq_wq]
root        33     2  0 15:40 ?        00:00:00 [kworker/1:1]
root        34     2  0 15:40 ?        00:00:00 [khungtaskd]
root        35     2  0 15:40 ?        00:00:00 [kswapd0]
root        36     2  0 15:40 ?        00:00:00 [vmstat]
root        37     2  0 15:40 ?        00:00:00 [ksmd]
root        38     2  0 15:40 ?        00:00:00 [khugepaged]
root        39     2  0 15:40 ?        00:00:00 [fsnotify_mark]
root        40     2  0 15:40 ?        00:00:00 [ecryptfs-kthrea]
root        41     2  0 15:40 ?        00:00:00 [crypto]
root        53     2  0 15:40 ?        00:00:00 [kthrotld]
root        55     2  0 15:40 ?        00:00:00 [scsi_eh_0]
root        56     2  0 15:40 ?        00:00:00 [scsi_eh_1]
root        58     2  0 15:40 ?        00:00:00 [scsi_eh_2]
root        59     2  0 15:40 ?        00:00:00 [scsi_eh_3]
root        61     2  0 15:40 ?        00:00:00 [kworker/u4:4]
root        62     2  0 15:40 ?        00:00:00 [kworker/0:2]
root        82     2  0 15:40 ?        00:00:00 [deferwq]
root        83     2  0 15:40 ?        00:00:00 [charger_manager]
root        85     2  0 15:40 ?        00:00:00 [kworker/0:3]
root        86     2  0 15:40 ?        00:00:00 [kworker/u5:1]
root       130     2  0 15:40 ?        00:00:00 [firewire]
root       132     2  0 15:40 ?        00:00:00 [firewire_ohci]
root       184     2  0 15:40 ?        00:00:00 [jbd2/sda1-8]
root       185     2  0 15:40 ?        00:00:00 [ext4-rsv-conver]
root       267     1  0 15:40 ?        00:00:00 /lib/systemd/systemd-udevd --daemon
root       272     1  0 15:40 ?        00:00:00 /lib/systemd/systemd-udevd --daemon
root       275     2  0 15:40 ?        00:00:00 [edac-poller]
root       306     2  0 15:40 ?        00:00:00 [kvm-irqfd-clean]
root      1147     2  0 15:41 ?        00:00:00 [krfcommd]
root      1556     2  0 15:41 ?        00:00:00 [kauditd]
root      1715     1  0 15:41 ?        00:00:00 /bin/sh /usr/lib/pm-utils/power.d/pci_devices false
root      2636     2  0 15:50 ?        00:00:00 [kworker/u4:1]
root      2664     1  0 15:51 ?        00:00:00 /bin/sh -e /proc/self/fd/9
root      2786     1  0 15:51 ?        00:00:00 @sbin/plymouthd --mode=shutdown
root      2790  2664  0 15:51 ?        00:00:00 /bin/sh /etc/init.d/rc 6
root      2960     2  0 15:51 ?        00:00:00 [kworker/0:0]
root      3048  2790  0 15:51 ?        00:00:00 /bin/sh /etc/rc6.d/S59Intercept stop
root      3049  3048  0 15:51 ?        00:00:00 ps -Af

I don't get it. Looks like it all ran up to the ps -Af, and that I assume is normal.

I didn't edit out the changes I made to umountroot. I copied the untouched file by the same name on my laptop downstairs to this machine. That way I was sure I had no errors or typos or whatever. Same version of Kubuntu from the same source disk with the same updates.

In the meantime I looked up info on this issue, and it seems to happen from time to time, but nothing I could find that was recent. One thread suggested some changes, then doing update-grub. I didn't make any other changes, but did to update-grub. As expected, no change. But, I thought I'd at least try.

Back to you.

[oshunluvr]

The problem is this issue has cropped up over several years and the solutions aren't consistent. One blog states the the --mode=shutdown switch on plymouthd is the problem but no one else did. Others report adding switches to the boot line and then running update-grub to make them take effect (the issue wasn't grub not being updated, but rather the act of running update-grub is required in order for the edit(s) to /etc/default/grub to take effect).

The only 14.04 machine I have is my headless server. I did the above changes also just to see what the effect was. No notable differences from your results. Interestingly, the machine would not complete shutdown. I had to manually power it down. Seems I might have the same issue, however my machine reboots without problem and I never power it down so if it's been there all along I have no way of knowing. You didn't say if your machine reboots OK or not...

At this point, I'm running out of ideas. I wouldn't go further without restoring the system to it's original state (remove all our edits and extra files).

The only two common causes are either network or ACPI related. You need to log into text mode, completely kill the GUI and all network related process and then attempt a manual shutdown. If that works, you'll know it may be network related. If it doesn't, then you can eliminate that as a cause.

Then try this: edit the file /etc/default/grub (as root) and change

GRUB_CMDLINE_LINUX_DEFAULT=

to

GRUB_CMDLINE_LINUX_DEFAULT="acpi=force reboot=acpi"

and run update-grub.

[Frank616]

The problem is this issue has cropped up over several years and the solutions aren't consistent.

Yeah, I've noticed that.

Interestingly, the machine would not complete shutdown. I had to manually power it down. Seems I might have the same issue, however my machine reboots without problem and I never power it down so if it's been there all along I have no way of knowing. You didn't say if your machine reboots OK or not...

No, it won't reboot either. Same problem It hangs before a restart.

The only two common causes are either network or ACPI related.

I wondered about ACPI. Seems I saw an issue with that in one of of the messages when shutting down. Any problem with me trying that first?

At this point, I'm running out of ideas. I wouldn't go further without restoring the system to it's original state (remove all our edits and extra files).

Will do. I'll let you know.

Rather than continue to play with this, what are the odds that a wipe and reinstall will get whatever it is? This machine has run 14.04 with no issues since 14.04 came out, and 12.04 before that, and 10.04 before that. Sometimes I find that weird things just happen, and while one can track them down, sometimes the effort is not worth it.

Frank.

[oshunluvr]

Rather than continue to play with this, what are the odds that a wipe and reinstall will get whatever it is? This machine has run 14.04 with no issues since 14.04 came out, and 12.04 before that, and 10.04 before that. Sometimes I find that weird things just happen, and while one can track them down, sometimes the effort is not worth it.

While I agree that a reinstall is sometimes the most expedient course, it's not always the solution. I would try some of the boot options like the one I suggested before starting from scratch unless you have no time at all invested in customizing this particular install.

[Frank616]

oshunluvr:

GRUB_CMDLINE_LINUX_DEFAULT="acpi=force reboot=acpi"

No change. (sigh)

As to killing network manager by going into text mode, I am not sure how to boot into text mode only.

Current boot options are 'advanced' which allows me to choose the recovery screen, and I can then choose grub, fsck, root shell, etc. from the recovery menu. I chose root shell after a clean boot and then ran the ps command to see which process I should try to kill. The output was pretty sparse:

root@irma:~# ps
PID TTY TIME CMD
441 tty1 00:00:00 recovery-menu
26035 tty1 00:00:00 root
26052 tty1 00:00:00 bash
26086 tty1 00:00:00 ps
root@irma:~# reboot

I typed in the reboot command, then wrote down what was on the screen before I tried the reboot. When I hit <enter> to reboot, I just went back to the recovery menu. I got called away on the phone at that point, and when I came back the screen was black. Touching the mouse or keyboard did nothing to awaken it, but touching the power button (not holding it) did bring up the recover menu again, but the machine was locked, and then I had to hold the power button.

So, is there some other mode I should be using here to be certain that no network services are running? There is a network entry on the recovery menu, but it only appears to turn it on. I can try to snap a picture of the output on that when it drops to command line for a second, but then it goes right back to the recovery menu again. Is NM respawning, like it did when I tried to shut it down from the gui process list?

Thanks.

Frank.

[oshunluvr]

Well, you could try booting up. Don't log in. Hit CTRL-ALT-F1 to get to the text terminal, log in there, kill the display manager, then switch to runlevel 1, then attempt shutdown.

sudo service sddm stop
sudo telinit 1
sudo shutdown -H

[Frank616]

oshunluvr:

Curiouser and curioser, as the expression goes....

Did as you said, and succeeded in logging into text mode. I didn't know how to do that before. It will now go into my troubleshooting notes for future reference.

[COLOR=#3E3E3E][FONT=sans-serif]sudo service sddm stop[/FONT][/COLOR]

Ran that. Unrecognized service was the report I got. So I continued with

[COLOR=#3E3E3E][FONT=sans-serif]sudo telinit 1[/FONT][/COLOR]

and got the output as per picture 1.

Then tried the next command

[COLOR=#3E3E3E][FONT=sans-serif]sudo shutdown -H[/FONT][/COLOR]

And got the output that it was expecting a time rather than -H

I then did sudo shutdown now, and got the output in picture 2

It stopped there, but did not shut down. As the command line was still active, I tried the ps command. Then tried:

sudo exit

as in picture 3. And, at that point, it locked. I tried several things, finally giving it the 3 finger salute Ctrl-Alt-delete, and got the output in picture 4, which is back to where I was before.

Now, I quite likely, in my ham-fistedness, did some things there that undid what you wanted. So, if you are still willing to put up with me, I'm ready to learn more.

Frank.

[oshunluvr]

Yeah, it should have been:

sudo shutdown -H now

sorry, -H means "halt"

Seems like it will shutdown. I sure wish we could figure out what is holding up the remount. No ideas at the moment...

[Frank616]

I also tried:

[COLOR=#3E3E3E][FONT=sans-serif]sudo service lightdm stop[/FONT][/COLOR]

As there appears to be no sddm running on this machine. I got a list of running services using

service --status-all

and sddm is not listed, but lightdm was. However, it made no real difference.

I also updated the machine this morning hoping the new kernel update would magically fix things. Nope.

I'm updating the nVidia video driver now as a shot in the dark.

Why does modemManager keep restarting? Is this thing respawning and leaving something open?

Frank.

[Frank616]

oshunluvr:

Anyway, time to throw in the towel. While updating the video driver, I cam across other issues. It keeps complaining that there is no opengl, when it is in fact installed, and then reinstalled. I am in the process of blowing the whole works away and reinstalling. Hopefully this one will work.

Thanks for all your kind help. I'll report back if the new install works or not.

Frank.

[oshunluvr]

Well yeah duh, lightdm was used in 14.04. I'm using 15.04 so I messed that up too. I always un-install modem manager and network manager on my desktop so I haven't had to deal with that.

Well, hopefully all this will go away when you re-install (fingers crossed).

[Frank616]

Nope. Still hangs on a brand-spanking new fresh install. Uninstalled modem manager, but no change.

I think maybe I'm going to try another distro on this machine. I don't know what else to do. It has run fine for years, but now that I have changed the drives, it just will NOT shut off anymore.

Any possibility of timing issues on the super-fast SSD? This is a core-2 Duo machine, and not sure if it is SATA 1 or 2.

In the meantime I'll download Mint and see if it will load and shut down.

Frank.