Unable to unmount /var

[rihad]

Hi, I need to lvextend the size of an LVM2 volume that /var is on. Obviously I need to unmount /var first, so I dropped to single user shell. Alas, some OS programs are accessing /var in single-user mode preventing me to umount it, according to fuser -vm /var. Killing them solves nothing, as init keeps respawning them. I guess this is a shortcoming. Why won't [K]Ubuntu step away and let me do what I need to?

[oshunluvr]

Re: Unable to unmount /var

I wouldn't call it a shortcoming - /var is a system directory and is used to store logs and program data by LOTS of programs.

Can you boot to a liveCD and do what you need?

BTW: when btrfs is ready for the masses we can stop using lvm and raid - preventing a lot of these problems! Soon I hope.

[rihad]

Re: Unable to unmount /var

I wouldn't call it a shortcoming - /var is a system directory and is used to store logs and program data by LOTS of programs.

I was speaking of single user mode. If admin can't take full control without resorting to workarounds (live CD in this case), then it's a shortcoming. This bug (being unable to unmount any partition, except for root, in single user mode) just isn't there in Debian 3.x-5.x or FreeBSD 4.x upward.

There are two programs accessing /var in SU mode: statd & rsyslogd. I was able to stop statd by simply commenting these two lines in /etc/init/statd.conf:

#expect fork
#respawn

but a similar trick in /etc/init/rsyslog.conf didn't help: init kept respawning rsyslogd even if I also commented all the "exec", "start", "stop" lines. So now I'm going to get a live CD, and boot from that.

[Snowhog]

Re: Unable to unmount /var

I was speaking of single user mode. If admin can't take full control without resorting to workarounds (live CD in this case), then it's a shortcoming.

Did you not understand oshuntervr's comment? In Kubuntu, /var is part of (a branch) the mounted 'root' filesystem). You can not unmount a 'branch'. This is not a bug.

[rihad]

Re: Unable to unmount /var

Did you not understand oshuntervr's comment? In Kubuntu, /var is part of (a branch) the mounted 'root' filesystem). You can not unmount a 'branch'. This is not a bug.

I'm afraid it's you not understanding. Being unable to unmount any FS except root in single-user mode is a strong deficiency compared to at least Debian and FreeBSD. Admin may want nothing to run, except for getty and shell, in single user mode.. Imho init should take this into account, and not keep /var, /var/run and /var/lock mounted.

[kubicle]

Re: Unable to unmount /var

In Kubuntu, /var is part of (a branch) the mounted 'root' filesystem).

By default yes, but it's not uncommon to have /var on a separate filesystem/partition, especially on servers.

So I'd consider this a valid bug.

EDIT
@rihad:
I can't offer any immediate help because I don't use rsyslog on any of my machines. I use syslog-ng for logging (switched to it from syslog before rsyslog was adopted in ubuntu), which doesn't start (by default) on Single user mode.

So I'd assume rsyslog shouldn't need to start either, could be a bug with either rsyslog (wrong configuration to start in runlevel S) or upstart (doesn't respect configuration)...or a combination of both.

Could you post the contents of your '/etc/init/rsyslog.conf' as I don't have one.

[rihad]

Re: Unable to unmount /var

Could you post the contents of your '/etc/init/rsyslog.conf' as I don't have one.

Thanks for supporting my bug report. The problem wasn't actually in rsyslog, but in several programs accessing /var or /var/run mounted therein: portmap, dd, all spawned by init according to /etc/init/* conf files. I was finally able to stop all of them, unmount /var, lvextend & resize2fs its ext4 partition. Here are the steps I took, some of them might have been redundant:
I commented these lines in /etc/init/gssd.conf:

#start on (started portmap
# or mount TYPE=nfs4 OPTIONS=sec=*krb5*)

and in /etc/init/statd.conf:

#start on (started portmap or mount TYPE=nfs)
and in pre-start script:
#status portmap | grep -q start/running || start portmap

and in /etc/init/portmap.conf:

#start on (local-filesystems
# and net-device-up IFACE=lo)

#expect fork
#respawn
and in script:
# exec portmap $OPTIONS

and in /etc/init/rsyslog-kmsg.conf:

#start on starting rsyslog
#respawn
#exec dd bs=1 if=/proc/kmsg of=/var/run/rsyslog/kmsg

After the edits I did initctl reload-configuration, and then tried killing the programs by PID (as determined by fuser -m $mountpoint -v where $mountpoint is either /var or /var/run.). If that didn't help, I did telinit 1 to enforce the new settings. Alas, I couldn't stop init from respawning portmap, which accessed /var, so a brute chmod -x /sbin/portmap luckily stopped init from respawning it indefinitely (of course I did a chmod +x /sbin/portmap afterwards). Finally no program was accessing /var or /var/run, as far as fuser or lsof could see, but /var still wouldn't unmount. So I ran a ps -ef and saw that dd was holding a file open for writing in /var/run. Commenting it out in /etc/init/* didn't help for some strange reason, so a chmod -x helped me again. And, finally, I did it! See how awkward that was? My box didn't crash or blow out on me for stopping those offending programs. They not only weren't necessary for what I was intending to do, Upstart's init tried its best to stop me from doing that.

[oshunluvr]

Re: Unable to unmount /var

Just a thought for future consideration:

Since runlevels 3,4, and 5 are not really used on a debian based system, wouldn't it be not-too-difficult to edit say, runlevel 3 to run the bare minimum?

I see now that a lot of unnecessary stuff is running for "trouble" mode is you're using run level 1.

/etc/rc1.d:
K20acpi-support
K20hddtemp
K20rsync
K20sysstat
K20cdemu-daemon
K20kerneloops
K20saned
K20vboxdrv
K20fancontrol
K20postfix
K20smartmontools
K20winbind
K21spamassassin
K25mdadm
K74bluetooth
K80cups
K99laptop-mode
S30killprocs
S70dns-clean
S70pppd-dns
S90single

Seems like statd is for NFS so should be killed

[kubicle]

Re: Unable to unmount /var

Upstart's init tried its best to stop me from doing that.

Instead of "fighting" upstart, you could try working with it (even though it was upstart that started the fight )

Instead of commenting out the start blocks, you could try adding stop blocks...something like
stop on runlevel [1S] (stop service in single user mode)
or
stop on runlevel [!2345] (stop service in non-multiuser runlevels)

You might also try adding runlevel requirements in those start blocks, something like
and runlevel [2345]

Haven't had a need to tinker with upstart confs much, so I can't tell for sure whether these mods would work as is, but if you're still interested to try them out they might be worth a shot (although I understood you already worked around your immediate issues)

Also, gssd and statd (judging by your configs) are likely triggered by portmap starting, so stopping portmap from starting might be enough to prevent those services from starting as well. Also preventing rsyslog from starting should also keep rsyslog-kmsg from firing up.

[mfburgo]

Re: Unable to unmount /var

Hi, I need to lvextend the size of an LVM2 volume that /var is on. Obviously I need to unmount /var first, so I dropped to single user shell. Alas, some OS programs are accessing /var in single-user mode preventing me to umount it, according to fuser -vm /var. Killing them solves nothing, as init keeps respawning them. I guess this is a shortcoming. Why won't [K]Ubuntu step away and let me do what I need to?

Confused, If using lvm just extend it while online. Why did you need to unmount the filesystem in the first place. I have extended filesystems online for a long time even var, tmp, even / without issue. I am not able to understand why you believed you needed to unmount var to do the lvextend < to the size you wanted > and do an resize2fs on the file system online resize has worked for years.

[rihad]

Re: Unable to unmount /var

I am not able to understand why you believed you needed to unmount var to do the lvextend < to the size you wanted > and do an resize2fs on the file system online resize has worked for years.

That's because of this part of "man resize2fs":

The resize2fs program will resize ext2, ext3, or ext4 file systems. It can be used to enlarge or shrink an unmounted file system
located on device. If the filesystem is mounted, it can be used to expand the size of the mounted filesystem, assuming the kernel
supports on-line resizing. (As of this writing, the Linux 2.6 kernel supports on-line resize for filesystems mounted using ext3
only.).

But I'm using ext4. Even if the on-line resize does work for ext4, I wouldn't want to try it for the first time on my desktop, you know.

[mfburgo]

Re: Unable to unmount /var

I am not able to understand why you believed you needed to unmount var to do the lvextend < to the size you wanted > and do an resize2fs on the file system online resize has worked for years.

That's because of this part of "man resize2fs":

The resize2fs program will resize ext2, ext3, or ext4 file systems. It can be used to enlarge or shrink an unmounted file system
located on device. If the filesystem is mounted, it can be used to expand the size of the mounted filesystem, assuming the kernel
supports on-line resizing. (As of this writing, the Linux 2.6 kernel supports on-line resize for filesystems mounted using ext3
only.).

But I'm using ext4. Even if the on-line resize does work for ext4, I wouldn't want to try it for the first time on my desktop, you know.

OK now I understand why you may have done this. I am letting you know that it does resize ext4. And I have not had issue's. <of course I always have a good backup to fall back on in the event something goes wrong>. If you have the space since your using lvm create a snapshot of the original do the resize if there is a problem fall back to the snapshot. Snapshot's are great for backup purpose in the event something goes wrong. ( also have your rescue CD ready )

Since you installed, I am assuming from a cd, you could also have booted into rescue mode on the cd opened a shell on your root filesystem which only mounts root and resize there if you are concerned. ( you may need to mount usr for that.

If you need more help let me know I will try it with sone test VM's for you and send you the steps.....