Announcement

Collapse
No announcement yet.

Unable to unmount /var

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Unable to unmount /var

    Hi, I need to lvextend the size of an LVM2 volume that /var is on. Obviously I need to unmount /var first, so I dropped to single user shell. Alas, some OS programs are accessing /var in single-user mode preventing me to umount it, according to fuser -vm /var. Killing them solves nothing, as init keeps respawning them. I guess this is a shortcoming. Why won't [K]Ubuntu step away and let me do what I need to?

    #2
    Re: Unable to unmount /var

    I wouldn't call it a shortcoming - /var is a system directory and is used to store logs and program data by LOTS of programs.

    Can you boot to a liveCD and do what you need?

    BTW: when btrfs is ready for the masses we can stop using lvm and raid - preventing a lot of these problems! Soon I hope.

    Please Read Me

    Comment


      #3
      Re: Unable to unmount /var

      Originally posted by oshunluvr
      I wouldn't call it a shortcoming - /var is a system directory and is used to store logs and program data by LOTS of programs.
      I was speaking of single user mode. If admin can't take full control without resorting to workarounds (live CD in this case), then it's a shortcoming. This bug (being unable to unmount any partition, except for root, in single user mode) just isn't there in Debian 3.x-5.x or FreeBSD 4.x upward.

      There are two programs accessing /var in SU mode: statd & rsyslogd. I was able to stop statd by simply commenting these two lines in /etc/init/statd.conf:
      #expect fork
      #respawn
      but a similar trick in /etc/init/rsyslog.conf didn't help: init kept respawning rsyslogd even if I also commented all the "exec", "start", "stop" lines. So now I'm going to get a live CD, and boot from that.

      Comment


        #4
        Re: Unable to unmount /var

        Originally posted by rihad
        I was speaking of single user mode. If admin can't take full control without resorting to workarounds (live CD in this case), then it's a shortcoming.
        Did you not understand oshuntervr's comment? In Kubuntu, /var is part of (a branch) the mounted 'root' filesystem). You can not unmount a 'branch'. This is not a bug.
        Windows no longer obstructs my view.
        Using Kubuntu Linux since March 23, 2007.
        "It is a capital mistake to theorize before one has data." - Sherlock Holmes

        Comment


          #5
          Re: Unable to unmount /var

          Originally posted by Snowhog
          Did you not understand oshuntervr's comment? In Kubuntu, /var is part of (a branch) the mounted 'root' filesystem). You can not unmount a 'branch'. This is not a bug.
          I'm afraid it's you not understanding. Being unable to unmount any FS except root in single-user mode is a strong deficiency compared to at least Debian and FreeBSD. Admin may want nothing to run, except for getty and shell, in single user mode.. Imho init should take this into account, and not keep /var, /var/run and /var/lock mounted.

          Comment


            #6
            Re: Unable to unmount /var

            Originally posted by Snowhog
            In Kubuntu, /var is part of (a branch) the mounted 'root' filesystem).
            By default yes, but it's not uncommon to have /var on a separate filesystem/partition, especially on servers.

            So I'd consider this a valid bug.

            EDIT
            @rihad:
            I can't offer any immediate help because I don't use rsyslog on any of my machines. I use syslog-ng for logging (switched to it from syslog before rsyslog was adopted in ubuntu), which doesn't start (by default) on Single user mode.

            So I'd assume rsyslog shouldn't need to start either, could be a bug with either rsyslog (wrong configuration to start in runlevel S) or upstart (doesn't respect configuration)...or a combination of both.

            Could you post the contents of your '/etc/init/rsyslog.conf' as I don't have one.

            Comment


              #7
              Re: Unable to unmount /var

              Originally posted by kubicle
              Could you post the contents of your '/etc/init/rsyslog.conf' as I don't have one.
              Thanks for supporting my bug report. The problem wasn't actually in rsyslog, but in several programs accessing /var or /var/run mounted therein: portmap, dd, all spawned by init according to /etc/init/* conf files. I was finally able to stop all of them, unmount /var, lvextend & resize2fs its ext4 partition. Here are the steps I took, some of them might have been redundant:
              I commented these lines in /etc/init/gssd.conf:
              #start on (started portmap
              # or mount TYPE=nfs4 OPTIONS=sec=*krb5*)
              and in /etc/init/statd.conf:
              #start on (started portmap or mount TYPE=nfs)
              and in pre-start script:
              #status portmap | grep -q start/running || start portmap
              and in /etc/init/portmap.conf:
              #start on (local-filesystems
              # and net-device-up IFACE=lo)

              #expect fork
              #respawn
              and in script:
              # exec portmap $OPTIONS
              and in /etc/init/rsyslog-kmsg.conf:
              #start on starting rsyslog
              #respawn
              #exec dd bs=1 if=/proc/kmsg of=/var/run/rsyslog/kmsg
              After the edits I did initctl reload-configuration, and then tried killing the programs by PID (as determined by fuser -m $mountpoint -v where $mountpoint is either /var or /var/run.). If that didn't help, I did telinit 1 to enforce the new settings. Alas, I couldn't stop init from respawning portmap, which accessed /var, so a brute chmod -x /sbin/portmap luckily stopped init from respawning it indefinitely (of course I did a chmod +x /sbin/portmap afterwards). Finally no program was accessing /var or /var/run, as far as fuser or lsof could see, but /var still wouldn't unmount. So I ran a ps -ef and saw that dd was holding a file open for writing in /var/run. Commenting it out in /etc/init/* didn't help for some strange reason, so a chmod -x helped me again. And, finally, I did it! See how awkward that was? My box didn't crash or blow out on me for stopping those offending programs. They not only weren't necessary for what I was intending to do, Upstart's init tried its best to stop me from doing that.

              Comment


                #8
                Re: Unable to unmount /var

                Just a thought for future consideration:

                Since runlevels 3,4, and 5 are not really used on a debian based system, wouldn't it be not-too-difficult to edit say, runlevel 3 to run the bare minimum?

                I see now that a lot of unnecessary stuff is running for "trouble" mode is you're using run level 1.

                /etc/rc1.d:
                K20acpi-support
                K20hddtemp
                K20rsync
                K20sysstat
                K20cdemu-daemon
                K20kerneloops
                K20saned
                K20vboxdrv
                K20fancontrol
                K20postfix
                K20smartmontools
                K20winbind
                K21spamassassin
                K25mdadm
                K74bluetooth
                K80cups
                K99laptop-mode
                S30killprocs
                S70dns-clean
                S70pppd-dns
                S90single

                Seems like statd is for NFS so should be killed

                Please Read Me

                Comment


                  #9
                  Re: Unable to unmount /var

                  Originally posted by rihad
                  Upstart's init tried its best to stop me from doing that.
                  Instead of "fighting" upstart, you could try working with it (even though it was upstart that started the fight )

                  Instead of commenting out the start blocks, you could try adding stop blocks...something like
                  stop on runlevel [1S] (stop service in single user mode)
                  or
                  stop on runlevel [!2345] (stop service in non-multiuser runlevels)

                  You might also try adding runlevel requirements in those start blocks, something like
                  and runlevel [2345]

                  Haven't had a need to tinker with upstart confs much, so I can't tell for sure whether these mods would work as is, but if you're still interested to try them out they might be worth a shot (although I understood you already worked around your immediate issues)

                  Also, gssd and statd (judging by your configs) are likely triggered by portmap starting, so stopping portmap from starting might be enough to prevent those services from starting as well. Also preventing rsyslog from starting should also keep rsyslog-kmsg from firing up.

                  Comment


                    #10
                    Re: Unable to unmount /var

                    Originally posted by rihad
                    Hi, I need to lvextend the size of an LVM2 volume that /var is on. Obviously I need to unmount /var first, so I dropped to single user shell. Alas, some OS programs are accessing /var in single-user mode preventing me to umount it, according to fuser -vm /var. Killing them solves nothing, as init keeps respawning them. I guess this is a shortcoming. Why won't [K]Ubuntu step away and let me do what I need to?

                    Confused, If using lvm just extend it while online. Why did you need to unmount the filesystem in the first place. I have extended filesystems online for a long time even var, tmp, even / without issue. I am not able to understand why you believed you needed to unmount var to do the lvextend < to the size you wanted > and do an resize2fs on the file system online resize has worked for years.

                    Comment


                      #11
                      Re: Unable to unmount /var

                      Originally posted by mfburgo
                      I am not able to understand why you believed you needed to unmount var to do the lvextend < to the size you wanted > and do an resize2fs on the file system online resize has worked for years.
                      That's because of this part of "man resize2fs":
                      The resize2fs program will resize ext2, ext3, or ext4 file systems. It can be used to enlarge or shrink an unmounted file system
                      located on device. If the filesystem is mounted, it can be used to expand the size of the mounted filesystem, assuming the kernel
                      supports on-line resizing. (As of this writing, the Linux 2.6 kernel supports on-line resize for filesystems mounted using ext3
                      only.).
                      But I'm using ext4. Even if the on-line resize does work for ext4, I wouldn't want to try it for the first time on my desktop, you know.

                      Comment


                        #12
                        Re: Unable to unmount /var

                        Originally posted by rihad
                        Originally posted by mfburgo
                        I am not able to understand why you believed you needed to unmount var to do the lvextend < to the size you wanted > and do an resize2fs on the file system online resize has worked for years.
                        That's because of this part of "man resize2fs":
                        The resize2fs program will resize ext2, ext3, or ext4 file systems. It can be used to enlarge or shrink an unmounted file system
                        located on device. If the filesystem is mounted, it can be used to expand the size of the mounted filesystem, assuming the kernel
                        supports on-line resizing. (As of this writing, the Linux 2.6 kernel supports on-line resize for filesystems mounted using ext3
                        only.).
                        But I'm using ext4. Even if the on-line resize does work for ext4, I wouldn't want to try it for the first time on my desktop, you know.
                        OK now I understand why you may have done this. I am letting you know that it does resize ext4. And I have not had issue's. <of course I always have a good backup to fall back on in the event something goes wrong>. If you have the space since your using lvm create a snapshot of the original do the resize if there is a problem fall back to the snapshot. Snapshot's are great for backup purpose in the event something goes wrong. ( also have your rescue CD ready )

                        Since you installed, I am assuming from a cd, you could also have booted into rescue mode on the cd opened a shell on your root filesystem which only mounts root and resize there if you are concerned. ( you may need to mount usr for that.

                        If you need more help let me know I will try it with sone test VM's for you and send you the steps.....

                        Comment

                        Working...
                        X