Announcement

Collapse
No announcement yet.

Saved by BTRFS and my auto-snapshot script

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Saved by BTRFS and my auto-snapshot script

    Last night I noticed my PC was acting different - my desktop seemed slow or jerky. It had been a couple days since I had re-booted and I had done several updates without rebooting so I figured before I spent time trouble-shooting I would reboot. It was dinner time, so I did not return to my office until this morning. On my monitor was about 4 seconds of boot text with "kernel panic" on the last line. The last time I rebooted I had installed 4.13.0-18 kernel and a new video driver after that and this was the result then - so I had booted into the older 4.13.0-17 kernel back then and it worked fine, but I had forgotten all about that. Without worry, I rebooted again and this time selected the -17 kernel. I saw the kernel panic again - now I worried a bit.

    If you read any of my other threads you know I'm a huge BTRFS proponent and write many posts about it's benefits. About a month ago, I wrote an auto-snapshot-and-backup script for my server so I didn't have to do any of that manually. The script runs as a cronjob and takes a daily snapshot, preserves the last three snapshots, and once a week sends a backup to two separate backup drives on the server a drive on my desktop PC - all hands off. Serendipitously, couple weeks ago, I modified the script a little and installed in on my desktop PC too.

    Along with the three daily snapshots, I have 5 other installs on my system - which all are on the same btrfs file system, no partitioning required - so I booted into Manjaro and rolled back my KDEneon install to the Tuesday snapshot, but it wouldn't boot either. Tried Monday, but also no-go. Finally booted to the Sunday snapshot - and I was back in business.

    Each rollback required:
    • Reboot to Manjaro
    • Renaming the subvolume that wouldn't boot
    • Renaming the next snapshot in the list to the subvolume name used in grub
    • Rebooting to the newly renamed subvolume.

    That's it. The whole process to rollback 3 times took less than 10 minutes.

    Additionally, every Sunday my system makes an automatic backup to another drive that is also bootable, but I didn't need to go that far this time. I keep two weekly backups and older one are automatically deleted.

    The main point here is: Without BTRFS, I might have been doing a re-install this morning. Instead, I went back to work in under 10 minutes and posted this thread. None of this - snapshots, 6 installs to the same partition, automated bootable backups - is possible without BTRFS.

    I have decided that, at least on my desktop, a smarter, more realistic snapshot schedule is in order. If I had waited one more day to reboot, the Sunday snapshot wouldn't have been available so easily. However, as luck would have it, Sunday is the backup snapshot day so it was there on the backup drive if I needed it, but I do not want to rely on luck! It is not uncommon for me to go a week or more without rebooting and if I had gone only one more day, I woudl have been restoring from a backup. A more logical approach is needed.

    I have decided a daily backup taken at the first successful login after a reboot is a better approach for my desktop. Then I will know for sure that what I'm leaving behind is bootable and my backups are as well. I am going to ponder the best way to accomplish this. Suggestions or ideas are welcome.

    Whatever I come up with, I will document it here.

    Please Read Me

    #2
    I don't know if I would go with "Daily" seems like a lot.

    weekly sounds better with the addition of just before a questionable install or large set of updates.

    and maybe keeping a known good and "sort" of recent one for backup , backup .

    I just use your service menu every now and then ,,,so my snapshots are ,,,,well older than yours , but totally usable

    VINNY
    i7 4core HT 8MB L3 2.9GHz
    16GB RAM
    Nvidia GTX 860M 4GB RAM 1152 cuda cores

    Comment


      #3
      Originally posted by vinnywright View Post
      I don't know if I would go with "Daily" seems like a lot.
      ... maybe weekly...
      A lot in what way? Clutter? It's just a a few control structures (I don't know what they're called... blocks? nodes?) and entries in directories.

      From a data perspective, seven daily snapshots take the same space as one a week ago. As would fourteen 12-hour snapshots.

      (Assuming only the latest, current, one is used and has changes. You could go back into old snapshots and muck around, maybe delete stuff, but that would be screwy.)

      Regards, John Little
      Regards, John Little

      Comment


        #4
        Accurate assessment about the use of space jlittle, besides, I have plenty of room - 460GB with around 190GB used. As far as mucking about inside snapshots - there's no problem with doing that. The results are that the snapshot grows to hold the changes you've made. I would be a silly thing to do if your intention was to retain a certain state of your OS, because obviously as soon as you muck about - it's no longer a true snapshot. it becomes a modified copy. To your comments, snapshots consume metadata space, which is reserved anyway, but zero data space until changes are made to the source subvolume. So realistically, a weeks worth of daily snapshots (just talking the OS here, not home) would only contain a weeks worth of updates and any modifications. Not really much in the way of data. A few hundred MB or even a GB isn't really much space theses days. I'd bet you could have a full Kubuntu install and a months worth of daily snapshots in 30-40GB.

        Vinny: Actually, I'm thinking rather than a time based type of schedule, something more related to actual use. Daily is really arbitrary and as my recent experience showed, insufficient to guarantee a usable snapshot. I did three dailies thinking that would protect me, but it didn't. Just blind luck that i rebooted when I did - one more day and I would have been restoring from two weeks prior - one more week and I would have been screwed. I'm thinking that if I snapshot after a successful log-in, then I'm reasonably sure that I can log in again. The question is how many to keep and how to trigger the action automatically. I can do it manually, but that defeats the point, because I'll get lazy, forgetful, or complacent eventually. Also, it seems likely that I would still want a daily because I might go a week or more without rebooting. I really only ever reboot when I install a new kernel, but htere are other things potentially going on that I might want to preserve. I'm still pondering this one.

        To the actual problem: I rolled back through a couple days of updates, but also during that time I had also removed several older kernels and added one program, PLUS updated my nvidia driver. Once I got booted back up to the Sunday snapshot, I re-did everything (AFTER making a snapshot, of course) except the nvidia driver update, and rebooted a couple time between various actions. It's working fine so either the nvidia driver caused the problem (I still haven't tried that update again) or it was a random corruption. I will eventually get around to trying the nvidia update, and if that's the cause - I'll bug report it and lock my current version. If not - I'll chalk it up to the "Ghost in the machine."

        Please Read Me

        Comment


          #5
          I don't do even do daily backups. I only backup before I do an update or install a package.
          I did a backup before I installed WPS but I didn't roll back because using dpkg -r purge worked fine. IF I encounter problems before the next major update I will roll back before I apply it.
          "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
          – John F. Kennedy, February 26, 1962.

          Comment


            #6
            The "Ghost in the machine." or the "Linux God's" seam to have been good to me over the years , as I can not remember the last time I had a catastrophic failure on one of my systems that I could not get out of .

            that said ,,,,,I have a hard time remembering last month ,,,,

            I do enjoy the warm fuzzy feeling that knowing just renaming a snapshot will give me a system back , gives one .

            VINNY
            i7 4core HT 8MB L3 2.9GHz
            16GB RAM
            Nvidia GTX 860M 4GB RAM 1152 cuda cores

            Comment


              #7
              Originally posted by vinnywright View Post
              The "Ghost in the machine." or the "Linux God's" seam to have been good to me over the years , as I can not remember the last time I had a catastrophic failure on one of my systems that I could not get out of .

              that said ,,,,,I have a hard time remembering last month ,,,,

              I do enjoy the warm fuzzy feeling that knowing just renaming a snapshot will give me a system back , gives one .

              VINNY
              So no sooner that I say this ,,,,than I get hit with a boo boo ,,,,Updates today on the Kubuntu-16.04/Neon-/dev/stable install gave me the Black screen for login , it's on an EXT4 formated partition.

              tried "mv .local .local.old1" and .config and .kde the same way ,,,,no good ,,,, upgraded the Nvidia driver ,,,,no good .

              have a working TTY and will hope for an update to fix it some time this weekend ,,,,,,But this just makes me more convinced to go ahead with my plan to give the hole system drive to BTRFS...exept for the swap and a smallish EXT4 boot partition ,,,as @oshunluvr has posted a plethora of info on

              Code:
              vinny@vinny-Bonobo-Extreme:~$ sudo parted -l
              [sudo] password for vinny: 
              Model: ATA HGST HTS725050A7 (scsi)
              Disk /dev/sda: 500GB
              Sector size (logical/physical): 512B/4096B
              Partition Table: msdos
              Disk Flags: 
              
              Number  Start   End    Size    Type      File system     Flags
              1      8225kB  323GB  323GB   primary   btrfs           boot
              3      323GB   379GB  56.3GB  primary   ext4
              4      379GB   496GB  117GB   extended
              6      379GB   436GB  57.0GB  logical   ext4
              5      436GB   496GB  59.8GB  logical   ext4
              2      496GB   500GB  4295MB  primary   linux-swap(v1)
              well I still have Kubuntu-14.04 & 17.10 , Neon-LTS & Debian-8-KDE sda3,sda1,sda1 & sda5 respectively.

              VINNY
              i7 4core HT 8MB L3 2.9GHz
              16GB RAM
              Nvidia GTX 860M 4GB RAM 1152 cuda cores

              Comment


                #8
                GRUB supports booting from BTRFS so you don't have to have a boot partition if you don't want, but it used to be that GRUB wouldn't boot from a multi-device btrfs file system. I'm not really sure if this is still true or not. If you're running from a single drive/partition, you can just install and boot.

                Since I use a multi-device btrfs file system (RAID0), my work-around was to create a dedicated GRUB partition. This solved the booting issue and facilitated the booting of an unlimited number of installations.

                I see you're using MBR drive partitioning. Interesting your first partition starts at 8225kB. I believe normally it's 1049kB. Regardless, you can stay with MBR and mash all the space together into a single BTRFS file system and multi-boot many installs from it. You probably should keep one spare partition separate in case you want to install a distro that doesn't support BTRFS or as a place to install for testing purposes. You could end up with just 3 partitions - one each of BTRFS, EXT4, SWAP.

                Please Read Me

                Comment


                  #9
                  Originally posted by oshunluvr View Post
                  GRUB supports booting from BTRFS so you don't have to have a boot partition if you don't want, but it used to be that GRUB wouldn't boot from a multi-device btrfs file system. I'm not really sure if this is still true or not. If you're running from a single drive/partition, you can just install and boot.

                  Since I use a multi-device btrfs file system (RAID0), my work-around was to create a dedicated GRUB partition. This solved the booting issue and facilitated the booting of an unlimited number of installations.

                  I see you're using MBR drive partitioning. Interesting your first partition starts at 8225kB. I believe normally it's 1049kB. Regardless, you can stay with MBR and mash all the space together into a single BTRFS file system and multi-boot many installs from it. You probably should keep one spare partition separate in case you want to install a distro that doesn't support BTRFS or as a place to install for testing purposes. You could end up with just 3 partitions - one each of BTRFS, EXT4, SWAP.
                  cool ,,,for some reason I was under the impression you had to have a boot partition with BTRFS ,,,,,that sounds like a plan , 1 each of BTRFS,EXT4 and swap .

                  thanks friend

                  VINNY
                  i7 4core HT 8MB L3 2.9GHz
                  16GB RAM
                  Nvidia GTX 860M 4GB RAM 1152 cuda cores

                  Comment

                  Working...
                  X