Announcement

Collapse
No announcement yet.

Would like some assistance

Collapse
X
Collapse
 
  • Page of 6
  • Filter
  • Time
  • Show
Clear All
new posts
Previous 1 2 3 4 5 6 template Next
    #16
    GreyGeek Okay, after changing the HDD IDs to reflect mine, and commenting out the mounting line for the external HDD (as already explained, I accept mounting when it is plugged in), I ran the script. The results:
    paul@barley-cat:~$ sudo -i
    root@barley-cat:~# cd /
    root@barley-cat:/# /make_snapshot.sh
    Mounting drives
    mount: /mnt: special device /dev/disk/by-uuid/67fab1ac-b53c-4e10-9faf-f9a434156e does not exist.
    Making today's snapshot
    Create a readonly snapshot of '/mnt/@' in '/mnt/snapshots/@202203041643'
    Finding previous snapshot as parent
    Attempting incremental backup
    btrfs send -p /mnt/snapshots/@202203041643 /mnt/snapshots/@home202203032257 | btrfs receive /backup
    At subvol /mnt/snapshots/@home202203032257
    At snapshot @home202203032257
    ERROR: parent subvol is not reachable from inside the root subvol
    Delete subvolume (commit): '/mnt/snapshots/@202203032257'
    ERROR: Not a Btrfs subvolume: Invalid argument
    Snapshots completed, oldest snapshots deleted
    umount: /backup: not mounted.
    Drives unmounted and 3 syncs done
    root@barley-cat:/#
    The unmount: /backup: not mounted is my bad. I didn't change the eval 'umount /backup' line.

    As to the to ERROR lines. Given my setup, what do I make of them?
    The mount: /mnt: error is my bad. Didn't write out the UUID correctly.
    Last edited by Snowhog; Mar 04, 2022, 05:13 PM.
    Windows no longer obstructs my view.
    Using Kubuntu Linux since March 23, 2007.
    "It is a capital mistake to theorize before one has data." - Sherlock Holmes​
    • Top
    • Bottom

    Comment

      #17
      I always used the UUID and never thought about using the label. I have given the label "BACKUP" to my internal NVMe SSD and to an external USB HDD as well. They both cannot be mounted at the same time as "BACKUP". So, before I plug in my external USB SSD labeled "BACKUP" the listing is:
      Code:
      $ vdir /dev/disk/by-label/
total 0
lrwxrwxrwx 1 root root 15 Mar 4 07:26 BACKUP -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Mar 4 07:26 DATA -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 10 Mar 4 07:26 EFI -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 4 07:26 ROOTFS -> ../../sda3
lrwxrwxrwx 1 root root 10 Mar 4 07:26 SWAP -> ../../sda2
      AFTER I plug in the external SSD the listing shows:
      Code:
      $ vdir /dev/disk/by-label/
total 0
lrwxrwxrwx 1 root root 10 Mar 4 16:46 BACKUP -> ../../sdb1
lrwxrwxrwx 1 root root 15 Mar 4 16:46 DATA -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 10 Mar 4 16:46 EFI -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 4 16:46 ROOTFS -> ../../sda3
lrwxrwxrwx 1 root root 10 Mar 4 16:46 SWAP -> ../../sda2
~$
      The NVMe is hidden. If I were to use

      Code:
      $ sudo -i
# mount BACKUP /backup                      
mount: /backup: special device BACKUP does not exist.
# mount /dev/disk/by-label/BACKUP /backup
# vdir /backup
total 0
drwxr-xr-x 1 root root 334 Feb 18 20:55 @202202182046
drwxr-xr-x 1 root root 334 Feb 23 19:11 @202202231903
drwxr-xr-x 1 root root 334 Feb 24 20:57 @202202242057
drwxr-xr-x 1 root root 334 Feb 25 19:18 @202202251913
drwxr-xr-x 1 root root 334 Feb 26 20:41 @202202262040
drwxr-xr-x 1 root root 334 Feb 28 16:10 @202202281609
drwxr-xr-x 1 root root 334 Mar  1 18:15 @202203011815
drwxr-xr-x 1 root root 334 Mar  2 22:13 @202203022213
      The external drive gets mounted.

      Code:
      /dev/sdb1 on /backup type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
      So, I could mount the internal BACKUP or the external backup using
      Code:
      mount /dev/disk/by-label/BACKUP /backup
      and which one gets mounted would be determined by plugging in the external SSD or not.
      Nice!


      PS: The UUID mistype plagued me until I restored to just using blkid in the Konsole and then C&P to put it onto the command line or into a script. Using the BACKUP label is going to change all of that. I can run the same script, first without the external SSD plugged in, and then with the external SSD plugged in.



      Last edited by Snowhog; Mar 04, 2022, 05:11 PM.
      "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
      – John F. Kennedy, February 26, 1962.
      • Top
      • Bottom

      Comment

        #18
        GreyGeek I gave my external USB HDD a label (BACKUP) and made the necessary reference changes to the script.

        I'm also unmounting the external USB HDD so I can then mount it at /backup

        This is the script:
        #!/bin/bash
        # make_snapshot.sh

        # Script to create backup snapshots to /mnt/snapshots and a differential backup to /backup
        # To be run as root from /
        #
        # Before executing this script:
        # Plug in external HDD. Select Mount in Disk & Devices pop-up
        # External drive will be mounted to /media/paul/BACKUP
        #
        echo "Mounting drives"
        # mount internal HDD to /mnt
        eval "mount /dev/disk/by-uuid/67fab1ac-b53c-4e10-9faf-f9a43941568e /mnt"
        # Unmount external HDD so it can be remounted to /backup
        eval "umount /media/paul/BACKUP"
        # mount external HDD to /backup
        eval "mount /dev/disk/by-label/BACKUP /backup"

        # capture the date and time
        NOW=$(date +%Y%m%d%H%M)
        echo "Making today's snapshot"
        MKSNAP='btrfs su snapshot -r /mnt/@ /mnt/snapshots/@'$NOW
        eval $MKSNAP
        eval 'sync'
        eval 'sync'

        echo "Finding previous snapshot as parent "
        PREVSNAP=""
        list=($(ls /mnt/snapshots/))
        PREVSNAP=${list[-2]}
        NOW=${list[-1]}
        echo "Attempting incremental backup"
        if [[ -s "/mnt/snapshots/"$PREVSNAP ]];
        then
        MKINC='btrfs send -p /mnt/snapshots/'$PREVSNAP
        MKINC=$MKINC' /mnt/snapshots/'$NOW
        MKINC=$MKINC' | btrfs receive /backup'
        echo $MKINC
        eval $MKINC
        eval 'sync'
        eval 'sync'
        eval "sync"
        DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
        eval $DELSNAP
        eval 'sync'
        eval 'sync'
        eval 'sync'
        list=''
        list=($(ls /backup))
        DELSNAP='btrfs subvol delete -C /backup'${list[0]}
        eval $DELSNAP
        eval 'sync'
        eval 'sync'
        eval 'sync'
        echo "Snapshots completed, oldest snapshots deleted"
        eval 'umount /backup'
        eval 'umount /mnt'
        eval 'sync'
        eval 'sleep 1'
        eval 'sync'
        eval 'sleep 1'
        eval 'sync'
        eval 'sleep 1'
        echo "Drives unmounted and 3 syncs done"
        else
        echo 'Incremental backup failed using '$PREVSNAP' and '$NOW
        echo 'Drives still mounted, ready to clean up'
        fi
        I start by plugging in my external USB HDD and selecting Mount in the Disk & Devices pop-up. Then I open a konsole and issue:

        sudo -i
        cd /
        /make_snapshot.sh

        This is the output from this morning:
        paul@barley-cat:/$ sudo -i
        root@barley-cat:~# cd /
        root@barley-cat:/# /make_snapshot.sh
        Mounting drives
        Making today's snapshot
        Create a readonly snapshot of '/mnt/@' in '/mnt/snapshots/@202203060645'
        Finding previous snapshot as parent
        Attempting incremental backup
        btrfs send -p /mnt/snapshots/@202203060645 /mnt/snapshots/@home202203032257 | btrfs receive /backup
        At subvol /mnt/snapshots/@home202203032257
        At snapshot @home202203032257
        ERROR: cannot find parent subvolume
        Delete subvolume (commit): '/mnt/snapshots/@202203042120'
        ERROR: Could not statfs: No such file or directory
        Snapshots completed, oldest snapshots deleted
        Drives unmounted and 3 syncs done
        Why am I getting those two ERROR's?
        Last edited by Snowhog; Mar 06, 2022, 09:17 AM.
        Windows no longer obstructs my view.
        Using Kubuntu Linux since March 23, 2007.
        "It is a capital mistake to theorize before one has data." - Sherlock Holmes​
        • Top
        • Bottom

        Comment

          #19
          Don't do this part:
          # Plug in external HDD. Select Mount in Disk & Devices pop-up
          # External drive will be mounted to /media/paul/BACKUP
          #
          All you have to do is plug the external HDD in for it to be registered in /dev/disk/by-label as "BACKUP"

          My script is for backing up just @, because I don't have a @home subvolume. I merged it into @/home and took out the line in fstab that mounted it. I wrote that script after I had merged the subvolumes.

          The script you modified and ran tried to this:
          btrfs send -p /mnt/snapshots/@202203060645 /mnt/snapshots/@home202203032257 | btrfs receive /backup
          Notice that the parent is @202203060645 BUT the child is @home202203032257
          That doesn't work because the list command:
          list=($(ls /mnt/snapshots/))
          doesn't not distinguish between @ and @home snapshots.

          IF you are going to keep both @ and @home then I would create two separate subdirectories:
          /mnt/snapshots/root
          /mnt/snapshots/home
          and store my @yyyymmddhhmm snapshots in the root subdir and @homeyyyymmddhhmm in the home subdir.
          Then, add a section of code that picks out the root parent and root child, and ditto for the home parent and the home child.
          That way the list command can properly identify the previous snapshot as parent.
          "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
          – John F. Kennedy, February 26, 1962.
          • Top
          • Bottom

          Comment

            #20
            Thank you for the explanation as to why "I" get those errors.

            The laptop is sacrificial; nothing critical on it. It's for playing/experimenting on.

            I already figured out the "Don't do this part". When I plug in my external USB HDD, the Disk & Devices pop-up appears, but I (now) just ignore it. The drive is powered and identified by the system, as you point out. So that area of the script now reads:
            # Before executing this script:
            # Plug in external HDD. Ignore Disk & Devices pop-up.
            # External drive will be powered and identified by system but not yet mounted.
            #
            echo "Mounting drives"
            # mount internal HDD to /mnt
            eval "mount /dev/disk/by-uuid/67fab1ac-b53c-4e10-9faf-f9a43941568e /mnt"
            # mount external HDD to /backup
            eval "mount /dev/disk/by-label/BACKUP /backup"
            Given my setup; separate @ and @home; how would the following part of the script be written so it all works?
            # capture the date and time
            NOW=$(date +%Y%m%d%H%M)
            echo "Making today's snapshot"
            MKSNAP='btrfs su snapshot -r /mnt/@ /mnt/snapshots/@'$NOW
            eval $MKSNAP
            eval 'sync'
            eval 'sync'

            echo "Finding previous snapshot as parent "
            PREVSNAP=""
            list=($(ls /mnt/snapshots/))
            PREVSNAP=${list[-2]}
            NOW=${list[-1]}
            echo "Attempting incremental backup"
            if [[ -s "/mnt/snapshots/"$PREVSNAP ]];
            then
            MKINC='btrfs send -p /mnt/snapshots/'$PREVSNAP
            MKINC=$MKINC' /mnt/snapshots/'$NOW
            MKINC=$MKINC' | btrfs receive /backup'
            echo $MKINC
            eval $MKINC
            eval 'sync'
            eval 'sync'
            eval "sync"
            DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
            eval $DELSNAP
            eval 'sync'
            eval 'sync'
            eval 'sync'
            list=''
            list=($(ls /backup))
            DELSNAP='btrfs subvol delete -C /backup'${list[0]}
            eval $DELSNAP
            eval 'sync'
            eval 'sync'
            eval 'sync'
            echo "Snapshots completed, oldest snapshots deleted"
            eval 'umount /backup'
            eval 'umount /mnt'
            eval 'sync'
            eval 'sleep 1'
            eval 'sync'
            eval 'sleep 1'
            eval 'sync'
            eval 'sleep 1'
            echo "Drives unmounted and 3 syncs done"
            else
            echo 'Incremental backup failed using '$PREVSNAP' and '$NOW
            echo 'Drives still mounted, ready to clean up'
            fi
            I know that's asking you to make the changes for me, and I hope you don't mind. It's that, or instruct me on how to merge (?) my @home into @ so my btrfs is setup like yours and the script would work as currently written.
            Windows no longer obstructs my view.
            Using Kubuntu Linux since March 23, 2007.
            "It is a capital mistake to theorize before one has data." - Sherlock Holmes​
            • Top
            • Bottom

            Comment

              #21
              The script I wrote for snapshotting @ would have to be duplicated and in the second part @ is changed to @home, pointing to the their respective subdirectories under /mnt/snapshots/...

              Here's the easy way.
              https://www.kubuntuforums.net/forum/...-only-root-sub
              My method was to sudo to root and mount /mnt. Then run mc and open @ /home in the left panel. In the right panel highlight jerry (@home/jerry) and use the copy command to copy jerry over to @/home. When it was done I edited /etc/fstab and put a # in front of the line that mounted @home to /home, and saved it. Then I unmounted everything and rebooted. Use your own account name, of course.

              In that thread Oshunluver posted his method, which I think is better.
              Last edited by GreyGeek; Mar 06, 2022, 03:59 PM.
              "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
              – John F. Kennedy, February 26, 1962.
              • Top
              • Bottom

              Comment

                #22
                Arrrgh...this thread is convincing me to stick with Timeshift for backups. When the new Kubuntu LTS version is out I well may install with btrfs - haven't decided. Timeshift can do btrfs backups.
                • Top
                • Bottom

                Comment

                #23
                GreyGeek I've been working on this (script) since last night, and have what I believe to be correct (for what I want "at this time"). I'd like your critical eye on it to see if you notice anything wrong.

                This script doesn't deal with incremental snapshots; maybe later; and it doesn't (yet) deal with cleaning up (deleting) older snapshots. That will be next. Baby Steps. First things first.

                #!/bin/bash
                # snapshot.sh
                #
                # To be run as root from /
                # Type: /snapshot.sh
                # To get the time taken to complete, use: time
                # Type: time /snapshot.sh
                #
                # Before executing this script:
                # Plug in external HDD. Ignore Disk & Devices pop-up. # External drive will be powered and identified by system but not yet mounted.
                #
                echo "Mounting drives"
                # mount internal HDD to /mnt
                eval "mount /dev/disk/by-label/LAPTOP /mnt"
                echo "Internal HDD mounted to /mnt"
                # mount external HDD to /backup
                eval "mount /dev/disk/by-label/BACKUP /backup"
                echo "External USB HDD mounted to /backup"
                echo ""
                # capture current date and time
                NOW=$(date +%Y%m%d%H%M)
                echo "Timestamp is $NOW"
                echo ""
                # Create snapshots
                echo "Making today's snapshot of @"
                MKSNAP1="btrfs su snapshot -r /mnt/@ /mnt/snapshots/@$NOW"
                eval $MKSNAP1
                eval 'sync'
                eval 'sync'
                echo "@$NOW successfully created"
                echo ""
                echo "Making today's snapshot of @home"
                MKSNAP2="btrfs su snapshot -r /mnt/@home /mnt/snapshots/@home$NOW"
                eval $MKSNAP2
                eval 'sync'
                eval 'sync'
                echo "@home$NOW successfully created"
                echo ""
                # Send snapshots
                SNDSNAP1="btrfs send /mnt/snapshots/@$NOW | btrfs receive /backup"
                SNDSNAP2="btrfs send /mnt/snapshots/@home$NOW | btrfs receive /backup"
                echo "Sending @$NOW and @home$NOW to external USB HDD"
                eval $SNDSNAP1
                eval 'sync'
                eval 'sync'
                eval $SNDSNAP2
                eval 'sync'
                eval 'sync'
                echo "@$NOW and @home$NOW successfully sent to external USB HDD"
                echo ""
                # Show snapshots on External USB HDD
                vdir /backup
                echo ""
                echo "Unmounting drives"
                eval 'umount /backup'
                eval 'umount /mnt'
                eval 'sync'
                eval 'sleep 1'
                eval 'sync'
                eval 'sleep 1'
                eval 'sync'
                eval 'sleep 1'
                echo ""
                echo "Drives unmounted and 3 syncs done"
                Added:
                I ran this and it completed without any errors. Sweet!
                Last edited by Snowhog; Mar 07, 2022, 04:25 PM.
                Windows no longer obstructs my view.
                Using Kubuntu Linux since March 23, 2007.
                "It is a capital mistake to theorize before one has data." - Sherlock Holmes​
                • Top
                • Bottom

                Comment

                  #24
                  Why all the "evals"? Using them stops your editor from properly syntax colouring the script. evals can be treacherous.

                  I wonder why
                  Code:
                  MKSNAP1="btrfs su snapshot -r /mnt/@ /mnt/snapshots/@$NOW"
eval $MKSNAP1
                  and not just
                  Code:
                  btrfs su snapshot -r /mnt/@ /mnt/snapshots/@$NOW
                  All those syncs look excessive. Some of my external drives are active for 20 or 30 seconds after my backup script finishes, while several GB in cache are flushed to the drives, but IIRC unmounts stall till the flush is complete. I wonder if a btrfs filesystem sync /backup is a good idea.
                  Regards, John Little
                  • Top
                  • Bottom

                  Comment

                    #25
                    That looks OK to me as for a coding is concerned.

                    However, your code still puts both @yyyymmddhhmm and @homeyyyyddhhmm into the same directory. Five days of snapshots could look something like this:

                    .../snapshots/@yyyymmd-5hhmm
                    .../snapshots/@yyyymmd-4hhmm
                    .../snapshots/@yyyymmd-3hhmm
                    .../snapshots/@yyyymmd-2hhmm
                    .../snapshots/@yyyymmd-1hhmm
                    .../snapshots/@yyyymmdhhmm
                    .../snapshots/@homeyyyymmd-5hhmm
                    .../snapshots/@homeyyyymmd-4hhmm
                    .../snapshots/@homeyyyymmd-3hhmm
                    .../snapshots/@homeyyyymmd-2hhmm
                    .../snapshots/@homeyyyymmd-1hhmm
                    .../snapshots/@homeyyyymmdhhmm
                    OR, hopefully, they could like something like this
                    .../snapshots/@yyyymmd-5hhmm
                    .../snapshots/@homeyyyymmd-5hhmm
                    .../snapshots/@yyyymmd-4hhmm
                    .../snapshots/@homeyyyymmd-4hhmm
                    .../snapshots/@yyyymmd-3hhmm
                    .../snapshots/@homeyyyymmd-3hhmm
                    .../snapshots/@yyyymmd-2hhmm
                    .../snapshots/@homeyyyymmd-2hhmm
                    .../snapshots/@yyyymmd-1hhmm
                    .../snapshots/@homeyyyymmd-1hhmm
                    .../snapshots/@yyyymmdhhmm
                    .../snapshots/@homeyyyymmdhhmm
                    The list commands in my code which identifies the oldest snapshot in a folder are
                    list=''
                    list=($(ls /mnt/snapshots ))
                    DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
                    ....
                    list=''
                    list=($(ls /backup))
                    DELSNAP='btrfs subvol delete -C /backup/'${list[0]}

                    Using the second listing the deletion code could be:
                    list=''
                    list=($(ls /mnt/snapshots ))
                    DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
                    list=''
                    list=($(ls /mnt/snapshots ))
                    DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
                    list=''
                    list=($(ls /mnt/snapshots ))
                    DELSNAP='btrfs subvol delete -C /backup/'${list[0]}
                    list=''
                    list=($(ls /mnt/snapshots ))
                    DELSNAP='btrfs subvol delete -C /backup/'${list[0]}
                    My list foo isn't good enough to remove the two oldest listings in the first set. Now you know why I recommend saving @ snapshots in .../snapshots/root/
                    and @home snapshots in
                    .../snapshots/home/




                    "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                    – John F. Kennedy, February 26, 1962.
                    • Top
                    • Bottom

                    Comment

                    • Snowhog
                      #25.1
                      Snowhog commented
                      Mar 07, 2022, 03:13 PM
                      Editing a comment
                      Yes, I'm aware of the problem my schema introduces in regards keeping both @ and @home, and not having separate directories for @ and @home snapshots in both, /mnt/snapshots and /backup. But for now, this is just for learning. Baby Steps.

                      I haven't included the process for creating incremental snapshots yet either. Again, Baby Steps.
                      Last edited by Snowhog; Mar 07, 2022, 03:15 PM.
                      • Top
                      • Bottom
                    #26
                    jlittle I'm just using what GreyGeek wrote as my starting point. As I said in my first post, btrfs is voodoo magic to me. I learn better when I have a working example. His script works for him, and my current rendition of it works for me. Can it be improved on? Most certainly! Can it be fine tuned to use the least amount of resources? Again, most certainly.

                    I'm all for having a 'lean, mean machine', so if you have suggestions on how I can improve my endeavor, please don't hesitate to suggest. (Examples most welcome!)
                    Windows no longer obstructs my view.
                    Using Kubuntu Linux since March 23, 2007.
                    "It is a capital mistake to theorize before one has data." - Sherlock Holmes​
                    • Top
                    • Bottom

                    Comment

                      #27
                      Unlike GreyGeeks use case, I don't have a need for such a detailed backup routine. The laptop isn't my production PC. What I'm really interested in is what I've accomplished so far, and that's creating full snapshots of @ and @home, and then sending them to an external storage device; my external USB HDD.

                      So, my current question is this: As I have successfully created and sent full snapshots to my external USB HDD, and I'm not (at this time anyway) going to be creating incremental snapshots, is there any reason (other than convenience and speed) for keeping the snapshots that are on my internal HDD?
                      Windows no longer obstructs my view.
                      Using Kubuntu Linux since March 23, 2007.
                      "It is a capital mistake to theorize before one has data." - Sherlock Holmes​
                      • Top
                      • Bottom

                      Comment

                        #28
                        Here is where I got my list foo from. It involves the use of curly brackets and square brackets on bash variables filled with the ls command.
                        https://www.linux.com/topic/desktop/...y-braces-bash/
                        "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                        – John F. Kennedy, February 26, 1962.
                        • Top
                        • Bottom

                        Comment

                          #29
                          GreyGeek Why do you only send incremental snapshots to /backup?
                          Windows no longer obstructs my view.
                          Using Kubuntu Linux since March 23, 2007.
                          "It is a capital mistake to theorize before one has data." - Sherlock Holmes​
                          • Top
                          • Bottom

                          Comment

                            #30
                            Because, typically, an incremental snapshot usually takes less than 20 seconds, while full snapshots can take 15-20 minutes. The result is the same. My internal backup is a 1Tb NVMe SSD. Its NVM Cntl chip has a maximum T of 70C and it begin throttling at 68.8C. The NVMe VNAND chip starts heating up slowly but continues heating up after the snapshot is done and can get as high as 68.8 as well. I use incremental snapshots to minimize the heating and because they take significantly less time. I have a heat sink on the NVMe but because of the restricted space it is only 2mm thick. I tried a 4mm thick heat sink and it kept the T at or below 52C, but it was so thick that I couldn't completely close the back cover.

                            BTW, comments do not show up on the "New Posts" or "Today's Posts" listings.
                            "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                            – John F. Kennedy, February 26, 1962.
                            • Top
                            • Bottom

                            Comment

                            Previous 1 2 3 4 5 6 template Next
                            Working...
                            X