Backing up my system completely (solved)

[eddieg780]

Re: Backing up my system completely

Thanks for your replies. Yes, sdb1 is USB. It's definitely slower. It's still backing up....

[eddieg780]

Re: Backing up my system completely

6 hours and still running backup.....

Also...How would I use this backup to restore my hard drive

[eddieg780]

Re: Backing up my system completely

It finally finished, almost 7 hours. The USB drive backup is larger than the original drive...strange?

[Qqmike]

Re: Backing up my system completely

Yes, strange would be one word that comes to mind.
Not sure what went wrong.

What do
sudo fdisk -lu
and df -h -T
tell you about both drives and partitions?

Sounds like the USB drive is using some other kind of sector size or something.
This didn't work very well.

[Qqmike]

Re: Backing up my system completely

http://www.linuxquestions.org/questi...06/page29.html

Start at Reply #423 (it's toward the end, in the last few pages of posts). There's some relevant stuff and other notes of interest.
I'm not sure how yet to fulyl sort out your experience with the USB external HDD and the time it took.

For anyone interested in the dd command, I think it would be worthwhile to read through every single reader post in that how-to. And I think I'll try to do so. Seems there are different experiences--an understatement. I do believe dd is safe, right, cool, but one has to experiment to really learn what it will do in a specific application & system. Lots of variables.

[eddieg780]

Re: Backing up my system completely

Here is what sudo fdisk -lu produced..

Disk /dev/sda: 100.0 GB, 100030242816 bytes
255 heads, 63 sectors/track, 12161 cylinders, total 195371568 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x000843e4

Device Boot Start End Blocks Id System
/dev/sda1 * 63 187333964 93666951 83 Linux
/dev/sda2 187333965 195366464 4016250 5 Extended
/dev/sda5 187334028 195366464 4016218+ 82 Linux swap / Solaris

Disk /dev/sdb: 100.0 GB, 100030242816 bytes
255 heads, 63 sectors/track, 12161 cylinders, total 195371568 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0xa315a315

Device Boot Start End Blocks Id System
/dev/sdb1 63 195366464 97683201 83 Linux

and df -h -t...

Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext3 88G 30G 55G 36% /
varrun tmpfs 1014M 116K 1013M 1% /var/run
varlock tmpfs 1014M 0 1014M 0% /var/lock
udev tmpfs 1014M 136K 1013M 1% /dev
devshm tmpfs 1014M 0 1014M 0% /dev/shm
lrm tmpfs 1014M 34M 980M 4% /lib/modules/2.6.22-14-generic/volatile

[Qqmike]

Re: Backing up my system completely

Yes, I guess we knew the geometries would be the same. You can use GParted Live CD to see if it will tell you how much of the sdb1 partition is "used." The problem--the showstopper--remains the transfer rate; or, more specifically, the imaging/backup rate, which is a function of many things (e.g., the disk controller, buffers, the USB connection, etc.).
You have done a lot of good work here. I'm not sure what to say next.
Maybe I will try this, if I can only get some time set aside to set it up...

[eddieg780]

Re: Backing up my system completely

Thank you again for your clear responses. Now I wonder how I would restore my backup if needed or say to install a larger drive. I know I would have to format the new drive, now that I know how to in linux, thanks to you, then I guess I would have to use a live CD to transfer the backup, but would the drive be bootable?

[rbrown3rd]

Re: Backing up my system completely

I am a total newbie but I ran across this. http://www.remastersys.klikit.org/ .

Probably not what you are looking for but I it might be of interest anyway.

[Qqmike]

Re: Backing up my system completely

Let me first interject this experiment:

I conducted two experiments, using dd to copy a 15 GB Kubuntu OS partition to
another 15 GB partition: internal to external, and internal to internal.
CONCLUSION:
Copy from internal HD to external USB HD ran 378 MB/minute.
Copy from internal HD to internal HD ran 888 MB/minute.
- - - - -
Details:

Experiment #1: dd copy an internal partition to an external partition

sdb6 Internal partition--15 GB (2.4 GB used)--ext3--Kubuntu 7.10
sdc2 External USB HDD partition—15GB--ext3
Copy sdb6 to sdc2:
sudo dd if=/dev/sdb6 of=/dev/sdc2 bs=3584 conv=noerror,notrunc
4388031+0 records in
4388031+0 records out
15726703104 bytes (16 GB) copied, 2502.68 seconds, 6.3 MB/s

Note: That's running 378 MB/minute; 41.71 minutes
- - - - -

Experiment #2: dd copy an internal partition to an internal partition

sdb6 Internal partition 15 GB--(2.4 GB used)--ext3--Kubuntu 7.10
sda8 Internal partition 15 GB--ext3
sudo dd if=/dev/sdb6 of=/dev/sda8 bs=3584 conv=noerror,notrunc
4388031+0 records in
4388031+0 records out
15726703104 bytes (16 GB) copied, 1059.67 seconds, 14.8 MB/s

Note: That's running 888 MB/minute; 17.66 minutes
- - - - -

Technical Note:
bs=3584
This value is close to the claimed “optimal” value of 4096; however, I chose it because it equals 7*512 => 7 sectors; a cylinder is 63*255 sectors and so is divisible by 7 sectors;
A partition contains a whole number of cylinders; at the end, if dd ends up without enough bytes to fill up a bs block size, it will stop; by choosing bs=7b (=7 sectors=3584 bytes), I ensure that dd will end on a whole number of bs block sizes (within the partition) => no bytes leftover uncopied.

- - - - -

Extrapolate to a 100 GB dd copy:

If you linearly extrapolate the dd copy rate between the internal partition and the external USB HDD partition, you'd get for 100 GB: 100 GB/ (378 MB/minute) = 264.55 minutes = 4.41 hours.

> Block size: I don't know the effect of using the smaller block size bs=512 (the default block size), but I'm under the impression (from my experience and from reading posts) that it would be slower.

> Effect of “used space” within the partition: I don't think that matters; a byte to be copied by dd is a byte to be copied by dd, and once it copies any filesystem structure, it shouldn't matter what's in a byte.


SYSTEM: Intel D915GAVL motherboard; Intel 530J processor (3 GHz); 1 GB RAM




rbrown3rd: no, not remastering (but we have good people here in this forum who work with that area )

eddieg780: I'm being pulled away here again; will return and try to address the issues you raised in your last post

[Qqmike]

Re: Backing up my system completely

Case 1: Restore the backup to sda1.

This should be a straight dd copy (just reverse the “if=” and the “of=”).
Worst case: when you re-boot, the filesystem has errors. In that case, I have seen where Kubuntu fixed them, then continued to boot. I have seen a message telling me to use, for example, sudo fsck /dev/sda1, to fix it, and it worked. In extremes, you might have to run the man pages on fsck (at a terminal, type man fsck) to learn to use some parameters for fsck, then run it. (You'll be dropped to a command line if/when this happens.) This restore image should be bootable (since you put Kubuntu right back where it was, exactly, and GRUB was set up to boot that Kubuntu.)

One of the best, to-the-point presentations of repairing K/Ubuntu filesystems I've seen:
Bigpond, home: http://users.bigpond.net.au/hermanzone/
(click to his Filesystem and Mounting page—wouldn't hurt to have access to that or print a copy)


Case 2: Instead of dd-ing sda1 to another partition, use dd to image sda1 to a (mounted) file, as we indicated above and as my how-to on dd command indicates; I've tested that and it works easily.


Case 3: Use the sdb1 image to build a new partition on a new drive.
I haven't done that yet; maybe I'll have time to do so with the image I just made in the experiments (above). This is the same issue you face with * any * backup/imaging software. When you restore that image sdb1 to a new hard drive/partition, it will not be bootable, BUT it will have GRUB files in it (since you copied them in your Kubuntu image). You will have to build the bootloader to boot the new OS installation (i.e., install GRUB to some MBR or install GRUB to the boot sector of sdb1 then boot the Kubuntu by chainloading it from any valid GRUB menu). You will also have to check the UUIDs that refer to partitions and change those UUIDs (in /boot/grub/menu.lst and /etc/fstab). Note: dd did copy the filesystem structure and format, byte for byte, so that's good.

UUIDs, listing:
From Live CD and HD: ls /dev/disk/by-uuid/ -alh
From HD: blkid

If I have time, I will try Case 3 with the internal image I made on sda8: I'll build a new partition on sdb and restore sda8 to it and then see what it takes to “hook up” the new partition to make it bootable.

What I * have * done successfully are the experiments listed in the second post of my dd command how-to. Note the example of cloning an entire, bootable flash drive (MBR and all): the clone boots right up! No problems, the sucker is ready to rock, right out of the dd copy.

My GRUB how-to would indicate the tools needed to hook GRUB up to a new Kubuntu installation:
How To GRUB Methods - Toolkit
http://kubuntuforums.net/forums/inde...opic=3081671.0

[Qqmike]

Re: Backing up my system completely

Restoring the backup and hooking up GRUB

eddieg780, here it is. It worked fine. I'll probably write this up as part of my dd how-to, but that may be awhile later. Here's the scoop--it worked just as I expected.

Restoring the cloned Kubuntu to a NEW partition and setting up GRUB * worked * perfectly well.

Re-cap:
Using dd we cloned the Kubuntu partition sdb6 to a partition sda8.

What I did next (in this post):

> Use my old GRUB setup to boot into the new Kubuntu using configfile. (If the hard drive is brand new, you can't do this--you must do all this using a live Kubuntu CD to set up GRUB as we did here.)
> Set up GRUB in the boot sector of the new sdb8 Kubuntu and boot into the new Kubuntu by chainloading.
> Set up GRUB in the Master Boot Record (MBR) of the drive sdb and boot into the new Kubuntu by chainloading.

NOTE: After doing so, in practice, you would then edit the appropriate GRUB boot menu (/boot/grub/menu.lst) to handle all this booting automatically for you.

So here we go with the details (this will help you when/if you do yours):

Next:
(I have an existing drive sdb; but this will work if sdb is a new hard drive.)

Create a new partition (using GParted):
sdb8, ext3, 17 GB
Then dd the Kubuntu image on sda8 to sdb8:

sudo dd if=/dev/sda8 of=/dev/sdb8 bs=3584 conv=noerror,notrunc
4973256+0 records in
4973256+0 records out
17824149504 bytes (18 GB) copied, 955.162 seconds, 18.7 MB/s

=> So the Kubuntu clone (on sda8) has been copied to a new partition sdb8 (=(hd1,7) in GRUB notation).

Re-boot to test it.
My old GRUB menu comes up.
Press “c” key to get a GRUB prompt grub>. Note that sdb8 = (hd1,7):
grub> root (hd1,7)
grub> configfile /boot/grub/menu.lst
then select Kubuntu from the menu.lst that pops up. (This menu.lst is from the cloned Kubuntu now on sda8. The configfile statement says, “show me the menu.lst located on the partition (hd1,7))
Error 15: File not found.
That's because that boot menu points at my “original” Kubuntu on sdb6 = (hd1,5). So, press any key to get the boot menu back again, then “e” key for edit, then highlight “root (hd1,5)” and press “e” key to edit it so it reads (hd1,7) (=sdb8). Press Enter to accept that edit. Press “b” to boot into it (as edited).
Then it boots but stops at error: “Unable to resolve UUID=etc.” (That's because our UUIDs for the partitions are all changed! Not a problem.)
Press Control+D and Enter to get past the error and boot up Kubuntu.
Log-in screen: I gave my password that I use for the original Kubuntu on sdb6.
Boot into Kubuntu.
My fstab points at the same /home partition on sdb7 (my /home is on its own separate partition).
Next steps:
Fix all incorrect UUIDs.
Change the /home for sdb8 Kubuntu if I want to.


NEXT experiment: re-boot and set up GRUB in the boot sector of the new sdb8 partition and boot into the cloned Kubuntu by chainloading.
Get the boot menu again. Set up GRUB in the boot sector of sdb8 = (hd1,7) and then boot into the cloned Kubuntu by chainloading as follows:
re-boot, get the GRUB boot menu.
Press “c” key to get a GRUB prompt.
grub> root (hd1,7)
grub> setup (hd1,7)
grub> chainloader +1
grub> boot

Then it boots into the cloned Kubuntu on sdb8 (after, again, I edit (hd1,5) to read (hd1,7) and press Control+D to get past the error “Unable to resolve UUID=etc”).


NEXT experiment: re-boot and set up GRUB in the MBR of the [new] hard drive sdb and boot into the cloned Kubuntu by chainloading.
re-boot, get the GRUB boot menu.
Press “c” key to get a GRUB prompt.
grub> root (hd1,7)
grub> setup (hd1)
grub> root (hd1) # because we want to test GRUB in the MBR (hd1)
grub> chainloader +1
grub> boot

And it worked fine (again, having to edit/Control+D past those two errors).
No problem.


To Do (TODO):
Edit relevant boot menu(s) (/boot/grub/menu.lst) to include correct entries for all devices (hdx,y) and all UUIDs and anything else you want.
If the drive sdb is brand new, after copying the sda8 Kubuntu into sdb8, you can not re-boot and get a GRUB boot menu. In this case, use Kubuntu live CD to set up GRUB in both the new Kubuntu boot sector and in the MBR of the sdb drive (using the GRUB in (hd1,7); that is, root (hd1,7)).

Piece of cake

[Qqmike]

Re: Backing up my system completely

aftermath

Just one problem: Now when I'm in "that" Kubuntu, I can't tell which it is, the original in sdb6 or the copied clone in sdb8 ... both operate on the same /home (on sdb7) ...

true, but just kidding -- I re-boot and carefully select the original Kubuntu on sdb6 to use

I'm also facing a good example of what happens with experimenting, re-partitioning, and moving OSs: I have GRUB & Kubuntu's set up * everywhere * on 2 internal hard drives (and elsewhere); but none of my GRUB boot menus work! (The device names (hdx,y) and UUIDs are all wrong because I've changed them but didn't edit menu.lst yet.)

The Rx, of course, is that I use the methods of that GRUB toolkit how-to to manually boot into Kubuntu (each time) by pressing "c" to get a grub> at the boot menu. Just proves how resilient, flexible, and easy Linux is to understand and deal with, even "on the fly."

[eddieg780]

Re: Backing up my system completely

This sure is a lot of great information. Thank you soooo much for answering my question. Looks like I have a lot of reading to do. I know it will be worth it in the end.

[Qqmike]

Re: Backing up my system completely

People (geeks, not normal people) like dd because it's a one-liner, clean, simple, powerful. To use it here, I rather prefer to make the backup of the root partition as an image * file * (not as a partition), as we discussed above at the start—like I did in the dd Command how-to (backing up your root partition in your /home directory). And you have to mount the directory into which you are making the image files (as we said above, where you got that error).

Remember also, the dcfl command from Helix Live CD—might be faster, safer, and has progress monitor on it.

And, again, there is Partimage, which a friend tells me is excellent but does require some start-up learning (as does dd).

One thing you might think about is keeping your /home as a separate partition. If you ever have to restore the root directory, you can do so easily and not wreck the home directory. Of course, you do have home backed up as part of the root, but it might be better to keep them separate (but then you have to manage the backups of both root and /home).
Partitioning—how to, Rog131:
http://kubuntuforums.net/forums/inde...opic=3090704.0