Announcement

Collapse
No announcement yet.

The dd Command

Collapse
This topic is closed.
X
This is a sticky topic.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    The dd Command

    The dd Command
    A basic guide, examples, tips & tricks, and links

    The dd command has been around for awhile but if you search on it, you'll keep running into just a handful of key references, and they all seem to be quoting each other or quoting what little is in the various man pages. The information is all spread about. What follows is a summary of sorts, where I've tried to give the subject some sort of organization. Learning dd is easy if you restrict your use to hard drives and flash drives and use the operand "bs=" (i.e., do not use separate input and output block sizes, ibs and obs), which is what I've done here--not floppies, not tapes, not optical media (dd does not write directly to optical, but there's tricks). What really counts is that you try some things using dd; that seems to be how you learn it. I have tested all the examples.

    Theory: You'll find that in Part 1. I've also tried to clarify some misconceptions about dd and to make certain conclusions definitive (where there seems to be confusion among other sources).
    Practice: You'll find examples in Part 2 and Part 3. Part 3 repeats six of the examples in Part 2, but Part 3 is expanded, more tutorial, and it is intended to support another of my how-to's: Privacy Cleanup 101 (see References).
    Privacy Cleanup 101 explains why it is sufficient to do a zero-fill (or a random number pass) on a file/partition/disk to prevent recovery of your sensitive data. You'll also find interesting additional references.
    Privacy Cleanup 101
    http://kubuntuforums.net/forums/inde....msg115011#new

    Some users may find this helpful:
    Commands at Konsole: Beginners: 3 parts
    http://kubuntuforums.net/forums/inde...opic=3091607.0

    = = = = = = = = = = = = = = = = = = = =

    New Added:
    dd: Showing the progress as dd executes
    See Post #20 below (in this thread):
    https://www.kubuntuforums.net/showth...l=1#post395805

    = = = = = = = = = = = = = = = = = = = =
    Table of Contents

    Part 1: The present post is Part 1 and contains the basics.
    Part 2 is in Post #2 and contains lots of applications/examples.
    Part 3 is in Post #3 and repeats 6 methods from Part 2, methods useful for
    my how-to Privacy Cleanup 101, and is expanded and more tutorial.

    Part 1 Basic dd tutorial and some useful Linux tools

    Section 1 Basic principles
    dd Definition
    dd Command statement
    A KEY point
    How dd works
    conv=notrunc, conv=sync, skip and seek, count=n, conv=noerror
    block size: bs bytes
    dd output: how to read it
    Diagnostics -- Examples (records in, records out)
    Tips/advice/opinions on using dd
    Notes about files (Technical)
    dd works relative to your current (working) directory you issue it from
    When to use sudo
    dcfldd
    Checking disk space and partitions: fdisk, ls, df, du
    Konsole: configure history (# lines to show)

    Section 2 Tools for doing your own experiments
    Some conclusions: result of dd copy and notrunc, sync
    Doing your own experiments to learn
    Creating test files, Looking at your results
    Useful Linux device files
    /dev/sdx, /dev/zero, /dev/null, /dev/urandom, /dev/mem
    Hexadecimal numbers
    Qalculate: a Hex calculator
    mc: Midnight Commander for reading your hex file
    Printing your Master Boot Record (MBR)
    Offsets, Print the MBR to a file, Using Midnight Commander to view MBR

    Part 2 Examples (Part 2 is in Reply #1)
    See Part 2, Reply #1, for an extensive list of examples, including:
    Zero-out the hard drive or flash drive, Zero-out the free space on a partition, Clean up your home, Clean a Partition Before Reusing It or Deleting It, Clean the Master Boot Record, Wipe a file before deleting it (to prevent recovery), Cloning a [bootable] flash drive or a hard drive, Master boot record: viewing and backup/restore, Print a partition boot sector, Creating files, Check a file for errors, Print bytes of a file to the screen, Look for non-zero entries on a drive (HDD or flash drive), Search for a string in your /home partition, Check a random part of a drive to see what's there, Write numbers to the end of a disk, md5sums and dd, Using the Bash command sync after a dd copy, What's at the end of YOUR hard drive?, Backup your Kubuntu OS to your separate /home partition, Bad disk sectors

    Part 3 dd -- Wipe partitions, home, files, disks, MBR (Part 3 is in Reply #2)
    (A tutorial for Privacy Cleanup 101)
    This duplicates material in Part 2 but is written in an expanded, tutorial style, intended to be used in conjunction with my how-to Privacy Cleanup 101.
    SECTION 1
    Introduction and philosophy
    When you delete a file, you do not delete the file: Now what?!
    SECTION 2
    How to Clean the Deleted/Unused Space on Your Kubuntu Partition or /Home
    SECTION 3
    How to Clean a File Before Deleting It
    SECTION 4
    How to Clean a Partition Before Reusing It or Deleting It
    SECTION 5
    How to Clean Your Hard Drive or Flash Drive Before Reusing It
    SECTION 6
    How to Clean the Master Boot Record, 1 sector; and 63 sectors

    = = = = = = = = = = = = = = = = = = = =
    Disclaimer
    (1) I've freely quoted, without attribution, where others have. See References!
    (2) WARNING: This is easy stuff except for one thing: Do NOT switch the input file (if) and the output file (of). If you do, you can wipe your files and your drives. dd will not ask you or warn you. No “Cancel” option in this game (other than stopping the Konsole command by issuing Control+c). Use sudo fdisk -lu to correctly identify your drive names /dev/sdx.
    = = = = = = = = = = = = = = = = = = = =

    Part 1 Basic dd tutorial and some useful Linux tools

    Section 1 Basic principles
    dd Definition
    dd Command statement
    A KEY point
    How dd works
    conv=notrunc, conv=sync, skip and seek, count=n, conv=noerror
    block size: bs bytes
    dd output: how to read it
    Diagnostics -- Examples (records in, records out)
    Tips/advice/opinions on using dd
    Notes about files (Technical)
    dd works relative to your current (working) directory you issue it from
    When to use sudo
    dcfldd
    Checking disk space and partitions: fdisk, ls, df, du
    Konsole: configure history (# lines to show)

    dd Definition: dd = convert and copy a file. The dd utility copies a specified input file to a specified output with possible conversions. dd copies from a “source” to a “target.” dd is (believed to be) an abbreviation for "data definition" in IBM's JCL.

    dd Command statement, the basic form:
    dd if=<source_file> of=<target_file> bs=<block size> skip=N1 seek=N2 conv=o1,o2,o3

    if = input file = source_file: is the data being read.
    of = output file = target_file: is where the data gets written to, gets copied to.
    bs = block size in bytes (see below); if not specified, the default is bs=1b=512 bytes (b=512 bytes).
    conv = conversion to be made (see below). conv=o1,o2,o3,... conversion options are separated by commas with no spacing.
    -- Default files: If the "if=" is omitted, dd gets its input file from the terminal screen (which is called "the default standard input"). If the "of=" is omitted, dd prints to the terminal screen (which is called "the default standard output").
    -- See man dd.

    A KEY point
    IMPORTANT: In this how-to, we are treating the special case of dd where "bs=" is used, NOT where separate input and output block sizes are used (ibs, obs). This is the common case most people would ever need dealing with files, partitions, and disks. With "bs=", dd will read an input block of size bs bytes, apply conversions (according to "conv="), and write the output block to the output file. When just the one block size is specified ("bs="), and not separate ones (ibs, obs), dd behaves just as you would expect it to behave and want it to behave. The wording of the rules (below) may at times sound confusing, but rest assured, dd (almost always) does exactly what you'd think it should do, no surprises.

    How dd works

    dd copies binary data, bit by bit, and exactly every bit, and uses buffers to get the source copied to the target; it stops at the end of the input file (when it gets to the maximum addressable sector of source_file). It reads the input one block at a time, using the block size bs. dd then processes the block of data actually returned, which could be smaller than the requested block size (e.g., if not enough data is left on the input file to make a full block of size bs bytes; or if it encounters errors in the input file). Then dd applies any conversions (conv=) that have been specified and writes the resulting data to the output in one block at a time (of size bs bytes; or less than bs bytes in case there's a partial block at the end and conv=sync is NOT specified).

    conv=notrunc Do not truncate the output

    This is not complicated at all! But because there is so much mis-information on it, I've dragged it out and addressed common questions about how it works. Basically, one can simply say that conv=notrunc means "do not truncate the output file."

    -- If you do not specify conv=notrunc, and if no "seek=" is specified, the output file will be truncated before the copy begins (which means that the output file starts with zero bytes). If you do not specify conv=notrunc, and if you do specify a "seek=N2", dd will preserve the blocks in the output file over which dd seeks (up to N2 bytes), but no other portion of the output file will be preserved. (Thus, if the size of the seek plus the size of the input file is less than the original size of the output file, the resulting output file is shortened by the dd copy when you don't use notrunc.)
    -- If you do specify conv=notrunc, it tells dd: do not truncate the output file. Thus, at the start of the dd copy operation, the output file starts out being exactly the file target_file originally specified in the dd statement. Then after dd finishes, the final resulting output file returned by dd will be the same size or larger than target_file but may be partially or totally overwritten by bytes from the input file. Thus, using conv=notrunc, dd does not truncate the output file when it begins its copying.
    -- The effect of conv=notrunc has nothing to do with the input file. notrunc is defined only in relation to the output file (i.e., whether or not to truncate target_file before the copy begins).
    -- With dd if=/dev/zero of=file2 notrunc bs=1 count=1, and when file2 already exists, dd will replace the first byte of file2 with zero and will retain the remaining bytes of file2 (beyond the first).
    With dd if=/dev/zero of=file2 bs=1 count=1, dd will replace the entire file2 with a single zero byte (so the resulting size of file2 will be 1 byte).
    -- If you omit of= in the dd statement, like this: dd if=source_file bs=<block size> [and possibly some operands for conv=], then dd will simply print to the screen what it reads from source_file (using bs and conv choices).
    -- If you write dd if=<source_file> of=<target_file> [and choices for bs and conv], but the specified target_file does NOT yet exist, then dd will create a file called target_file and perform its copy operation from source_file. The source_file is copied byte for byte to the file called target_file, the resulting size of target_file will be the same as source_file or larger (if there is a partial block copied and conv=sync is used to pad it to a full block size bs).
    -- If target_file does not yet exist before issuing dd, and so you have dd if=<source_file> of=<target_file> [and choices for bs and conv] , using conv=notrunc does not make sense, it would have no effect. notrunc is defined only in relation to the output file.
    -- If target_file exists before issuing dd, and so you have dd if=<source_file> of=<target_file> [and choices for bs and conv], and target_file is smaller than source_file, when dd is done, target_file will be identical to source_file, with or without using conv=notrunc.
    -- If target_file exists before issuing dd, and so you have dd if=<source_file> of=<target_file> [and choices for bs and conv], and target_file is larger than source_file, when dd is done, target_file will be identical to source_file if you did NOT use conv=notrunc; if you used conv=notrunc, target_file will be the same size as it was before dd, but its contents (bytes) at the beginning will be overwritten by the contents of source_file (and target_file will be the same as it was beyond those bytes from source_file).
    -- Everything said here applies to disk devices /dev/sdx and partitions /dev/sdxn. For example, if disk sdx is 500 MB and has unallocated space except for a 100 MB partition, when you copy sdx to an image file, the entire 500MB will be copied, even though 400 MB are empty or unused or unallocated, and whether or not you use conv=notrunc. (It doesn't matter, 500 MB gets copied IF there is room; i.e., the unused parts of sdx also get copied with or without notrunc). If you copy a 500 MB disk to an existing 700 MB file called imagefile, without using notrunc, all 500 Mb wil get copied and the final size of imagefile will be only 500 MB; if you use notrunc, the final size of imagefile will be 700 MB but the first 500 MB will be identical to the disk. There's a lot of confusion about this, but you can test it for yourself (using the methods we discuss below). (If you try to copy a big partition to a small partition, with or without notrunc, there will not be enough room and the copy will stop when the small partition is full. Same goes for copying disks. But for "regular" files (not device files), if you copy a big file to a small file, the small file will expand and fill up with a copy of the input file.)

    conv=sync
    Upon reading the input file, dd may return less than a full block of bs bytes. This could happen if there is not enough data left on the input file to fill up a block of bs bytes. It can also happen even when the input file has plenty of bytes (e.g., /dev/zero has infinitely many zeros), and the target_file is almost full toward the end of the copy but requires only a partial input block to fill it up completely. It could also happen if dd encounters an error (in the input file) while reading the input file and thus can not read a full block size of bs bytes.
    -- If dd returns less than a full block (from reading the input file) and the conv=sync conversion is not specified, dd will write whatever it gets to the output file (something less than a full bs-byte block size).
    -- If dd returns less than a full block (from reading the input file) and the conv=sync conversion is specified, null bytes are appended to the input data up to the block size bs bytes. Any remaining conversions and output will include the pad characters as if they had been read from the input. Thus, when conv=sync is used, all output blocks written to the output file will always be bs bytes in size (even if they have to be padded with null bytes to make that happen).

    skip and seek
    skip=n tells dd to skip n blocks of size bs bytes at the beginning of the source_file before starting to copy.
    seek=n tells dd to skip n blocks of size bs bytes from the beginning of the output file before copying (i.e., seek skips over n blocks on the output target_file before writing to target_file).

    count=n
    Copies only n input blocks (each of size bs bytes). (You may append this anywhere in the dd statement.)

    conv=noerror
    Noerror means to keep going if there is an error. Normally dd stops at any error. When an input error occurs, a diagnostic message is written on standard error (i.e., the terminal screen), followed by the current input and output block counts in the same format as used at completion. If the sync conversion is specified, the missing input is replaced with null bytes and processed normally.
    If you omit conv=noerror: The "bad" part of the input block will be omitted from the output. Any partial output block is written to the output file, a diagnostic message is written, and the copy operation stops.
    Advanced note: If you are copying a filesystem, using conv=noerror could give you problems since the output file may then have a damaged filesystem.

    block size: bs bytes
    If omitted, the default is 512 bytes = 1b, where b=512.

    bs can be any number of bytes; often, a power of 2 is used: 512, 1024 (=2b), 2048, 4096, etc; a number may end with k, b, or w to specify multiplication by 1024, 512, or 2, respectively, (w = two bytes = a "word"). Numbers may also be separated by x to indicate multiplication (e.g., 2bx8). b: 512, kB: 1000, K: 1024, MB: 1000x1000, M: 1024x1024, GB: 1000x1000x1000, G: 1024x1024x1024. You can use any block size bs you wish. If bs is too big, it may slow dd down, having to read, work with its buffers, then write large blocks. Of course, if bs is too small, dd may also slow down because it has so many read-write's to do.
    > Some block sizes to try are bs=512 (the default: 512 bytes = 1 sector of the disk); bs=4096 (= 8 sectors); and bs=63x255b (=63x255x512 = 8,225,280 bytes = 1 "cylinder").

    dd output: how to read it
    -- Exit status
    Exit values are returned: 0 The input file was copied successfully. >0 An error occurred. If an input error is detected and the noerror conversion isn't specified, any partial output block is written to the output file, a diagnostic message is written, and the copy operation stops. If some other error is detected, a diagnostic message is written and the copy stops.

    -- Diagnostics
    f+p records in (out)
    => number of full (f) blocks read and partial (p) blocks (written)
    (a "record" is the same as a block as specified by" bs=")

    Examples
    For each example, assume test1 is a file of size 100 bytes; test2 is 1000 bytes
    dd if=test1 of=test2 conv=notrunc bs=20
    5+0 records in
    5+0 records out
    100 bytes (100 B) copied, 5.5989e-05 s, 1.8 MB/s
    Interpretation: 5 full blocks of 20 bytes each (and no partial blocks) were read from test1; 5 full blocks (and no partial blocks) were written to test2. Total: 100 bytes copied (to test2). Result: test2 is still 1000 bytes because of notrunc, but its first 100 bytes are identical to the 100 bytes of test1.

    dd if=test1 of=test2 conv=notrunc bs=30
    3+1 records in
    3+1 records out
    100 bytes (100 B) copied, 5.5989e-05 s, 1.8 MB/s
    Interpretation: 3 full blocks of 30 bytes each (and one partial block) were read from test1; 3 full blocks (and 1 partial block) were written to test2. Total: 100 bytes copied (to test2). The partial block was 10 bytes. Result: test2 is still 1000 bytes because of notrunc, but its first 100 bytes are identical to the 100 bytes of test1.

    test3 is a file of 1000 bytes.
    test4 is a file of 100 bytes.
    dd if=test3 of=test4 bs=30 conv=sync
    33+1 records in
    34+0 records out
    1020 bytes (1.0 kB) copied, 0.000238582 s, 4.3 MB/s
    Interpretation: 33 full blocks of 30 bytes each (and one partial block) were read from test3; 34 full blocks (and 0 partial blocks) were written to test4. Since, for test3 (which is a 1000-byte file), 33x30=990 bytes, the partial block was 10 bytes. Since conv=sync was used, the partial block of 10 bytes was padded using 20 null bytes to make it a full block of 30 bytes. Thus, 34 blocks were then written to test4. Total: 1020 bytes copied (to test4) (since 34x30=1020). Result: test4 is now 1020 bytes: the first 1000 bytes are identical to the 1000 bytes of test3; the remaining 20 bytes are the (padded) null bytes (hex zeros).

    Tips, advice, opinions on using dd
    dd is used for many things, it's a very handy tool. Experts may be able to make it do whatever they want. This how-to will give you plenty of interesting and useful applications. However, be aware of the following:
    > dd is not filesystem-aware. Furthermore, there are some technical/logistical issues when copying a partition to another partition, even if the target is the same size as the source. These issues involve metadata, truncation (or it reverse, padding block devices with nulls and the effect that has on filesystems), what happens when copying a larger partition to a smaller one (and vice versa), and issues involving the partitions tables and the bootloader. I would not recommend using dd to copy one partition to another; however, people do it. See the References for some help on this. For copying filesystems, I would use a filesystem-aware utility (e.g., Gparted, Partimage).
    > However, dd is very nice to use when cloning an entire disk. A good example is using dd to clone a flash drive (even a bootable one). dd will copy everything exactly: the filesystem, the Master Boot Record, the bootloader setup. The target will be ready-to-go.
    > When cloning a disk, (in my opinion) you should do so to an identical, same-size disk:
    dd if=/dev/sdx of=/dev/sdy [and if you wish to use it: conv=noerror], where sdx and sdy are two same-size disks (hard drives or flash drives).

    Notes about files

    Some of this is too technical for now, but make a note of it and return to it later after you study dd.
    > Do not use dd to copy files between file systems having different block sizes (the filesystem block size is different from the bs we are talking about in dd).
    > dd does not work with directories; it copies files. (Note that a Linux block device is a file; e.g., /dev/sda.)
    > You can't write a file directly to a device. You have to write the file to a file in a directory. For example, to write file myfile to /dev/sdc2, you need to mount sdc2, as say /media/sda2, then do something like dd if=/path/to/myfile of=/media/sdc2/myfile. (In particular, if you write dd if=/path/to/myfile of=/media/sdc2/, or try to use of=/dev/sdc2, dd will complain that /media/sdc2 or /dev/sdc2 is a directory (i.e., it is not a file).)
    > When working on device files (/dev/sdxn) using dd, they need not be mounted (except see the previous point). You can copy bytes from sdx without mounting it (e.g., its MBR). Sometimes, if you are getting errors, it is recommended you unmount it if it is mounted.
    > Device files obey just as other files when using dd to copy. To get a file image of a disk:
    dd if=/dev/sdx of=diskimagefile. If diskimagefile doesn't already exit, it will be created (in your current working directory; or use a path: /path/to/diskimagefile). If diskimagefile exists before issuing the dd command, and if you don't use conv=notrunc, then after dd, the resulting file diskimagefile will be the same size as sdx. If diskimagefile exists before issuing the dd command, and if diskimagefile is larger than sdx, and if you use conv=notrunc, then after dd, the resulting file diskimagefile will be larger than sdx, and its contents at the beginning will be replaced by the contents of sdx, and the bytes of diskimagefile not affected by the dd will be preserved (following the image of sdx embedded in diskimagefile).

    dd works relative to your current (working) directory you issue it from
    For example, if you use
    cd /
    and then issue
    dd if=file1 of=file2
    then file2 will be located directly under root (at the top level of the root filesystem: /file2).
    If you use
    cd ~ (or: cd /home/your_name)
    to change into your home directory, and then issue
    dd if=file1 of=file2
    then file2 will be located in your home. If you do: cd Desktop and issue dd if=file1 of=file2, then file2 will be located in your Desktop (/home/your_name/Desktop/file2).
    Now suppose you are at the level of / (cd /), and you want file2 to be saved to your Desktop. Then issue it this way:
    dd if=file1 of=/home/your_name/Desktop/file2
    That is, give the absolute path (the full path) to file2.

    When to use sudo
    Use sudo whenever you receive the dd error: Permission denied. That means you are trying to read, write, access some file for which you do not have permissions (usually owned by root). Examples include using device files (/dev/zero, /dev/urandom), reading the MBR, writing to devices (not owned by you), etc. Form: sudo dd if=file1 of=file2.

    dcfldd
    dcfldd is based on dd and is an enhanced version of dd. It was developed at USA Dept. of Defense.
    dcfldd includes progress reporting and verification, as well as various methods for ensuring accuracy and speed. To get dcfldd, use your package manager, or:
    sudo apt-get install dcfldd
    See: man dcfldd. Syntax and many command-line options are the same as for dd.
    Suggestion: If you are doing a lot of zero-fills of large drives, you might want to try dcfldd. You can run it just as you do dd, from a live CD, from GParted Live CD, from Kubuntu, even from a flash drive. In place of dd in the recipes, substitute dcfldd. Use dcfldd, also, if you find that dd dropped the last sector of a source file.

    dcfldd: references
    > dcfldd(1) - Linux man page
    http://linux.die.net/man/1/dcfldd
    > See also, dev notes re verify:
    http://www.networksecurityarchive.or.../msg00004.html
    > dcfldd - Latest version 1.3.4-1
    http://dcfldd.sourceforge.net/

    Checking disk space and partitions
    fdisk, ls, df, du

    To check your partitions and their sectors:
    sudo fdisk -lu
    To check disk space, use ls, df, du:
    ls and du: work relative to the directory you issue them from
    df: works from any directory you are presently in

    ls -al See man ls.
    List all files (in the present working directory) in long form and shows their size. A quick way to check things. First, change directories (cd) into the directory (folder) you wish to check.

    df -hT See man df.
    A quick way to see your mounted filesystems, the space Used and Available, your root (/) and /home directories. You can issue this from any directory and get the same result.

    du See man du.
    Shows you the disk usage of each file. Change into the directory you wish to check. du will check it and subdirectories recursively. So you run it, study the output, pick a directory from the output to investigate further, change directories into that directory (using cd), run du again, study the output, etc. Try these now, first changing into the root directory (cd /), then after changing into your home directory (cd ~ , or: cd /home/your_name).
    du -sh is a summary (-s) only, in human readable form (-h)
    du -h shows directory details (and subdirectories).
    du -ah shows (all = -a) files
    See next topic about configuring Konsole.

    Konsole: configure history (# lines to show)
    When you issue a command that prints a large number of lines, you can scroll up to see lines that have scrolled off the screen; or use Up arrow or Page Up. The default is to show you 1000 lines this way. If you need more (e.g., using du in certain cases), you can configure Konsole to show them all:
    Open Konsole, Settings, History, History Configuration, Set Unlimited, OK.

    Section 2 Doing your own experiments
    Some conclusions: result of dd copy and notrunc, sync
    Doing your own experiments to learn dd
    Creating test files
    Useful Linux device files
    Hexadecimal numbers
    Qalculate: a Hex calculator
    mc: Midnight Commander for reading your hex file
    Printing your MBR: dd, hex numbers, offsets, midnight commander

    Some conclusions: result of dd copy and notrunc, sync
    A good way to learn dd is to practice, using some simple test files. You will quickly get the feel of it. If you study the basics in Section 1, you'll see how dd works in common situations. Here's a few patterns you should be able to think through. If you'd rather not read this, that's OK! Skip to the next subsection: "Doing your own experiments to learn."

    Let's say you have
    dd if=fileA of=fileB bs=? conv=?,?
    where A and B are regular files of finite size.
    If you do NOT use conv=notrunc, then the result of dd will be a fileB of the same size of larger than fileA:
    If conv=sync is NOT used, fileB will be identical to fileA.
    If conv=sync is used, and bs divides evenly into the size of fileA,
    then fileB will be identical to fileA.
    If conv=sync is used, and bs does NOT divide evenly into the size of fileA,
    then there will be a partial block copied to fileB, and
    the last block copied will be padded with nulls and copied to fileB
    making fileB somewhat larger than fileA.
    fileB will be an even multiple of bs, but will contain some nulls at the end.
    If you use conv=notrunc, then the result of dd will depend:
    If fileA is smaller than fileB, it's bytes will be copied to fileB.
    If there is a partial block at the end of fileA,
    and if you used conv=sync,
    it will be padded with nulls and copied to fileB.
    fileB will contain its original bytes except the bytes at the beginning
    will have been overwritten with bytes from fileA,
    possibly with some nulls added (if you used sync and
    if there was a partial block at the end of the fileA read).
    If fileA is larger than fileB,
    file B will be overwritten with the contents of fileA
    possibly with some nulls added (if you used sync and
    if there was a partial block at the end of the fileA read).

    Doing your own experiments to learn
    Creating test files
    Looking at your results

    On your Desktop (or wherever you like to work where things will be clear to you), create some test files to play with. Say file1 with 100 bytes, file2 with 500 bytes. Then try various statements:
    dd if=file1 of=file2 bs=? conv=?,?; or try a copy from file2 to file1 (big to small),
    testing different values for bs and different combinations of conv=notrunc and conv=sync.
    Choose bs values that are large, small, some that divide evenly into the size of file1, and some that do not divide evenly into the size of file1.

    Creating test files
    To create a file of 100 zero bytes:
    Change into the directory you want to be in (e.g., cd Desktop).
    dd if=/dev/zero of=testfile bs=1 count=100
    To create a file of 100 random (numbers) bytes:
    dd if=/dev/urandom of=testfile bs=1 count=100
    Another useful file, especially for experimenting with skip and seek, might be this:
    On your Desktop, right-click to Create a New text file. Then type lines into it that you will be able to identify while testing:
    1111111111
    2222222222
    3333333333
    etc.

    Looking at your results
    In some cases, simply looking at the files on your Desktop will be revealing about the result of a dd copy. (Remember to right-click, Refresh Desktop.) Otherwise, use Midnight Commander (see below), keeping it open as you experiment.

    Useful Linux device files
    The /dev directory contains the special device files for the Linux devices. The device files are created during installation.

    /dev/sdx (e.g., sda, sdb, sdc, ...)
    These are your drives (hard drive, external drive, flash drive, etc.). There may also be /dev/hda, /dev/hdb, etc.

    /dev/zero
    This is a simple way of getting many 0s. Every time you read from this device it will return 0. This can be useful sometimes, for example when you want a file of fixed length but don't really care what it contains. It is a character device.

    /dev/null
    The bit bucket. A black hole where you can send data for it never to be seen again. Anything sent to /dev/null will disappear. This can be useful if, for example, you wish to run a command but not have any feedback appear on the terminal. It is a character device. You can use null to find the size (in bytes) of a drive or flash drive:
    sudo dd if=/dev/sdc of=/dev/null bs=4096
    123007+1 records in
    123007+1 records out
    503840256 bytes (504 MB) copied, 41.9472 s, 12.0 MB/s
    Thus, the drive has 503,840,256 bytes.
    dd read the drive sdc, reading in blocks of 4096 bytes at a time, recorded that fact (in its own registers), dumped those bytes into the /dev/null black hole, then grabbed another block of 4096 bytes to read and count (as a record in - record out). No harm is done to the drive /dev/sdc; it remains the way it was before dd. The dd output shows us how many bytes were successfully read, and we take that as the number of bytes we have to work with in sdc.

    Random number generators (see man urandom)

    /dev/random (“true” random numbers; it uses and is limited to current entropy pool)
    /dev/urandom (uses pseudo random numbers, unlimited)

    To make a file of 100 random bytes, many ways to do it:
    dd if=/dev/urandom of=/home/my_name/myrandomfile1 bs=100 count=1
    or, dd if=/dev/urandom of=/home/my_name/myrandomfile2 bs=1 count=100

    /dev/mem is your system memory. mem is a character device file that is an image of the main memory of the computer. See man mem.

    See: The Linux System Administrator's Guide, Section 3.4
    http://tldp.org/LDP/sag/html/index.html

    Hexadecimal numbers

    You run into these numbers quite often in Linux. At the very least you should recognize them and be able to convert them to familiar decimal numbers. In Linux, they usually are represented by the prefix 0x for numeric constants represented in hex; as in 0x64A2, where the hexadecimal number is 64A2. Then, you should be able to convert it to decimal if necessary, and for that most people use a hex calculator or hex converter. See the following topic: Qalculate.

    Optional: What is a hexadecimal number?
    You are familiar with decimal numbers, also called base 10 numbers. The basic numerals are the 10 numbers 0, 1, 2, ..., 9. Other decimal numbers are products of these and powers of 10 and sums of such products. The powers of 10 are (where ^ indicates exponent or power): 10^0 = 1, 10^1 = 10, 10^2 = 10x10 =100, 10^3 = 1000, 10^4 = 10000, 10^5 = 100000, and so on. The decimal number 126 (as an example), is representable as (starting from the rightmost digit) 126 = 6*(10^0) + 2*(10^1) + 1*(10^2) = 6*1 + 2*10 + 1*100 = 6+20+100 = 126.

    The hexadecimal system works the same way, exactly, except instead of "10" we use "16" as the base. The basic numerals are the 16 numbers 0, 1, 2, 3, ... 9, 10, 11, 12, 13, 14, 15. However, using 10-15 is confusing, so those numbers are represented by letters: A (10), B (11), C (12), D (13), E (14), and F (15), or their lower case a, b, c, d, e, f (so 5EB52 is the same as 5eb52). Powers of 16 are: 16^0 = 1, 16^1 = 16, 16^2 = 16x16 = 256, 16^3 = 16x16x16 = 4096, 16^4 = 65536, 16^5 = 1,048,576, and so on. A hexadecimal number is also an expansion of powers of 16, starting from the right. Example: Hexadecimal 21B = B* (16^0) + 1*(16^1) + 2*(16^2) = 11*1 + 1*16 + 2*256 = 539 (in decimal equivalent).

    By the way, all bases work this way. For example, binary numbers are base 2, the basic numerals are 0 and 1, and a binary number represents an expansion using powers of 2. Example: 1111 (a binary number) = (starting from the rightmost "1") 1*(2^0) + 1*(2^1) + 1*(2^2) + 1*(2^3) = 1*1 + 1*2 + 1*4 + 1*8 = 1 + 2 + 4 + 8 = 15.
    In binary, 0000 through 1111 represent decimal numbers 0 to 15 (the basic hex numbers). Thus each hexadecimal digit can be represented by four binary digits. You can easily convert binary to hexadecimal using this fact. Example: the binary number 01011110101101010010 (binary) = 0101 1110 1011 0101 0010 = 5 14 11 5 2 = 5EB52 (base 16). Using a hex calculator (see below), the hexadecimal number 5EB52 = 387922 (decimal).
    See
    http://en.wikipedia.org/wiki/Hexadecimal

    Qalculate: a Hex calculator
    You can use Qalculate to quickly convert between the number bases 16 (hexadecimal) and 10 (decimal) (and also octal base 8 and binary base 2). Use your package manager to install Qalculate or use sudo apt-get install qalculate. Then find it in K > > Applications > Science [or Math] > Qalculate. Open Qalculate and then File > Convert Number Bases.

    mc: Midnight Commander for reading your hex file
    If you goof around experimenting enough, it is handy to be able to view a file written in hexadecimal representation as a "regular" (ASCII) file. A quick way to do this is to use the file manager Midnight Commander. Install it using your package manager. To use it, open Konsole, type its command name and press Enter:
    mc
    You'll see right and left panes, each showing your file system. You may type a pathname after your prompt (at the bottom of the screen); or simply use arrow keys to navigate to your a file or double-click on a directory (folder) to open it (or hit Enter to open it). Your mouse also works in mc. To return to the previous menu, highlight Up--Dir and hit Enter. When you find the file you wish to view, highlight it, click View (at the bottom), then click Hex or Ascii (try going back and forth). The addresses (offsets) are given down the leftmost column under View > Hex. When you are done with the file, click Quit (perhaps twice, depending on which Quit you use). If you've changed a file, to view its new version in mc, navigate to the window containing the file (say in the left pane), then click Left > Rescan. See the next example: Printing your MBR.
    When doing this work, you may want to open two instances of Konsole. Use mc to open one of them and keep it open to mc (using Left > Rescan as necessary). Open another instance of Konsole to use for your dd commands.

    Printing the Master Boot Record (MBR)
    using dd, hex numbers, offsets, midnight commander
    Print MBR to the screen; print MBR to a file

    Master Boot Record = MBR = the first 512 bytes (i.e., the first sector) of the disk.
    446 bytes is for the IPL (Initial Program Loader: usually GRUB in Linux)
    64 bytes is for the partition table (= 16 bytes for each of the four primary partitions)
    2 bytes for the “55 AA” signature (the "end of file" for the MBR)
    Total = 512 bytes

    Since you are accessing your MBR (on a device owned by root), you'll need sudo:
    To print the MBR to the screen (the default output file of=target_file):
    sudo dd if=/dev/sdx count=1 | hexdump -C

    where /dev/sdx is your hard drive (or flash drive).
    To find sdx, use sudo fdisk -lu.
    (If you forget to write count=1, you can stop the output using Control+c.)
    The dd statement reads just one sector (count=1, and the default bs=512 is used). It then pipes (that's the vertical bar) that output to be the input of the hexdump program (see man hexdump), and hexdump prints its result to the default standard output, the screen. This is a standard technique you will see often.

    Note the output Diagnostics:
    1+0 records in
    1+0 records out
    512 bytes (512 B) copied, 0.00650396 s, 78.7 kB/s
    (makes sense--see Section 1: How to read dd output)
    The eight-digit numbers running down the left column are offsets
    There are 32 rows, 16 columns, and so 512 entries in the table, each entry representing one byte (= 8 bits = 1 character; note that each of the 512 8-bit entries is represented as two hex digits). The 512 characters in the table are in hexadecimal; their "normal" ASCII equivalents are shown in the right column (which DOES rather run together!). Notice the last two bytes 55 aa. They signify "end-of-file" or EOF for the MBR. You can find such EOF's all over your disk if you have many partitions. You may be able to make out some familiar words in the printout (right column). Compare this to using Midnight Commander (next, below). By the way, don't worry if you see something like
    | G |
    | RUB .Geom.Hard D |
    | |isk.Read. Error . |
    (It seems to be normal. Note "GRUB.")

    Offsets
    The offset of the first line is 00000000, indicating zero bytes from the start of the disk. Then count 16 bytes in that line, from left to right (1 through 16).
    The offset of the last line is 000001f0, a hexadecimal number. Use Qalculate to convert it to decimal 496 (bytes), which indicates that the first entry in the last row is at 497 bytes; the last entry, aa, is at 512 bytes (from the start of disk sdx).

    To print the MBR to a file
    sudo dd if=/dev/sdx of=/home/your_name/mbr_copy bs=512 count=1

    The dd statement reads the MBR and writes it to the file in your home named mbr_copy.
    Note that for fun this time, we explicitly indicated the block size bs=512; but since that's the default, we can also omit writing that. If you look in your home, you'll see that file. Right-click > Properties shows it is 512 bytes. Try to open it using Kate: right-click > Open with > Utilities > Kate, and you'll be warned it is a binary file and not to mess with it. Go ahead and have a look and see if you see anything familiar, then close it.

    Using Midnight Commander, mc to view the file
    Open Midnight Commander by typing at Konsole mc.
    Using arrow keys, mouse, or by typing the filename /home/your_name/mbr_copy at the prompt in mc (toward the bottom of the screen), highlight the file, click View, then Hex, then try Ascii. This is the same output as above. Midnight Commander enables you to "read" (sort of!) the binary file.

    = = = = = = = = = =
    References

    Commands at Konsole: Beginners: 3 parts
    http://kubuntuforums.net/forums/inde...opic=3091607.0

    Part 3 is intended to support Privacy Cleanup 101 and is referenced by it.
    Privacy Cleanup 101
    http://kubuntuforums.net/forums/inde....msg115011#new

    dd
    http://en.wikipedia.org/wiki/Dd_(Unix)

    IEEE specs, dd
    http://www.opengroup.org/onlinepubs/...lities/dd.html

    Excellent man page:
    Sun Microsystems Documentation (one that everyone quotes, along with man dd)
    http://docs.sun.com/app/docs/doc/817...=logins&a=view

    An excellent source, with links (choppy, incomplete, but lots of info):
    http://www.softpanorama.org/Tools/dd.shtml

    Learn the dd Command (Author: AwesomeMachine)
    http://www.linuxquestions.org/questi...ommand-362506/
    See this site if you want to copy a partition to another partition. The how-to is in the first two posts. Also read the posts of the thread where users have raised some good issues (use the search feature on your browser). The thread contains many dozens of examples and good discussion. Site is maintained/updated.

    dd is part of GNU Coreutils:
    Coreutils, dd
    http://www.gnu.org/software/coreutil...nvocation.html

    Icewalkers, man pages
    http://www.icewalkers.com/Linux/ManPages/dd-1.html

    Hexadecimal number base
    http://en.wikipedia.org/wiki/Hexadecimal

    Linux file system and device files
    See: The Linux System Administrator's Guide, Section 3.4
    http://tldp.org/LDP/sag/html/index.html

    dcfldd
    > dcfldd(1) - Linux man page
    http://linux.die.net/man/1/dcfldd
    > See also, dev notes re verify:
    http://www.networksecurityarchive.or.../msg00004.html
    > dcfldd - Latest version 1.3.4-1
    http://dcfldd.sourceforge.net/

    dd_rescue: Variation of dd for rescuing data off defective hard drive with bad sectors. http://www.garloff.de/kurt/linux/ddrescue/

    Damaged disks, damaged MBR:
    http://www.cgsecurity.org/wiki/Damag...using_.27dd.27
    http://www.cgsecurity.org/wiki/TestDisk

    Two excellent papers on bad sectors and repairing them are:

    Bad Sector Blues:
    The Dreaded "error=0x40 { UncorrectableError }"
    http://www2.uic.edu/~aciani1/sector_blues.html

    Bad block HOWTO for smartmontools
    http://smartmontools.sourceforge.net/badblockhowto.html
    Last edited by Qqmike; Nov 27, 2016, 08:52 PM.
    An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

    #2
    Re: dd Command

    PART 2 dd Applications: Examples

    Table of Contents

    Example 1 Zero-out the hard drive or flash drive
    Example 2 Zero-out the free space on a partition
    Example 3 Clean up your home
    Example 4 Clean a Partition Before Reusing It or Deleting It
    Example 5 Clean the Master Boot Record, 1 sector; or Clean 63 sectors
    Example 6 Wipe a file before deleting it (to prevent recovery)
    Example 7 --blank for now--
    Example 8 Cloning a [bootable] flash drive or a hard drive
    Example 9 Master boot record (MBR): viewing and backup/restore
    Example 10 Print a partition boot sector
    Example 11 --blank for now--
    Example 12 Creating files
    Example 13 Check a file for errors
    Example 14 Print bytes of a file to the screen
    Example 15 --blank for now--
    Example 16 Find the size of a drive or flash drive
    Example 17 Look for non-zero entries on a drive (HDD or flash drive)
    Example 18 Search for a string in your /home partition
    Example 19 Check a random part of a drive to see what's there
    Example 20 --blank for now--
    Example 21 Write numbers to the end of a disk
    Example 22 md5sums and dd
    Example 23 Using the Bash command sync after a dd copy
    Example 24 What's at the end of YOUR hard drive? -> See Reply #3 of this thread
    Example 25 Backup your Kubuntu OS to your separate /home partition
    Example 26 --blank for now--
    Example 27 Bad disk sectors

    Tips
    > For each application, open Konsole and change to the directory in which you want to work (use cd). See Part 1, Section 1: "dd works relative to the your current directory" For example, to change to your Desktop: cd Desktop. Or, use complete paths; to change to a directory you set up in your home called TestFolder: cd /home/your_name/TestFolder.
    > Midnight Commander (mc): see Part 2 for using it.
    > Manual pages. To learn about command xyz, open Konsole and issue the command: man xyz.
    > For drives, flash drives, we use /dev/sdx. You should run sudo fdisk -lu to see your drives and identify the correct one to use in your dd commands.
    > Be careful! Don't zero-out or overwrite the wrong drive!


    Example 1 Zero-out the hard drive or flash drive
    -> Clears it to like-new state
    From Live CD or from Kubuntu/Linux in another partition:

    sudo dd if=/dev/zero of=/dev/sdx conv=notrunc

    Details
    Use the command fdisk -lu to check the drives, identify the drive to zero-out. Suppose it is sdx.
    To write zeros to the sdx drive:
    sudo dd if=/dev/zero of=/dev/sdx conv=notrunc
    It took about 45 minutes to complete this on a 160 GB SATA drive. You will not get a progress meter from dd. If you use a variation of dd called dcfldd, you will get a progress meter. Install dcfldd using your package manager in the Live CD session or simply by: sudo apt-get install dcfldd. See Part 1, Section 1 "Basic principles -- dcfldd."
    -> Afterwards, use GParted Live CD to re-partition the drive, beginning with setting a new Disk Label (select the standard default MSDOS Label when prompted, unless you are using GPT for 2 TB drives).

    Advanced note: I can see no reason why you need conv=notrunc, but many authors use it. I think there's a lot of misinformation out there. Doesn't hurt to leave it in, but I don't use it. The statement is the same as sudo dd if=/dev/zero of=/dev/sdx bs=512 conv=notrunc, since bs=512 is the default. When using bs (instead of separate input and output block sizes), the dd command does what you expect it to do in this example: it will copy zeros all the way throughout the drive sdx; it will not truncate /dev/sdx; and it will fill up /dev/sdx fully (possibly using a partial block for the last copy).


    = = = = = = = = = =
    Example 2 Zero-out the free space on a partition
    (Also: Clean up the root partition of your Kubuntu OS)

    From live CD:
    sudo dd if=/dev/zero of=/media/sdxn/bigjunkfile
    sudo rm /media/sdxn/bigjunkfile

    Details
    We overwrite all the free (available or unused) space on a partition; this prevents recovery of past-deleted files. It does not disturb any existing files you are still using. Suppose you are working in a Kubuntu OS on your hard drive or from your Kubuntu Live CD. Assume the target drive is seen as sdx; assume the partition you want to clean up is sdxn; and that sdxn is mounted as /media/sdxn.
    (Check the disks: sudo fdisk -lu; Check the partition: cd /media/disk and ls -al to see all files and their sizes in partition sdxn.)
    Create a file named bigjunkfile in sdxn and write zeros to it until it fills up all the unused space in sdxn:

    sudo dd if=/dev/zero of=/media/sdxn/bigjunkfile
    (The output will say "dd: writing to `/media/disk/bigjunkfile': No space left on device" plus details.)
    Now remove the file:
    sudo rm /media/sdxn/bigjunkfile

    Clean up the root partition of your Kubuntu OS
    -- Do it as explained above from a live CD.
    Or,
    -- Do it while booted into your Kubuntu OS:
    sudo dd if=/dev/zero of=bigjunkfile
    sudo rm bigjunkfile


    = = = = = = = = = =
    Example 3 Clean up your home

    Similar to Example 2.
    -- If your home is on the same partition as the root partition of Kubuntu, use Example 2 ("Clean up the root partition of your Kubuntu OS").
    -- If your home is on a separate partition sdxn, then exactly as in Example 2, either from a live CD or while booted into your Kubuntu OS (you don't need sudo since it is your home):

    dd if=/dev/zero of=/media/sdxn/bigjunkfile
    rm /media/sdxn/bigjunkfile


    = = = = = = = = = =
    Example 4 Clean a Partition Before Reusing It or Deleting It

    From another partition or from a live CD, to zero-out partition sdxn:
    sudo dd if=/dev/zero of=/dev/sdxn conv=notrunc,noerror

    Advanced note: I can see no reason why you need conv=notrunc, but many authors use it. I think there's a lot of misinformation out there. Doesn't hurt to leave it in, but I don't use it.


    = = = = = = = = = =
    Example 5 Clean the Master Boot Record, 1 sector; or Clean 63 sectors

    Zero-fill the MBR of drive /dev/sdx:
    sudo dd if=/dev/zero of=/dev/sdx bs=512 count=1

    Zero-fill the first 63 sectors of drive /dev/sdx
    sudo dd if=/dev/zero of=/dev/sdx bs=512 count=63

    (The first partition usually begins in LBA sector 63; use sudo fdisk -lu to confirm this. Since LBA counts sectors starting from zero, the first partition starts in the 64th sector. dd counts fom 1. )


    = = = = = = = = = =
    Example 6 Wipe a file before deleting it (to prevent recovery)

    Let's get rid of the file testfile. Change into the directory containing testfile. (E.g., if it is on your Desktop, cd Desktop)

    ls -al
    Output:
    -rwxr-xr-x 1 mike mike 23996 2008-01-17 14:37 testfile

    sudo dd if=/dev/zero of=testfile bs= 23996 count=1 conv=notrunc
    sudo rm testfile

    (We see its size: 23,996 bytes, write zeros to it, then remove it.)

    Advanced note: This will not remove all file metadata which may include the name of the file. Normally, that shouldn't be a problem. If it is for you, get in the habit of carefully selecting file names of sensitive files. After removing it, you can search for any remains of the file on partition sdxn:

    sudo dd if=/dev/sdxn bs=4096 | hexdump -C | grep 'testfile'


    = = = = = = = = = =
    Example 7 --blank for now--


    = = = = = = = = = =
    Example 8 Cloning a [bootable] flash drive or a hard drive

    dd if=/dev/sdx of=/dev/sdy bs=4096 conv=notrunc,noerror

    sdx and sdy are physically identical (same size) drives. The result is that sdy will be identical in every way to sdx. If sdx is bootable with a MBR (and GRUB), a file system, data and programs, so will sdy.


    = = = = = = = = = =
    Example 9 Master boot record (MBR): viewing and backup/restore

    --- To see the MBR of drive sdx:
    sudo dd if=/dev/sdx count=1 | hexdump -C

    This prints the MBR to the screen. Count=1 means to copy only one block size (in this case bs=512=the default, and 1 sector =512 bytes). Note the end of file signature 55 aa.
    -> For more on this example, see Part 1, Section 2 on the MBR.

    --- Make a backup of your MBR
    sudo dd if=/dev/sdx count=1 of=MBR_copy

    (That puts the copy in your home; use: of=/path/to/MBR_copy to place the copy where you want it.)

    --- To restore your MBR
    sudo dd if=MBR_copy of=/dev/sdx count=1

    (Or sudo dd if=/path/to/MBR_copy of=/dev/sdx count=1)


    = = = = = = = = = =
    Example 10 Print a partition boot sector

    To see the boot sector of partition sdxn, n>1 :

    sudo dd if=/dev/sdxn count=1 | hexdump -C

    Prints it to the screen. Count=1 means to copy only one block size (in this case bs=512= 1 sector= the default). The last row of output shows an offset of 1f0 in hexadecimal (or 0x1f0 or 1f0h) which is 496 bytes in decimal (or 496d). Then there are 16 entries across the final row, which would take us to 512 bytes (1 sector) from the start of sdxn. Note the end of file signature 55 aa.
    To print it to a file located at /path/to/boot_sector_sdx_copy:
    sudo dd if=/dev/sdxn count=1 of=/path/to/boot_sector_sdx_copy

    The hard way when n=1 (but it demonstrates some things):
    The above method works for the first partition, but here's another (harder) way. To see the boot sector of the first partition on the disk, you have to skip over the MBR (the first sector) and other bytes until you get to the first sector of the first partition. In Logical Block Addressing (LBA), sectors start from sector zero on a disk. If you run sudo fdisk -lu, you'll (probably) see the first partition of your drive starts at sector 63. Since LBA counting starts at zero (0, 1, ..., 63), sector 63 is actually the 64th sector, and to see it, you have to skip over 63 sectors; thus, to print the boot sector of sdx1:
    dd if=/dev/sdx count=1 skip=63 | hexdump -C


    = = = = = = = = = =
    Example 11 --blank for now--


    = = = = = = = = = =
    Example 12 Creating files

    Create a 1 megabyte file (1,000,000 bytes) containing only zeros
    dd if=/dev/zero of=megfile bs=1MB count=1
    (If you want 1024x1024, use bs=1024x1024 or bs=1M.)

    Create a file of size 250,000 bytes containing random numbers
    dd if=/dev/urandom of=bigranfile bs=250000 count=1

    Create a file named file1 of size 1000 bytes that contains nothing (but metadata)
    dd if=/dev/zero of=file1 bs=1000 count=0 seek=1
    You will see that it contains 1000 bytes; however, if you view it in Midnight Commander (mc), you'll see there's nothing in it.


    = = = = = = = = = =
    Example 13 Check a file for errors

    Check the partition /dev/sdxn for errors:
    dd if=/dev/sdxn of=/dev/null

    You can have dd copy through a file to see if it encounters any errors. The file can be any file, including device files like drives /dev/sdx and partitions /dev/sdxn. dd will copy its "reads" to the null device (see Part 1, Section 2, "Useful Linux device files"). Of course, do not use noerror since you want dd to inform you if it encounters an error. We used the default block size bs=512, but you can use any value.

    Use skip and count to narrow down your probe.
    Example: dd if=/dev/sdc10 of=/dev/null bs=100 skip=1M count=5000
    will skip 1,000,000 bytes on partition sdc10, then copy 100 bytes at a time until it copies 500,000 bytes (5000x100) and dumps them in the black hole /dev/null. If it encounters an error, it will stop and report it.


    = = = = = = = = = =
    Example 14 Print bytes of a file to the screen

    You can't read a binary file unless you use a hexdump or Midnight Commander.
    --- To see the binary file binaryfile, change directory (cd) into the directory containing binaryfile, then
    dd if=binaryfile | hexdump -C
    or pipe it to hexdump and maybe to less:
    dd if=binaryfile | hexdump -C
    dd if=binaryfile | hexdump -C | less
    See man less.
    Of course, instead of using cd, you can use the full path name from any directory:
    dd if=/path/to/binaryfile | hexdump -C

    --- Print the first byte of a file to the screen:
    dd if=myfile bs=1 count=1
    (A byte is eight bits and represents a character.)
    --- Print more bytes using count; e.g., print 5 bytes: dd if=myfile bs=1 count=5
    --- Skip some bytes using skip.
    Print the third, fourth and fifth bytes: dd if=myfile bs=1 skip=2 count=3


    = = = = = = = = = =
    Example 15 --blank for now--



    = = = = = = = = = =
    Example 16 Find the size of a drive or flash drive
    You can use null to find the size (in bytes) of a drive or flash drive:
    sudo dd if=/dev/sdc of=/dev/null bs=4096
    123007+1 records in
    123007+1 records out
    503840256 bytes (504 MB) copied, 41.9472 s, 12.0 MB/s
    Thus, the drive has 503,840,256 bytes.


    = = = = = = = = = =
    Example 17 Look for non-zero entries on a drive (HDD or flash drive)

    dd if=/dev/sdx | hexdump -C | grep [^00]

    You might run this after you zero-out your drive, to verify all zeros on it.

    Check for non-zero entries on a disk beyond 1,031,760,000 bytes:
    dd if=/dev/sdx skip=10317600 bs=100 | hexdump -C | grep [^00]

    See man grep: The brackets means "find anything inside them." The ^ means "not." Thus "find anything that is not 00. The 00 is two zeros, or one byte in hexadecimal, and one byte is enough to represent one character (a letter, number of symbol). Thus a zero would show up in a hex printout as 00 (using 4 bits for each zero: 1 byte = 8 bits = 0000 0000 binary = 00 hexadecimal).


    = = = = = = = = = =
    Example 18 Search for a string in your /home partition

    I have just one instance of the string "urmyvalentine" in my home. It is the name of an empty folder on my Desktop. Let's see how many references come up on it.

    sudo dd if=/dev/sdb7 bs=4096 | hexdump -C | grep 'urmyvalentine'
    Output:
    381e8fab0 20 27 75 72 6d 79 76 61 6c 65 6e 74 69 6e 65 27 | 'urmyvalentine'|
    45437e040 3d 75 72 6d 79 76 61 6c 65 6e 74 69 6e 65 0a 32 |=urmyvalentine.2|
    45691b040 3d 75 72 6d 79 76 61 6c 65 6e 74 69 6e 65 0a 0a |=urmyvalentine..|
    5120710+1 records in
    5120710+1 records out
    20974431744 bytes (21 GB) copied, 4770.11 s, 4.4 MB/s
    (It took 79.5 minutes to search the 21 GB partition.)

    This suggests that if I'm going to protect my privacy and do some cleanups of my personal files, it won't be enough to simply "delete" that folder. It also suggests that perhaps you should be careful choosing names for your files and folders for sensitive data.
    You can also play with the offsets using a hex calculator (Qalculate); for example, 381e8fab0 hex = 1.5064431E10 decimal = 15,064,431,000.


    = = = = = = = = = =
    Example 19 Check a random part of a drive to see what's there

    dd if=/dev/sdx bs=1 skip=100000 count=100 | hexdump -C

    This skips the first 100,000 bytes of drive sdx, then prints the next 100 bytes to the screen (using hexdump -C so you can see it). You might also print the result to a file (say, checkfile) and then view it using Midnight Commander:
    dd if=/dev/sdx of=checkfile bs=1 skip=100000 count=100

    See also Example 16 Find the size of a drive or flash drive.


    = = = = = = = = = =
    Example 20 --blank for now--


    = = = = = = = = = =
    Example 21 Write numbers to the end of a disk

    In this example, we write random numbers to the disk starting at 1,031,760,000 bytes and beyond to the end of the disk:
    dd if=/dev/urandom of=/dev/sdx seek=10317600 bs=100
    (We used a small block size bs=100 bytes knowing we were near the end of the disk (a "1 GB" flash drive that "ends" at 1,031,798,784 bytes).

    See also Example 15 Find the size of a drive or flash drive.


    = = = = = = = = = =
    Example 22 md5sums and dd

    Calculate the md5sum of drive sdx:
    md5sum /dev/sdx > /path/to/mywork/sdx_md5sum.txt

    /path/to/mywork is the directory where you are keeping results as you work; it may be your home or a mounted directory on another partition or disk, such as /media/sdy/mywork/sdx_md5sum.txt; sdx_md5sum.txt is a file created by the above command that contains the md5sum of drive sdx.

    Write sdx to a file using dd:
    dd if=/dev/sdx of=/path/to/mywork/sdx_image

    Calculate the md5sum of this image:
    md5sum /path/to/mywork/sdx_image > /path/to/mywork/sdx_image_md5sum.txt

    This command redirects (>) the output of md5sum to the file sdx_image_md5sum.txt (which is created by this command). Now you can compare the two md5sums: sdx_md5sum.txt and sdx_image_md5sum.txt to see that they match (which implies the dd copy was correct).


    = = = = = = = = = =
    Example 23 Using the Bash command sync after a dd copy

    (This is not the same as the dd sync option in conv=sync.)
    Man sync:
    sync - flush file system buffers
    Force changed blocks to disk, update the super block.
    You'll see some writers use sync after using dd. The idea is that invoking sync will ensure that the effect of dd is written to disk promptly, thus reducing the risk that the result of dd will be lost if the system should fail or be interrupted. Some writers use it for security.
    For example, suppose testfile is 26478 bytes and you wish to write zeros to it (zero-out the file):
    dd if=/dev/zero of=testfile bs=26478 count=1 conv=notrunc
    sync
    [optional: then remove testfile: rm testfile]
    (We use notrunc because we don't want dd to truncate testfile before doing the copy; truncating testfile would mean that we lose its "place" or its bytes--we don't want those bytes returned to the file system before we write the zeros.)


    = = = = = = = = = =
    Example 24 What's at the end of YOUR hard drive?

    A full example of this is in Reply #3 below.


    = = = = = = = = = =
    Example 25 Backup your Kubuntu OS to your separate /home partition

    Example Suppose your home is on sda4, and your Kubuntu is on sda5.
    If you want to make a backup of your root partition in your home partition, allow room for that when planning the size of your home partition.

    Backup
    To make a file containing the image of sda5 in your home, at Konsole, issue the following command while you are in Kubuntu on sda5:
    sudo dd if=/dev/sda5 of=/home/your_name/sda5image bs=4096 conv=noerror,notrunc

    To make this backup from a live CD:
    sudo dd if=/dev/sda5 of=/media/sda4/your_name/sda5image bs=4096 conv=noerror,notrunc
    where we are assuming your home partition is mounted as /media/sda4.

    Restore
    To restore your backup image of sda5 to partition sda5, issue the following command from a live Kubuntu CD:
    sudo dd if=/media/sda4/your_name/sda5image of=/dev/sda5 bs=4096 conv=noerror,notrunc

    Troubleshooting

    -- “Permission denied” Use sudo dd.
    -- “/dev/sdxn is not a directory” Use the mount point corresponding to the device /dev/sdxn (which may be something like /media/sdxn).
    Errors upon rebooting
    -- “Unable to resolve UUID=” “Press Control+D to end this shell etc...”
    Ignore it; press Control+D to continue (or Control+D, then Enter). You got this error if you made a new partition for the root system backup but forgot to include the new partition's UUID in the /etc/fstab file of your Kubuntu OS. Try to remember to fix it.
    -- Upon re-booting: “...filesystem error(s)...”
    Hopefully, the system can fix those upon re-booting--it will tell you so on the screen. Otherwise, you may have to manually run the command given to you on the screen(e.g., fsck /dev/sda5). These usually resolve themselves quickly. If not, re-boot with a live Kubuntu CD, and re-try to restore the OS root partition (using the commands given to you above or try using dcfldd instead of dd).
    -- Example of actual error: Dropped to a terminal prompt with messages: “sda5 contains a filesystem with errors; Inode 526117 has illegal blocks; unexpected inconsistency; run fsck manually w/o -a or -p options.” Action: To repair the filesystem, you might then type sudo fsck /dev/sda5. Then reboot with sudo shutdown -r now.

    Advice (based on dozens of experiments)
    -- dcfldd is best (safest, fastest). The fastest, safest way to do work like this is using the special dd command version dcfldd. See Part 1, Section 1. dcfldd is faster (about twice as fast for me). dcfldd is safe: For me, no errors; perfect reboot.
    -- We did this work while booted into the OS we were making a backup of, and that's OK. Next best is to simply use your live Kubuntu CD and the dd or dcfldd command. Safe: I never got an error using the live CD method; perfect re-boot.
    -- You can also do this work from another operating system partition on the same hard drive as the partition you are trying to backup & restore. I've had excellent results doing it this way(errors encountered just once, and then easily fixed automatically upon re-boot.)
    -- Lastly, I got the most errors and the most serious errors when doing this work from a hard drive Kubuntu OS partition that was NOT on the same hard drive as the Kubuntu OS I was making the backup of. Is there a reason for this? I don't know; but the pattern, for me, on my system was unmistakable. Errors were easily fixed upon re-boot automatically or using fsck.

    References
    -- Fixing filesystems:
    http://members.iinet.net.au/%7Eherman546/p10.html
    -- GParted Live CD capabilities: It also has capability to copy and restore a partition
    GParted: http://gparted.sourceforge.net/ Make a partition the same size and filesystem as the one you are copying; copy Kubuntu to that partition; restore using GParted again (copy/paste) or using dd.
    GParted Live CD also has a Terminal (double-click the icon at the top of the Gparted screen) you can use as you would use any terminal, running commands like fdisk -lu and dd; you can also install dcfldd to your Live GParted session (sudo apt-get install dcfldd).
    -- Partimage: A good program for making image backups, like what we are doing here
    http://www.debianadmin.com/backup-an...partimage.html
    http://www.partimage.org/Main_Page


    = = = = = = = = = =
    Example 26 --blank for now--


    = = = = = = = = = =
    Example 27 Bad disk sectors
    There are six ways a disk can develop bad sectors. Two of them may be fixable (and maybe not). If the ECC fails by detecting an error but not fixing it, you'll get a sector read error (or bad sector error, or 0x40 error). By the time you start seeing bad sectors appear consistently, it it probably too late to hope to save the drive; the hard disk should have been taking care of bad sectors all along, so apparently something is amiss now. Your hard disk deals with sectors, writing whole sectors at a time, even if only one byte is changed. Disks map bad sectors to good ones, setting the bad sector aside. The file system (e.g., ext3) doesn't work with sectors but with groups of sectors called blocks or clusters. A file system can mark bad sectors so they aren't used; or remap bad blocks, say by using fsck with the badblock option.

    Where does dd fit in? iflag=direct and oflag=direct

    You can use dd to write zeros to a hard drive (or flash drive), and that will wipe it clean and "freshen" it up: dd if=/dev/zero of=/dev/sdx. If dd hits a bad sector, it may be able to write zeros to it and things will be OK; it may cause the disk to reallocate the bad sector, which is also a good thing. Then again, dd may not be able to fix the problem. dd must be able to read and write to the bad sector(s) (sometimes to a block of 4096 bytes), but that will not be possible if there is unrecoverable physical damage to the disk, in which case this issue is then rendered moot: you simply write zeros to as much of the disk as you can, then to prevent recovery of any remaining personal data, physically destroy the disk (burning by blow torch, physically shredding, pulverizing).

    Your kernel reports hard disk errors in /var/log/messages or at Konsole by running dmesg. You might then identify a bad sector and test it using dd and try to write zeros to it using dd. For one thing, certainly, unmount the file system (from which the bad sector is being reported) because you don't want anything written to the bad area and cause loss of your data. Unfortunately for dd, your filesystem "buffers reads," so if you want to read a sector, the file system will process the request in a block of sectors; if a bad sector is in the block, that will cause dd to fail. New versions of dd have two input/output options, iflag=direct and oflag=direct, that will give an unbuffered read. See man dd or dd --help.

    To give you the flavor of how dd may be useful here, suppose you learn from the kernel that LBA sector 12345 is bad. Since LBA counts sectors from zero, that means dd has to skip 12345 sectors to hit LBA 12345 (dd counts from one); then the next sector is the bad one, and this will tell you if that sector is bad (by giving you an Input/Output error, or I/O error):
    dd if=dev/sdx of=/dev/null count=1 bs=512 skip 12345
    (The default block size is used = 512; the data read is sent to the null black hole device so you don't have to see all that stuff on the screen.)
    To write zeros to that sector and maybe cause it be reallocated ...
    CAUTION: you may now lose some data or get file corruption! ... Hope you have backups? ...
    dd if=/dev/zero of=/dev/sdx count=1 bs=512 seek 12345
    sync
    (See man sync; see Example 21 above.)
    If dd gives you an error, and the kernel log says it's a write error, you may be out of luck, unless you can identify the filesystem block containing the bad sector and overwrite the entire block (of, say, 4096 bytes or whatever your filesystem uses) ... Or, hit it directly with the iflag=direct oflag=direct flags in your dd statement (see dd --help). It may turn out that you have a bad file (or more) or a bad superblock. Run SMART to investigate further. It may also happen that you can not use standard Linux tools to do the work because the bad sector(s) may belong to an unassigned block and thus they are not in any file (so debugfs tells you that no file contains that sector). You can force the disk to allocate unassigned sectors to one big file by writing zeros to a very big file that fills up the unused space on the partition: dd if=/dev/zero of=/path/mountpoint/mybigfile bs=4096, and then proceed using SMART.

    The format for using iflag=direct and oflag=direct.
    To read:
    dd if=/dev/sdx iflag=direct skip=X of=somefile bs=Y count=Z
    and to write it back:
    dd if=somefile of=/dev/sdx seek=X oflag=direct bs=Y count=Z

    For exact details on how to do all this, see these two papers:

    Bad Sector Blues:
    The Dreaded "error=0x40 { UncorrectableError }"
    http://www2.uic.edu/~aciani1/sector_blues.html

    Bad block HOWTO for smartmontools
    http://smartmontools.sourceforge.net/badblockhowto.html


    = = = = = = = = = =
    Last edited by Qqmike; Apr 22, 2015, 05:36 PM.
    An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

    Comment


      #3
      Re: dd Command

      Part 3

      dd -- Wipe partitions, home, files, disks, MBR

      How to clean up (wipe) your partition, home, sensitive files, hard drive, flash drive, and MBR

      TABLE of CONTENTS

      Summary Sheet: Quick Reference Guide
      The Summary Sheet is for experienced users. Each method is explained in detail in the text (under the section headings).

      SECTION 1
      Introduction and philosophy
      When you delete a file, you do not delete the file: Now what?!
      SECTION 2
      How to Clean the Deleted/Unused Space on Your Kubuntu Partition or /Home
      SECTION 3
      How to Clean a File Before Deleting It
      SECTION 4
      How to Clean a Partition Before Reusing It or Deleting It
      SECTION 5
      How to Clean Your Hard Drive or Flash Drive Before Reusing It
      SECTION 6
      How to Clean the Master Boot Record, 1 sector; and 63 sectors

      -> CAUTION <-
      Before running dd commands, make a BACKUP of all important files.

      = = = = = = = = = = = = = = = = = = = =
      = = = = = = = = = = = = = = = = = = = =

      SUMMARY SHEET: Quick Reference Guide

      The Summary Sheet is for experienced users. Each method is explained in detail in the text (under the section headings).

      SECTION 2
      How to Clean the Deleted/Unused Space on Your Kubuntu Partition or /Home

      > Delete unwanted personal files. > Use Privacy Cleanup 101 to clean up your Kubuntu partition. Or use a cleaner, like Bleachbit.
      > Empty the Trash. > Re-boot (to clear /tmp directories)
      Boot into Kubuntu. Open Konsole. Change into your home directory:
      cd ~ (or: cd /home/your_name)
      dd if=/dev/zero > bigjunkfile
      [Optional: check things with df -hT, ls -al, du -sh, du -ah, Konqueror/Dolphin.]
      rm bigjunkfile
      That cleans up your /home.
      It also cleans up your root partition if your home is on the same partition as root (/).
      If your Kubuntu root partition is separate from your /home partition, then:
      Clean up the root /, as follows:
      Boot into Kubuntu, at Konsole (note the sudo):
      cd /
      sudo dd if=/dev/zero > bigjunkfile
      sudo rm bigjunkfile
      That cleans up your root / .
      Re-boot to clear out any /tmp directories and ensure that your OS works OK.

      Working from Kubuntu Live CD
      Start your Kubuntu Live CD. Open Konsole.
      See your partitions: sudo fdisk -lu
      Look for your Kubuntu root / and your /home partition(s).
      Example: your /home is on /dev/sdxn
      sudo mkdir /media/sdxn
      sudo mount /dev/sdxn /media/sdxn
      cd /media/sdxn/home/your_name
      sudo dd if=/dev/zero of=bigjunkfile
      [Optional: check things with df -hT, ls -al, du -sh, du -ah, Konqueror/Dolphin.]
      sudo rm bigjunkfile
      cd /
      sudo umount /media/sdxn
      That cleans up your /home.

      Technical note:
      Instead of changing directory into your home (using cd /media/sdxn/home/your_name), you could simply run dd as
      sudo dd if=/dev/zero of=/media/sdxn/home/your_name/bigjunkfile
      (That is, in the Live CD session, supply full, absolute path names to the dd command.)
      And then to remove bigjunkfile, use: sudo rm /media/sdxn/home/your_name/bigjunkfile)

      -> If /home is on the root partition, that also cleans up your Kubuntu root.
      Or:
      If Kubuntu root is on /dev/sdxm (where m is different from n), to clean up sdxm:
      sudo mkdir /media/sdxm
      sudo mount /dev/sdxm /media/sdxm
      sudo dd if=/dev/zero of=/media/sdxm/bigjunkfile
      [Optional: check things with df -hT, ls -al, du -sh, du -ah, Konqueror/Dolphin.]
      sudo rm /media/sdxm/bigjunkfile
      sudo umount /media/sdxm
      That cleans up your root / partition.
      You are done. Exit Konsole with exit command; exit the live CD session, re-boot (to clean up /temp directories).

      Troubleshooting tip: if "Cannot unmount, device busy," then exit Konsole by typing exit, then open Konsole, and do again sudo umount /media/sdxm or sudo umount /media/sdxn .


      SECTION 3
      How to Clean a File Before Deleting It
      To clean the file oldfile and then remove it:
      Assume the path to the file is /path/to/oldfile.
      Open Konsole.
      cd /path/to
      ls -al
      From the output, get the size of oldfile in bytes, say it is B.
      dd if=/dev/zero of=oldfile bs=B count=1 conv=notrunc
      rm oldfile
      Done.
      (Note: Use notrunc to hold the location and bytes of the oldfile.)

      To use pseudo random numbers (instead of zeros):
      dd if=/dev/zero of=oldfile bs=B count=1 conv=notrunc
      rm oldfile


      SECTION 4
      How to Clean a Partition Before Reusing It or Deleting It
      You can do this from a (Kubuntu) live CD, from another Kubuntu partition, or from GParted Live CD. If you use GParted, after you zero-out the partition, you can immediately re-format it. In Kubuntu, you use Konsole and you must use sudo dd. In GParted Live CD, open a terminal by double-clicking Terminal icon at the top, that gives you a terminal with a root prompt, so you do not have to use sudo and can simply use dd.
      Here it is using sudo:

      Find out exactly how the partition is named:
      sudo fdisk -lu
      Suppose it is /dev/sdxn.
      sudo dd if=/dev/zero of=/dev/sdxn conv=notrunc

      The partition /dev/sdxn will now be zerod-out (and so recovering your data would be quite impossible). Note that the partition is now FULL--full of the zeros you wrote. If you use Gparted Live CD to reformat it, it will now be mostly empty (except for filesystem overhead) and ready to use again.

      SECTION 5
      How to Clean Your Hard Drive or Flash Drive Before Reusing It
      By writing zeros to a hard drive or flash drive, you restore it to like-new state.
      sudo fdisk -lu
      Identify the drive. Example: Let's say it is /dev/sdx. CAUTION: Make sure about this!
      sudo dd if=/dev/zero of=/dev/sdx
      The drive is now clean but FULL--full of the zeros you just wrote to it!
      You must re-partition the drive, starting with a new partition table (or MBR or Disk Label).


      SECTION 6
      How to Clean the Master Boot Record, 1 sector; and 63 sectors

      Zero-fill the MBR of drive /dev/sdx:
      sudo dd if=/dev/zero of=/dev/sdx bs=512 count=1

      Zero-fill the first 63 sectors of drive /dev/sdx
      sudo dd if=/dev/zero of=/dev/sdx bs=512 count=63

      [End of Summary Sheet: Quick Reference Guide]

      = = = = = = = = = = = = = = = = = = = =
      = = = = = = = = = = = = = = = = = = = =

      SECTION 1
      Introduction and philosophy

      It comes down to this:
      > When you delete a file you do not actually destroy it. You only destroy the data structure (filesystem) reference to it. The bits and bytes of file information are still on your disk.
      > When you delete a partition, you do not actually destroy it. You only destroy the data structure (MBR partition table) reference to it. The bits and bytes of information comprising the partition are still on your disk.

      Proof? Easy as pie. Delete a bunch of your files (documents, photos, movies), and then before messing with that disk, use PhotoRec to find them and retrieve them for you. Or, use GParted Live CD to delete your Kubuntu or /home partition, then before messing with that disk, use TestDisk to find and restore the partition(s). Or, delete a file, then use dd and grep to search for key words in that file.
      => When you delete a file, you do not delete the file

      ->-> My Philosophy for this how-to is simple:
      I go along with Starman's conclusions regarding the literature on this and feel that simply writing zeros (or random numbers) to a disk is enough to prevent recovery in almost all normal cases. ->-> Please note what I'm talking about: We are not talking about hiding illegal activities or data. If that's your gig, you're on your own there, pal (hint: state-of-the-art computer forensics is probably ahead of you). We are talking here about deleting data that you consider to be private, confidential, sensitive or personal in some (legal) way. Examples might be the following: financial records/data, health/medical notes, sensitive business information, personal correspondence to friends, your personal diaries, family photos, and personal interest items you've downloaded from the Internet. Wiping a disk or MBR is also used to eliminate malicious code.
      ->-> Your philosophy about data deletion may differ from mine. You may feel it is necessary to make multiple passes of zero writes or random-number writes to your drives to prevent data recovery. That's fine. This how-to applies: you simply apply the how-to multiple times


      = = = = = = = = = = = = = = = = = = = =
      SECTION 2
      How to Clean the Deleted/Unused Space on Your Kubuntu Partition or /Home

      -> CAUTION <-
      When you fill up your Kubuntu root partition or your /home partition, strange things may happen when you try to re-boot. Thus, follow these recipes carefully and do NOT re-boot until deleting the file we have called bigjunkfile!!!

      -> CAUTION <-
      Before running dd commands, make a BACKUP of all important files.

      First:
      > Delete unwanted personal files
      > Use Privacy Cleanup 101 to clean up your Kubuntu partition. Or use a cleaner, like Bleachbit.
      > Empty the Trash
      > Re-boot (to clear /tmp directories)

      Next: Now you will have a bunch of empty space in your Kubuntu OS; however, some of that empty space contains files you deleted from your filesystem but are still on your disk in bits and bytes. They could be recovered by, say, PhotoRec. So you will fill all that empty deleted/unused space with zeros. That should clean things up for you and get rid of all the data and stuff laying around in your empty space and make it very difficult--probably quite impossible--to retrieve your deleted files..

      Case 1: Suppose your /home is on same partition as the Kubuntu root partition

      Open Konsole.
      Change into your home directory:
      cd ~ (or: cd /home/your_name)
      dd if=/dev/zero > bigjunkfile

      (Home Full Message: ignore it.
      If you get a message saying that your home (or partition) is almost full, Do you want to open Konqueror/Dolphin to check it?, say no (Cancel), and let dd continue doing its thing until done.)

      The dd statement creates the file bigjunkfile in your present working directory (/home/your_name) and fills it up as far as it will go with zeros. The file bigjunkfile will grow until it fills up all the empty space in your Kubuntu partition (since in Case 1, your /home is located on the same partition as the Kubuntu root OS). Your Kubuntu partition is now FULL. You can check it using
      df -hT
      and
      du -sh
      du -ah
      and
      ls -al (which will show you how big bigjunkfile is), or use Konqueror or Dolphin to check things (keeping Konsole open).
      Beginner note: Instead of bigjunkfile, you can name that file anything you wish, like goodbyejunk or whatever. Doesn't matter what we call it as long as you can recognize what it is as you work.

      CAUTION: Before you get into any trouble, better delete bigjunkfile.
      If you are not still in your home directory, change into it (cd ~) and delete the file:

      rm bigjunkfile

      That's it, you're done: your empty partition space is pretty clean, filled with zeros, bit by bit.
      Recommended:
      -> Now re-boot to clear out any /tmp directories and ensure that your OS works OK.

      Case 2: Suppose your /home is on a separate partition (from the Kubuntu root partition)
      Case 2a: Clean up empty space only on your /home partition
      Case 2b: Clean up empty space in the Kubuntu root partition

      Case 2a: Same as above. That is,
      Open Konsole. Change into your home directory:
      cd ~ (or: cd /home/your_name)
      dd if=/dev/zero > bigjunkfile
      [Optional: check things with df -hT, ls -al, du -sh, du -ah, Konqueror/Dolphin.]
      Remove bigjunkfile:
      rm bigjunkfile

      Case 2b: Clean up empty space in the Kubuntu root partition.
      After doing Case 2a, if you wish also to clean up empty space in your Kubuntu root partition, do so as follows:
      Open Konsole
      Change into your root directory:
      cd /
      sudo dd if=/dev/zero > bigjunkfile
      [Optional: check things with df -hT, ls -al, du -sh, du -ah, Konqueror/Dolphin.]
      The file bigjunkfile will now reside directly under the root OS directory: /bigjunkfile. That's why we need to use root privileges with sudo.
      Remove bigjunkfile:
      sudo rm bigjunkfile

      Recommended:
      -> Now re-boot to clear out any /tmp directories and ensure that your OS works OK.


      Working from a live CD
      The topic continues, How to Clean the Deleted/Unused Space on Your Kubuntu Partition or /Home.
      Working from a live CD is another way to do this. Some users might feel it is safer, working outside the OS, or they just prefer to do such work from live CD. This is done the same as above, except you must mount the partition(s) you wish to work with.

      (Advanced note: Chrooting into a partition to work as root. See my how-to in References, Commands at Konsole, Part 3, for chrooting. It is NOT necessary to chroot to do this work.)

      Case 1: Suppose your /home is on same partition as the Kubuntu root partition.

      Start your Kubuntu Live CD. Select the option to try the OS without changing anything (that is, do NOT select Install). Open Konsole.
      Example: Assume your root partition is named sda9. If you are not sure, run
      sudo fdisk -lu
      to see your partitions.
      Make a mount point for your root partition-- a directory (a folder) on which to mount the partition you wish to work on.
      sudo mkdir /media/sda9
      Mount your root partition sda9 on the mount point
      sudo mount /dev/sda9 /media/sda9
      Change directories into your home (see Technical note that follows):
      cd /media/sda9/home/your_name
      Then run the familiar dd command (with the same logic as discussed above):
      sudo dd if=/dev/zero of=bigjunkfile
      (you need sudo)
      [Optional: check things with df -hT, ls -al, du -sh, du -ah, Konqueror/Dolphin.]
      Now remove bigjunkfile:
      sudo rm bigjunkfile
      Change directories back to root (of the CD session):
      cd /
      Unmount the mount point folder (note the spelling of the command umount):
      sudo umount /media/sda9

      Troubleshooting tip: if "Cannot unmount, device busy," then exit Konsole by typing exit, then open Konsole, and do again
      sudo umount /media/sda9

      For Case 1, you are done.
      You may leave Konsole by typing exit, then exit the live CD session and re-boot.

      Recommended:
      -> Now re-boot to clear out any /tmp directories and ensure that your OS works OK.

      Technical note
      Instead of changing directory into your home (using cd /media/sda9/home/your_name), you could simply run dd as
      sudo dd if=/dev/zero of=/media/sda9/home/your_name/bigjunkfile
      (That is, in the Live CD session, supply full, absolute path names to the dd command.)
      And then to remove bigjunkfile, use: sudo rm /media/sda9/home/your_name/bigjunkfile)

      Case 2: Suppose your /home is on a separate partition (from the Kubuntu root partition)
      Case 2a: Clean up empty space only on your /home partition.
      Case 2b: Clean up empty space in the Kubuntu root partition.

      Continuing with the same example, we assume your home on /dev/sda9 is separate from your root partition, and the root partition is named /dev/sda8:
      root /dev/sda8
      /home /dev/sda9

      First, clean up your /home just as we did above in Case 1 using the Kubuntu Live CD.
      For Case 2a, you are done:
      Exit Konsole with the exit command; then exit the live CD session and re-boot.
      For Case 2b, continue:
      You now have the option of entering your root partition where you can zero-fill its unused space. If you wish to do so, that is done as follows (assuming your Kubuntu root partition is on sda8):

      sudo mkdir /media/sda8
      sudo mount /dev/sda8 /media/sda8
      sudo dd if=/dev/zero of=/media/sda8/bigjunkfile
      [Optional: if you wish, you can change directories into /media/sda8 (cd /media/sda8) and check things with df -hT, ls -al, du -sh, du -ah, Konqueror/Dolphin.]
      sudo rm /media/sda8/bigjunkfile
      sudo umount /media/sda8
      Troubleshooting tip: if "Cannot unmount, device busy," then exit Konsole by typing exit, then open Konsole, and do again
      sudo umount /media/sda8

      You are done. Exit Konsole with the exit command; then exit the live CD session and re-boot.

      Recommended:
      -> Now re-boot to clear out any /tmp directories and ensure that your OS works OK.


      = = = = = = = = = = = = = = = = = = = =
      SECTION 3
      How to Clean a File Before Deleting It

      Let's say you have a file you wish to delete, and (with high probability) you do not want it to be retrievable . Before deleting it in Kubuntu, you should "fill it" with zeros (or random numbers)--replace the bytes of the file with zeros (or random numbers). Actually, we will use dd to write zeros to a file of identical size and with the same name as the file you wish to be rid of. That is, the file you wish to be rid of will be overwritten with zeros.

      Example
      I have a file on my Desktop named discardfile.odt.
      Open Konsole. Change directories to where the file is (to the Desktop).
      cd Desktop
      List all your files under Desktop:
      ls -al
      The (partial) output is
      -rwxr-xr-x 1 mike mike 44425 2009-12-03 21:15 discardfile.odt
      The size of discardfile.odt is 44,425 bytes.
      Use dd to create a file discardfile.odt of 44,425 bytes and containing only zeros, and since discardfile.odt already exists, the latter will be overwritten by the former:
      dd if=/dev/zero of=discardfile.odt bs=44425 count=1 conv=notrunc
      The output is
      1+0 records in
      1+0 records out
      44425 bytes (44 kB) copied, 0.000122628 s, 362 MB/s

      When I look at it in Konqueror, it is now a Plain Text Document; when I open it with Kate it is a binary file showing all zeros (........).

      Next, remove it (I'm still in the /home/mike/Desktop directory where the file is):
      rm discardfile.odt

      It is gone in this sense:
      It is deleted from your filesystem, and its contents (on your disk in bits and bytes) has been replaced by zeros; thus you could not recover the original file (say, by using PhotoRec). (You could recover the new file discardfile.odt which contains all zeros.)
      The file name Note that we used the exact file name, including its extension .odt. Had it been a photo called myhouse.jpg, you would use myhouse.jpg as the name in the dd statement.
      Technical note: Although the file is gone, the file's metadata may not be gone, including the name of the file. This suggests that for sensitive files, you may wish to choose the file name carefully.

      Random numbers
      The following statement would write random numbers "over" discardfile.odt:
      dd if=/dev/urandom of=discardfile.odt bs=44425 count=1 conv=notrunc


      = = = = = = = = = = = = = = = = = = = =
      SECTION 4
      How to Clean a Partition Before Reusing It or Deleting It

      Suppose you have been using a partition for something; for example, for data, for an operating system, or for your /home, or for a combination of these. Now you're done with it but would like to re-use the exact same partition for something else (e.g., to install a new operating system; or for a new /home partition); or you are done with it and now wish to delete the partition (so you can re-partition your drive perhaps). Here's how to wipe any evidence of past used files.

      You can do this from a (Kubuntu) live CD, from another Kubuntu partition, or from GParted Live CD. If you use GParted, after you zero-out the partition, you can immediately re-format it. In Kubuntu, you use Konsole and you must use sudo dd. In GParted Live CD, open a terminal by double-clicking Terminal icon at the top, that gives you a terminal with a root prompt, so you do not have to use sudo and can simply use dd.
      Here it is using sudo:

      Find out exactly how the partition is named:
      sudo fdisk -lu
      Suppose it is /dev/sdxn.
      sudo dd if=/dev/zero of=/dev/sdxn conv=notrunc

      => Now, you may re-use that partition knowing that it has been cleaned up (wiped or filled with zeros); thus you would begin by formatting it (in GParted). Or, you may delete the partition (in GParted), knowing that the data on it has been wiped (overwritten by zeros).
      CAUTION: The partition will be FULL after you issue the dd command! It will be full of zeros! If you re-format it in GParted Live CD, it will be "empty" again (but clean).


      = = = = = = = = = = = = = = = = = = = =
      SECTION 5
      How to Clean Your Hard Drive or Flash Drive Before Reusing It

      By writing zeros to a hard drive or flash drive, you restore it to like-new state. Zeros are written to every writable byte. This makes it extremely difficult if not impossible to recover any data you once had on the drive. You can do this from an OS on another hard drive attached to your PC or from a live CD or live flash drive. Either way, it is easy to do and the steps are the same and apply to wiping any drive (internal HD, external HD, or flash drive):

      sudo fdisk -lu
      Identify the drive. Example: Let's say it is /dev/sdx. CAUTION: Make sure about this!
      Then, simply do
      sudo dd if=/dev/zero of=/dev/sdx

      (It may take awhile, depending on your setup. For example, on an Intel D915GAVL board with a 3 GHz CPU, it took 45 minutes to zero-fill a 160 GB Seagate SATA drive.)

      Next:
      You may re-partition the drive, making new partitions and formatting each new partition.
      When you do this, say by GParted Live CD, you will get the message that the drive needs a partition table (or MBR or Disk Label). Accept this, select the standard PC type--MSDOS partition table or Disk Label--then proceed to make new partitions. Note: Some of you may be using the newer GPT partition table type for 2 TB drives. GParted supports GPT: click the drop list arrow (in the message) to see your choices.

      CAUTION: After running sudo dd if=/dev/zero of=/dev/sdx , the drive will be FULL because it is full of the zeros you just put in it. Thus, if you try to use it in some programs (e.g., usb-creator), you will get a message telling you the drive is full. To correct this and start new, you must re-partition the drive.


      = = = = = = = = = = = = = = = = = = = =
      SECTION 6
      How to Clean the Master Boot Record, 1 sector; and 63 sectors

      The Master Boot Record (MBR) is the first 512 bytes of the flash drive; that's the first sector of the flash drive. (A sector is 512 bytes.) The MBR is very important; it is the first thing your BIOS looks for when booting.

      The traditional PC MBR contains 446 bytes of bootloader code; then a 64-byte partition table: enough for four primary partitions, where the fourth one may be an Extended partition; then a 2-byte end-of-file boot record signature (in hexadecimal: 55 aa). To see your MBR on drive /dev/sdx, do this:
      sudo dd if=/dev/sdx count=1 | hexdump -C
      (The "|" is the vertical bar which is Shift and \, or Shift+\)

      Your MBR may be empty or it may have some past bootloader code in it or just some useless data. When GRUB is used, sectors following the MBR are also used for GRUB code. And when partitioning is done, usually the first partition starts at sector 63--after all this other stuff. In LBA addressing, the first sector is sector 0 (in older C/H/S addressing, the first sector is sector 1). Thus, in LBA, 63 sectors precede the first sector of the first partition. (My dd Command how-to explains more about this.)

      Before re-using an old drive, you may want to zero it out (see SECTION 5, How to Clean Your Hard Drive or Flash Drive Before Reusing It). However, some people simply want to ensure that they have a clean MBR to start with, and therefore want to zero-out the MBR of the drive. Let's do that first.
      usb-creator (and other programs): If you run usb-creator on a flash drive that has bootloader code on it (in its MBR), you will have problems (since usb-creator uses Syslinux code in the first partition, and the MBR code you have will confuse it). Thus, before running usb-creator on this flash drive, clear out its MBR.

      Zero-out the MBR of drive /dev/sdx
      sudo dd if=/dev/zero of=/dev/sdx bs=512 count=1

      The block size bs=512 is actually the default block size, but it is nice to write it out so we can see it explicitly here. Then count=1 tells dd to read just one block from the input file (if) and write that block to the output file (of). Thus, the dd statement writes zeros to the first 512 bytes (= the MBR) of sdx.

      (CAUTION: Use sudo fdisk -lu to identify the correct drive sdx. Where do you think dd got its nickname of "data destroyer"?)

      The first 63 sectors are used for the MBR (the first sector) and other code (e.g., GRUB Legacy embeds Stage_1.5 bootloader code in 16 sectors of that space). Malicious hackers may put code in the space, too. Other programs may dump code there. Thus, you may wish to zero-out the first 63 sectors of a drive.

      Zero-out the first 63 sectors of drive /dev/sdx
      sudo dd if=/dev/zero of=/dev/sdx bs=512 count=63



      ------------------------------------------------------
      ------------------------------------------------------
      References

      Commands at Konsole: Beginners: 3 parts
      http://kubuntuforums.net/forums/inde...opic=3091607.0

      Privacy Cleanup 101
      http://kubuntuforums.net/forums/inde....msg115011#new

      RESCUE Partitions & Data, TestDisk--PhotoRec--Knoppix
      http://kubuntuforums.net/forums/inde...opic=3091847.0 )
      Last edited by Qqmike; Apr 22, 2015, 05:42 PM.
      An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

      Comment


        #4
        Re: dd Command

        dd Application:
        What's at the end of YOUR hard drive?

        I've read in a few places that there could be anything at the end of your hard drive (phantom MBRs, spies, info plants, and malware). I doubt it. But just for some dd practice, it's easy to have a look:

        And what better drive to look at than my XP drive (just in case, of course... ); that would be drive sda.
        At Konsole,
        sudo fdisk -lu
        yields this output for hard drive sda (Seagate 160 GB SATA):

        Disk /dev/sda: 160.0 GB, 160041885696 bytes
        255 heads, 63 sectors/track, 19457 cylinders, total 312581808 sectors
        Units = sectors of 1 * 512 = 512 bytes
        Disk identifier: 0x90439043
        Device Boot Start End Blocks Id System
        /dev/sda1 * 16065 204812684 102398310 7 HPFS/NTFS (Windows XP)
        /dev/sda2 204812685 205021529 104422+ 83 Linux
        /dev/sda3 205021530 312576704 53777587+ 5 Extended
        /dev/sda5 205021593 219351509 7164958+ 83 Linux
        /dev/sda6 219351573 233681489 7164958+ 83 Linux
        /dev/sda7 233681553 254164364 10241406 83 Linux

        The plan -- What to look for:
        A partition contains a whole number of cylinders (not any partial cylinder).
        => Find out what's stored in the partial cylinder at the end of the drive.
        Fact: 1 sector = 512 bytes
        Fact: 1 cylinder = 255*63 = 16065 sectors (= 8,225,280 bytes)

        The drive sda has 160,041,885,696 bytes = 312,581,808 sectors
        There are 19457 cylinders listed (in the fdisk -lu output above).
        19457 cylinders * 16065 sectors/cylinder = 312,576,705 sectors “addressable”
        => 312,581,808 sectors total - 312,576,705 sectors addressable = 5103 sectors leftover
        => 5103 sectors leftover at the end of the disk
        (not enough for a cylinder since a cylinder is 16065 sectors; so the 5103 sectors will not be part any partition)


        Next, look to see what's in the 5103 leftover sectors.

        First, do a quick check to see if there are any non-zero characters in there:
        sudo dd if=/dev/sda skip=312576705 | hexdump -C
        That gave the following output:
        00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
        *
        5103+0 records in
        5103+0 records out
        2612736 bytes (2.6 MB) copied, 0.165139 seconds, 15.8 MB/s

        => all zeros in the leftover 5103 sectors.
        Note: The dd command does the copy using the default block size of 512 bytes = 1 sector (bs=512 is the default).
        Note: skip=312576705 causes the first 312576705 blocks of size 512 bytes each to be skipped before dd reads the sda drive.
        Note: 5103 sectors = 5103 sectors * 512 bytes/sector = 2612736 bytes (2.6 MB) copied. 1 record = 1 block size = bs=512 bytes (by default since we didn't specify another value for bs= ); i.e., in our case, 1 record = 512 bytes = 1 sector.
        Note: In the dd output, the single * on the second line indicates that the rest of the file consists of zeros.
        Note: On the first line of the dd output, the |................| is the ASCII part of the hexadecimal dump and indicates all zeros in the first line.
        Note: The dd command skips 312576705 bytes, then copies the rest of sda to the screen (standard output) and that output is piped (using the pipe symbol | ) to hexdump -C which displays its output on the screen.


        If there had been nonzero characters in the dd output, then you must investigate what's in there.
        You will see some of it in the ASCII part of the hexdump (the part that has “regular” ASCII character conversions (shown on the right between | and |) of the hexadecimal characters).

        Another way to investigate more closely is to have dd copy the 5103 sectors to a file, then use Midnight Commander to view that file.
        Here's an outline of how to do that:
        In Kubuntu, at Konsole Terminal program,
        sudo dd if=/dev/sda of=/home/mike/Desktop/myFile skip=312576705
        5103+0 records in
        5103+0 records out
        2612736 bytes (2.6 MB) copied, 0.0996157 seconds, 26.2 MB/s

        That put the 2612736 (leftover) bytes into a file called myFile on my Desktop.
        Install Midnight Commander using Adept Manager (K > System > Adept Manager).
        Indicate this in Adept by typing either Midnight Commander or mc in the search window.
        Open Midnight Commander in Konsole by typing mc and press Enter.
        In mc, click File, cursor down to View, Enter, type the location of the file (e.g., /home/mike/Desktop/myFile), and it'll come up in mc. You can play with mc using the options at the bottom (e.g., press various function keys Fn for View, hexadecimal, ASCII, etc.).


        (Edited for several typos on 3-23-08...)
        An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

        Comment


          #5
          Re: dd Command

          Very informative. Thank you. I ran through this on my laptop:

          Disk /dev/sda: 120.0 GB, 120034123776 bytes
          255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors
          Units = sectors of 1 * 512 = 512 bytes
          The addressable sectors equals 234,436,545, leaving (interestingly) 5103 'left over' sectors as well. Running the dd command, skipping the first 234436545 bytes, revealed the following:
          Code:
          00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
          *
          00107c00 eb 52 90 4e 54 46 53 20 20 20 20 00 02 08 00 00 |.R.NTFS  .....|
          00107c10 00 00 00 00 00 f8 00 00 3f 00 ff 00 00 e8 2e 00 |........?.......|
          00107c20 00 00 00 00 80 00 80 00 ff 57 ca 0d 00 00 00 00 |.........W......|
          00107c30 00 00 0c 00 00 00 00 00 10 00 00 00 00 00 00 00 |................|
          00107c40 f6 00 00 00 01 00 00 00 7a f6 ee 42 05 ef 42 08 |........z..B..B.|
          00107c50 00 00 00 00 fa 33 c0 8e d0 bc 00 7c fb 68 c0 07 |.....3.....|.h..|
          00107c60 1f 1e 68 66 00 cb 88 16 0e 00 66 81 3e 03 00 4e |..hf......f.>..N|
          00107c70 54 46 53 75 15 b4 41 bb aa 55 cd 13 72 0c 81 fb |TFSu..A..U..r...|
          00107c80 55 aa 75 06 f7 c1 01 00 75 03 e9 d2 00 1e 83 ec |U.u.....u.......|
          00107c90 18 68 1a 00 b4 48 8a 16 0e 00 8b f4 16 1f cd 13 |.h...H..........|
          00107ca0 9f 83 c4 18 9e 58 1f 72 e1 3b 06 0b 00 75 db a3 |.....X.r.;...u..|
          00107cb0 0f 00 c1 2e 0f 00 04 1e 5a 33 db b9 00 20 2b c8 |........Z3... +.|
          00107cc0 66 ff 06 11 00 03 16 0f 00 8e c2 ff 06 16 00 e8 |f...............|
          00107cd0 40 00 2b c8 77 ef b8 00 bb cd 1a 66 23 c0 75 2d |@.+.w......f#.u-|
          00107ce0 66 81 fb 54 43 50 41 75 24 81 f9 02 01 72 1e 16 |f..TCPAu$....r..|
          00107cf0 68 07 bb 16 68 70 0e 16 68 09 00 66 53 66 53 66 |h...hp..h..fSfSf|
          00107d00 55 16 16 16 68 b8 01 66 61 0e 07 cd 1a e9 6a 01 |U...h..fa.....j.|
          00107d10 90 90 66 60 1e 06 66 a1 11 00 66 03 06 1c 00 1e |..f`..f...f.....|
          00107d20 66 68 00 00 00 00 66 50 06 53 68 01 00 68 10 00 |fh....fP.Sh..h..|
          00107d30 b4 42 8a 16 0e 00 16 1f 8b f4 cd 13 66 59 5b 5a |.B..........fY[Z|
          00107d40 66 59 66 59 1f 0f 82 16 00 66 ff 06 11 00 03 16 |fYfY.....f......|
          00107d50 0f 00 8e c2 ff 0e 16 00 75 bc 07 1f 66 61 c3 a0 |........u...fa..|
          00107d60 f8 01 e8 08 00 a0 fb 01 e8 02 00 eb fe b4 01 8b |................|
          00107d70 f0 ac 3c 00 74 09 b4 0e bb 07 00 cd 10 eb f2 c3 |..<.t...........|
          00107d80 0d 0a 41 20 64 69 73 6b 20 72 65 61 64 20 65 72 |..A disk read er|
          00107d90 72 6f 72 20 6f 63 63 75 72 72 65 64 00 0d 0a 42 |ror occurred...B|
          00107da0 4f 4f 54 4d 47 52 20 69 73 20 6d 69 73 73 69 6e |OOTMGR is missin|
          00107db0 67 00 0d 0a 42 4f 4f 54 4d 47 52 20 69 73 20 63 |g...BOOTMGR is c|
          00107dc0 6f 6d 70 72 65 73 73 65 64 00 0d 0a 50 72 65 73 |ompressed...Pres|
          00107dd0 73 20 43 74 72 6c 2b 41 6c 74 2b 44 65 6c 20 74 |s Ctrl+Alt+Del t|
          00107de0 6f 20 72 65 73 74 61 72 74 0d 0a 00 00 00 00 00 |o restart.......|
          00107df0 00 00 00 00 00 00 00 00 80 9d b2 ca 00 00 55 aa |..............U.|
          00107e00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
          *
          5103+0 records in
          5103+0 records out
          2612736 bytes (2.6 MB) copied, 0.182902 seconds, 14.3 MB/s
          0027de00
          This HD came with Windoze Vista pre-installed, which I eventually dumped in favor of Linux. I used GParted to remove the existing Vista partitions (two of them) and then set up my partitions for use with Linux (eight partitions in all - three primary, one extended, and five logical). I don't believe I formatted the HD before repartitioning with GParted, although, of course, the root and separate home partitions were formatted using GParted.

          First question: Does formatting a drive only affect the addressable sectors? Based on my output above, this would seem to be the case.

          Second question: As I have only Linux on my HD now (Windoze runs in a VM), is it safe to zero out these bytes in the unaddressable sectors?
          Windows no longer obstructs my view.
          Using Kubuntu Linux since March 23, 2007.
          "It is a capital mistake to theorize before one has data." - Sherlock Holmes

          Comment


            #6
            Re: dd Command


            “First question: Does formatting a drive only affect the addressable sectors? Based on my output above, this would seem to be the case.”

            I think so. There's a matter of terminology here, which is always confusing unless you program this stuff: “addressable” by the filesystem? But, yes, it certainly seems so.


            “Second question: As I have only Linux on my HD now (Windoze runs in a VM), is it safe to zero out these bytes in the unaddressable sectors?”

            Yes, you can zero it out. Probably no harm is keeping it, though, in your case – it looks like Vista put a phantom MBR there (note the 55 aa end-of-file signature at the end). Very strange message it put there, too.

            I really don't like giving dd recipes, especially after a day of making typos as I have!
            But it seems it would be this (>>>> please check the seek= bytes):
            dd if=/dev/zero of=/dev/sda bs=512 seek=234436545

            (and as you know from the dd stuff, you use seek rather than skip on the output file (“of”) end of the equation; so before writing zeros to sda, the dd command will skip 234436545 bytes on sda; and, as you know, if anything slips here, you will write zeros to that drive ...)

            I heard somewhere that Vista plants such a thing at the end of the drive ...

            Your post is informative and a good example for others to see and try.
            An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

            Comment


              #7
              Re: dd Command

              Originally posted by Qqmike
              Your post is informative and a good example for others to see and try.
              Thank you. Not one to simply 'leap before looking', I'll leave the data in these last bytes alone (for now). As you stated, "Probably no harm is keeping it,"

              The usefulness of this comparison - that my 'unaddressable sectors' do contain 'data' and yours does not - is that stuff can be written there! While not a Windoze user anymore, outside of keeping Bill trapped within a VM, this might explain why a Windoze user hit with a particularly nasty virus, can't seem to get rid of it, even after reformatting the HD and reinstalling the OS. mbr (Master Boot Record) viri are common enough, but I wonder about eod (End of Disc) varieties??
              Windows no longer obstructs my view.
              Using Kubuntu Linux since March 23, 2007.
              "It is a capital mistake to theorize before one has data." - Sherlock Holmes

              Comment


                #8
                Re: dd Command

                S-H: "... but I wonder about eod (End of Disc) varieties??"

                Yes, and spyware? trojans? various informational plants? who knows!

                It's a good reason, after infection/suspicion, before reformatting and re-installing, to wipe the drive by a zero-fill (or zero-out the drive). If the drive is sdb, that would be, from a live CD:
                dd if=/dev/zero of=/dev/sdb
                or using the DoD version of dd from Helix Live CD:
                dcfldd if=/dev/zero of=/dev/sdb
                or, use a zero-fill program like the SeaTools utility.
                An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

                Comment


                  #9
                  Re: dd Command

                  dd ( data destroyer ) is great as long as you are very careful. Have a backup, right?

                  txWingMan

                  Comment


                    #10
                    Re: dd Command

                    I think by now, that goes without saying, in reference to your anachronism for dd.
                    We did read the first post, right?
                    Anyone using these methods is smart enough to take precautions, just as they do when editing system files like fstab, menu.lst, X.org files, sources.list, ...
                    Or, smart enough to fix things/recover when they don't make those backups, which I'm sure is more often the case.
                    And, there will always be people who will never get if= and of= straight.
                    An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

                    Comment


                      #11
                      Re: dd Command

                      Originally posted by Qqmike
                      Anyone using these methods is smart enough to take precautions, just as they do when editing system files like fstab, menu.lst, X.org files, sources.list, ...
                      Or, smart enough to fix things/recover when they don't make those backups, which I'm sure is more often the case.
                      LOL... Well, I am just not so quick to make assumptions about people.
                      txWingMan

                      Comment


                        #12
                        Re: dd Command

                        Update: dcfldd on Helix CD

                        dcfldd no longer available on free Helix CD:
                        The Helix reference for free CD with dcfldd and how-to's:
                        Helix: http://www.efense.com/helix
                        No longer available free; by membership only.
                        Helix3 vs Helix3 Pro -- changes in the program and products
                        http://www.youtube.com/watch?v=soBNib2ydt0

                        To get dcfldd:
                        sudo apt-get install dcfldd
                        man dcfldd


                        dcfldd
                        > dcfldd(1) - Linux man page
                        http://linux.die.net/man/1/dcfldd
                        > See also, dev notes re verify capability:
                        http://www.networksecurityarchive.or.../msg00004.html
                        > dcfldd - Latest version 1.3.4-1
                        http://dcfldd.sourceforge.net/


                        I have modified the reference section of the OP to reflect this change in Helix products.
                        An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

                        Comment


                          #13
                          NEW-&gt; The dd Command

                          The dd Command

                          Brand new, more tutorial, more complete, tons of examples, much improved, completely rewritten.

                          How to use the dd command

                          In three parts:
                          Part 1: Contains the basics, plenty of discussion and guidance.
                          Part 2 is in Reply #1 and contains lots of applications/examples.
                          Part 3 is in Reply #2 and repeats 6 methods from Part 2, methods useful for
                          my how-to Privacy Cleanup 101, and is expanded and even more tutorial for beginners.

                          You can scroll up to the first post and read the Table of Contents for all three parts.

                          This one is loaded for the beginning-to-intermediate level.
                          16,500 words.

                          Not to say that I won't improve it and add a bit more here and there, I will. But it may be the most complete guide (I know of) on the Internet, mainly because there isn't much out there. Just a fact. I know, because I searched for stuff to read, and much of it is either incorrect or impossible to read or both. It also clarifies issues that are not correctly reported elsewhere, issues people seem to be confused about. (It no doubt needs some proofing and copyediting for some typos, but all the examples are fully tested.)

                          Could be fun stuff for folks wanting to learn some more Linux methods.

                          You don't have to use dd for everything, but almost everyone could find an application or two that would make their work easier.


                          An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

                          Comment


                            #14
                            Use dd to create a swap file.

                            Here is one for your examples section:
                            Create a 1GB swap file.

                            This command creates the empty file
                            $ sudo dd if=/dev/zero of=/boot/swap.img bs=1024 count=1000000
                            1000000+0 records in
                            1000000+0 records out
                            1024000000 bytes (1.0 GB) copied, 3.41333 s, 300 MB/s

                            This command turns the empty file into a swap file
                            $ sudo mkswap /boot/swap.img
                            Setting up swapspace version 1, size = 999996 KiB
                            no label, UUID=795987b3-d0d5-4c5b-8737-0a29199716b7

                            This command tells the system to start using the swap file
                            $ sudo swapon /boot/swap.img

                            This command verifies that the swap file is being used
                            $ swapon -s
                            Filename Type Size Used Priority
                            /dev/sdb5 partition 975868 16 -1
                            /boot/swap.img file 999996 0 -2

                            Lastly you need to edit /etc/fstab and add the swapfile entry
                            $ sudo nano /etc/fstab

                            /boot/swap.img none swap sw 0 0

                            Now I can reclaim my /dev/sdb5 partition by removing the entry from fstab. Or I can leave it as is. If you increase your memory you don't have to fdisk your drive and add more swap space, just create a swap file.

                            <edit> Corrected count in dd command.

                            Comment


                              #15
                              Re: The dd Command

                              Nice example.

                              Let's just check one thing:
                              $ sudo dd if=/dev/zero of=/boot/swap.img bs=1024 count=500000

                              Should that be count=1000000? (To get 1 GB) (Or, use bs=2048?)
                              An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

                              Comment

                              Working...
                              X