The dd CommandA basic guide, examples, tips & tricks, and links
The dd command has been around for awhile but if you search on it, you'll keep running into just a handful of key references, and they all seem to be quoting each other or quoting what little is in the various man pages. The information is all spread about. What follows is a summary of sorts, where I've tried to give the subject some sort of organization. Learning dd is easy if you restrict your use to hard drives and flash drives and use the operand "bs=" (i.e., do not use separate input and output block sizes, ibs and obs), which is what I've done here--not floppies, not tapes, not optical media (dd does not write directly to optical, but there's tricks). What really counts is that you try some things using dd; that seems to be how you learn it. I have tested all the examples.
Theory: You'll find that in Part 1. I've also tried to clarify some misconceptions about dd and to make certain conclusions definitive (where there seems to be confusion among other sources).
Practice: You'll find examples in Part 2 and Part 3. Part 3 repeats six of the examples in Part 2, but Part 3 is expanded, more tutorial, and it is intended to support another of my how-to's: Privacy Cleanup 101 (see References).
Privacy Cleanup 101 explains why it is
sufficient to do a zero-fill (or a random number pass) on a file/partition/disk to prevent recovery of your sensitive data. You'll also find interesting additional references.
Privacy Cleanup 101
http://kubuntuforums.net/forums/index.php?topic=3090100.msg115011#newSome users may find this helpful:
Commands at Konsole: Beginners: 3 parts
http://kubuntuforums.net/forums/index.php?topic=3091607.0= = = = = = = = = = = = = = = = = = = =
Table of ContentsPart 1: The present post is Part 1 and contains the basics.
Part 2 is in Reply #1 and contains lots of applications/examples.
Part 3 is in Reply #2 and repeats 6 methods from Part 2, methods useful for
my how-to Privacy Cleanup 101, and is expanded and more tutorial.
Part 1 Basic dd tutorial and some useful Linux toolsSection 1 Basic principles dd Definition
dd Command statement
A KEY point
How dd works
conv=notrunc, conv=sync, skip and seek, count=n, conv=noerror
block size: bs bytes
dd output: how to read it
Diagnostics -- Examples (records in, records out)
Tips/advice/opinions on using dd
Notes about files (Technical)
dd works relative to your current (working) directory you issue it from
When to use sudo
dcfldd
Checking disk space and partitions: fdisk, ls, df, du
Konsole: configure history (# lines to show)
Section 2 Tools for doing your own experiments Some conclusions: result of dd copy and notrunc, sync
Doing your own experiments to learn
Creating test files, Looking at your results
Useful Linux device files
/dev/sdx, /dev/zero, /dev/null, /dev/urandom, /dev/mem
Hexadecimal numbers
Qalculate: a Hex calculator
mc: Midnight Commander for reading your hex file
Printing your Master Boot Record (MBR)
Offsets, Print the MBR to a file, Using Midnight Commander to view MBR
Part 2 Examples (Part 2 is in Reply #1)See Part 2, Reply #1, for an extensive list of examples, including:
Zero-out the hard drive or flash drive, Zero-out the free space on a partition, Clean up your home, Clean a Partition Before Reusing It or Deleting It, Clean the Master Boot Record, Wipe a file before deleting it (to prevent recovery), Cloning a [bootable] flash drive or a hard drive, Master boot record: viewing and backup/restore, Print a partition boot sector, Creating files, Check a file for errors, Print bytes of a file to the screen, Look for non-zero entries on a drive (HDD or flash drive), Search for a string in your /home partition, Check a random part of a drive to see what's there, Write numbers to the end of a disk, md5sums and dd, Using the Bash command sync after a dd copy, What's at the end of YOUR hard drive?, Backup your Kubuntu OS to your separate /home partition, Bad disk sectors
Part 3 dd -- Wipe partitions, home, files, disks, MBR (Part 3 is in Reply #2)
(A tutorial for Privacy Cleanup 101)
This duplicates material in Part 2 but is written in an expanded, tutorial style, intended to be used in conjunction with my how-to Privacy Cleanup 101.
SECTION 1
Introduction and philosophy
When you delete a file, you do not delete the file: Now what?!
SECTION 2
How to Clean the
Deleted/Unused Space on Your Kubuntu Partition or /Home
SECTION 3
How to Clean a File Before Deleting It
SECTION 4
How to Clean a Partition Before Reusing It or Deleting It
SECTION 5
How to Clean Your Hard Drive or Flash Drive Before Reusing It
SECTION 6
How to Clean the Master Boot Record, 1 sector; and 63 sectors
= = = = = = = = = = = = = = = = = = = =
Disclaimer(1) I've freely quoted, without attribution, where others have. See References!
(2)
WARNING: This is easy stuff except for one thing: Do NOT switch the input file (if) and the output file (of). If you do, you can wipe your files and your drives. dd will not ask you or warn you. No “Cancel” option in this game (other than stopping the Konsole command by issuing Control+c). Use sudo fdisk -lu to correctly identify your drive names /dev/sdx.
= = = = = = = = = = = = = = = = = = = =
Part 1 Basic dd tutorial and some useful Linux toolsSection 1 Basic principles dd Definition
dd Command statement
A KEY point
How dd works
conv=notrunc, conv=sync, skip and seek, count=n, conv=noerror
block size: bs bytes
dd output: how to read it
Diagnostics -- Examples (records in, records out)
Tips/advice/opinions on using dd
Notes about files (Technical)
dd works relative to your current (working) directory you issue it from
When to use sudo
dcfldd
Checking disk space and partitions: fdisk, ls, df, du
Konsole: configure history (# lines to show)
dd Definition: dd = convert and copy a file. The dd utility copies a specified input file to a specified output with possible conversions. dd copies from a “source” to a “target.” dd is (believed to be) an abbreviation for "data definition" in IBM's JCL.
dd Command statement, the basic form:
dd if=<source_file> of=<target_file> bs=<block size> skip=N1 seek=N2 conv=o1,o2,o3
if = input file = source_file: is the data being read.
of = output file = target_file: is where the data gets written to, gets copied to.
bs = block size in bytes (see below); if not specified, the default is bs=1b=512 bytes (b=512 bytes).
conv = conversion to be made (see below). conv=o1,o2,o3,... conversion options are separated by commas with no spacing.
-- Default files: If the "if=" is omitted, dd gets its input file from the terminal screen (which is called "the default standard input"). If the "of=" is omitted, dd prints to the terminal screen (which is called "the default standard output").
-- See man dd.
A KEY pointIMPORTANT: In this how-to, we are treating the special case of dd where "bs=" is used, NOT where separate input and output block sizes are used (ibs, obs). This is the common case most people would ever need dealing with files, partitions, and disks. With "bs=", dd will read an input block of size bs bytes, apply conversions (according to "conv="), and write the output block to the output file. When just the one block size is specified ("bs="), and not separate ones (ibs, obs), dd behaves just as you would expect it to behave and want it to behave. The wording of the rules (below) may at times sound confusing, but rest assured, dd (almost always) does exactly what you'd think it should do, no surprises.
How dd worksdd copies binary data, bit by bit, and exactly every bit, and uses buffers to get the source copied to the target; it stops at the end of the input file (when it gets to the maximum addressable sector of source_file). It reads the input one block at a time, using the block size bs. dd then processes the block of data actually returned, which could be smaller than the requested block size (e.g., if not enough data is left on the input file to make a full block of size bs bytes; or if it encounters errors in the input file). Then dd applies any conversions (conv=) that have been specified and writes the resulting data to the output in one block at a time (of size bs bytes; or less than bs bytes in case there's a partial block at the end and conv=sync is NOT specified).
conv=notrunc Do not truncate the output
This is not complicated at all! But because there is so much mis-information on it, I've dragged it out and addressed common questions about how it works. Basically, one can simply say that conv=notrunc means "do not truncate the output file."
-- If you do not specify conv=notrunc, and if no "seek=" is specified, the output file will be truncated before the copy begins (which means that the output file starts with zero bytes). If you do not specify conv=notrunc, and if you do specify a "seek=N2", dd will preserve the blocks in the output file over which dd seeks (up to N2 bytes), but no other portion of the output file will be preserved. (Thus, if the size of the seek plus the size of the input file is less than the original size of the output file, the resulting output file is shortened by the dd copy when you don't use notrunc.)
-- If you do specify conv=notrunc, it tells dd: do not truncate the output file. Thus, at the start of the dd copy operation, the output file starts out being exactly the file target_file originally specified in the dd statement. Then after dd finishes, the final resulting output file returned by dd will be the same size or larger than target_file but may be partially or totally overwritten by bytes from the input file. Thus, using conv=notrunc, dd does not truncate the output file when it begins its copying.
-- The effect of conv=notrunc has nothing to do with the input file. notrunc is defined only in relation to the output file (i.e., whether or not to truncate target_file before the copy begins).
-- With dd if=/dev/zero of=file2 notrunc bs=1 count=1, and when file2 already exists, dd will replace the first byte of file2 with zero and will retain the remaining bytes of file2 (beyond the first).
With dd if=/dev/zero of=file2 bs=1 count=1, dd will replace the entire file2 with a single zero byte (so the resulting size of file2 will be 1 byte).
-- If you omit of= in the dd statement, like this: dd if=source_file bs=<block size> [and possibly some operands for conv=], then dd will simply print to the screen what it reads from source_file (using bs and conv choices).
-- If you write dd if=<source_file> of=<target_file> [and choices for bs and conv], but the specified target_file does NOT yet exist, then dd will create a file called target_file and perform its copy operation from source_file. The source_file is copied byte for byte to the file called target_file, the resulting size of target_file will be the same as source_file or larger (if there is a partial block copied and conv=sync is used to pad it to a full block size bs).
-- If target_file does not yet exist before issuing dd, and so you have dd if=<source_file> of=<target_file> [and choices for bs and conv] , using conv=notrunc does not make sense, it would have no effect. notrunc is defined only in relation to the output file.
-- If target_file exists before issuing dd, and so you have dd if=<source_file> of=<target_file> [and choices for bs and conv], and target_file is smaller than source_file, when dd is done, target_file will be identical to source_file, with or without using conv=notrunc.
-- If target_file exists before issuing dd, and so you have dd if=<source_file> of=<target_file> [and choices for bs and conv], and target_file is larger than source_file, when dd is done, target_file will be identical to source_file if you did NOT use conv=notrunc; if you used conv=notrunc, target_file will be the same size as it was before dd, but its contents (bytes) at the beginning will be overwritten by the contents of source_file (and target_file will be the same as it was beyond those bytes from source_file).
--
Everything said here applies to disk devices /dev/sdx and partitions /dev/sdxn. For example, if disk sdx is 500 MB and has unallocated space except for a 100 MB partition, when you copy sdx to an image file, the entire 500MB will be copied, even though 400 MB are empty or unused or unallocated, and whether or not you use conv=notrunc. (It doesn't matter, 500 MB gets copied IF there is room; i.e., the unused parts of sdx also get copied with or without notrunc). If you copy a 500 MB disk to an existing 700 MB file called imagefile, without using notrunc, all 500 Mb wil get copied and the final size of imagefile will be only 500 MB; if you use notrunc, the final size of imagefile will be 700 MB but the first 500 MB will be identical to the disk. There's a lot of confusion about this, but you can test it for yourself (using the methods we discuss below). (If you try to copy a big partition to a small partition, with or without notrunc, there will not be enough room and the copy will stop when the small partition is full. Same goes for copying disks. But for "regular" files (not device files), if you copy a big file to a small file, the small file will expand and fill up with a copy of the input file.)
conv=syncUpon reading the input file, dd may return less than a full block of bs bytes. This could happen if there is not enough data left on the input file to fill up a block of bs bytes. It can also happen even when the input file has plenty of bytes (e.g., /dev/zero has infinitely many zeros), and the target_file is almost full toward the end of the copy but requires only a partial input block to fill it up completely. It could also happen if dd encounters an error (in the input file) while reading the input file and thus can not read a full block size of bs bytes.
-- If dd returns less than a full block (from reading the input file) and the conv=sync conversion is not specified, dd will write whatever it gets to the output file (something less than a full bs-byte block size).
-- If dd returns less than a full block (from reading the input file) and the conv=sync conversion is specified, null bytes are appended to the input data up to the block size bs bytes. Any remaining conversions and output will include the pad characters as if they had been read from the input. Thus, when conv=sync is used, all output blocks written to the output file will always be bs bytes in size (even if they have to be padded with null bytes to make that happen).
skip and seekskip=n tells dd to skip n blocks of size bs bytes at the beginning of the source_file before starting to copy.
seek=n tells dd to skip n blocks of size bs bytes from the beginning of the output file before copying (i.e., seek skips over n blocks on the output target_file before writing to target_file).
count=nCopies only n input blocks (each of size bs bytes). (You may append this anywhere in the dd statement.)
conv=noerrorNoerror means to keep going if there is an error. Normally dd stops at any error. When an input error occurs, a diagnostic message is written on standard error (i.e., the terminal screen), followed by the current input and output block counts in the same format as used at completion.
If the sync conversion is specified, the missing input is replaced with null bytes and processed normally.
If you omit conv=noerror: The "bad" part of the input block will be omitted from the output. Any partial output block is written to the output file, a diagnostic message is written, and the copy operation stops.
Advanced note: If you are copying a filesystem, using conv=noerror could give you problems since the output file may then have a damaged filesystem.
block size: bs bytesIf omitted, the
default is 512 bytes = 1b, where b=512.
bs can be any number of bytes; often, a power of 2 is used: 512, 1024 (=2b), 2048, 4096, etc; a number may end with k, b, or w to specify multiplication by 1024, 512, or 2, respectively, (w = two bytes = a "word"). Numbers may also be separated by x to indicate multiplication (e.g., 2bx8). b: 512, kB: 1000, K: 1024, MB: 1000x1000, M: 1024x1024, GB: 1000x1000x1000, G: 1024x1024x1024. You can use any block size bs you wish. If bs is too big, it may slow dd down, having to read, work with its buffers, then write large blocks. Of course, if bs is too small, dd may also slow down because it has so many read-write's to do.
> Some block sizes to try are bs=512 (the default: 512 bytes = 1 sector of the disk); bs=4096 (= 8 sectors); and bs=63x255b (=63x255x512 = 8,225,280 bytes = 1 "cylinder").
dd output: how to read it-- Exit status
Exit values are returned: 0 The input file was copied successfully. >0 An error occurred. If an input error is detected and the noerror conversion isn't specified, any partial output block is written to the output file, a diagnostic message is written, and the copy operation stops. If some other error is detected, a diagnostic message is written and the copy stops.
-- Diagnostics
f+p records in (out)
=> number of full (f) blocks read and partial (p) blocks (written)
(a "record" is the same as a block as specified by" bs=")
ExamplesFor each example, assume test1 is a file of size 100 bytes; test2 is 1000 bytes
dd if=test1 of=test2 conv=notrunc bs=20
5+0 records in
5+0 records out
100 bytes (100 B) copied, 5.5989e-05 s, 1.8 MB/s
Interpretation: 5 full blocks of 20 bytes each (and no partial blocks) were read from test1; 5 full blocks (and no partial blocks) were written to test2. Total: 100 bytes copied (to test2). Result: test2 is still 1000 bytes because of notrunc, but its first 100 bytes are identical to the 100 bytes of test1.
dd if=test1 of=test2 conv=notrunc bs=30
3+1 records in
3+1 records out
100 bytes (100 B) copied, 5.5989e-05 s, 1.8 MB/s
Interpretation: 3 full blocks of 30 bytes each (and one partial block) were read from test1; 3 full blocks (and 1 partial block) were written to test2. Total: 100 bytes copied (to test2). The partial block was 10 bytes. Result: test2 is still 1000 bytes because of notrunc, but its first 100 bytes are identical to the 100 bytes of test1.
test3 is a file of 1000 bytes.
test4 is a file of 100 bytes.
dd if=test3 of=test4 bs=30 conv=sync
33+1 records in
34+0 records out
1020 bytes (1.0 kB) copied, 0.000238582 s, 4.3 MB/s
Interpretation: 33 full blocks of 30 bytes each (and one partial block) were read from test3; 34 full blocks (and 0 partial blocks) were written to test4. Since, for test3 (which is a 1000-byte file), 33x30=990 bytes, the partial block was 10 bytes. Since conv=sync was used, the partial block of 10 bytes was padded using 20 null bytes to make it a full block of 30 bytes. Thus, 34 blocks were then written to test4. Total: 1020 bytes copied (to test4) (since 34x30=1020). Result: test4 is now 1020 bytes: the first 1000 bytes are identical to the 1000 bytes of test3; the remaining 20 bytes are the (padded) null bytes (hex zeros).
Tips, advice, opinions on using dddd is used for many things, it's a very handy tool. Experts may be able to make it do whatever they want. This how-to will give you plenty of interesting and useful applications. However, be aware of the following:
> dd is not filesystem-aware. Furthermore, there are some technical/logistical issues when copying a partition to another partition, even if the target is the same size as the source. These issues involve metadata, truncation (or it reverse, padding block devices with nulls and the effect that has on filesystems), what happens when copying a larger partition to a smaller one (and vice versa), and issues involving the partitions tables and the bootloader. I would not recommend using dd to copy one partition to another; however, people do it. See the References for some help on this. For copying filesystems, I would use a filesystem-aware utility (e.g., Gparted, Partimage).
> However,
dd is very nice to use when cloning an entire disk. A good example is using dd to clone a flash drive (even a bootable one). dd will copy everything exactly: the filesystem, the Master Boot Record, the bootloader setup. The target will be ready-to-go.
> When cloning a disk, (in my opinion) you should do so to an identical, same-size disk:
dd if=/dev/sdx of=/dev/sdy [and if you wish to use it: conv=noerror], where sdx and sdy are two same-size disks (hard drives or flash drives).
Notes about filesSome of this is too technical for now, but make a note of it and return to it later after you study dd.
> Do not use dd to copy files between file systems having different block sizes (the filesystem block size is different from the bs we are talking about in dd).
> dd does not work with directories; it copies files. (Note that a Linux block device is a file; e.g., /dev/sda.)
> You can't write a file directly to a device. You have to write the file to a file in a directory. For example, to write file myfile to /dev/sdc2, you need to mount sdc2, as say /media/sda2, then do something like dd if=/path/to/myfile of=/media/sdc2/myfile. (In particular, if you write dd if=/path/to/myfile of=/media/sdc2/, or try to use of=/dev/sdc2, dd will complain that /media/sdc2 or /dev/sdc2 is a directory (i.e., it is not a file).)
> When working on device files (/dev/sdxn) using dd, they need not be mounted (except see the previous point). You can copy bytes from sdx without mounting it (e.g., its MBR). Sometimes, if you are getting errors, it is recommended you unmount it if it is mounted.
> Device files obey just as other files when using dd to copy. To get a file image of a disk:
dd if=/dev/sdx of=diskimagefile. If diskimagefile doesn't already exit, it will be created (in your current working directory; or use a path: /path/to/diskimagefile). If diskimagefile exists before issuing the dd command, and if you don't use conv=notrunc, then after dd, the resulting file diskimagefile will be the same size as sdx. If diskimagefile exists before issuing the dd command, and if diskimagefile is larger than sdx, and if you use conv=notrunc, then after dd, the resulting file diskimagefile will be larger than sdx, and its contents at the beginning will be replaced by the contents of sdx, and the bytes of diskimagefile not affected by the dd will be preserved (following the image of sdx embedded in diskimagefile).
dd works relative to your current (working) directory you issue it fromFor example, if you use
cd /
and then issue
dd if=file1 of=file2
then file2 will be located directly under root (at the top level of the root filesystem: /file2).
If you use
cd ~ (or: cd /home/your_name)
to change into your home directory, and then issue
dd if=file1 of=file2
then file2 will be located in your home. If you do: cd Desktop and issue dd if=file1 of=file2, then file2 will be located in your Desktop (/home/your_name?Desktop/file2).
Now suppose you are at the level of / (cd /), and you want file2 to be saved to your Desktop. Then issue it this way:
dd if=file1 of=/home/your_name/Desktop/file2
That is, give the absolute path (the full path) to file2.
When to use sudoUse sudo whenever you receive the dd error: Permission denied. That means you are trying to read, write, access some file for which you do not have permissions (usually owned by root). Examples include using device files (/dev/zero, /dev/urandom), reading the MBR, writing to devices (not owned by you), etc. Form: sudo dd if=file1 of=files.
dcfldddcfldd is based on dd and is an enhanced version of dd. It was developed at USA Dept. of Defense.
dcfldd includes progress reporting and verification, as well as various methods for ensuring accuracy and speed. To get dcfldd, use your package manager, or:
sudo apt-get install dcfldd
See: man dcfldd. Syntax and many command-line options are the same as for dd.
Suggestion: If you are doing a lot of zero-fills of large drives, you might want to try dcfldd. You can run it just as you do dd, from a live CD, from GParted Live CD, from Kubuntu, even from a flash drive. In place of dd in the recipes, substitute dcfldd. Use dcfldd, also, if you find that dd dropped the last sector of a source file.
dcfldd: references
> dcfldd(1) - Linux man page
http://linux.die.net/man/1/dcfldd> See also, dev notes re verify:
http://www.networksecurityarchive.org/html/Computer-Forensics/2005-05/msg00004.html> dcfldd - Latest version 1.3.4-1
http://dcfldd.sourceforge.net/Checking disk space and partitionsfdisk, ls, df, du
To check your partitions and their sectors:
sudo fdisk -lu
To check disk space, use ls, df, du:
ls and du: work relative to the directory you issue them from
df: works from any directory you are presently in
ls -al See man ls.
List all files (in the present working directory) in long form and shows their size. A quick way to check things. First, change directories (cd) into the directory (folder) you wish to check.
df -hT See man df.
A quick way to see your mounted filesystems, the space Used and Available, your root (/) and /home directories. You can issue this from any directory and get the same result.
du See man du.
Shows you the disk usage of each file. Change into the directory you wish to check. du will check it and subdirectories recursively. So you run it, study the output, pick a directory from the output to investigate further, change directories into that directory (using cd), run du again, study the output, etc. Try these now, first changing into the root directory (cd /), then after changing into your home directory (cd ~ , or: cd /home/your_name).
du -sh is a summary (-s) only, in human readable form (-h)
du -h shows directory details (and subdirectories).
du -ah shows (all = -a) files
See next topic about configuring Konsole.
Konsole: configure history (# lines to show)When you issue a command that prints a large number of lines, you can scroll up to see lines that have scrolled off the screen; or use Up arrow or Page Up. The default is to show you 1000 lines this way. If you need more (e.g., using du in certain cases), you can configure Konsole to show them all:
Open Konsole, Settings, History, History Configuration, Set Unlimited, OK.
Section 2 Doing your own experiments Some conclusions: result of dd copy and notrunc, sync
Doing your own experiments to learn dd
Creating test files
Useful Linux device files
Hexadecimal numbers
Qalculate: a Hex calculator
mc: Midnight Commander for reading your hex file
Printing your MBR: dd, hex numbers, offsets, midnight commander
Some conclusions: result of dd copy and notrunc, syncA good way to learn dd is to practice, using some simple test files. You will quickly get the feel of it. If you study the basics in Section 1, you'll see how dd works in common situations. Here's a few patterns you should be able to think through. If you'd rather not read this, that's OK! Skip to the next subsection: "Doing your own experiments to learn."
Let's say you have
dd if=fileA of=fileB bs=? conv=?,?
where A and B are regular files of finite size.
If you do NOT use conv=notrunc, then the result of dd will be a fileB of the same size of larger than fileA:
If conv=sync is NOT used, fileB will be identical to fileA.
If conv=sync is used, and bs divides evenly into the size of fileA,
then fileB will be identical to fileA.
If conv=sync is used, and bs does NOT divide evenly into the size of fileA,
then there will be a partial block copied to fileB, and
the last block copied will be padded with nulls and copied to fileB
making fileB somewhat larger than fileA.
fileB will be an even multiple of bs, but will contain some nulls at the end.
If you use conv=notrunc, then the result of dd will depend:
If fileA is smaller than fileB, it's bytes will be copied to fileB.
If there is a partial block at the end of fileA,
and if you used conv=sync,
it will be padded with nulls and copied to fileB.
fileB will contain its original bytes except the bytes at the beginning
will have been overwritten with bytes from fileA,
possibly with some nulls added (if you used sync and
if there was a partial block at the end of the fileA read).
If fileA is larger than fileB,
file B will be overwritten with the contents of fileA
possibly with some nulls added (if you used sync and
if there was a partial block at the end of the fileA read).
Doing your own experiments to learn Creating test files
Looking at your results
On your Desktop (or wherever you like to work where things will be clear to you), create some test files to play with. Say file1 with 100 bytes, file2 with 500 bytes. Then try various statements:
dd if=file1 of=file2 bs=? conv=?,?; or try a copy from file2 to file1 (big to small),
testing different values for bs and different combinations of conv=notrunc and conv=sync.
Choose bs values that are large, small, some that divide evenly into the size of file1, and some that do not divide evenly into the size of file1.
Creating test filesTo create a file of 100 zero bytes:
Change into the directory you want to be in (e.g., cd Desktop).
dd if=/dev/zero of=testfile bs=1 count=100
To create a file of 100 random (numbers) bytes:
dd if=/dev/urandom of=testfile bs=1 count=100
Another useful file, especially for experimenting with skip and seek, might be this:
On your Desktop, right-click to Create a New text file. Then type lines into it that you will be able to identify while testing:
1111111111
2222222222
3333333333
etc.
Looking at your resultsIn some cases, simply looking at the files on your Desktop will be revealing about the result of a dd copy. (Remember to right-click, Refresh Desktop.) Otherwise, use Midnight Commander (see below), keeping it open as you experiment. (Remember to refresh the panes after changing files: click Left > Rescan.)
Useful Linux device filesThe /dev directory contains the special device files for the Linux devices. The device files are created during installation, and later with the /dev/MAKEDEV script.
/dev/sdx (e.g., sda, sdb, sdc, ...)
These are your drives (hard drive, external drive, flash drive, etc.). There may also be /dev/hda, /dev/hdb, etc.
/dev/zero
This is a simple way of getting many 0s. Every time you read from this device it will return 0. This can be useful sometimes, for example when you want a file of fixed length but don't really care what it contains. It is a character device.
/dev/null
The bit bucket. A black hole where you can send data for it never to be seen again. Anything sent to /dev/null will disappear. This can be useful if, for example, you wish to run a command but not have any feedback appear on the terminal. It is a character device. You can use null to find the size (in bytes) of a drive or flash drive:
sudo dd if=/dev/sdc of=/dev/null bs=4096
123007+1 records in
123007+1 records out
503840256 bytes (504 MB) copied, 41.9472 s, 12.0 MB/s
Thus, the drive has 503,840,256 bytes.
dd read the drive sdc, reading in blocks of 4096 bytes at a time, recorded that fact (in its own registers), dumped those bytes into the /dev/null black hole, then grabbed another block of 4096 bytes to read and count (as a record in - record out). No harm is done to the drive /dev/sdc; it remains the way it was before dd. The dd output shows us how many bytes were successfully read, and we take that as the number of bytes we have to work with in sdc.
Random number generators (see man urandom)
/dev/random (“true” random numbers; it uses and is limited to current entropy pool)
/dev/urandom (uses pseudo random numbers, unlimited)
To make a file of 100 random bytes, many ways to do it:
dd if=/dev/urandom of=/home/my_name/myrandomfile1 bs=100 count=1
or, dd if=/dev/urandom of=/home/my_name/myrandomfile2 bs=1 count=100
/dev/mem is your system memory. mem is a character device file that is an image of the main memory of the computer. See man mem.
See: The Linux System Administrator's Guide, Section 3.4
http://tldp.org/LDP/sag/html/index.htmlHexadecimal numbersYou run into these numbers quite often in Linux. At the very least you should recognize them and be able to convert them to familiar decimal numbers. In Linux, they usually are represented by the prefix 0x for numeric constants represented in hex; as in 0x64A2, where the hexadecimal number is 64A2. Then, you should be able to convert it to decimal if necessary, and for that most people use a hex calculator or hex converter. See the following topic: Qalculate.
Optional: What is a hexadecimal number?
You are familiar with decimal numbers, also called base 10 numbers. The basic numerals are the 10 numbers 0, 1, 2, ..., 9. Other decimal numbers are products of these and powers of 10 and sums of such products. The powers of 10 are (where ^ indicates exponent or power): 10^0 = 1, 10^1 = 10, 10^2 = 10x10 =100, 10^3 = 1000, 10^4 = 10000, 10^5 = 100000, and so on. The decimal number 126 (as an example), is representable as (starting from the rightmost digit) 126 = 6*(10^0) + 2*(10^1) + 1*(10^2) = 6*1 + 2*10 + 1*100 = 6+20+100 = 126.
The hexadecimal system works the same way, exactly, except instead of "10" we use "16" as the base. The basic numerals are the 16 numbers 0, 1, 2, 3, ... 9, 10, 11, 12, 13, 14, 15. However, using 10-15 is confusing, so those numbers are represented by letters: A (10), B (11), C (12), D (13), E (14), and F (15), or their lower case a, b, c, d, e, f (so 5EB52 is the same as 5eb52). Powers of 16 are: 16^0 = 1, 16^1 = 16, 16^2 = 16x16 = 256, 16^3 = 16x16x16 = 4096, 16^4 = 65536, 16^5 = 1,048,576, and so on. A hexadecimal number is also an expansion of powers of 16, starting from the right. Example: Hexadecimal 21B = B* (16^0) + 1*(16^1) + 2*(16^2) = 11*1 + 1*16 + 2*256 = 539 (in decimal equivalent).
By the way, all bases work this way. For example, binary numbers are base 2, the basic numerals are 0 and 1, and a binary number represents an expansion using powers of 2. Example: 1111 (a binary number) = (starting from the rightmost "1") 1*(2^0) + 1*(2^1) + 1*(2^2) + 1*(2^3) = 1*1 + 1*2 + 1*4 + 1*8 = 1 + 2 + 4 + 8 = 15.
In binary, 0000 through 1111 represent decimal numbers 0 to 15 (the basic hex numbers). Thus each hexadecimal digit can be represented by four binary digits. You can easily convert binary to hexadecimal using this fact. Example: the binary number 01011110101101010010 (binary) = 0101 1110 1011 0101 0010 = 5 14 11 5 2 = 5EB52 (base 16). Using a hex calculator (see below), the hexadecimal number 5EB52 = 387922 (decimal).
See
http://en.wikipedia.org/wiki/HexadecimalQalculate: a Hex calculatorYou can use Qalculate to quickly convert between the number bases 16 (hexadecimal) and 10 (decimal) (and also octal base 8 and binary base 2). Use your package manager to install Qalculate or use sudo apt-get install qalculate. Then find it in K > Science and Math > Qalculate. Open Qalculate and then File > Convert Number Bases.
mc: Midnight Commander for reading your hex fileIf you goof around experimenting enough, it is handy to be able to view a file written in hexadecimal representation as a "regular" (ASCII) file. A quick way to do this is to use the file manager Midnight Commander. Install it using your package manager. To use it, open Konsole, type its command name and press Enter:
mc
You'll see right and left panes, each showing your file system. You may type a pathname after your prompt (at the bottom of the screen); or simply use arrow keys to navigate to your a file or double-click on a directory (folder) to open it (or hit Enter to open it). Your mouse also works in mc. To return to the previous menu, highlight Up--Dir and hit Enter. When you find the file you wish to view, highlight it, click View (at the bottom), then click Hex or Ascii (try going back and forth). The addresses (offsets) are given down the leftmost column under View > Hex. When you are done with the file, click Quit (perhaps twice, depending on which Quit you use). If you've changed a file, to view its new version in mc, navigate to the window containing the file (say in the left pane), then click Left > Rescan. See the next example: Printing your MBR.
When doing this work, you may want to open two instances of Konsole. Use mc to open one of them and keep it open to mc (using Left > Rescan as necessary). Open another instance of Konsole to use for your dd commands.
Printing the Master Boot Record (MBR)using dd, hex numbers, offsets, midnight commander
Print MBR to the screen; print MBR to a file
Master Boot Record = MBR = the first 512 bytes (i.e., the first sector) of the disk.
446 bytes is for the IPL (Initial Program Loader: usually GRUB in Linux)
64 bytes is for the partition table (= 16 bytes for each of the four primary partitions)
2 bytes for the “55 AA” signature (the "end of file" for the MBR)
Total = 512 bytes
Since you are accessing your MBR (on a device owned by root), you'll need sudo:
To print the MBR to the screen (the default output file of=target_file):
sudo dd if=/dev/sdx count=1 | hexdump -Cwhere /dev/sdx is your hard drive (or flash drive).
To find sdx, use sudo fdisk -lu.
(If you forget to write count=1, you can stop the output using Control+c.)
The dd statement reads just one sector (count=1, and the default bs=512 is used). It then pipes (that's the vertical bar) that output to be the input of the hexdump program (see man hexdump), and hexdump prints its result to the default standard output, the screen. This is a standard technique you will see often.
Note the output Diagnostics:
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00650396 s, 78.7 kB/s
(makes sense--see Section 1: How to read dd output)
The eight-digit numbers running down the left column are offsets
There are 32 rows, 16 columns, and so 512 entries in the table, each entry representing one byte (= 8 bits = 1 character; note that each of the 512 8-bit entries is represented as two hex digits). The 512 characters in the table are in hexadecimal; their "normal" ASCII equivalents are shown in the right column (which DOES rather run together!). Notice the last two bytes 55 aa. They signify "end-of-file" or EOF for the MBR. You can find such EOF's all over your disk if you have many partitions. You may be able to make out some familiar words in the printout (right column). Compare this to using Midnight Commander (next, below). By the way, don't worry if you see something like
| G |
| RUB .Geom.Hard D |
| |isk.Read. Error . |
(It seems to be normal. Note "GRUB.")
OffsetsThe offset of the first line is 00000000, indicating zero bytes from the start of the disk. Then count 16 bytes in that line, from left to right (1 through 16).
The offset of the last line is 000001f0, a hexadecimal number. Use Qalculate to convert it to decimal 496 (bytes), which indicates that the first entry in the last row is at 497 bytes; the last entry, aa, is at 512 bytes (from the start of disk sdx).
To print the MBR to a filesudo dd if=/dev/sdx of=/home/your_name/mbr_copy bs=512 count=1 The dd statement reads the MBR and writes it to the file in your home named mbr_copy.
Note that for fun this time, we explicitly indicated the block size bs=512; but since that's the default, we can also omit writing that. If you look in your home, you'll see that file. Right-click > Properties shows it is 512 bytes. Try to open it using Kate: right-click > Open with > Utilities > Kate, and you'll be warned it is a binary file and not to mess with it. Go ahead and have a look and see if you see anything familiar, then close it.
Using Midnight Commander, mc to view the fileOpen Midnight Commander by typing at Konsole mc.
Using arrow keys, mouse, or by typing the filename /home/
your_name/mbr_copy at the prompt in mc (toward the bottom of the screen), highlight the file, click View, then Hex, then try Ascii. This is the same output as above. Midnight Commander enables you to "read" (sort of!) the binary file.
= = = = = = = = = =
ReferencesCommands at Konsole: Beginners: 3 parts
http://kubuntuforums.net/forums/index.php?topic=3091607.0Part 3 is intended to support Privacy Cleanup 101 and is referenced by it.
Privacy Cleanup 101
http://kubuntuforums.net/forums/index.php?topic=3090100.msg115011#newdd
http://en.wikipedia.org/wiki/Dd_(Unix)
IEEE specs, dd
http://www.opengroup.org/onlinepubs/009695399/utilities/dd.htmlExcellent man page:
Sun Microsystems Documentation (one that everyone quotes, along with man dd)
http://docs.sun.com/app/docs/doc/817-0708/6mgg6t7bs?q=logins&a=viewAn excellent source, with links (choppy, incomplete, but lots of info):
http://www.softpanorama.org/Tools/dd.shtmlLearn the dd Command (Author: AwesomeMachine)
http://www.linuxquestions.org/questions/linux-newbie-8/learn-the-dd-command-362506/See this site if you want to copy a partition to another partition. The how-to is in the first two posts. Also read the posts of the thread where users have raised some good issues (use the search feature on your browser). The thread contains many dozens of examples and good discussion. Site is maintained/updated.
dd is part of GNU Coreutils:
Coreutils, dd
http://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.htmlIcewalkers, man pages
http://www.icewalkers.com/Linux/ManPages/dd-1.htmlHexadecimal number base
http://en.wikipedia.org/wiki/HexadecimalLinux file system and device files
See: The Linux System Administrator's Guide, Section 3.4
http://tldp.org/LDP/sag/html/index.htmldcfldd
> dcfldd(1) - Linux man page
http://linux.die.net/man/1/dcfldd> See also, dev notes re verify:
http://www.networksecurityarchive.org/html/Computer-Forensics/2005-05/msg00004.html> dcfldd - Latest version 1.3.4-1
http://dcfldd.sourceforge.net/dd_rescue: Variation of dd for rescuing data off defective hard drive with bad sectors.
http://www.garloff.de/kurt/linux/ddrescue/Damaged disks, damaged MBR:
http://www.cgsecurity.org/wiki/Damaged_Hard_Disk#The_old_and_slow_method_using_.27dd.27http://www.cgsecurity.org/wiki/TestDiskTwo excellent papers on bad sectors and repairing them are:
Bad Sector Blues:
The Dreaded "error=0x40 { UncorrectableError }"
http://www2.uic.edu/~aciani1/sector_blues.htmlBad block HOWTO for smartmontools
http://smartmontools.sourceforge.net/badblockhowto.html
Author
