The dd Command

[#!/bin/bash]

Re: The dd Command

Nice example.

Let's just check one thing:
$ sudo dd if=/dev/zero of=/boot/swap.img bs=1024 count=500000

Should that be count=1000000? (To get 1 GB) (Or, use bs=2048?)

Thanks.

You are right. I ran the command twice and created a 500Mb file the first time then created the 1Gb file. I guess I copied the wrong command. I think the rest of it is correct though.

[Qqmike]

Edits to:

The dd Command Parts 1,2, and 3:
https://www.kubuntuforums.net/showth...The-dd-Command

https://www.kubuntuforums.net/showth...l=1#post110611

https://www.kubuntuforums.net/showth...l=1#post117130

I went through all three parts and caught some typos, nothing really significant (except one statement, where b2=4096 should be bs=4096): Edits include Parts 1, 2, 3 in Posts #1, #2, #3.

Boy, if you are content with how your Kubuntu OS is running, bored, looking for something to do, don't know dd or are rusty, try this stuff ;-) (if you're nuts enough to do so).

At one time, when I wrote this, and did all the experiments/examples, I was hot on it, suitably obsessed. But then didn't look it, except to do occasional tasks with dd. Now I see there's some neat things here I had forgotten and that one could play with some more:

-- Cloning a flash drive to an identical one (which I do find myself doing on occasion).
-- Using sync after dd (I notice S-R doing this on occasion).
-->> Backup your Kubuntu OS root partition to your separate /home partition (as a file). Which, at one time, I used to do, and it worked well--do see the discussion in Part 1 on this, though.
-- A reminder about using Qalculate.
-- Using dcfldd (it's listed in Muon Package Manager)--pretty neat.
-- And using dd to clean stuff for privacy, before reusing it, like unused space on your home or root partition; a file; a partition; a flash drive or HDD; a MBR/partition table, or the first 63 sectors of a MBR scheme. For example, after running Bleachbit, how to clean the empty space leftover.


The examples involving BIOS are based on the traditional MBR scheme, its 512 bytes, the first 63 sectors, and so on. Thought about extending the examples to the GPT scheme, and I may do that sometime. But I realize that the reader can easily do it simply by applying the exact principles in the how-to. Easy enough: just get your skips, seeks, and counts right ... and your bs (that would be, of course, block size). It would involve exploring (uhmmm, playing with) the many features of a GPT setup:

The Protective Master Boot Record
1 MiB gap and alignment
Partition Table Header
Partition entry array
The last sector of the disk (LBA -1) contains the secondary or backup Partition Table Header
and the sector preceding the last sector (LBA -2) contains the secondary or backup partition table.
And possibly, in some setups, the Bios Boot Partition.

For this stuff, see the study guide and references therein:
GPT, UEFI -- Study Guide
https://www.kubuntuforums.net/showth...l=1#post346604

Added:
I just noticed that TestDisk has been extended for EFI GPT, very nice. Thought of this with the dd experiments, messing up a partition (intentionally, or as a test) and recovering it using TestDisk.

[jlittle]

<nostalgia mode="old fart">
I can remember solving problems using dd in the middle 1980's. With half inch tape, I think. Block sizes really were block sizes.
</nostalgia>

[Qqmike]

jlittle, and so, with tapes, and with dd, that's where they used the more technical (and, to me, confusing) ibs and obs (input, output block sizes) rather than the single bs (block size).

[Qqmike]

dd: Showing the progress as dd executes


As dd runs, it does not show you its progress; it only reports at the end when it is done--and that can take quite awhile for some jobs. There are a couple easy (non-expert) ways to see the progress of dd as it copies.

Do you have GNU Coreutils 8.24+ (Ubuntu 16.04 and newer)?

(1) If so, you can use status=progress in your dd statement. See below.

(2) If not, then you can use the utility pv in your dd statement. See below.


What version of coreutils do you have?
Here are four ways to check:
-- Muon, type coreutils, left-click on coreutils entry to highlight it, click the version tab.
-- At Konsole, run apt-cache policy coreutils
-- At Konsole, run sudo dpkg -s coreutils
-- Check the Ubuntu packages:
http://packages.ubuntu.com/search?keywords=coreutils
It shows the version according to each K(U)buntu version; for example:
Package coreutils
precise (12.04LTS) (utils): GNU core utilities, 8.13-3ubuntu3.3
trusty (14.04LTS) (utils): GNU core utilities 8.21-1ubuntu5.1
xenial (16.04LTS) (utils): GNU core utilities 8.25-2ubuntu2
yakkety (16.10) (utils): GNU core utilities 8.25-2ubuntu2
zesty (utils): GNU core utilities 8.25-2ubuntu2


Btw, what is coreutils?
https://en.wikipedia.org/wiki/GNU_Core_Utilities
http://www.gnu.org/software/coreutils/coreutils.html



(1) dd progress for GNU Coreutils 8.24+ (K(U)buntu 16.04 and newer)


--> Use the status=progress option in your dd statement.
See man dd (at Konsole in your K(U)buntu 16.04 or newer version OS).

Example To zero-out a 16 GB flashdrive and see the progress as dd executes:

 [FONT=Liberation Serif][COLOR=#000000]mike@mike-All-Series:~$ [/COLOR][SIZE=3][COLOR=#000000][B]sudo dd if=/dev/zero of=/dev/sdb bs=16M status=progress[/B][/COLOR][/SIZE][/FONT]
 [COLOR=#000000][FONT=Liberation Serif]...[/FONT][/COLOR]
 [COLOR=#000000][FONT=Liberation Serif]it will show you the #bytes completed and the rate (MB/s) as it goes along; and when done, it shows you the result:[/FONT][/COLOR]
 [FONT=monospace]
[FONT=Liberation Serif]1593835520 bytes (1.6 GB, 1.5 GiB) copied, 46.8188 s, 34.0 MB/s
...[/FONT][/FONT]
 [FONT=Liberation Serif][COLOR=#000000]1610612736 bytes (1.6 GB, 1.5 GiB) copied, 71.1188 s, 22.6 MB/s[/COLOR]
...[/FONT]
 [FONT=Liberation Serif][COLOR=#000000]2164260864 bytes (2.2 GB, 2.0 GiB) copied, 248.283 s, 8.7 MB/s [/COLOR]
...
[COLOR=#000000]6325010432 bytes (6.3 GB, 5.9 GiB) copied, 1425.35 s, 4.4 MB/s [/COLOR]
...[/FONT]
 [FONT=Liberation Serif][COLOR=#000000]15602810880 bytes (16 GB, 15 GiB) copied, 4003.56 s, 3.9 MB/s   [/COLOR]
dd: error writing '/dev/sdb': No space left on device[/FONT]
[FONT=Liberation Serif][COLOR=#000000]931+0 records in [/COLOR]
930+0 records out 
15610576896 bytes (16 GB, 15 GiB) copied, 4508.64 s, 3.5 MB/s[/FONT]

Note: We used a block size of 16 MiB, bs=16M, which is kind of arbitrary, and up to the user. You can omit bs= and it will use the default bs=512. Or try bs=4096. See man dd.


(2) dd progress for GNU Coreutils older than GNU Coreutils 8.24

--> Use the pv utility in your dd statement.


First, get the pv package.
For example, run at Konsole: sudo apt-get install pv. Or, open Muon, type pv in the search window, mark it for Installation, and click Apply.
Then see man pv.

From the package description:

pv - Pipe Viewer - is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.

ref.: https://www.ivarch.com/programs/pv.shtml


Example Zero-out a 16 GB flashdrive and see the progress as dd executes

dd occurs twice in the statement below, and each occurrence requires sudo (sudo dd). Instead of doing it that way (which would also work), we will first get a root terminal using sudo su; then proceed (without sudo's); when done, issue an exit so we return to the normal user (non-root) prompt. The | is a pipe symbol. Like this:

mike@mike-desktop:~$ sudo su
root@mike-All-Series:/home/mike# [B][SIZE=3]dd if=/dev/zero | pv -s 16G  -pterb |  dd  of=/dev/sdb bs=16M[/SIZE][/B]

 ...

as it goes along, it will show you its progress from time to time
 ...
and then it will finish:
0+188740 records in 
0+188739 records out 
14.5GiB 1:11:08 [3.49MiB/s] [====================================>     ] 90%             

 15610576896 bytes (16 GB, 15 GiB) copied, 4268.09 s, 3.7 MB/s 
root@mike-All-Series:/home/mike# exit 
exit 
mike@mike-All-Series:~$

Note about pv

See man pv. Thus, we used the five default options -pterb, along with our estimate of the size of the copy, which we set at 16 GiB: -s 16G . If you want progress reports on percent complete, you must supply an estimate of the size; i.e., you must use the -s option.