A new privacy and log scrubber for KDE4, Firefox, & Flash

[Qqmike]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

"I don't think there is any way to guarantee no leakage on journaled filesystems, though, especially because you don't know where else the fs may have stored pieces of the file before srm even got to it."

That's probably the best argument, rendering moot further analysis of the effects of journaling.


http://www.freesoftwaremagazine.com/...isks_gnu_linux
-- yes, that's one of the articles that got me started on this.


"Certainly not. rm only marks the space consumed by the file as 'available', it doesn't remove the data. (That's why it's so much faster than srm, and why kscrubber took an hour to run on sithlord's system)."
Misunderstanding on that one, yes I realize what rm is vs srm. But the first point renders this issue moot anyway.


"A really clean option is to delete files you don't want, copy the rest to a backup, overwrite or wipe the original partition completely with dd, format it, then copy the files back. Short of dropping the drive into a bucket of acid to dissolve it and using a new one, that's probably the best you can do. It's a good idea when setting up a new OS."

Yes, agreed, almost verbatim from something I'm writing now.
(where I conclude, "You can't clean up your system perfectly, you can't get it all, so you MUST regularly do wiping and re-installs on fresh partitions/disks (Section XYZ).")

And see my recent post on it where I used dd to do exactly that:
Privacy Cleanup 101
http://kubuntuforums.net/forums/inde...opic=3090100.0
Replies 7 & 8 (re dd methods, wiping)


I think we may differ about modern data recovery vs "number of passes.".
After a lot of reading-research, I pretty much agree with Starman's position about safe wiping: one pass of zeros is sufficient with modern drives. Gutman's work has been refuted by Freedman and by Gutman himself in his added prologue. Modern analysis supports the one wipe (on high-density ECC drives). I'm writing something on it now for a revision of my 101 how-to (which will also incorporate a lot of your stuff and kscrubber). Not knowing where to post it so you can see it, for now I put it in
Reply #6, privacy Cleanup 101, as a "Draft":
http://kubuntuforums.net/forums/inde...9244#msg109244
(with references)
In particular, the 38-pass thing is obsolete, as is the so-called "government's 7-pass requirement" (at least the logic for it is unclear).


Thanks again for your feedback.
I should be posting my 101 update in a week or so, I hope.

[IgnorantGuru]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

Yes, agreed, almost verbatim from something I'm writing now.
(where I conclude, "You can't clean up your system perfectly, you can't get it all, so you MUST regularly do wiping and re-installs on fresh partitions/disks (Section XYZ).")

I think that's good advice. Anytime you need to reinstall it's good to wipe the partition first as a matter of routine maintenance.

Swap partitions should also be wiped - that's what sswap provides. In many cases swap can simply be turned off - most modern systems have enough memory and don't need it, unless you're doing very memory intensive things.

And see my recent post on it where I used dd to do exactly that:
Privacy Cleanup 101
http://kubuntuforums.net/forums/inde...opic=3090100.0

I think that info deserves to be a wiki - maybe find a home for it on Ubuntu or Kubuntu wiki.

I think we may differ about modern data recovery vs "number of passes.".
After a lot of reading-research, I pretty much agree with Starman's position about safe wiping: one pass of zeros is sufficient with modern drives. Gutman's work has been refuted by Freedman and by Gutman himself in his added prologue. Modern analysis supports the one wipe (on high-density ECC drives). I'm writing something on it now for a revision of my 101 how-to (which will also incorporate a lot of your stuff and kscrubber). Not knowing where to post it so you can see it, for now I put it in
Reply #6, privacy Cleanup 101, as a "Draft":
http://kubuntuforums.net/forums/inde...9244#msg109244
(with references)
In particular, the 38-pass thing is obsolete, as is the so-called "government's 7-pass requirement" (at least the logic for it is unclear).

First, the link to starman's article is broken, so I couldn't read that. Nor was there any epilogue to Gutmann's piece. Frankly, I find Gutmann's analysis more believable than Feenberg's rebuttal, which has all the signs of being a paid debunking piece. Neither contain hard info though, as in what is really being done in recovery labs (many of which guard their secrets). Given the engineering tolerances required to make modern drives as reliable as they are, I find it VERY hard to believe that there is no multiple memory effect from overwriting. And overwriting just zeros would seem a very poor choice over random as you are creating a constant change. Those are just my opinions - the truth probably lies somewhere between one layer of zeros and 38 layers of random data. There are even those who say no amount of overwriting is a guarantee against some fragments being recoverable, which is why there are all kinds of devices for destroying drives physically.

Chaos theory in general makes it very hard to make order vanish completely. It's amazing what can be recovered from randomized systems. Even just coming up with strong random numbers is a task.

And I think the 'don't believe everything you read' advice applies here. Remember there are those seeking to read secrets, and they don't want people wiping drives well, so they'll drag out an unqualified 'debunker' to convince people. Just as in cryptography, where no one knows what different agencies are capable of doing or not doing, no one really knows what the state of the art is for recovery. Any claim that the bare minimum is sufficient is highly unqualified and unprovable.

Aside from an interesting discussion, this isn't really relevant to most of us. It's very expensive to open a drive and put it under a microscope, and there are few labs that do it, so until you have REALLY sensitive data (as in governments or corporations will spend thousands of dollars to get it), a pass or two is sufficient. For the average user, the main issue is overwriting all the little fragments that the filesystem writes and forgets about, since these are recoverable with software - no microscope involved. One pass over the drive with dd takes care of that, and is the only sure-bet method.

[Qqmike]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

I'll try this again with the Starman link, I was just there this morning.
This should do:
http://mirror.href.com/thestarman/asm/mbr/WIPE.html
Include the footnote marked with a * at the very bottom.

[IgnorantGuru]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

@Qqmike
Okay that link worked. (The geocities one on your Privacy 101 needs to be updated.) Starman seems reasonable.

I also found Gutmann's prologue from there... http://mirror.href.com/thestarman/as...nnEpilogue.txt

For any modern PRML/EPRML drive, a few passes of random scrubbing is the best
you can do. As the paper says, "A good scrubbing with random data will do
about as well as can be expected". This was true in 1996, and is still
true now.

That I would agree with.

[IgnorantGuru]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

And as Starman points out, I think the greater danger to most users are all the programs that claim to "delete", "remove", and "destroy" data, while leaving much behind. That's why dd is a great method - it simply overwrites every byte of the drive.

[IgnorantGuru]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

Gosh I don't know what I was thinking when I wrote reply #89, but I forgot the partition numbers in my little dd how-to. Should of course be sda1 not sda. I've edited it.

Really I was just testing to see who caught the error.

[Qqmike]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

#89 I did scan it quickly but blew past it as I quickly got the idea.
Besides, all that very fine print (Ctrl,+).

Three things:

> Bad sectors on a drive that are shut out but yet may contain sensitive data.
(If discarding the drive, no problem: hammer, burn, acid, shred, etc.; but if keeping the drive or selling it, then a potential issue.).

> Extra copies/artifacts of a file that got strewn around the system somewhere.

> And the ext3 thing, journaled file systems.
All the reading I've done suggests that dd and srm should be OK with at least two of the three types of journaled file systems: The ordered and the writeback types. Maybe not the Journal type.
http://en.wikipedia.org/wiki/Ext3
The ordered type of journaled ext3 system is the Linux default.

The issue would be whether extra copies of the original file are being generated during the zero-write process, and whether the file is actually zeroed in place. Never thought about this with dd, except to ASSUME that the file is in place as dd is writing zeros to it, bit by bit. Spent a lot of time this morning researching all this.

[Qqmike]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

continuing to work on this and think about it, been using kscrubber and all is well


kscrubber
I wonder if "delete Firefox cookies" should be an option rather than default. Of course, again, one can always comment that line out in the script. Some cookies are useful (e.g., to log you in to a site, like your home page).


deletions ... thoughts

I think doing deletions on ext3 journeled systems is fine unless it is the "Journal" type. If it is the Linux default type (the "ordered" type), then no problems.

One thing that I don't think gets deleted is the metadata on file. I don't think it gets deleted entirely.
For example, I have searched on the name of a (unique) file on my Desktop and found four references to it. I have deleted a file, searched on its name, and got results.
Using, like:
sudo dd if=/dev/sdxn bs=4096 | hexdump -C | grep "some_string"

where sdxn is my home partition. (Thus, I searched the entire 20 GB partition for 90 minutes for deleted file names and found them).

If true (which I think it is), it suggests that one should be careful choosing file names of sensitive files.

Even after secure deletion (which I haven't yet fully tested), I'll bet some of that file metadata (e.g., the name) can be found somewhere.

[IgnorantGuru]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

continuing to work on this and think about it, been using kscrubber and all is well

Good to hear. I use it regularly too on both Kubuntu and Arch.

kscrubber
I wonder if "delete Firefox cookies" should be an option rather than default. Of course, again, one can always comment that line out in the script. Some cookies are useful (e.g., to log you in to a site, like your home page).

The general behavior of cleaners/scrubbers is to remove browser cookies by default, which is probably a good idea, cookie cleaning being a classic use for a cleaner. If you are cleaning a system, leaving automatic logins behind would be problematic. But as you say, those lines can be commented out.

One thing that I don't think gets deleted is the metadata on file. I don't think it gets deleted entirely.
For example, I have searched on the name of a (unique) file on my Desktop and found four references to it. I have deleted a file, searched on its name, and got results.
Using, like:
sudo dd if=/dev/sdxn bs=4096 | hexdump -C | grep "some_string"

That's an interesting approach. To my knowledge, metadata (info about files) is handled by the OS and programs, and would thus be stored in files. You might try a grep search over that same drive, which will also look inside binary files, and see if you find it. Or you can use kscrubber's --check function for this.

If you don't find it in a file, it could be on the drive as a remnant of a deleted or fragmented metadata file. As we said, wiping files may not be 100% on a journaled fs, so these pieces can be left behind. Not much to be done about this except wiping the whole drive, wiping the free space on the drive (sfill), or using an encrypted filesystem.

Even after secure deletion (which I haven't yet fully tested), I'll bet some of that file metadata (e.g., the name) can be found somewhere.

secure-delete's srm does do its best to scramble filenames when removing a file, and kscrubber does its best to wipe metadata. If you find any in a file let me know. Not much I can do about data which lies outside of any file.

On this subject, I found that the database for the locate command stores filenames in /var/lib, so the most recent update to kscrubber will wipe /var/lib/mlocate/mlocate.db, and will later run updatedb to rebuild the database. (Most systems update the database once per day at night, which means until then the database can contain filenames of files which were deleted. So if you're not already using the latest version of kscrubber that is a good reason to upgrade.

Thanks for your feedback.

[Qqmike]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

Just noticing your new versions of kscrubber vs previous, and how the former use wipe( ) versus the latter that used a straight srm in the deletions. From my programming days, seems you've defined wipe as a function? (I'm not real clear on all the details inside wipe ( ), but I have the general idea.) Any reason for this change in versions/style? to include options for running srm, perhaps?

Btw, I know you are linking to my Privacy Cleanup 101.
I posted a fully new version this morning. Your kscrubber is rather strongly promoted in it. Just so you know. I don't know how long you will maintain kscrubber, but I'm convinced it's the best option right now for most Kubuntu users (which they may supplement with their own lines or script or manual deletion routine, as needed). Just an fyi, since you have a link on your blog page to it.

Privacy Cleanup 101
http://kubuntuforums.net/forums/inde...opic=3090100.0

I feel my 101 is a good, basic intro for beginning users (with some intermediate-level stuff tossed in) to make them aware of the many issues and their options. Ultimately, it would seem that a user should (1) devise a custom plan (perhaps using kscrubber as a base), and (2) periodically (zero-) wipe root and home partitions and re-install. At the very least, now and then, run kscrubber (and various others deletions) then do a wipe of the unused partition space (root and home partitions).

Replies #7 and #8 under 101 are also new (justification for the zero-fill one-pass (which you've read), and a discussion of "Disk Full" and a decision tree for dealing with it).
Somewhat related, my dd how-to is completely new (revised):
The dd Command
http://kubuntuforums.net/forums/inde...opic=3090824.0
where Part 3 is specifically written as a dd tutorial to support the 101 wiping tasks.
--another fyi.

[IgnorantGuru]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

Just noticing your new versions of kscrubber vs previous, and how the former use wipe( ) versus the latter that used a straight srm in the deletions. From my programming days, seems you've defined wipe as a function? (I'm not real clear on all the details inside wipe ( ), but I have the general idea.) Any reason for this change in versions/style? to include options for running srm, perhaps?

Yes wipe is a function. That was precipitated by the addition of the simulation capability - otherwise every srm call would have to be placed in an if block. Most of the complexity in the wipe function is in simulating the recursion and globbing normally handled by srm and bash. Also, the edit function replaces the sed calls similarly. This was one of the costs of simulation - now the script is less readable. But I think the simulation command works well, and the wipe calls can be treated similarly to an srm call.

A few things to note if you add wipe commands. Always put the file or glob in quotes. If a glob isn't in quotes then bash will expand it when it passes to the function, which is what the "internal error" in wipe() is designed to catch, as it can't handle that. Instead you want globs to be passed unexpanded to srm. Also, spaces and other troublesome characters in filenames must be escaped, even quoted. This is because kscrubber calls srm without quotes (to enable globbing). To escape a character, just put a backslash \ in front of it. So to wipe "xy z.txt":

wipe "xy\ z.txt"

I think that will work, though I haven't tested it. kscrubber doesn't actually wipe any file or folder names that have a space so there are no examples of this in the script. You can also just add raw srm calls like before, but remember that they will NOT be simulated in simulation mode, they'll always be done for real.

I'll take a look at your updated 101 - thanks for the update.

[Qqmike]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

Thanks for the notes re wipe, sim, srm usage.

[Snowhog]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

@IG

Ran kscrubber --clean for the first time this evening. Before I did, I stopped both, Nepomuk Semantic Desktop and Strigi Desktop File Indexer via System Settings > Advanced > Desktop Search.

Two things I noted. First, sudo kscrubber [option] doesn't work because the installed location (/opt/scripts) isn't part of sudo's path. I had to run it as sudo /opt/scripts/kscrubber --clean. Second, when launched, I'm told that nepomuk is running

WARNING: The following programs are currently running. kscrubber may not
thoroughly clean the files of running programs. For best results,
quit these programs before proceeding:
nepomuk

even though I stopped it before launching kscrubber. On finishing, kscrubber reports:

WARNING: nepomuk is running - cannot clean thoroughly

[Snowhog]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

A third thing I noticed after running kscrubber - my Bookmarked items 'lost' their URL icons. I think that over time they will be restored, but...

This is a minor annoyance.

[IgnorantGuru]

Re: A new privacy and log scrubber for KDE4, Firefox, & Flash

First, sudo kscrubber [option] doesn't work because the installed location (/opt/scripts) isn't part of sudo's path. I had to run it as sudo /opt/scripts/kscrubber --clean.

Correct. This is discussed in the script installation instructions, but I just added a reminder to the Usage line as well. If you want to run it without the full path, you can move the script into /usr/bin, or put a link to it there. eg

sudo ln -s /opt/scripts/kscrubber /usr/bin

Second, when launched, I'm told that nepomuk is running

kscrubber should not check that akonadi, nepomuk, and soprano are running, so I have corrected that in the update. Normally kscrubber does nothing with these programs, unless the "--killindex" option is used, in which case it shuts them down for you.

However, if kscrubber reported nepomuk running then it was. kscrubber gets this info directly from ps. That is one of the many virus-like qualities of nepomuk. It's a very odd program - IMO one of the shadier aspects of KDE4. Consider that there is no package named or containing "nepomuk" - it is installed with KDE itself, and cannot be removed by the package manager. Yet KDE runs fine without it - just a few apps require it. It's also a potential security risk and resource hog. This is why kscrubber has the ability to disable it.

A third thing I noticed after running kscrubber - my Bookmarked items 'lost' their URL icons. I think that over time they will be restored, but...

What browser are you referring to? If you mean Firefox, I don't believe it was kscrubber which removed these. When you cleared the history within Firefox prior to running kscrubber, you may have opted to remove Site Preferences, which is probably what removed those icons. And they will indeed be restored as you visit those sites. If you don't want to lose those, you can uncheck Site Preferences in Firefox's Clear Recent History dialog. Just be advised that this leaves a history of sites visited (and not just bookmarked ones).

If you still believe it was kscrubber that removed these, or if you're talking about a different browser, then let me know and we can try to track that down. Thanks for contributing.