Secure Deletion

[Admiral Valdemar]

I'm looking for a secure way of deleting files totally on Ubuntu 5.10 Breezy Badger, but so far I see that using any journaling filesystem means secure deletion isn't guaranteed (I'm using Ext3 as default).

In that case, is it best to encrypt files tagged for deletion before sending them to the waste bin? Assuming someone wanted to get at my old password keys or financial documents etc., they'd have to not only recover the file from deletion, but crack a 4 kilobyte cipher. Would that do it?

[Alferd Packer]

That sounds like a reasonable approach, especially since you want to keep the computer usable. As long as it's definitely encrypting the file and not just sticking a password in front of it, and as long as you're not using a broken encryption scheme, you should be solid.

[Admiral Valdemar]

I'm using the GnuPG tool which can go up to 4 kb keys, so that should be more than enough (GCHQ and the NSA are always pissed because of how hard 128 bit is, letalone this). Magnetic force microscopy is useless if you can't read the file after recovering it anyway.

[Xon]

You need to encrypt the files when they are in use. Otherwise you have a copy somewhere on disk which isnt encrypted.

[Admiral Valdemar]

Then what's the best strategy for that, given that I'm encyrpting the original files and the temporary one should be removed after the process?

[Faram]

Well the only really secure way is this.

remove the hard drive.

Open it, no need to be careful

Throw it into a volcano or somthing.

Or encrypt the entire drive.

[Beowulf]

Well the only really secure way is this.

remove the hard drive.

Open it, no need to be careful

Throw it into a volcano or somthing.

Or encrypt the entire drive.

You don't necessarily need to frag the drive. Just need to do a secure wipe of the entire drive (number of utilities will repeatedly overwrite the drive with random data. Depending on size and speed of drive, this could take anywhere from hours to days.)

However, DOD standard for classified data is complete destruction of drive. Thermite is preferred. Shotgun to the platter also works (while drive is spinning). Combination of overwritting and drive destruction is best.

[Admiral Valdemar]

I actually know the DoD wipes their drive with Hades' fury, but I actually quite like my hard disk, so that's out.

I just wanted to know if there was any way on a journaling FS to make it hard enough to recover a file short of being a gov't institution.

[Spacebeard]

I'm using the GnuPG tool which can go up to 4 kb keys, so that should be more than enough (GCHQ and the NSA are always pissed because of how hard 128 bit is, letalone this). Magnetic force microscopy is useless if you can't read the file after recovering it anyway.

An aside: those 4,096-bit keys are for assymetric (i.e., public key) cryptography, such as RSA or DSA. The 128-bit keys you mention are for symmetric cryptography, such as AES or Blowfish. You can't really directly compare them based on key length, as an assymetric algorithm requires much, much longer keys for equivalent security. At any rate, GPG doesn't encrypt the data with a public-key cipher anyway since they are insanely slow; it generates a key for a symmetric cipher that, probably 128-512 bits, with the public-key algorithm and the rest with a symmetric algorithm.

As for the original question, multiple passes of "dd if=/dev/urandom of=<whatever>" when you dispose of the disk will be fine against anyone but motivated and well-funded professionals. Without reformatting the disk, ensuring that no plaintext remains on it is tricky both because of the delayed writes done by some filesystems, as you mentioned, and because any programs that read those files at any point may have been paged out, leaving data in your swap partition.

[Admiral Valdemar]

An aside: those 4,096-bit keys are for assymetric (i.e., public key) cryptography, such as RSA or DSA. The 128-bit keys you mention are for symmetric cryptography, such as AES or Blowfish. You can't really directly compare them based on key length, as an assymetric algorithm requires much, much longer keys for equivalent security. At any rate, GPG doesn't encrypt the data with a public-key cipher anyway since they are insanely slow; it generates a key for a symmetric cipher that, probably 128-512 bits, with the public-key algorithm and the rest with a symmetric algorithm.

I was contemplating using Blowfish symmetric like I used to on my Windows box, but it was easier setting up the asymmetric system for starters. I'll consider weighing up which is best if it really means a big difference, I was wary about the software to start with given it seemed overly complex next to the app. I've used elsewhere.

As for the original question, multiple passes of "dd if=/dev/urandom of=<whatever>" when you dispose of the disk will be fine against anyone but motivated and well-funded professionals. Without reformatting the disk, ensuring that no plaintext remains on it is tricky both because of the delayed writes done by some filesystems, as you mentioned, and because any programs that read those files at any point may have been paged out, leaving data in your swap partition.

I've heard you could always encrypt the swap partition and use Wipe to get anything left in the open. Is that any doable or just a waste of time? I'd not have this problem with Ext2, but I'm not about to revert to an older FS just to make sure some confidential files are out of view.

[Spacebeard]

An aside: those 4,096-bit keys are for assymetric (i.e., public key) cryptography, such as RSA or DSA. The 128-bit keys you mention are for symmetric cryptography, such as AES or Blowfish. You can't really directly compare them based on key length, as an assymetric algorithm requires much, much longer keys for equivalent security. At any rate, GPG doesn't encrypt the data with a public-key cipher anyway since they are insanely slow; it generates a key for a symmetric cipher that, probably 128-512 bits, with the public-key algorithm and the rest with a symmetric algorithm.

I was contemplating using Blowfish symmetric like I used to on my Windows box, but it was easier setting up the asymmetric system for starters. I'll consider weighing up which is best if it really means a big difference, I was wary about the software to start with given it seemed overly complex next to the app. I've used elsewhere.

I wasn't criticizing your choice in encryption; I was just pointing out that it's not correct to gush about an assymetric cipher with a 4,096-bit key being vastly superior to a symmetric cipher with a 128-bit key solely due to the key length.

As for the original question, multiple passes of "dd if=/dev/urandom of=<whatever>" when you dispose of the disk will be fine against anyone but motivated and well-funded professionals. Without reformatting the disk, ensuring that no plaintext remains on it is tricky both because of the delayed writes done by some filesystems, as you mentioned, and because any programs that read those files at any point may have been paged out, leaving data in your swap partition.

I've heard you could always encrypt the swap partition and use Wipe to get anything left in the open. Is that any doable or just a waste of time?

Encrypting swap is a possibility. I have no idea how easily it would be accomplished on Linux, though; the only operating system I use that encrypts swap is OpenBSD, which does it by default.

Also, remember that in addition to any saved copies of the file, you'd have to wipe out any temporary files created by programs with which you edited the file: vi leaves recovery journals lying around, for example.

I'd not have this problem with Ext2, but I'm not about to revert to an older FS just to make sure some confidential files are out of view.

If you have any spare disk slices, you could always create a new filesystem, maybe an encrypted one, for storing sensitive files. Or you could use removable media

In general, though, it's extraordinarily difficult to be absolutely certain that a disk is clean of plaintext without entirely overwriting or destroying it. Since an attacker would need root or physical access to the machine in order to recover any of this data, though, I wouldn't be too worried about it unless it's a laptop at risk of theft.

[Uraniun235]

You don't necessarily need to frag the drive. Just need to do a secure wipe of the entire drive (number of utilities will repeatedly overwrite the drive with random data. Depending on size and speed of drive, this could take anywhere from hours to days.)

However, DOD standard for classified data is complete destruction of drive. Thermite is preferred. Shotgun to the platter also works (while drive is spinning). Combination of overwritting and drive destruction is best.

The DOD acts in a fashion that would be paranoid against the full resources of the NSA, let alone against any civilian attempt. One overwrite is more than enough to effectively destroy the data.

[Stark]

Yeah, I've never been clear on how anyone could recover the data if you low-level set everything to 1, then low-level set everything to 0. It's all gone.

[Uraniun235]

Technically, it's not a "low-level" format; true low-level formatting involves setting the parameters for the physical geometry of the drive itself.

Furthermore, you don't even need to write everything to 1 and then 0; simply zeroing out the hard drive is sufficient.

[Stark]

I meant changing the drive state - it's possible, at a 'high' level, to believe you've removed everything from a drive with data recoverable, hence my suggestion that you'd need to make sure it was a 'low' level procedure. I haven't even heard the term 'low level format' since the 386 days.

[Beowulf]

You don't necessarily need to frag the drive. Just need to do a secure wipe of the entire drive (number of utilities will repeatedly overwrite the drive with random data. Depending on size and speed of drive, this could take anywhere from hours to days.)

However, DOD standard for classified data is complete destruction of drive. Thermite is preferred. Shotgun to the platter also works (while drive is spinning). Combination of overwritting and drive destruction is best.

The DOD acts in a fashion that would be paranoid against the full resources of the NSA, let alone against any civilian attempt. One overwrite is more than enough to effectively destroy the data.

The DoD is paranoid about these things. Classified material must be destroyed in a verifiable manner. Physical destruction of the drive works. BTW, did you know it's possible to destroy a CD by hand. Apparently it's a valid way to destroy one, if you need to do it in a hurry.

And no, writing over the data in a single pass with zeros will not make it impossible for a data recovery expert to retrieve the data. A "1" that gets overwritten by a "0" has a lower field strength than a "0" than gets written over by a "0". Ditto the other way. This can be used to determine what used to be on the drive. Multiple overwrites with random data, or special patterns is required to assure data destruction.

[Uraniun235]

And no, writing over the data in a single pass with zeros will not make it impossible for a data recovery expert to retrieve the data. A "1" that gets overwritten by a "0" has a lower field strength than a "0" than gets written over by a "0". Ditto the other way. This can be used to determine what used to be on the drive. Multiple overwrites with random data, or special patterns is required to assure data destruction.

Maybe if you're using an ancient 120 MB hard drive.

link

Mid-level format - this requires a third-party tool and will overwrite every byte on the disk with a particular character or sequence of characters. Zero is a popular choice, and this is why this format level is often referred to as "zeroing the drive".
Note that this format is often erroneously called a low-level format.
On this subject, a good choice for a mid-level format is something like Darik's Boot and Nuke (DBAN). Download it for free from http://dban.sourceforge.net/ and do a single pass over your target disk. As you can read in the following paragraphs, a single wipe pass is enough to wipe your data for good.

...

From these descriptions you should be able to see that if only a high-level format has been performed (either quick or full) then your data has remained untouched.
Now, there are some people out there who claim that data can be recovered even from a mid-level format.
A popular paper on this topic is written by Gutmann, and can be found here:
http://www.cs.auckland.ac.nz/~pgut001/p ... e_del.html

This document has been examined and a rather good rebuttal on it can be found here:
http://www.nber.org/sys-admin/overwritt ... ttman.html
I am of the opinion that the arguments put forward by the gentleman in the rebuttal document are accurate. I have yet to locate anyone on the planet who is capable of recovering useful data that has been overwritten. Having said that, there are a number of people in the field who have successfully recovered overwritten data under certain limiting conditions.
Firstly, the person has to know the nature of the data to begin with. I must admit that I'm not certain as to what extent this familiarity must be, but it sounds to me like you must know what the data is in order to make a determination on what it should be when recovered.
Secondly, the process is very slow - of the order of around 1 kilobyte per hour. Work out how long that would take for you to recover all your mp3 files off a 120GB drive.
Finally, this process is only capable of being performed on low-density drives, such as the MFM encoded drives mentioned earlier.
Professor Gomez at the University of Maryland in the US seems to be the leading expert in this area, and he got a mention in New Scientist magazine some years ago. The limitations mentioned earlier pertain to work he has done with scanning tunnelling electron microscopes and/or magnetic force microscopes (that's another MFM acronym that is often confused with the elderly drive type. For drives, it stands for "modified frequency modulation".)

These issues mean that the recovery of overwritten data is impossible in the real world.

[note: since I wrote this article it has been brought to my attention that Guttman has added an epilogue to his report. It basically covers the fact that his analyses were based on older drive technology. He still believes multiple passes are needed for wiping, but tends to put it down to 'a few' rather than 35. He ends with stating 'the chances of an adversary being able to find the erased traces of [some small amount of data] in [, say,] 80GB of other erased traces are close to zero.']

Now some of you are asking "if you can't recover data that has been overwritten just once, why do companies sell software that does multiple overwrites?"
I have an opinion on this, but I can't back it up with any facts. Here it is anyway:
Company A brings out DataDeathstar, a program that will eradicate your rebel files by overwriting them once. This is all you need.
Company B makes a similar product, perhaps without such a copyright-infringing name, but in order to sound better than Company A, they claim they can do multi-pass overwrites. Perhaps they back this decision up with the Gutmann article mentioned earlier.
Now if the cost is the same, Joe User will choose the program with more features - the version that does multi-pass overwrites.
This then precipitates an escalation in the number of wipes any package will perform, to make them sound better than their competitors. Eventually we end up with the Department of Defense 35-pass "standard", or the Bilbo-level Eleventy-billion Insano-wipe.

So why does the Department of Defense specify that huge multi-pass overwrite if one is enough? Once again I can only theorise, as I don't know anyone in that industry who could speak about this topic. Here goes:
Decisions are made by people far above the technical guys on the ground. That is, management types with no techie knowhow. I'm not berating this issue, as it is the same the world over.
At the weekly meeting, one of the subordinate guys points out he read a report from Gutmann about recovering data. It may have mentioned the MFM-issue but that's all techie-speak. The boss decides that he'd rather not risk his career on an issue he can't understand and doesn't have the resources to examine in any depth.
To be safe, he makes sure the standard is some huge amount of overkill, so he can never be determined to be a traitor by allowing data to get into the wrong hands.
This all seems fairly reasonable to me - everyone errs on the side of caution in a field they don't understand.
Also, the military has had loads of data on old MFM technology in their time, and recovery MAY be possible on this gear. Why make multiple standards for different types of drives when your staff may not be able to tell the difference between them?
They also have plenty of manpower, and would be quite happy letting some guys spend their days just wiping data, whether it's a waste of time or not.

Just remember one thing - one overwrite pass is enough to stop anyone recovering your data. If anyone tells you otherwise, tell them to put up or shut up. It's quite simple to get a floppy disk (or hard disk if they prefer), put some files on it and then wipe them so that they can be recovered with some magical system this person says exists. Make it easy for them and tell them what the file types are if you like - it won't help.
There is just too much money to be made in the private sector if some firm were capable of doing overwrite recovery - you would have heard of it being done if it were possible. People often state that perhaps the NSA or US military can do it but aren't telling anyone. Well, those particular organisations outsource all their data recovery to a private company, so the services would be offered to anyone at the right price.

One other thing has been brought up, and makes a good point: modern drives automagically remap bad sectors onto spare sectors on the disk. This way you don't really notice anything, and something like the SMART system keeps an eye on it.
These bad sectors are tracked by a process that is not affected by a zeroing of the drive. This means that your overwrite pass will not touch any data that is residing in those sectors prior to them being marked as bad. If you feel that it's likely important data resides in those bad sectors, then you'll need to physically destroy the disk to be safe, or reset the bad sector map in some way and hope you can wipe them afterwards.
Having said that, there don't appear to be any tools out there to access those sectors or reset the bad sector map. Presumably the drive manufacturers can do it, but they don't make the software available as far as I can tell.
Obviously this condition is unaffected by the number of wipe passes you perform - those blocks marked as bad will be untouched no matter what. What are the chances those bad blocks contain useful/incriminating data? Who knows. I'd personally say it's unlikely, but this is pretty much the only reason I'd use to physically destroy a drive.

Re-quoted for emphasis:

Just remember one thing - one overwrite pass is enough to stop anyone recovering your data. If anyone tells you otherwise, tell them to put up or shut up. It's quite simple to get a floppy disk (or hard disk if they prefer), put some files on it and then wipe them so that they can be recovered with some magical system this person says exists. Make it easy for them and tell them what the file types are if you like - it won't help.
There is just too much money to be made in the private sector if some firm were capable of doing overwrite recovery - you would have heard of it being done if it were possible.

[Pu-239]

Store things in encfs

Why do you need this encryption anyway?

[Admiral Valdemar]

Store things in encfs

Why do you need this encryption anyway?

I store personal passwords to various sites that have my financial details on among other things, plus, I dislike my personal work folders being open access along withfamily photos. I'd like all that to remain under lock and key, even if this is a desktop, I have a high-level firewall running and virus scanner and don't run as root. Getting it sorted now would save time for if any real delicate files from a future job need protecting.

Thanks for that link though, I'll consider making a partition with that on to store any sensitive data.

[phongn]

The DoD is paranoid about these things. Classified material must be destroyed in a verifiable manner. Physical destruction of the drive works. BTW, did you know it's possible to destroy a CD by hand. Apparently it's a valid way to destroy one, if you need to do it in a hurry.

The data layer of a CD is exposed (the reverse side of the top label) so it is pretty easy.

[Admiral Valdemar]

Microwave ovens are just as quick and have a nice effect too as the aluminium burns away inside.

[Beowulf]

The DoD is paranoid about these things. Classified material must be destroyed in a verifiable manner. Physical destruction of the drive works. BTW, did you know it's possible to destroy a CD by hand. Apparently it's a valid way to destroy one, if you need to do it in a hurry.

The data layer of a CD is exposed (the reverse side of the top label) so it is pretty easy.

Actually, you just start snapping the CD. Take it in both hands, bend until it breaks. Repeat until you can't get a good enough grip on the pieces.

[Admiral Valdemar]

The ease can depend on what quality of disc you have too. I have some CDs with software or music from magazines that will bend so each end touches and not snap. Some of my CD-RWs are extremely rigid, however.

I expect scissors would help cutting the halves into smaller pieces too, reminds me of the new shredders that cut documents into diamonds rather than strips (which most intelligence agencies will painstakingly work on).

[General Zod]

The ease can depend on what quality of disc you have too. I have some CDs with software or music from magazines that will bend so each end touches and not snap. Some of my CD-RWs are extremely rigid, however.

I expect scissors would help cutting the halves into smaller pieces too, reminds me of the new shredders that cut documents into diamonds rather than strips (which most intelligence agencies will painstakingly work on).

Not unless they're industrial strength scissors. Having attempted to break several CDs out of sheer general maliciousness (they were AOL disks), I can say that they're not the easiest material to damage with your bare hands or simple scissors in the world.

[GrandMasterTerwynn]

The ease can depend on what quality of disc you have too. I have some CDs with software or music from magazines that will bend so each end touches and not snap. Some of my CD-RWs are extremely rigid, however.

I expect scissors would help cutting the halves into smaller pieces too, reminds me of the new shredders that cut documents into diamonds rather than strips (which most intelligence agencies will painstakingly work on).

Not unless they're industrial strength scissors. Having attempted to break several CDs out of sheer general maliciousness (they were AOL disks), I can say that they're not the easiest material to damage with your bare hands or simple scissors in the world.

Yes. The quickest way to actually safely render a CD completely unreadable is to sand the side with the reflective layer on it. Do it with a sander, or throw it on rough concrete and grind vigorously with your foot.

Using a microwave to help destroy a CD is fun, but only if it's somebody else's microwave.