Announcement

**vinnywright** · Aug 28, 2009, 09:16 AM

Re: Disappearing Free Space on USB Drives

have you ben unpluging them withought clicking safley remove?

VINNY

**josefko** · Aug 28, 2009, 10:31 AM

Re: Disappearing Free Space on USB Drives

I see you have tried a lot, but I miss trying to read data on your external ext3 disk from Windows. To do this you need to download ext2 driver for example Ext2Fsd-0.46a.exe and install it on Windows.

**GreyGeek** · Aug 28, 2009, 01:41 PM

Re: Disappearing Free Space on USB Drives

High USB drive failure rates is becoming a concern:
http://news.techworld.com/storage/11...failure-rates/

http://www.buzzle.com/articles/usb-d...a-failure.html

The Merits of USB Flash Drives:

-USB drives are very economical as they are very cheap and easily accessible.
-USB drives have zero possibility of dust and scratches of the saved data as these devices have zero moving parts.
-The data storing capability is enormously as compared to CD/DVD’s, floppy disk and other storage devices.
-They are extremely handy and lightweight so they are a good portable device.
-As compared to hard drives these USB drives use very little power for storing and removing data.
-Booting up a computer is possible with flash drives.

The Demerits of USB Flash Drives:

-Every flash drives depending upon its memory and function has limited cycles of data writes and removes. Thus, they cannot be made a primary storage device as they will render useless over time.
-Although these drives are very handy, the risk of virus attack and data failure is a major problem seen in flash drives.
-USB drives are made fragile, they are lightweight. Thus, the possibility of external damages and getting crushed is unquestionable.
-Being small in size the risk of misplacing, theft or lost is likely.
-Most flash drives ignore the write-protect mechanism.
-Being small and portable the risk of storing confidential information is not applicable in these devices because they can be easily stolen and smuggled. Huge business organizations have to deal with this issue.

There seems to be a pattern emerging: the bigger the drive the shorter it lasts.

Your pen drive may be failing. Time to move your data to a fresh one and reformat the old one, taking care to notice how many GP the final size is compared with its original size.

**jtree** · Aug 29, 2009, 09:56 AM

Re: Disappearing Free Space on USB Drives

Originally posted by vinnywright

have you been unplugging them without clicking safely remove?

VINNY

Yes for the flash drives.
No for the 500G Hard Drive.

I had thought that was the problem too. Until I noticed that it happened on my external 500G hard drive. The external hard drive stays plugged in all the time.

But, even if that were the problem for the 500G drive doesn't linux have a way of fixing it? Especially on a journaled filesystem like ext3?

_________________________________________________

Originally posted by josefko

I see you have tried a lot, but I miss trying to read data on your external ext3 disk from Windows. To do this you need to download ext2 driver for example Ext2Fsd-0.46a.exe and install it on Windows.

I just heard about that recently. Do you know if this would allow me to run a windows chkdsk on the drive as well? And would it be safe? :-X

_________________________________________________

Originally posted by Teunis

The ultimate way to check for disk space/ use oddities is to run KDirStat, it'll give a very nice graphical representation of all things on your disk.

KDirStat is a KDE implementation of an original Windows program called SequoiaView.

Thanks for the reminder. I will probably install that too, although I much prefer the radial pie chart format used by filelight. http://www.methylblue.com/filelight/ A windows equivalent program is called Scanner (http://www.steffengerlach.de/freeware/)

I do use filelight to figure out what to delete to clean my hard drive. But in this case the hard drive has 17+GB of actual free space reported by partitioning utilities, but is reporting 0 Bytes Free in file managers.

**josefko** · Aug 29, 2009, 11:17 AM

Re: Disappearing Free Space on USB Drives

I see you have tried a lot, but I miss trying to read data on your external ext3 disk from Windows. To do this you need to download ext2 driver for example Ext2Fsd-0.46a.exe and install it on Windows.
I just heard about that recently. Do you know if this would allow me to run a windows chkdsk on the drive as well? And would it be safe?

I didn't try to run chkdsk on such disk, I doubt it is possible, I use e2fsck in Kubuntu for that. I thought, when you had tried the disk with some Linux distros, it would be interesting to see the content of the disc in other operating system. It doesn't hide hidden files, maybe it doesn't hide anything.

I personally would look at the size of all directories with some software for example Krusader or KDirStat to see what is going on.

Or maybe this would be interesting

Originally posted by dibl

Here's something I saw the other day that might be of interest. The new Parted Magic 4.4 Live CD has a graphical version of smartmontools. I keep thinking one night before I go to bed I'm going to boot it and run the long test on the hdd that I'm most nervous about.

http://partedmagic.com/

**dibl** · Aug 29, 2009, 12:02 PM

Re: Disappearing Free Space on USB Drives

You said you mounted it on Mandrake and Parted Magic. Do the "file browser" utilities on those systems also give the "wrong" information, or is this problem limited to Kubuntu?

You said you use filelight -- does it show the free space correctly?

I'm trying to figure out where the "wrong" information is being generated -- seems like it could be KDE or KDE packages, if GParted and other non-KDE packages are seeing it correctly.

**jtree** · Aug 29, 2009, 03:53 PM

Re: Disappearing Free Space on USB Drives

Originally posted by dibl

You said you mounted it on Mandrake and Parted Magic. Do the "file browser" utilities on those systems also give the "wrong" information, or is this problem limited to Kubuntu?

You said you use filelight -- does it show the free space correctly?
...

The free space displays wrong on any file browser I have tried on Mandrake, PartedMagic, KDE3.5, KDE 4.2.2, Dolphin, Konqueror.
Likewise Filelight displays the wrong free space as well.

They all display the correct USED space however.

I know the free space really is free, because I was able to successfully resize the partition smaller and larger in GParted. Plus when I delete files, the "free" space remains at 0 Bytes.

I don't believe this is specific to KDE or (K)ubuntu. But I thought I would start here first. I am only assuming that PartedMagic doesn't use any KDE packages.

**dibl** · Aug 29, 2009, 04:51 PM

Re: Disappearing Free Space on USB Drives

Interesting. It almost sounds like the problem is with the drive's own firmware or the USB interface to it. Possibly Windows has a superior driver (since we know hardware OEMs write their drivers for a Windows interface.).

This is not a problem that you hear about with PATA or SATA drives. I don't know if anyone else has studied it over a period of time like you have, to observe the gradual disappearance of free space.

It would be interesting (for me, at least) to see what would happen with a different filesystem type, like ext4 or JFS or XFS. Since you are using it for data storage and not for running the OS, XFS or JFS should work. Or, maybe it was designed for NTFS and Windows -- I dunno.

I've been using an 8GB USB memory stick for the better part of a year, for data transfers between home and office. It also has a bootable OS on it (E-Live). But the data partition seems to be holding up fine, adding and deleting files weekly with no apparent change in its capacity.

So, I dunno -- I guess I'm not helping much here ...

**josefko** · Aug 30, 2009, 02:00 AM

Re: Disappearing Free Space on USB Drives

I tried something for you with my 2 GB Ext2 flash disk:
Dolphin, Krusader ... 463 MB free from 1,9 GB; Used space 1315(1378)
KDirStat the same used space.
ds -h the same free space
About 100 MB is surprisingly missing.
Windows ... 559 MB free from 1,87 GB; Used space 1,32 GB
Nothing is missing.
No more files than in Kubuntu there. I used spacemonger.exe to check it out.

In Kubuntu I use 160 GB WD Passport with Ext2 and 250 GB IOMEGA Presstige with NTFS(NTFS is fine, but cannot be used for the whole Kubuntu backup).
I don't know if there are some space leaks. I have still some disk space free.

**josefko** · Aug 30, 2009, 11:31 AM

Re: Disappearing Free Space on USB Drives

The rest of tests:
Passport ext2 first partition free 58,1 GB from 134,6 GB used 47 GB leak 29,5 GB
Windows results are similar
Passport ext2 second partition free 3,6 GB from 11,3 GB used 6,9 GB leak 0,8 GB
Iomega NTFS free 53,7 GB from 232,9 GB used 179 GB leak 0 GB

From time to time I had some problems with Passport, which was not clean and needed to be repaired by fsck. I had never problems with Iomega.

@jtree: so I see am in the same troubles as you are, but 46 GB is simple to move and reformat disk.

**jtree** · Aug 31, 2009, 03:50 PM

Re: Disappearing Free Space on USB Drives

I am glad that someone else has been able to observe this as well. At least I'm not crazy.

Some additional info...
My offending external flash drive is formatted fat16 and is mounted with vfat.
My offending mp3 player is formatted fat32 and is mounted with vfat.

Some thoughts...
When a user requests drive usage percentage it comes up very fast. Apparently the OS and file managers do NOT need to scan the entire contents of the drive and add up the sums to calculate free space. This leads me to suspect that the 'free space' number is stored in a separate location.
Some searching found this quote:

Originally posted by FAT File System [url

http://www.yale.edu/pclt/BOOT/fat.htm][/url] The directory is allocated at the start of the partition and it contains the table of freespace.....
....
When the system crashes, no data is lost. However, a FAT system may have removed disk area from the chain of free space, but may not yet have assigned it to any permanent new dataset. The CHKDSK (or on newer systems the SCANDISK) utility examines the FAT table to determine the status of every record on disk. The records which are not part of any dataset may be returned to the free space chain.

On Windows, chkdsk and scandisk examine everything and update the free space record accordingly. I don't know what tools to use on linux, or maybe I haven't used e2fsck correctly yet.

**josefko** · Sep 01, 2009, 12:46 AM

Re: Disappearing Free Space on USB Drives

After some googling I have an additional info about ext2/ext3 formats.

During formatting, Linux allocates about 5% disk space as left behind for root. It makes a difference between free and used space. If you want to delete this root space (for free and used space to have equal values) you can run

Code:

sudo tune2fs -m 0 /dev/external

or you can use -m option while formatting.
But it is not reasonable to use a disk with ext2/ext3 file systems near to 0% free space. It is not NTFS, where it is usual. Some recommend to use with -m at least 1% and to leave the real free space not less than 5%.

Another thought from Ubuntu forums

... to get all the external usb hard drive space to show up i had to change formats to NTFS.

For revealing free space some recommend using -S option in mke2fs on a partition with data, in manual it is described as "This is useful if all of the superblock and backup superblocks are corrupted, and a last-ditch recovery method is desired. There is no guarantee that any data will be salvageable." At least one guy claimed he was successful, but he backuped before.

I tried to add and delete files on both my external drives and free space shows as expected. I am not sure if the difference between free and used space on my Passport is not due to 400000 small files on it.

**jtree** · Sep 09, 2009, 10:54 AM

Update - Solved

I am marking this post as SOLVED because my main problem has been diagnosed.

Problem 1:
Problem: Missing hard drive space on external ext3 drive.
Symptom: Can delete files, but free space remains at zero for all users.
Suspected Cause: Misappropriation of free space table on drive.
Actual Cause: Somehow the drive was allowed to fill up well past the limit of reserving 5% free space reserved for root. When deleting files the used space was still above 95% so the system was reporting '0 Bytes Free' to maintain it's 5% free reserved space.
Solution: I kept deleting a mass amount of files until the actual free space fell below the 95% threshold. Then the remaining free space below the 95% threshold was reported properly.
Comment: The last program I remember using before the drive was full was rsync. I am still unsure why rsync was allowed to fill the drive past the 5% free space reserve.

Problem 2:
Problem: Missing drive space on external usb flash drives formatted to FAT16 and FAT32.
Symptom: Can delete files, but there remains unaccounted for missing free space. This eventually gets bigger until too much is missing and I am forced to re-format.
Suspected Cause: Misappropriation of free space table on drive.
Actual Cause: A friend of mine sent me this...

...it's common with FAT filesystems that failing to properly unmount the volume will lead to sectors being marked as used even when the files have been unlinked from the directories...

Solution: Probably just unmount them first. Then run check on them to correct problems.

It appears that they were separate problems with similar symptoms.

**Snowhog** · Sep 09, 2009, 02:36 PM

Re: [mostly SOLVED] - Disappearing Free Space on USB Drives

Nice deduction and documented explanation.