USB File Transfer corruption with Dolphin

[enginemusic]

Okay, this is a combination of frustrating and scary. I've experienced mp3 corruption while transferring over USB. Corruption manifests itself as songs being cut off abruptly, songs with part of the title song then another song or songs appended to it, digital glitches, and misreporting of song length. It has happened on the following transfers:

11.10 desktop > USB key
11.10 desktop > Nokia cellphone
and interestingly,
WD external drive > Lubuntu netbook using PCManFM 0.9.9 (default GUI file manager)

So the only constants are that I'm using GUI file managers over USB. Has anyone heard of this issue before? One thing I know I've done on at least a couple of occasions is to do drag-and-drops while the system is already copying something else but I'm not sure if this has been the case every time.

I know file transfer works well over USB at least some of the time as I have luckybackup verify my backups and haven't had any issues.

I should note that I'm plugging the USB right into the ports on the (stock Dell) box, not using some shonky aftermarket USB hub. I've also gone back and verified that the source files are not corrupt.

[snappwrench]

I have had problems copying video files while another transfer was in progress myself which basically gave the same results but also I have had the same issue with my Seagate external drive and that problem wound up being a bad data cable so not sure if this helps but basically I now only make one transfer at a time before I drag and drop anything else and changing the data cable also helped. Sorry I couldn't give any more help than that, but I would start there at least if you haven't already.

[dibl]

USB file transfers take longer than they appear to -- probably you are terminating the transfer (closing Dolphin or PCmanFM) before the transfer is fully complete. To test it, try using "cp" in a terminal, to copy from one device to another. The transfer is not complete until you get the prompt back.

[bra|10n]

...songs with part of the title song then another song or songs appended to it, ...

I have experienced something similar whilst editing my conkyrc's, finding snipets of code in other scripts with somewhat similar filenames.

[enginemusic]

...problem wound up being a bad data cable...

The cable isn't being used when the USB stick is plugged straight into the box (admittedly there is a cable inside the box to the mobo) so I can't point my finger at that. Also, with LuckyBackup I'm getting verified backups with the same cable I used to Xfer partially corrupt files to the netbook. I think Dibl might be on to something as I can be a bit impatient. I do have a habit of "safely ejecting" the device before unplugging it. Don't know if that forces writes to complete or not. I'll just to have to spend some time copying files with Dolphin and comparing them from the command line to try to narrow it down.

I thought I read somewhere that Dolphin transfers take longer than using the cp command - I was just curious if there was a known accuracy issue as well. I suppose if there was we'd know a lot more about it by now!

[quarkslot]

I've experienced mp3 corruption while transferring over USB. Corruption manifests itself as songs being cut off abruptly, songs with part of the title song then another song or songs appended to it, digital glitches, and misreporting of song length.

I've experienced all of this when copying files to my audio player, one time I had to reformat the drive as it was left read-only because I disconnected it too soon.

The problems is that Dolphin (or whatever program you are using) indicates the transfer is complete when it isn't. Makes it a bit of a lottery.

Other than copying from the command line I don't know how to tell when the transfer is complete.

[SteveRiley]

One other tip: after dragging and dropping in Dolphin, open a command window and run

sync

When the prompt returns, the file transfer is complete.

One of the handiest things, if you like to do things at the command line, is to install yakuake. This package places a command window directly at a fingertip: namely, the one closest to the [F12] key.

[Snowhog]

One is apt to just 'unplug' the USB drive when one thinks that the process of copying is complete. With Linux, that process may not actually be complete, as the actual process is being performed (if not all, at least in part) in RAM, and the write cache process has not actually been executed. This is why, when working with 'removable' devices, you must use the right-click 'Safely remove' option. This action flushes the cache and any unwritten data is then written to the device. Only then is it safe to remove/unplug the device.

[Ole Juul]

One of the handiest things, if you like to do things at the command line, is to install yakuake. This package places a command window directly at a fingertip: namely, the one closest to the [F12] key.

Although I always have my first desktop reserved for a stack of terminals, it always seems that I need yet another one for some little task - this is perfect! Thanks for the tip.

[enginemusic]

I've used "sync" on the last few transfers and have had no further issues. I'm fairly sure I had used "safely remove" option as Snowhog suggests when I had these issues, but can't swear to it. One question I DO have - is this unique to GUI file transfers? I.E. if I did the transfer using the command line does the OS close off the transfer as soon as it's done or do I need to use the sync command there as well?

One other tip: after dragging and dropping in Dolphin, open a command window and run

sync

When the prompt returns, the file transfer is complete.

One of the handiest things, if you like to do things at the command line, is to install yakuake. This package places a command window directly at a fingertip: namely, the one closest to the [F12] key.

[SteveRiley]

I've used "sync" on the last few transfers and have had no further issues. I'm fairly sure I had used "safely remove" option as Snowhog suggests when I had these issues, but can't swear to it. One question I DO have - is this unique to GUI file transfers? I.E. if I did the transfer using the command line does the OS close off the transfer as soon as it's done or do I need to use the sync command there as well?

I don't have a definitive answer for your question. I suppose it depends whether the file transfer is direct-to-disk or through some buffers. There's also the file system's journal to contend with, and how you've configured journaling (it can happen before or after actual data writes). I've pretty much made it habit to always sync before removing a drive. I've aliased it to s so I can literally get it done in four keystrokes: [F12] + s + [Enter] + [F12].

[james147]

I think the problem is that you are removing the disk too soon, even after selecting the "eject" button you have to wait until the data is actually written to the disk (this can take some time). Before unplugging the device (after clicking eject) make sure that the led on it is no longer flashing (indicating it is no longer writing) or wait for the device to register as unmounted, I noted that the device remover list the device as "removing" until the device is synced are unmounted properly.

Note that the sync command will force a flush to the disk return when all data is actually written to the disk but this is not needed if you are unmounting (as unmounting forces a sync before actually unmounting) and can be seen by issuing the umount <device> command from terminal (it will return when the device is properly synced and unmounted - ie safe to remove). With the gui make sure you check the device has finished being unmounted as the notifications to this are more subtle.

USB file transfers take longer than they appear to -- probably you are terminating the transfer (closing Dolphin or PCmanFM) before the transfer is fully complete. To test it, try using "cp" in a terminal, to copy from one device to another. The transfer is not complete until you get the prompt back.

Even cp is not guaranteed to return before all data is written (it will return when all data is written to the kernel/device buffers).

The problems is that Dolphin (or whatever program you are using) indicates the transfer is complete when it isn't. Makes it a bit of a lottery.

The problem is actually that dolphin reports when it has finished writing the file, but not when the kernel has finished writing the file to disk. Command line tool can also suffer from this behaviour.

One is apt to just 'unplug' the USB drive when one thinks that the process of copying is complete. With Linux, that process may not actually be complete, as the actual process is being performed (if not all, at least in part) in RAM, and the write cache process has not actually been executed. This is why, when working with 'removable' devices, you must use the right-click 'Safely remove' option. This action flushes the cache and any unwritten data is then written to the device. Only then is it safe to remove/unplug the device.

Only after the flush has completed, which can take some time depending on the amount of data being written, don't assume because you clicked safely remove that it has been removed, wait for it to be listed as unmounted.

I've used "sync" on the last few transfers and have had no further issues. I'm fairly sure I had used "safely remove" option as Snowhog suggests when I had these issues, but can't swear to it. One question I DO have - is this unique to GUI file transfers? I.E. if I did the transfer using the command line does the OS close off the transfer as soon as it's done or do I need to use the sync command there as well?

It is not guaranteed to flush all the data after the command returns from command line. You should always unmount properly before unplugging the device.

This problem is caused by how linux handles disk io, it generally does disk io asynchronously so it can return control to the application as soon as it can while it deals with actually making sure the data is written to disk letting the application continue with that ever it wants. This feature also lets the kernel spindown the disk for longer periods as it can cache the changes in memory till it decides to spin the disk back up again resulting in lower battery usage in laptops.

As mentioned above the sync command (and unmounting the device) forces a flush of cache to the disk, but this can take time depending on the amount of data. There is however a way to make sure that a device is never buffered in memory, which can harm application performance in favour of making sure its always safe to remove the device (as soon as the application says its finished that is) and that is to mount the device with the "sync" option. The sync option forces the kernel to block on disk transfers rather then returning as soon as it can.

This option can be added when mounting manually by:

sudo mount -o sync /dev/sdXY /path/to/mount

though I am not sure how to automatically do it in udev for removable media.

[quarkslot]

Great post summarising the issue.

In my experience "safely removing" USB thumb drives in Dolphin is enough as long as you wait until the device icon indicates that it is no longer mounted.

Unfortunately the same cannot be said for my MP3 player, if (when) it goes wrong I can usually copy the files again but in the worst-case I had to re-format the device because the file system was left in a read-only state. The belt-and-braces approach is to give the transfer plenty of time to finish, then sync and then safely remove. Of course, all this shouldn't really be necessary.

though I am not sure how to automatically do it in udev for removable media.

I looked at this a few months ago having decided the problem must be in the way the system was mounting removable devices but I wasn't smart enough to consider udev, just some feeble attempts messing about with fstab which, of course, has no effect at all.