Weird problem when copying/syncing

[Quaxo76]

Hello,
I have a NAS (Network Attached Storage) that I use to share files and documents between my home's computers. I also keep a backup of all the NAS's files on an external USB drive.
A few days ago I had to reformat the NAS, and now I want to copy the backup files from the USB drive back to the NAS.
At first, I did the regular "copy-paste" using Konqueror. It took more than 24 hours (it's 160 GB of data and the NAS is slow) but seemed to work OK. Then, just to be safe, I checked the size of the transferred folder - and the size on the NAS was several gigabytes smaller than the original. Lots of subfolders were totally ignored while copying.
How could this happen?
And then, to try to fix things, I ran

rsync -v -r /media/Elements/Store/Public/Mp3 /home/cristian/smb4k/NAS/public

and

cp -p -r -v -u /media/Elements/Store/Public/Mp3 /home/cristian/smb4k/NAS/public

(the souce dir is /media/Elements/Store/Public/Mp3/ and the destination is /home/cristian/smb4k/NAS/public/Mp3/)

but none of these commands will transfer any file. But the files ARE there in the source drive, but they ARE NOT (or at least, seem to be not) on the NAS!!!

What's happening here? This is driving me mad!

I mount the NAS using smb4k, it gets mounted in /home/user/sbm4k.

Thanks, Cristian

[Quaxo76]

Re: Weird problem when copying/syncing

An addition: if I give the above "cp" command *without* the -u option, it starts copying the files, in the correct locations. So why not with the -u?!?

Cristian

[Quaxo76]

Re: Weird problem when copying/syncing

Things get more and more absurd.
Let's call the origin folder /SRC and the destination /DEST (yeah, I have fantasy...)

In Konqueror, I right-click /SRC, select Properties: I get "22712 files, 1700 subfolders, 93.4GB".
Doing the same on /DEST, I get "21562 files, 1700 subfolders, 88.7 GB".

That's a difference of almost 5GB. So, I thought maybe some files have weird permissions or filenames and don't get copied... So I did what follows:

I entered

cd /SRC
ls -aR > list1.txt

and

cd /DEST
ls -aR > list2.txt

then:

diff -i /SRC/list1.txt /DEST/list2.txt > diff1.txt

And guess what? The diff1.txt file is empty. If I omit the -i option, then the diff file is not empty - it reports a few case changes (which I have no idea why happened), anyway, this proves that the diff command works. And that there are no differences between the two folders.

What's going on? I'm wasting whole afternoons on this issue...

[Quaxo76]

Re: Weird problem when copying/syncing

OK. I touched the bottom.
In Konqueror, I right-click the folder, click on "Properties", to check the size and file count of the folder... Guess what... The result is different every time I check. And not by a couple of files, but by 1000-2000 files. And no program is accessing the folder in the meanwhile. I have no idea what's going on here. And no one here seems to have an idea either - or be interested. I had never seen such a weird behavior in Linux...

Cristian

[dibl]

Re: Weird problem when copying/syncing

- or be interested. I had never seen such a weird behavior in Linux...

Au contraire, I'm fascinated!

I just have no idea what is causing that behavior -- I'm hoping you'll discover the cause and then tell us.

If it were me, I'd try using a different file manager, to sort out whether it is something about the Kubuntu system or something about the file manager.

[Quaxo76]

Re: Weird problem when copying/syncing

That's a good idea. I just couldn't think that Konqueror could do such a thing - I mean, if it's a bug, it's pretty serious - now I'll try to install Dolphin or Thunar and see what happens...
What makes me worried now is that maybe I only ended up with a partial backup! And with 80000+ files, I wonder how I can check... And I don't even know if I can trust the console!

I'll report back when I have installed other file managers and made some experiments...

Thanks
Cristian

[dibl]

Re: Weird problem when copying/syncing

Wow, that is a huge number of files! Are they in a lot of different sub-directories?

On a far smaller scale (a few hundred music files and folders) I sometimes get my reassurance by opening side-by-side file manager windows of the same width, say 4 or 5 thumbnail icons wide. Then just scroll down to the bottom of each. If the backup folder is identical to the source folder, the icons should look the same at the end of the list (i.e. if there are only two showing on the bottom row, it should be the same in both source and backup). If there's a difference, you know something isn't matching and you can go hunting for what's missing.

[Qqmike]

Re: Weird problem when copying/syncing

Quaxo76: “And no one here seems to have an idea either - or be interested...”

Well, I'm not jumping in with anything, but am certainly interested, and am certainly watching.
--Mike
(who reads the details/techniques you have reported using to investigate this and who is still learning a lot and has begun to mess with copying OSs in different ways: dd, cp, and next, rsync would be a candidate—e.g., Rog131 here has written a how-to using rsync to duplicate a Kubuntu OS.)

[Quaxo76]

Re: Weird problem when copying/syncing

Wow, that is a huge number of files! Are they in a lot of different sub-directories?

Yes, it's a lot of files! It's all my documents - music, photos, ebooks, everything... And yes, they're in several hundreds of separate sub-folders. After all I think the "diff" method should be the only viable check...


I have made some experiments. I tested the file managers on a folder with a complex structure of sub-folders, containing more than 20000 files.
I tested both the "original" folder - on the internal hard disk - and the "clone" of that folder on the NAS. I took several readings with each file manager. Here are the results (the numer is the file count):

Konqueror, NAS folder:
25139
24698
24730
24643
24730
24730
24730

Konqueror, HDD folder:
25388
25388
25388
25388


Dolphin, NAS folder:
24730
24730
24754
24730

Dolphin, HDD folder:
25388
25388
25388
25388


Thunar, NAS folder:
25664
25664
25664
25664
25664

Thunar, HDD folder:
25664
25664
25664
25664
25664


Does that make any sense?!? I have to say though, that when checking the HDD folder, I could only hear the disk heads' noises the first times - as if the folder structure was kept in a cache somewhere. With the NAS instead, the "activity" light would flash every time.

Cristian

[Jean]

Re: Weird problem when copying/syncing

I am curious what the outcome of this will be.

Are there differences in the mounting of src and dest?
I had a problem where characters in filenames could not be read because of the codepage in the mount statement was different.

[dibl]

Re: Weird problem when copying/syncing

Interesting that Thunar counts them both the same (but gets a different answer than the other file managers). My little "hobgoblin" mind likes that consistency!

The Dolphin results were almost consistent, but with a different number on the backup side than the source side. Both Dolphin and Konqueror were very consistent on the source side, but had problems and inconsistency on the backup side. That implies that there is something about the network operation or the network filesystem that is affecting the counting function in both of them.

That's my very inexpert observation. :P

[Quaxo76]

Re: Weird problem when copying/syncing

That implies that there is something about the network operation or the network filesystem that is affecting the counting function in both of them.

But then why does Thunar show a different count on the HD folder too?
If this is a bug indeed (and it seems so) where does it lie? The file managers, the kernel, the network... ?
In the next days I'll try to access the NAS from another computer, and see the results - to see if it's something weird with my pc or it's a problem with Linux!

Cristian

[Quaxo76]

Re: Weird problem when copying/syncing

OK, I tried (briefly) to access the NAS from a laptop. I only tried Dolphin, but I get similar behavior. The file count varies. Obviously I couldn't test the local HDD folder, but I think this has to be a "long-term" bug - as on my main PC I have Kubuntu 7.10, with several added "unofficial" packages; but on the laptop I have a "clean" 8.04 beta, just installed and updated.
So. What should I do? Is this a bug to be reported? If yes, to who?

Thanks
Cristian

[dibl]

Re: Weird problem when copying/syncing

While it's a desirable and virtuous thing to report bugs when they are found, I would not know what to "accuse", in this case. While a NAS should behave like a hard drive partition, there's something different going on with yours. I know from experience that Konqueror and Dolphin will count files consistently for backups made on different hard drives on the same computer. I dunno -- something clearly isn't handling the data correctly. :P

[Quaxo76]

Re: Weird problem when copying/syncing

Hmmm... and besides the NAS, there's still the problem of different file managers giving different file counts on a "common", local HDD! That must have nothing to do with the NAS or networking...

Cristian