FInding a cross-system program(2 questions)

[errol]

Hi
I'm looking for a program that will find duplicate files (not OS files). I'll explain. I am a photographer and have many photos all over my computer. I have 2 large hard drives and when I make changes to my system (eg. install a new version of Linux) I do a backup of everything I fear I may loose. photos, docs etc. I am paranoiac in that way :-) , I guess. As I have a dual boot system, Windows & Linux (I use mainly Linux) I want to organize my photos and burn them onto DVD's in an orderly fashion. For this I want to find all duplicate photos (& other files if possible) and make some order in my computing life. Is there such a program which can go through ALL my partitions (even if they are NTFS), find duplicate files and allow me to delete or make order. It was suggested (on another forum) that I try FSLint but it seemed only to find duplicate Linux OS files.

2nd question
The other program I am looking for is a photo organizer that will "see" all the photos on my computer and enable me to catalogue them. I've tried digikam, fotoxx, Gimp. I'm sure it's a "mapping" problem because I can see the photos when using Dolphin.
Mmm.. Maybe I should open two threads for two different questions - though they are related, I think.
TIA
Errol

[Rog131]

Re: FInding a cross-system program(2 questions)

Duplicates:

Commandline:
fdupes - finds duplicate files in a given set of directories

GUI:
KleanSweep

File cleaner for KDE
KleanSweep allows you to reclaim disk space by finding unneeded files.
It can search for files basing on several criterias; you can seek for:
- empty files
- empty directories
- backup files
- broken symbolic links
- broken executables (executables with missing libraries)
- dead menu entries (.desktop files pointing to non-existing executables)
- duplicated files
- orphaned files -- files not found in RPM (for rpm-based distros, e.g.
Fedora Core, Suse) or DPKG (for dpkg based distros, e.g. Debian and Ubuntu)
database
- obsolete thumbnails (thumbnails conforming to freedesktop.org standard,
pointing to non-existing images)

Homepage: http://linux.bydg.org/~yogin/

It was suggested (on another forum) that I try FSLint but it seemed only to find duplicate Linux OS files.

Hmm - strange, i have thought that these programs are using MD5 signatures etc..

Hmm #2 man fdupes tells:

DESCRIPTION
Searches the given path for duplicate files. Such files are found by
comparing file sizes and MD5 signatures, followed by a byte-by-byte
comparison.

The other program I am looking for is a photo organizer that will "see" all the photos on my computer and enable me to catalogue them. I've tried digikam, fotoxx, Gimp.

Have you tried kphotoalbum ?

Package: kphotoalbum

tool for indexing, searching and viewing images by keywords for KDE

KPhotoAlbum lets you index, search, group and view images by keywords, date, locations and persons. It provides a quick and elegant way to lookup groups of images when you have thousands of pictures on your hard disk.

The information associated with each photo is stored in an XML file. Together with its keywords, KPhotoAlbum stores each picture's MD5 sum, so it will recognize them even if you move them to another directory. KPhotoAlbum can also create HTML galleries with the images you select.

KPhotoAlbum can also make use of the KIPI image handling plugins to extend its capabilities. The kipi-plugins package contains many useful extensions. Among others, it contains extensions for photo manipulation, importing, exporting and batch processing.

[errol]

Re: FInding a cross-system program(2 questions)

Thanks Rog131
I see now that my problem is "reading" the hard drives from within Linux. I'll try to explain. When I open Dolphin and go to storage media I can see and enter any of the partitions there - even if they are NTFS partitions. When going through one of the programs you suggested (or any program I've tried) I get to the "storage media" icon but nothing is seen. No list of drives, nothing. Just the word "media:/" comes up in the dialogue box and that's where it ends. The program (say Kphotoalbum) isn't able to scan the media and find the photos. As I said in my original request, it may be a "mapping" problem (for want of a better term). On the other hand when using Dolphin all is visible, so I'm stuck.
Errol

[Rog131]

Re: FInding a cross-system program(2 questions)

I see now that my problem is "reading" the hard drives from within Linux. I'll try to explain. When I open Dolphin and go to storage media I can see and enter any of the partitions there - even if they are NTFS partitions.

Are you mounting your ntfs/fat/ms windows partitions at the start ?

Do you have in the /etc/fstab something like this:

# fat32
# /dev/sda6
UUID=2A3D-1CD4 /media/sda6 vfat utf8,umask=007,gid=46 0 0
# /dev/sda7
UUID=333B-1CDD /media/sda7 vfat utf8,umask=007,gid=46 0 0

That is mounting two of my fat partitions to the /media.


fstab links

> fstab wiki
> How to fstab

[errol]

Re: FInding a cross-system program(2 questions)

Below is a copy of my /etc/fstab. I don't know if that is how it should be.
What I am looking for is a program (or a setting that will enable a program) to sweep all these partitions and do two things.
1. Gather all my photos is an album and display the thumbnails. KPhohtoalbum or the likes (maybe digikam)
2. Find duplicate photos so I can organize and delete the duplicates. kleensweep or the likes
This may be two separate programs. Meanwhile I can't get any program to "sweep" all my drives.
Errol



# -- This file has been automaticly generated by ntfs-config --
#
# <file system> <mount point> <type> <options> <dump> <pass>

proc /proc proc defaults 0 0
# Entry for /dev/sdb2 :

UUID=085fdea8-5be4-4d4a-aee2-9a579ed28588 / ext3 relatime,errors=remount-ro 0 1
# Entry for /dev/sdc5 :
UUID=373c7ca4-b426-41e3-80ea-f59f4c400a7a /home ext3 relatime 0 2
# Entry for /dev/ !! UNKNOW DEVICE !! :
UUID=9ca2d064-6a02-4b50-9fdf-4f351bfa233c none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,auto,exec,utf8 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
/dev/sdc6 /media/WinXP ntfs-3g defaults,locale=en_US.UTF-8 0 0
/dev/sdc2 /media/DATA ntfs-3g defaults,locale=en_US.UTF-8 0 0
/dev/sdc1 /media/PROGRAMS ntfs-3g defaults,locale=en_US.UTF-8 0 0
/dev/sdb1 /media/DRV1_VOL1 ntfs-3g defaults,locale=en_US.UTF-8 0 0
/dev/sda1 /media/New\040Volume ntfs-3g defaults,locale=en_US.UTF-8 0 0

[Rog131]

Re: FInding a cross-system program(2 questions)



Here

sudo fdisk -l

tells:

Device Boot Start End Blocks Id System
/dev/hdb1 * 1 9963 80027766 7 HPFS/NTFS

and i'm mounting it:

sudo mount /dev/hdb1 /media/tmp

I can use pictureviewer/duplicate search normally.


What happens if you / do you get error messages ?

1) make a small directory with few pictures

Here:

:~$ ls /media/tmp/tmp
2003-10-b-full_jpg.jpg
2003-10-e-full_jpg.jpg
2003-10-c-full_jpg.jpg
2004-04-a-full_jpg.jpg
2003-10-d-full_jpg.jpg
Duplicate_2003-10-b-full_jpg.jpg

2) run fdupes

Here:

:~$ fdupes /media/tmp/tmp/
/media/tmp/tmp/Duplicate_2003-10-b-full_jpg.jpg
/media/tmp/tmp/2003-10-b-full_jpg.jpg

[errol]

Re: FInding a cross-system program(2 questions)

I didn't quite understand what I should do. Sorry. I am not too well up on commands.
Errol

[Troberg]

Re: FInding a cross-system program(2 questions)

If I may hijack the thread a little:

I have a similar need, but on a larger scale. Does the listed programs work with a larger data set, such as millions of files, too much to scan it all every time new files are added?

Also, is there any software which can detect and remove or move logical image dupes as well as bit by bit dupes? In other words, to find dupes of an image, even if it's in another format or has been resized, and which is also able to select the "best" image for keeping (usually the largest pixel count and largest file size, possibly with a rule for "best" image format as well to keep BMP's from getting preference over PNG's and so on).

I once made such a program (it was even better, as it could detect dupes with added logos and stuff as well), but that was for Windows, so it's of little use now, and I'm far from up to speed on Linux development, so a rewrite is not imminent. The algorithm was almost ridicously simple, though, if anyone is interested.

[Rog131]

Re: FInding a cross-system program(2 questions)

To Errol:

- make a folder at your ntfs partition
- put there a picture, copy it to another name
- run the fdupes command:

fdupes /path/to/your/folder

Is the fdupes commad working ?
Do you get an error messge ?


To Troberg:

I have a similar need, but on a larger scale. Does the listed programs work with a larger data set, such as millions of files, too much to scan it all every time new files are added

Yes, i think, that duplicate finding is working but it will take much more time. I have searched duplicates from max 2000 pics.

Also, is there any software which can detect and remove or move logical image dupes as well as bit by bit dupes? In other words, to find dupes of an image, even if it's in another format or has been resized, and which is also able to select the "best" image for keeping (usually the largest pixel count and largest file size, possibly with a rule for "best" image format as well to keep BMP's from getting preference over PNG's and so on).

Yes, there are but those programs (image recognition) are usually commercial applications with hefty price tags.

There are some free/shareware applications, one of the good ones was a windows program called Dup Detector:
> Dup Detector
> Dup Detector site #2

[errol]

Re: FInding a cross-system program(2 questions)

Rog131, I'll get onto it tomorrow and get back to this thread. Once again, thanks for your efforts.
Errol

[Troberg]

Re: FInding a cross-system program(2 questions)

There are some free/shareware applications, one of the good ones was a windows program called Dup Detector:

Thanks, but I've tried it, and it didn't handle large image libraries well, and it detected quite a few false dupes.

The program I'm looking for really needs a database to keep track of already scanned images. The last time I dumped the database my program use (due to a crashed disk which made it inaccurate), it took me almost two months to make a complete rescan, running 24/7 on a 3 GHz machine...

[errol]

Re: FInding a cross-system program(2 questions)

Hi rog131
Sorry it took a while to test fdupes. Whatever media I tried I got the following error message
"could not chdir to /media/****
Something is wrong, but what?
Errol

[Snowhog]

Re: FInding a cross-system program(2 questions)

Sorry it took a while to test fdupes. Whatever media I tried I got the following error message
"could not chdir to /media/****

What was the exact command that you typed that resulted in the error message above?

[errol]

Re: FInding a cross-system program(2 questions)

This is what I typed & the reply
errol@errol-linux:~$ fdupes /media/New\040Volume
fdupes: could not chdir to /media/New040Volume
errol@errol-linux:~$
I based the command on the last line of the fstab file which is the volume I made a duplicate of a photo in another file. Below is the fstab.

# /etc/fstab: static file system information.
#
# -- This file has been automaticly generated by ntfs-config --
#
# <file system> <mount point> <type> <options> <dump> <pass>

proc /proc proc defaults 0 0
# Entry for /dev/sdb2 :
UUID=085fdea8-5be4-4d4a-aee2-9a579ed28588 / ext3 relatime,errors=remount-ro 0 1
# Entry for /dev/sdc5 :
UUID=373c7ca4-b426-41e3-80ea-f59f4c400a7a /home ext3 relatime 0 2
# Entry for /dev/ !! UNKNOW DEVICE !! :
UUID=9ca2d064-6a02-4b50-9fdf-4f351bfa233c none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,auto,exec,utf8 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
/dev/sdc6 /media/WinXP ntfs-3g defaults,locale=en_US.UTF-8 0 0
/dev/sdc2 /media/DATA ntfs-3g defaults,locale=en_US.UTF-8 0 0
/dev/sdc1 /media/PROGRAMS ntfs-3g defaults,locale=en_US.UTF-8 0 0
/dev/sdb1 /media/DRV1_VOL1 ntfs-3g defaults,locale=en_US.UTF-8 0 0
/dev/sda1 /media/New\040Volume ntfs-3g defaults,locale=en_US.UTF-8 0 0

[Snowhog]

Re: FInding a cross-system program(2 questions)

This is what I typed & the reply
errol@errol-linux:~$ fdupes /media/New\040Volume
fdupes: could not chdir to /media/New040Volume

This is what you need to type:

fdupes /media/New\ Volume

The 040 is not really part of the name. It represents the 'space' you have in it: New Volume

Try again using the syntax I've shown above and the command will execute properly,