ExifTool Forum

ExifTool => Newbies => Topic started by: lumiere on July 13, 2020, 02:35:41 PM

Title: Sorting out photos with content verification (md5 / sha256)
Post by: lumiere on July 13, 2020, 02:35:41 PM
Hello all,

I have several family pictures scattered across many, many folders often with clones around.
Some of them were taken with burst mode (many pictures in the same second) so YYYY/MM/DD/HH/MM/SS is not the right pattern.
I would like to copy them over to final folder destination sorting them by:
YYYY/MM/DD/HH-MM-SS.[ext]
But if there is already a picture with particular name in the target folder then I would like to compare md5/sha256 of their content to really make sure they are the same.
If they are the same then skip it. If they are not then create new file with [name]-1.jpg, [name]-2.jpg etc
Is there any way I can achieve this with exiftool ?
Also is there any way I can check the md5/sha256 in parallel ? I have quite good CPU with many threads and can burn it to speed up the check.
Title: Re: Sorting out photos with content verification (md5 / sha256)
Post by: Phil Harvey on July 13, 2020, 02:44:31 PM
ExifTool doesn't have the ability to do a checksum of the entire file.  I would suggest using an md5 utility to generate checksums for all of your files, then removing the files from the list that have duplicate checksums, then sending the remaining file names to ExifTool with a command like this:

exiftool "-filename<createdate" -d %Y/%m/%d/%H-%M-%S%%-c.%%e -ext jpg -r DIR

- Phil
Title: Re: Sorting out photos with content verification (md5 / sha256)
Post by: lumiere on July 13, 2020, 06:17:18 PM
Thanks @Phil !
Could it be more generic command working not only with jpg but all jpg/png/gif at once ?
Also can it be sorted based on exif data stored inside each file ?
Title: Re: Sorting out photos with content verification (md5 / sha256)
Post by: Phil Harvey on July 13, 2020, 08:20:15 PM
Yes.  Just add more -ext options with whatever files you want to move.  Or leave out the -ext option entirely to move anything in the exiftool -listwf extension list (provided it has a CreateDate tag).

- Phil
Title: Re: Sorting out photos with content verification (md5 / sha256)
Post by: lumiere on July 14, 2020, 01:51:37 PM
Thanks Phil again :-)
WIll it sort based on exif data stored inside those files or based on ... filename ?
I am not sure I understand what the syntax "-filename<createdate" means ...
Title: Re: Sorting out photos with content verification (md5 / sha256)
Post by: Phil Harvey on July 15, 2020, 06:27:25 AM
Read here (https://exiftool.org/filename.html) for more details about setting the FileName.

- Phil
Title: Re: Sorting out photos with content verification (md5 / sha256)
Post by: TaToToMeK on March 04, 2023, 05:48:47 AM
Hi
following lumiere's question:

I am looking for a method to calculate the MD5 hash for image only - not for the whole file with metadata.
I sometimes correct exiftags manually (e.g. date, GPSloc). And I need a method to find all copies of the same image whether their metadata are modified or not.
What is your recommendation?
Title: Re: Sorting out photos with content verification (md5 / sha256)
Post by: Phil Harvey on March 04, 2023, 07:06:45 AM
For JPEG files, an MD5 for the image only may be calculated like this:

exiftool -all= -o - FILE | md5

Other formats are more problematic because metadata may not be completely removed.

- Phil
Title: Re: Sorting out photos with content verification (md5 / sha256)
Post by: Phil Harvey on March 15, 2023, 02:18:38 PM
As of ExifTool 12.58 there is a new Extra (https://exiftool.org/TagNames/Extra.html) ImageDataMD5 tag which returns an MD5 digest of the image data only for JPEG and TIFF-based files.

- Phil