First, apology and request for forgiveness if this exact same question has been asked before : )
Hoping to use ExifTool to solve a problem that is described by the following:
- Multiple hard drives
- Each hard drive has folders containing pictures, videos, and other files and documents
- Duplicated folders of pictures and videos are scattered everywhere, a result of many failed organizing attempts
- Scared to let a program de-duplicate and organize photos and videos in case of loss, but surely want to because there are way too many to do this manually, by hand...
The end goal:
- Have a hard drive of pictures and videos organized according to year and month
- Remove all duplicates by performing a bit-by-bit comparison of all files for 100% confidence that only true duplicates are being removed (no deleting any pictures that are not duplicates)
- Use rsync on Linux to create a backup of that hard drive
- Same as above, create an additional backup
The approach:
- All organizing work done using a Linux computer
- Gather all the pictures and videos, including duplicates, into folders organized by year and month
- Don't include other documents, just pictures and videos
- Identify and remove duplicates after verifying 100% match of all duplicate files
- Once organized, rename some files as needed, keeping custom names that were manually assigned to some items years before
- Never get myself into a mess like this again : )
This goal may end up using ExifTool as the main workhorse, which is great : )
I am also ok with using some additional Linux tools as needed, and would like to avoid proprietary software in this process.
Can it be done? If so, how best to proceed, and also how to ask additional questions as needed for this topic?
Thank you! Also a big thank you for ExifTool : )
Search through the forums. This is one of the most common questions and there are litterally hundreds of posts about it. You can use the forum search, but sometimes is also good to use your favorite search engine and add
site:exiftool.org. The forum search only searches exact words, but Google/Bing/etc, will include similar words and sometimes pull up better results.
Quote from: h on June 15, 2025, 12:01:19 PMRemove all duplicates by performing a bit-by-bit comparison of all files for 100% confidence that only true duplicates are being removed (no deleting any pictures that are not duplicates)
One thing to take note of is that you can have the exact same image, but a bit-by-bit comparison will fail because the metadata does not match. It's better to use a search program that does image comparison (Perceptual Hash/PHash). This can be either through a duplicate finder program such as DupeGuru or Czkawka (both also do bit-by-bit comparison) or through a Digital Asset Management (DAM) program such as Lightroom.
Exiftool does have the ability to create a hash of just the image data, ignoring the metadata, but it can't search between multiple files. You could use exiftool to create a CSV file, load that into a spreadsheet program, and sort/search for duplicates.
exiftool -ImageDataHash -csv /path/to/files/ >ImageHash.csv