Performance question

Started by philbond87, October 06, 2021, 11:48:10 AM

Previous topic - Next topic

philbond87

I have a nested directory structure containing several hundred files, from which I need to read from several different meta-data tags.

Reading the tags I need from each file individually, using individual calls to exiftool, is much slower than building one recursive call to gather the data.

What I am wondering, though, is what the speed/performance difference is between making one recursive call and building in ARGFILE of individual file names, along with thee one line exiftool command.

StarGeek

The performance hit of individual calls to exiftool is always going to be significantly greater than one or two calls, especially over hundreds or thousands of files.  This is Common Mistake #3 for this reason.

If you can build up the list in some way, either using exiftool with the -p (-printFormat) option for example, or the even with the find command on linux, it will be much faster.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

philbond87

@StarGeek, thanks.
I know it's much slower making individual calls so I make every effort to avoid it, whenever possible (which is why so many of my questions on this forum relate to cramming whatever logic I can into the exiftool call!)

I can build the list of file paths (I already do this in a couple of my apps).
I was just wondering if running exiftool with such an argfile, plus a single exiftool command, ends up being as fast (or nearly as fast) as running the same command with the -r option along with the directory path.

StarGeek

I've never tested it, but I doubt there's a significant difference between reading the list from an arg file and just passing the directory.

I would think there would be more of a performance hit based upon the how fast the app used to generate the list in the first place is.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

Phil Harvey

Generating a list of files to process is very fast unless the -if option is being used.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

f-stop

Quote from: Phil Harvey on October 07, 2021, 07:38:37 AMGenerating a list of files to process is very fast unless the -if option is being used.

- Phil

Any way to speed this up (the folder contains >10,000 files) ?

exiftool -progress -if '$DateTimeOriginal ge "2023:01:18"' -p '$FileName' *.jpg *.heic
Thanks!

Phil Harvey

You could add -fast2 and -e to speed it up a bit, but it probably won't have a great effect.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).