Any way to speed up this command on a directory with 50,000+ files?

Started by theprof, October 31, 2024, 03:56:02 PM

Previous topic - Next topic

theprof

ExifTool version is 12.76 and it's scanning a network file share directory with a 1GPS connection.

exiftool -F -q -q -f -s3 -json -api QuickTimeUTC -api LargeFileSupport -r -FileType -CompressorName -CompressorId -FileTypeExtension -MIMEType -Duration -CoverArt "$DIRECTORY"

I basically need FileType (Container), CompressorName (Encoder), CompressorId (Codec Name), File Type Extension, Mime Type, Duration, and Cover Art (basically I check if its empty or not in the results).

It takes about an hour or so for exiftool to output the results, is there a way to speed it up.

Phil Harvey

I would add this to the command:

-api ignoretags=all

Also, some form of -fast may help.  Read here for more ideas.

But basically, this command will run slowly over a network connection no matter what you do.

You say it takes an hour for 50000 files...  That's 0.072 seconds per file.  I'm actually surprised it is that fast.  The videos must not be too long.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

theprof

Technically, it's in a virtual machine and the host is the file server so that may explain it.

FYI, not sure if this is a bug, when I add the "ignoretags=all" option, I get the following error in the console output from ExifTool:

Use of uninitialized value in string ne at /usr/share/perl5/Image/ExifTool/Text.pm line 194, <EXIFTOOL_FILE> line 1.

For reference, the full command I ran was:
exiftool -F -q -q -f -json -api QuickTimeUTC -api LargeFileSupport -api ignoretags=all -r -FileType -CompressorName -CompressorID -FileTypeExtension -MIMEType -Duration -CoverArt "$SOURCE_DIR"

Phil Harvey

A great, thanks.  That is a bug -- part of the code requires MIMEEncoding to be set, but it is being ignored.  I'll fix this in 13.01.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).