Need to *significantly* speed up Exif-fetching. However how to do that !??

[louis]

Hello,

As the title suggest, I have to significantly speed up ExifFile generation. My Idea is to switch from a ^per photo/video call^ towards one ^batch call^.

An (automatic generated) batch file, containing all files to be processed together which a per file specified xml-filename, should be send to Exiftool.
That should result in a directory with all the generated xml-files, having the ^specified name^.
 

At this moment I am using ExifTool to fetch Exif-info from my photo- and video-collection. Lots of mainly photo's. The goal is to put the exif data in a database.

I do that using ms-access in combination with VBA (visual basic for applications)
- step-1: generating a list of all files (in many directory's)
- step-2: calling Exiftool via the shell generating a xmp file per photo/video
Shell (ExifTool & " " & ExifCmd & " " & SQ & FullFileName & SQ), vbHide
- step-3: parsing the xmp-file and placing the info in the DB

That works, however one BIG-problem! Exiftool takes much much to much time to generate the xmp-file
(about 1,5 seconds!!      per file)

So that must be two magnitudes faster. The only way to archive that seems to be:
1) to prepare a file containing a list of files to be processed. Containing:
a path to source filename
b path / name xml-output file 
2) define how each file (photo/video) should be processed actual:
^ExifCmd = "-X -o "^
3) send the command file to ExifTool, in order to fill my ^ExifFileDirectory^ with ^xml-exif-files^
4) parse those files and put the exif data in the DB (note that the xml-file name starts with the DB-key)

That is the idea, which hopefully will take care of a dramatic speedup.
However how to implement this !?

Idea's / help would be appreciated!
 

Louis
PS. an API to access ExifTool directly from VBA, so that the workaround via the shell could be bypassed,
would be highly appreciated
 



       

[StarGeek]

My Idea is to switch from a ^per photo/video call^ towards one ^batch call^.

This is Common Mistake #3.  Exiftool's startup time is its biggest performance hit.

An (automatic generated) batch file, containing all files to be processed together which a per file specified xml-filename, should be send to Exiftool.
That should result in a directory with all the generated xml-files, having the ^specified name^.

Generate your list of files, save it as a text file, and then pass that list to exiftool with the -@ (Argfile) option.
exiftool <commands> -@ /path/to/FileList.txt

Or even pipe it directly to exiftool.  Here's an example which takes the output of Linux's find command as exiftool's input.
 

At this moment I am using ExifTool to fetch Exif-info from my photo- and video-collection. Lots of mainly photo's. The goal is to put the exif data in a database.

I always suggest using already existing Digital Asset Management (DAM) programs rather than writing your own.  Programs such as DigiKam (free), LightRoom, and IMatch (both paid) already keep track of metadata as well as writing it to the correct locations without having to learn of the maddening details of metadata.  Myself, I use IMatch, which uses exiftool on the back end, as it is extremely powerful for dealing with metadata.

- step-2: calling Exiftool via the shell generating a xmp file per photo/video
Shell (ExifTool & " " & ExifCmd & " " & SQ & FullFileName & SQ), vbHide

See Metadata Sidecar Files example #13.  You would just pass your list of files via the -@ option and exiftool will run through the list, creating XMP files as needed.

But since you want XML files instead of XMP, you would have to tweak the command a bit.  Probably use Example #11 as a starting point.

[louis]

Some reactions:

- I only partly agree with "common mistake 3^. Yep it is a mistake as seem from the ExifTool as it is designed, but calling the application per file. is is the logical way to do it from the usage point of view. IMHO it is crazy that you can not fix ExifTool in memory as running application have to be loaded for every call. I understand that using a batch process is a work around for that (MHO)

- I understand your batch command line, but it will not work since I also have to specify the name of the generated ExifFile (I need a more precise/exact key to correlate image file and ExifFile)

- I understand your preference for a out of the box tool (has advantages I agree), but I prefer my own database having full control

- I do not really mind the exact output format, as long as it can be parsed (json, xml, xmp). They have a lot of similarities.

- As stated at the end, I would really like to see a program api

       



       

[StarGeek]

- As stated at the end, I would really like to see a program api

Use the -stay_open option.

[louis]

Hum,

I do not completely understand the stay_open function. It feels like ExifTool is listening with a UNIX tail command to to a textfile and become active every time a line is added to that file.

However
- I am using windows
- and can I pass both 'commandpart' and 'filepart' in each line of that control file !?

I will do some trails, but I would prefer to have working examples. That would save me the trail and error method to see what is happening ....

[StarGeek]

It feels like ExifTool is listening with a UNIX tail command to to a textfile and become active every time a line is added to that file.

That's the basic idea, though there isn't a need for any other command line program such as tail

- I am using windows

Works perfectly fine on Windows.  For example, Photools IMatch runs up to four instances of exiftool in the background.

- and can I pass both 'commandpart' and 'filepart' in each line of that control file !?

It uses the -@ (Argfile) option formatting (see also FAQ #29).  Each argument needs to be on a separate line, as shown in the example in that FAQ.  No trailing spaces, no quotes are used and you can't use file redirection, as that is a property of the command line.

Once you pass all your arguments, then the -execute option is passed to do the processing. 

It's not something I've taken time to figure out, so I can't help there, but seach these forums for "-Stay_Open", there should be some examples.  Example

I don't know if it would help, but if you know Python, you can take a look at the source code for PyExiftool to see how it's implemented there.  Or the C++ Interface for ExifTool if you know that.