Help with speeding up ExifTool

[Phil Harvey]

However it's difficult to understand the relation between this wrong parameter for extension selection and the fact that over 1077 files analysed a little bit less than 30 have been with such warning message.

If you request specific tags to be extracted, ExifTool will only return a warning if "Warning" was one of the tag you requested, or if none of the requested tags existed.

2/The warnings/errors go to stderr.  If this is the same file as stdout, the buffering on some systems may cause the outputs to be out of order in relationship to each other.
Agreed. May be a flush at the end of each analysed file can solve this issue. With extra time cost.

You can use -v0 to enable output flushing after each line, which will likely solve this problem, but may affect performance slightly as you mentioned.

3/Your "-stay_open\nfalse\n" problem could be 2 things:  1) extra space somewhere in an argument, or 2) an old version of ExifTool.
Agreed. It was an extra space issue. (I'm running version 9.20)

Seems that exiftool is sensible to extra space... From now I'll use only one space and only when required.

In the args file, all spaces are significant.  In this file, the newlines are the separator, not white space as on the command line.

Remark : We do not receive an acknowledgement after a -stay_open false request.
Could it be possible to have a message like this

Code Select Expand

Exiftool will terminate soon...
{ready}

You can add any terminating messages you want by using a -echo option after the -stay_open true -@ ARGFILE on the command line.

- Phil

[cuagn]

Magic! (or well analysed and delivered!)

Thank you

Marc

[jgro]

Phil (or anyone),

I'm running Windows 7 and using the command: exiftool -stay_open True -@ args.txt

Initially, my args.txt file is created, but blank.  As I append entries to the file, exiftool always seems to be behind. 

For example, when I add and save the following lines to args.txt, nothing happens:

test.JPG
-execute123


However, when I add another line to args.txt, exiftool processes the test.JPG file.  Weird!!  Can anyone explain why this is?  (Sorry for bringing up an old post, but thought it'd be better than creating a new one).

-jgro

[Phil Harvey]

It sounds like you aren't ending the last line with a newline character.

- Phil

[jgro]

Hmm, yeah it's definitely something weird with the newlines on Windows. Looks like it depends on how you write to the arguments file.  When I open the file with notepad++ or notepad, the output always seems to lag behind the latest addition to the file.  However, when I use a Python script to write to the arguments file, it works just fine! Go figure.

And of course it runs great in Linux.  Thanks for the tip, Phil.

-jgro

[Phil Harvey]

OK.  Now that sounds like a flushing problem.  For the -stay_open documentation:

            3) Write "-execute\n" to ARGFILE, where "\n" represents a newline
            sequence.  (Note: You may need to flush your write buffers here if
            using buffered output.)

- Phil

[martenzi]

Hi Phil,
Above documentation is good enough... I mean, -stay_open isn't easy to explain and it took a while before I "got it" 
How I started was, I ran Exiftool from console window. That's the best way to understand -stay_open, because results (and mistakes) can be seen immediately. Once I understood it, I only needed to transfer that knowledge into my app -and if results weren't as expected, I knew only my app was resposible for that.
If I recall corectly, it goes like this:

1. create empty txt file, for example MyArgs.txt and save it.
2. Open console window and execue:
exiftool -stay_open True -@ MyArgs.txt
..and nothing will happen.
It can't, because ExifTool was only loaded in memory and is waiting to execute some command, which is expected to be inside MyArgs.txt file.
3. While console window is open, open MyArgs.txt file and write inside:
-ver
-execute
..and re-save the file. At the momment you do this, ExifTool will execute commands in that file and result will be written into console window.... and ExifTool keeps waiting for next command(s) inside MyArgs file.
4. Open MyArgs.txt file again, delete all lines and add* (for example):
-Exif:CreateDate
MyPhoto.jpg
-execute
..and re-save file again. And again, you'll get requested result from ExifTool. Repeat that as long you wish: ExifTool is just waiting for your commands 
When you're done, open MyArgs file for the last time, delete all lines and add* inside:
-stay_open
False
..and re-save file. Now, ExifTool will exit "waiting state" and you'll get prompt in console window again.

I firmly think this practical guide should make the documentation as an example. I´m somewhere in between noob and power user (relating to cmd usage). I have some additional info to relay to other users and a problem I´m trying to debunk. I´m on High Sierra and generally using Sublime Text for editing argfiles. For some reason, saving in ST does not give any results to the console window. I tried "Textedit" but it does not allow me to save in plain "txt" files, it forces RTF. I looked briefly in ST setting/config and could not find anything that seemed related to some "automatic save" feature or "stay open" options, as I presumed ST had some feature that allowed me to automatically have a file open in ST and not save until verified. (I´m speculating here about this potential feature).

When I read Bogdans instructions I was led to believe that I can use the actual console window to write directly while stay open. This results in nothing. I now use "Atom" and the writing inside argfile.txt works and gives results in console window. Thus:

1. does bogdan mean that one can type commands directly into console (not ref to regular exiftool invokation in terminal) ? - I don´t understand where I would type out my arguments, if I choose not to use argfile.

2. I havent figured out why saving in ST does not imply direct save to argfile.txt and thus show results in console window.

[Phil Harvey]

Saving with a text editor will only work if it writes in append mode, which is definitely not guaranteed.

You can write in append mode from the Terminal like this:

echo "ARG" >> MyArgs.txt

- Phil

[Hubert]

I tried "Textedit" but it does not allow me to save in plain "txt" files, it forces RTF.

You can change Textedit's default format from RTF to Plain text in Preferences. TextWrangler (free) is a much more powerful option that also writes plain text files.

[cowwoc]

The biggest gains (roughly 60x) would come if you can pre-launch exiftool.exe with the -stay_open option instead of executing exiftool for each file.

- Phil

Hi Phil,

Out of curiosity, if one mixes -stay_open with binary output how do we identify the end of output after each execution? I've never managed to successfully switched back and forth between parsing a process' output as text and binary. Is there an easier way?

[cowwoc]

The biggest gains (roughly 60x) would come if you can pre-launch exiftool.exe with the -stay_open option instead of executing exiftool for each file.

- Phil

Hi Phil,

Out of curiosity, if one mixes -stay_open with binary output how do we identify the end of output after each execution? I've never managed to successfully switched back and forth between parsing a process' output as text and binary. Is there an easier way?

I got this working. ExifTool prints "{ready}\r\n" at the end of each command's output, regardless of whether it is outputting text or binary.

Instead of switching back and forth between text and binary mode parsing, I treat just the output as if it is always binary.

1. Convert the string "{ready}\r\n" into a byte[] once at the beginning of the program.
2. For every command invoked by the user, read the output as binary (even if it contains text).
3. Search the output for the end-of-stream byte sequence from step 1. Return all the bytes/text up until this point.

[Phil Harvey]

The output of exiftool should always be treated as binary.

- Phil