stay_open and stdout caching under Windows

[Mac2]

Hi,

I today tried to make stay_open work with my application. But it seems that Windows stdout caching plays havoc with my experiments.

I use args files to drive ExifTool. The arg files produces one MIE file and then -executes. It then produces an XML output file using the -X option and two (!) input files, the original image and the MIE file produced earlier, something like:

Code Select Expand

-tagsfromfile
...
original file name
name of temp MIE file
-execute
-G
-X
...
name of temp MIE file
original file name
-execute



The expected result is one XML file with two XML sets, one for each file. This works without stay_open.

When I use stay_open, some part of the XML file remain somewhere in the Windows stdout buffers.
I only get the remaining data when I close my app and thus close the stdout and stderr pipes. which flushes them.

Furthermore, my app is counting the {ready} statements. But ExifTool issues only one {ready} statement even if there are two -executes in the args file?

When I add -v0 to force a flush of the file buffers, it works and I get all the XML data in stdout.
The second {ready} after the second execute is still missing.

The problem with this approach is that -v0 causes ExifTool to also write the file names into the resulting XML file, rendering it invalid.

[Phil Harvey]

I guess I don't understand the problem.  Since you are already filtering the "{ready}" messages, why is it a problem to also filter the file names when -v0 is used?

- Phil

Edit: Oh wait.  You said the 2nd "{ready}" is missing even if you use -v0.  I don't understand this because exiftool's stdout should be flushed by the newline after the "{ready}" when -v0 is used.

[Phil Harvey]

In case it helps, there is are undocumented -echo and -echo2 options which do nothing but send a message to stdout and stderr respectively.  Each option takes a single parameter which is the message string.

These may be useful for diagnostics, command/response synchronization, or maybe even help with flushing the buffers.

- Phil

[Mac2]

Hi, Phil

thanks for taking the time to look into this.
I've made several experiments and a simple additional -execute seems to do the trick:
...
-X
temporary MIE file
original file name
-execute
-execute

I then get a 2nd {ready} in stdout after the XML content was streamed and the buffers are flushed.
And I cannot step through this in the debugger, which somehow seems to interfere with the stdout processing. Strange.
Oh wonders of Windows stdout 

In case it helps, there is are undocumented -echo and -echo2 options

Yeah, I've played with these too. But removing -v0 and adding the 2nd execute works as well and the output stays "clean".

To aid synchronizing when using stay_open it would be cool to have an optional parameter for execute that is mirrored by {ready}:

-execute some_id

which results in

{ready some_id}

This would make it easier and safer to match a {ready} to a specific -execute.

[Phil Harvey]

To aid synchronizing when using stay_open it would be cool to have an optional parameter for execute that is mirrored by {ready}:

-execute some_id

which results in

{ready some_id}

This would make it easier and safer to match a {ready} to a specific -execute.

This is a really good idea, but I can't add an optional parameter to -execute without being able to recognize it somehow.  However, something like -execute# would work, where # is echoed in the ready message.  ie) -execute962 gives "{ready962}".

- Phil

Edit:  I have added this new feature and it will appear in version 8.64

[Mac2]

Fabulous!

[Mac2]

I've made some further tests.
On my Windows 7 64-Bit machine it seems that ExifTool does not emit the second {ready} when running an argument file with two -execute commands like this:

-overwrite_original
-m
-tagsfromfile
c:\image.jpg
-all
-all:all
f:\temp\imt1084050914771.xmp
-execute
-G
-X
-D
-f
-X
-t
-q
-q
-l
-lang
en
f:\temp\imt1084050914771.xmp
c:\image.jpg
-execute


Even when I close the pipe handles to force a "flush" or add -v0, the second {ready} never shows up in the ExifTool output.
Part of the resulting XML output remain in the stdout pipe, there seems to be no flush. Maybe this is related to the second {ready} missing?

(I tried -echo but i does not work in args files (gives "no file specified" message)).

I'm really pulling my hair out over this...

[Phil Harvey]

The answer may be found in the -stay_open documentation:

3) Write "-execute\n" to ARGFILE, where "\n" represents a newline
            sequence.  (Note: You may need to flush your write buffers here if
            using buffered output.)  Exiftool will then execute the command
            with the arguments received up to this point, send a "{ready}"
            message to stdout when done (unless the -q option is used), and
            continue trying to read arguments for the next command from
            ARGFILE.

It sounds like maybe you want this feature of -q disabled.  Is there an easy work-around for you, or should I consider changing this in exiftool?

- Phil

[Phil Harvey]

I tried using -echo and it works fine for me.  Remember to put argument for -echo on a separate line.

However, I just realized that the work-around is fairly straighforward.  If you really need to use the -q option, then you can recover the "{ready}" message by adding another -execute with an empty command afterward.  ie)

-overwrite_original
-m
-tagsfromfile
c:\image.jpg
-all
-all:all
f:\temp\imt1084050914771.xmp
-execute
-G
-X
-D
-f
-X
-t
-q
-q
-l
-lang
en
f:\temp\imt1084050914771.xmp
c:\image.jpg
-execute
-execute

- Phil

[Mac2]

Aaahhhhh.... 3 x

The -q is a left-over from my initial implementation, which worked without stay  stay_open. I wanted to keep the output as clean as possible, thus the two -q -q and the -m.
I obviously have missed the mention of -q in the stay_open description. Note to self: Always read the fine print.

I've read all that stuff several times, honest. But it's a ton. And switching my existing code to use stay_open is quite a larger task than I had anticipated and such dumb mistakes have to creep up. Sigh.

The -q explains perfectly why the problem appeared only for some of the tasks I do with ExifTool. I used it only for this specific argument segment.

Phil, I apologize for wasting your time!

I will now go and bang my head against a wall for some time. And then sit down tonight and implement the changes. Hopefully this not only solves the missing {ready} but also the data remaining in the buffer. If all else fails I flush it with echos (I got this one working too. Options always on a separate line in Args files  ).

[Phil Harvey]

I'm glad this answered some questions.

Please don't bang your head too hard.  I know how easy it is to miss things like this in the documentation.

- Phil