stay_open never sends {ready}

[cowwoc]

Hi,

I am using exiftool 12.13 under Windows.

I keep on getting hangs trying to use "stay_open" where I'm sending a command, getting back some output but never get back {ready}. I wasted a day chasing one case where I was only reading stdout but not stderr. Although I don't see anything being written to stderr, the minute I started draining it I started getting {ready} on stdout. Now I am reading both stdout and stderr but I am getting a different kind of problem.

I am trying strip all metadata from image/video files prior to comparing them. So I invoked "exiftool -stay_open true -@ -" and then piped the following command to stdin:

-all=\n
-o\n
-\n
-b\n
<path>\n
-execute\n

exiftool sends some output to stdout but also sends "Warning: ICC_Profile deleted. Image colors may be affected - <path>". The warning causes a problem because I am expecting the output to only contain the file contents. The <path> differs between files although their content is identical. So the comparison fails when it should not.

I tried adding:

-m\n
-q\n
-q\n

to the above commands to suppress the warning. Now I get some output on stdout but {ready} is no longer being received on stdout. stderr doesn't seem to contain anything.

So I added:

-v0\n

to the above commands and now I get "Nothing changed in <path>" at the end of stdout but still no "{ready}".

It almost sounds like -q -q causes exiftool to not flush its stdout properly. Any ideas?

[cowwoc]

Well, that was painful... This seems to be by design:

https://github.com/exiftool/exiftool/blob/master/windows_exiftool#L418

Code Select Expand


# flush console and print "{ready}" message if -stay_open is in effect
if ($stayOpen >= 2) {
    if ($quiet and not defined $executeID) {
        # flush output if possible
        eval { require IO::Handle } and STDERR->flush(), STDOUT->flush();
    } else {
        eval { require IO::Handle } and STDERR->flush();
        my $id = defined $executeID ? $executeID : '';
        my $save = $|;
        $| = 1;     # turn on output autoflush for stdout
        print "{ready$id}\n";
        $| = $save; # restore original autoflush setting
    }
}


I was expecting {ready} to get sent even if quiet mode was enabled. If you plan to maintain this behavior, can you please document it and indicate that we need to use -execute<number> or -echo to force {ready} to get sent?

[cowwoc]

Is there a way to direct all warnings/errors to stderr instead of suppressing them in stdout? That would simplify parsing a lot for me.

[Phil Harvey]

I keep on getting hangs trying to use "stay_open" where I'm sending a command, getting back some output but never get back {ready}. I wasted a day chasing one case where I was only reading stdout but not stderr. Although I don't see anything being written to stderr, the minute I started draining it I started getting {ready} on stdout.

That is odd.  I wouldn't have expected this.  I doubt it works like this on Mac/Linux.

-q does suppress the {ready} message unless a number is used in the -execute option (eg. -execute99).

...but I don't understand why you think the warnings are going to stdout.  Here is an example on my system (sh on Mac):

Code Select Expand

% exiftool c.jpg -o d.jpg  -artist=me >t1 2>t2
% cat t1
    1 image files created
% cat t2
Warning: XMP format error (no closing tag for x:xmpmeta) - c.jpg
Warning: [minor] Empty XMP - c.jpg

As you can see, the warnings went to t2 (stderr).

- Phil

[cowwoc]

My mistake. Warnings/errors really do go to stderr. The next problem I am having is that "-echo4 Exit code: ${status}" sends data to stderr before flushing stdout.

As soon as my code gets "Exit code" in stderr it expects to terminate the output from one command and begin output from the next command. I can see based on https://github.com/exiftool/exiftool/blob/b9d3dee6b9eaeb6d236a0dcadb8eb5471874f140/windows_exiftool#L420 that this is by design. Is there a way to make this work without sleep()ing for some undetermined time before attempting to read stdout? This seems fragile. It would have been better from my perspective to always flush stdout before stderr, not the other way around.

Remember, I'm trying to get only the binary contents of the file in stdout. Ideally, I don't want to see any other text thrown in such as {ready}. I am pushing those into stderr.

[Phil Harvey]

I have to either flush stdout or stderr first, and the normal use is for the {ready} message to be in stdout, so it really makes sense to flush stdout with the {ready} message after everything else is complete.  Your use case is a bit different.  Offhand I can't think of a quick way to satisfy both cases, but I'll think about this more when I get some time.

- Phil

[cowwoc]

Is there maybe a no-op command I could invoke? This way I would:

1. Receive "exit code" in stderr.
2. Invoke "no-op" in your stdin.
3. Wait for a second "exit code" in stderr and I'd be guaranteed that between the first and second "exit code" you would have flushed stdout.

No-op would output nothing to stdout, but flushing stderr followed by stdout would still take place.

Thinking about this further, wouldn't it be sufficient for me to invoke:

-echo4 "Status code: ${status}"
-execute

and this would result in the no-op operation I asked for? I'm going to try this now.

[Phil Harvey]

Yes.  Putting the -echo4 in a separate command should do it.

There is an undocumented -nop option btw, but I don't think you need to use it.

- Phil

[cowwoc]

FYI, "-q\n-echo4 stdout flushed\n-execute\n" ended up working for me.

Thank you for your help!

[Phil Harvey]

FYI, "-q\n-echo4 stdout flushed\n-execute\n" ended up working for me.

I think you forgot a newline.  It should be:

Code Select Expand

-q\n-echo4\nstdout flushed\n-execute\n

[cowwoc]

Yes, you're right.