Stay_open via Standard input

[BogdanH]

Hi Phil,
Usage of -stay_open via ArgFile seems to be relative easy to learn, because there's always some feedback on screen (either success or error message). But, I would like to feed data to exiftool via Standard input -to avoid managing ArgFile(s).
I've created a procedure where Pipe is used for that purpose, but working with these things (pipes) can be quite tricky... That is, I've tried few things, but my app always "freezes" on attempt to write data via StdIn. I can imagine  two reasons why that happens:
-my "piping" is buggy, or
-data I send via StdIn is wrong (and Exiftool just "waits" for more).

Now the question.. What's the proper call, to set ExifTool in stay_open state? My thoughts:
-exiftool -stay_open true <-won't work, because -@ is needed,
-exiftool -stay_open true -@ <-won't work, because ArgFile is expected (which I don't use/have),
-exiftool -stay_open true -@ - <-here, ExifTool seems to wait for further input (at least in case, it's used in console window).

Let's say, I use last command (from above):
-exiftool -stay_open true -@ -
-to initialize ExifTool.
Now, I would like to execute command equivalent to:
exiftool -ver
-but via StdIn. What do I now send via StdIn?
-ver\n-execute\n <-something like this?

That is, to elimininate errors in my code, first I must be sure, that I'm using stay_open properly...

Thanks for answering,
Bogdan

[Phil Harvey]

Hi Bogdan,

Yes, "exiftool -stay_open true -@ -" is correct to use stdin for the argfile input.

And "-ver\n-execute\n" is correct for the argfile syntax.

Looks like you are good to go.  You must just be sure to either disable the output buffering or flush your output after "-execute\n".  Otherwise you may have a problem with exiftool waiting for input.

- Phil

[BogdanH]

Thanks for answering fast. I will tryout tomorrow.. it's past 9PM here -I don't even dare to start trying now

Bogdan

[BogdanH]

Here I am... embarrassed -I simply can't get this thing to work. I don't pretend being an expert in programming (especially using pipes), but it can't be that complicated!
As mentioned above, I "normally" start ExifTool, by executing:
exiftool -stay_open true -@ -
-because ExifTool doesn't send any output at this stage, I suspect it is waiting to get further data (commands actually) via pipe -btw. I've checked that ExifTool process is actually running.
Then I send following string (via StdInput):
-ver\n-execute\n
-where \n is substituded with #13#10 characters (for newline sequence:CR+LF). That is, above string has length of 16 characters and my function confirms 16 bytes have been sent successfully. But, nothing happens! -as if ExifTool would wait for more data.

After my ideas ran out, I've started using StdInput only (no StdOutput pipe created, for reading output) and I made ExifTool's console window visible, so I could (hopefully) see some ExifTool output there. But, nothing.. except cursor blinking.
I must admit, that I'm writting this to clear my mind in first place. And it's not that I need "stay_open" right now.. what drives me crazy is, that I'm not able to solve such "basic" task. I probably need a break 

Bogdan

[Phil Harvey]

If you have properly sent your string to exiftool's stdin, then the only remaining problem is the buffering I mentioned.  Presumably you are writing to some filehandle that you attached to exiftool's stdin.  Did you try flushing or closing this filehandle?

- Phil

[BogdanH]

Succeed 

What a spaghetti code.. there's nothing I haven't tried. To tell the truth, I started to believe there's something wrong with ExifTool and thus even tried to "decypher" ExifTool source -imagine that 

Of course, after cleaning up, I need to do some further tests.. but in general, it works.

Greetings,
Bogdan

[Phil Harvey]

Hi Bogdan,

So did you figure out what the specific problem was?

- Phil

[BogdanH]

Hi Phil,

Yes, I figured out the reason. Above I said, it looked like (after sucessfully sending data via StdIn and flushing buffer) ExifTool would still wait for "something" -it was reasonable to assume that, because there was no response from ExifTool (by reading StdOut).

And before continuing: even at that time, I knew, the reason is my limited knowledge -but admitting that doesn't solve problems, though.

Anyway, to keep story short: The reason was, I was reading StdOut to soon after sending data to StdIn. That is, data from ExifTool wasn't there yet! -and according to Windows docs, attempt to read "empty" StdOut causes endless waiting (=simplified).
Ok, those, who know more than I do, will say: hey, there's "PeekNamedPipe" API function ment exactly for such cases. This function always return immediately and tells if there's any (how much, actually) data ready in StdOut. The problem is/was, that this function always returns "no data there" -if called too soon after sending data to StdIn.

Notice, that at that time, I didn't know I'm reading StdOut (or using PeekNamedPipe) too fast after sending to StdIn -all I knew was, there's no data in StdOut.

And solution (finally):
1. send data to StdIn,
2. wait few milliseconds,
3. check if there's something in StdOut,
4. if there's nothing, repeat to step 2 few times
5. if there is something, read StdOut, else.. blame ExifTool 
-yes, I know... It's sounds simple now.

Bogdan

[Phil Harvey]

Hi Bogdan,

Thanks for the explanation.  I'm glad you understand things now.  I didn't think of this, but it does make sense as you said.  So you just have to use a wait/peek/read loop until you get the "{ready}" message back again.  Sounds good.

And yes, interprocess communication can be a bit tricky sometimes.

- Phil

[BogdanH]

I hope, some will find this thread informative -that's the main reason I'm writting this.

And yes, interprocess communication can be a bit tricky sometimes.

Indeed. And one can't avoid some time penalty. For example (simplistic):
Write(StdIn)
repeat
  Wait 20msec
  Peek(StdOut,BytesReady)
until BytesReady
Read(StdOut)
-Wait is used before Peek, because I know, StdOut isn't ready immediately. And one can't know exactly, how much waiting will be needed (20msec minimum is just a good guess). Now, if data isn't ready after first 20msec, another 20msec will pass, and another... And shortening Wait interval (say, to 5msec) seems to be of no benefit: I've noticed, that repeated Peek-ing in short time intervals, actually adds some time penalty. Ok, could be, my approach wasn't the best.
Too bad, there's no such thing like Peek(StdOut,ReturnOnBytesReady) API call.
Right now, I'm not sure how much speed can I gain by using -stay_open. I mean, Windows loads (cached) ExifTool quite fast...

One question: Is there any order how ExifTool is writting to StdOut and StdErr? For example, are first all errors (if any) written to StdError and then normal data to StdOutput? -that would mean StdError is ready for readout before StdOutput.

Bogdan

[Phil Harvey]

I would think there should be a way to block until input is available.  You can do this in all the languages I know.  In C and Perl, this is done with the select() function in the standard library.

The last thing that exiftool writes is "{ready}\n" to StdOutput.  So when you receive this you know that StdError must be complete for this command.

- Phil

[BogdanH]

Hi Phil,

You're right: such function exist. After reading some Win docs (again), it's usage (preparation to use it) is quite complicated and I'm affraid, I would need to spend too much additional (learning) time for experimenting. That is, I'll try to avoid to use that, until I haven't tried everything in my "classic" approach. And as it seems, I've been rewarded by insisting on this.

The magic word was:

The last thing that exiftool writes is "{ready}\n" to StdOutput.

Right now, I can get data from ExifTool really fast, because no "Wait/Peek" is used. So, I believe, it's time to sumarize this thread with:
I can confirm, that -stay_open mode via pipes works "as advertized" 

Thank you for hints and suggestions,
Bogdan

[Phil Harvey]

Hi Bogdan,

I'm glad it is working.  Just so I understand, the problem you were having was when using a file instead of a pipe for the argfile input?  If so, I definitely understand because saw a similar thing when reading the argfile from exiftool.  I didn't think about this because I thought you were using pipes.

For what it's worth, I'm attaching the Perl code that I use to read the argfile when the -stay_open option is used.  Here I am using select() for a simple delay, and not to wait on a filehandle as it can also be used to do.  With pipes, the Perl sysread() function will wait automatically until new data is available. But it is different for files, and always returns immediately for these (which is different than the problem you described), so I use a read/wait loop with a 1/100 second delay for file input.

- Phil

Code Select Expand

#------------------------------------------------------------------------------
# Read arguments from -stay_open argfile
# Inputs: 0) argument list ref
# Notes: blocks until -execute, -stay_open or -@ option is available
#        (or until there was an error reading from the file)
sub ReadStayOpen($)
{
    my $args = shift;
    my (@newArgs, $processArgs, $result);
    my $lastArg = '';
    my $unparsed = length $stayOpenBuff;
    for (;;) {
        if ($unparsed) {
            # parse data already read from argfile
            $result = $unparsed;
            undef $unparsed;
        } else {
            # read more data from argfile
            # - this read may block (which is good) if reading from a tty device
            $result = sysread(STAYOPEN, $stayOpenBuff, 65536, length($stayOpenBuff));
        }
        if ($result) {
            my $pos = 0;
            while ($stayOpenBuff =~ /\n/g) {
                my $len = pos($stayOpenBuff) - $pos;
                my $arg = substr($stayOpenBuff, $pos, $len);
                $pos += $len;
                $arg =~ s/(^\s+|\s+$)//g; # remove leading/trailing white space
                # remove white space before, and single space after '=', '+=', '-=' or '<='
                $arg =~ s/^([-\w]+)\s*([-+<]?=) ?/$1$2/;
                unless ($arg eq '' or $arg =~ /^#/) {
                    push @newArgs, $arg;
                    $arg = lc $arg;
                    # process args after -execute, -stay_open or -@ option
                    # (note: we really don't know for sure if these are the options
                    #  because we aren't fully processing the arguments so these may
                    #  just be arguments for other options, but this is OK because
                    #  the main code handles all of the parsing when we exit here)
                    if ($arg =~ /^-execute\d*$/ or $lastArg eq '-stay_open' or $lastArg eq '-@') {
                        $processArgs = 1;
                        last;   # process arguments up to this point
                    }
                    $lastArg = $arg;
                }
            }
            next unless $pos;   # nothing to do if we didn't read any arguments
            # keep unprocessed data in buffer
            $stayOpenBuff = substr($stayOpenBuff, $pos);
            if ($processArgs) {
                # process new arguments after -execute or -stay_open option
                unshift @$args, @newArgs;
                last;
            }
        } elsif ($result == 0) {
            # sysread() didn't block (ie. when reading from a file),
            # so wait for a short time (1/100 sec) then try again
            select(undef,undef,undef,0.01);
        } else {
            Warn "Error reading from ARGFILE\n";
            close STAYOPEN;
            $stayOpen = 0;
            last;
        }
    }
}


[BogdanH]

Hi Phil,

No, the whole thread I am/was working on having complete communtication with ExifTool via pipes only (without argfiles inbetween). That is, since beginning, using argfiles wasn't that usefull option for me.

And as it came out, the problems I had, were because I didn't know how to use pipes properly. I still can't say, I'm an expert on pipes usage, but I have achieved what I wanted.
Now I can use ExifTool in stay_open mode from my application (almost) the same way as if passing parameters in cmd-line mode (or right now, calling ExifTool from GUI). But response time is much faster -which is the whole point.

I'm sure example code you've posted will be usefull for some programmers, but speaking for me.. I'm just amazed when I see Perl code

Thank you for showing interest on what I do.

Bogdan

[jmccall]

Bogdan,
You say

Right now, I can get data from ExifTool really fast, because no "Wait/Peek" is used.

but I don't see/understand how you finally achieved that.  Could you please post (psuedo)code.
Thanks,
James