Capturing Exiftool StdOut from Powershell script (asynchroneous communication)

[PatE]

I needed to read some custom tags from a list of files, as fast as possible.
The best way seems to load Exiftool in memory with stay_open. But how to communicate with it ?
I wonder why noone had a description of how to do this.
It took me several hours to cobble it together, below is a working solution, I hope this can be useful for others.
(credits to several authors).

1. Define this function to start exiftool.exe resident in memory and allow capturing its StdOut:

function Start-Proc  {
     param (
             [string]$exe = $(Throw "An executable must be specified"),
             [string]$arguments,
             [switch]$hidden,
             [switch]$waitforexit,
             [switch]$redirectStdOut
             )   
     
     # Build Startinfo and set options according to parameters
     $startinfo = new-object System.Diagnostics.ProcessStartInfo
     $startinfo.FileName = $exe
     $startinfo.Arguments = $arguments
     if ($hidden){
         $startinfo.WindowStyle = "Hidden"
         $startinfo.CreateNoWindow = $true
     }
     if($redirectStdOut){
        $startinfo.RedirectStandardOutput=$true
        $startinfo.UseShellExecute=$false
     }
     $process = [System.Diagnostics.Process]::Start($startinfo)
     if ($waitforexit) {$process.WaitForExit()}
     $process 
}


2. pre-create empty argfile, invoke the function to startup exiftool residently, redirecting StdOut away from console

if (Test-Path exifargfile) { del exifargfile }           # delete any existing argfile
$null | Out-File exifargfile -Append -Encoding Ascii;    # and create an empty new one

$p = start-proc "exiftool.exe" "-stay_open True -@ exifargfile" -redirectStdOut -hidden      #start exiftool, make it resident monitoring exifargfile, capturing stdout


3. process your list of files in a loop.
I wanted to receive from exiftool the filename and a custom tag MyTag that had been written to my files previously.

# process files

foreach ($f in $files)
{
    # send exiftool the command to execute
    "-filename"        | Out-File exifargfile -Append -Encoding Ascii;         # return me the filename of the processed file
    "-xmp-dc:MyTag"    | Out-File exifargfile -Append -Encoding Ascii;         # return me the value of MyTag
    "$f"               | Out-File exifargfile -Append -Encoding Ascii;         # file to process
    "-execute`n"       | Out-File exifargfile -Append -Encoding Ascii;         # execute it

    # read exiftool response from stdout 
    # Readline() will wait for input if there is none (yet), effectively pausing script execution, giving exiftool time to process the command
    $resultFilename = $p.StandardOutput.Readline()      # this returns a string in the form : "Filename        : somefilename"
    $resultMyTag    = $p.StandardOutput.Readline()      # this returns a string in the form : "MyTag           : somevalue
    $dummy          = $p.StandardOutput.Readline()      # read exiftool's termination string "{ready}"
}

[Phil Harvey]

Thanks for the note.  But don't you have to create an empty exifargfile before step 2?  (and not delete it in step 3)

- Phil

[PatE]

Hello Phil, thanks for your reaction.

Yes, thanks for the remark, I corrected the code.
Note: the -append flag creates the file if it does not exist.

I may have left out another important bit: stop exiftool after processing, and make sure that no other instance is present in memory before starting processing. This to avoid the potential complication of having multiple instances in memory, all reading from the same argfile, which could produce unpredictable and unwanted results. Especially useful during the code-test-debug development cycle.
(Now that I think of it: maybe that's an idea to be exploited for implementing some kind of multithreading inside a Powershell script that has to manipulate 10.000's of files as in my case. I presume every exiftool.exe runs in it's own address space, has it's own argfile and stdout redirection and thus exif commands can be sent to several exiftool instances in parallel... interesting)

Before the body of my program, and before program exit, I stop any remaining exiftool processes as follows (there may be more elegant ways to accomplish this):
while ((get-process -erroraction silentlycontinue exiftool) -ne $null){
    stop-process -name exiftool -erroraction silentlycontinue
}

Initially I did try stopping exiftool in a controlled fashion by sending the command
"-stay_open`nFalse`n" | Out-File exifargfile -Append -Encoding Ascii;
but this not seem to work out consistently, hence the stop-process brute-force approach.

[Phil Harvey]

(Now that I think of it: maybe that's an idea to be exploited for implementing some kind of multithreading inside a Powershell script that has to manipulate 10.000's of files as in my case. I presume every exiftool.exe runs in it's own address space, has it's own argfile and stdout redirection and thus exif commands can be sent to several exiftool instances in parallel... interesting)

Yes, multiple exiftools can run simultaneously.  The C++ interface for ExifTool uses this technique.

Before the body of my program, and before program exit, I stop any remaining exiftool processes as follows (there may be more elegant ways to accomplish this):
while ((get-process -erroraction silentlycontinue exiftool) -ne $null){
    stop-process -name exiftool -erroraction silentlycontinue
}

Initially I did try stopping exiftool in a controlled fashion by sending the command
"-stay_open`nFalse`n" | Out-File exifargfile -Append -Encoding Ascii;
but this not seem to work out consistently, hence the stop-process brute-force approach.

This is worrisome.  ExifTool should exit with "-stay_open\nFalse\n".  If not, it isn't receiving commands properly.

Pulling the plug by killing the process has a chance of corrupting files that ExifTool is currently writing.

Is there any way that writes to the argfile could be buffered?  If so, be sure the buffer is flushed after your final write.

- Phil

[PatE]

I've checked again, exiftool is correctly terminating after  "-stay_open\nFalse\n". I was just not giving it enough time to react.


I have incorporated exiftool resident mode inside a larger program, it works real fluently to read tags from JPG's I need to process, rather slick actually.

But I have a problem writing custom tags to my files in stay_open mode, am at a total loss, would appreciate if you could offer your insight on this ?

When specified in a command prompt, this works OK:
exiftool -config c:\test\exifconfig.cfg -charset filename=Latin -xmp-dc:MyTag1="ABC" -xmp-dc:MyTag2="123" c:\test\test.jpg -overwrite_original

However, sending the same command piecewise to exiftool in stay_open mode does not write tags to my file.
This is the argfile that is being produced by my sequential writes to the file :
-config
c:\test\exifconfig.cfg
-charset
filename=Latin
-xmp-dc:MyTag1=ABC
-xmp-dc:MyTag1=123
c:\test\test.jpg
-overwrite_original
-execute
\n

When reading exiftool stdout response after writing these commands to the argfile, the only response I receive is {ready}, but the tags are not being written to the file.
Am I overlooking some subtle requirement ?
Can I debug exiftool in resident mode ? -v[NUM] does not seem to produce extra exiftool output o StdOut

Many thanks and greetings from Belgium-Europe,
Patrick

[Phil Harvey]

Hi Patrick,

However, sending the same command piecewise to exiftool in stay_open mode does not write tags to my file.
This is the argfile that is being produced by my sequential writes to the file :
-config
c:\test\exifconfig.cfg
...

From the documentation:

      -config CFGFILE
            Load specified configuration file instead of the default
            ".ExifTool_config".  If used, this option must come before all
            other arguments on the command line. [...]

So you can't use this option from within an argfile.

- Phil

[PatE]

Hello Phil,

thanks again.
I first tried specifying the custom config file as first option in the stay_open startup command, but it still did not work, tags were not being written :
   $p = start-proc "exiftool.exe" "-configfile c:\test\exifconfig.cfg -stay_open True -@ exifargfile" -redirectStdOut -hidden

Then I put the contents of my custom config file inside a file named .Exiftool_config file in the home directory, starting exiftool with the previous command line :
   $p = start-proc "exiftool.exe" "-stay_open True -@ exifargfile" -redirectStdOut -hidden
and then it started working, tags were written.

(note to others: in Windows, .Exiftool_config must be created by renaming it in command line. A filename with nothing in front of "." is not allowed in Explorer, as per the documentation)

So, should I conclude that it is not possible to have exiftool resident using stay_open, and at the same time specify a custom config file ? I would really like to keep the contents of my config file in a separate file, for the sake of portability and code isolation.

Patrick

[Phil Harvey]

Hi Patrick,

The problem must be that ExifTool is not getting the proper arguments.  Are you looking at the stderr output from exiftool?  Do you get a message saying "Config file not found"?

I suspect that the backslashes in your directory name may be the problem (does powershell see them as escape sequences maybe?).  Try using forward slashes instead.

- Phil

[PatE]

Hi Phil,

I'm confused, it is working now, with backslashes as originally.
Can't reconstruct why it didn't yesterday, unfortunately, lost a learning opportunity.

It also works with forward slashes as you suggested.
And when I experimented with escaped names like  \\\\server\\dir\\exiftool and `\`\server`\dir`\exiftool I saw indeed an exiftool "Config file not found" message coming in via StdErr (using $p.StandardError.ReadLine() )

My problems are solved now, thank you again for the direct feedback, a very valuable approach !

Patrick

[PatE]

Hello Phil,

I have another yet graver problem.
How to get filenames with special characters like é, à, ç etc. pass on to exiftool using argfile ?
I read a lot about it but can't seem to distill from it what I need to do, maybe you could give me a pointer.

If from Powershell I output a filename with accented characters using ASCII encoding to argfile, the accents get replaced by other characters. This is to be exspected during the translation Unicode (= Powershell internal encoding) -> ASCII. Exiftool processes the file OK, communicating the normal messages via StdOut, but of course returns me a distorted filename.

If from Powershell I output the entire argfile in Unicode, the special characters are conserved (verified with a text editor), but after the "execute\n", I get no reply from Exiftool, not on StdOut, not on StdErr.

If I output the options preceding the filename in ASCII, the filename in Unicode, and the "execute\n" back in ASCII, the argfile stays in ASCII format, but looks something like :
-charset
filename=Latin
-m
-filename
-xmp-dc:Bprocessed
-xmp-dc:Bquality
-xmp-dc:Boriginalsize
D : \ _ d i r \ s u b d i r \ f i l e n a m e w i t h l o s t a c c e n t e d c h a r a c t e r s

Again no respons from exiftool on StdOut nor StdErr.

Also tried outputting the argfile in UTF8 with option -charset filename=UTF8. This retains the accents OK in the passed argfile, but now Exiftool reponds  : "0 images read", "3 files could not be read", although only 1 file was specified in argfile. Tried finding out if exiftool was now treating the accented chars as separator, but was not able to confirm.

Thanks for any hands-on advice you could give about this,
Patrick

[Phil Harvey]

I think you may be able to get the filename in the proper format like this:

exiftool -filename -s3 DIR > out.txt

I can't test this in Windows right now, but I think that reading this back should work:

exiftool -charset filename=utf8 -@ out.txt

Sorry I don't have more time right now, but that may give you something to work with.

- Phil

[PatE]

Thanks, but in my scenario exiftool is resident (stay_open, and using argfile).

Per your suggestion, I added -charset filename=utf8 to the startup command:
exiftool -config configfile -charset filename=utf8 -stay_open True -@ exifargfile

and send the file processing commands to exiftool via argfile in UTF8, like:
...
"-m"                   | Out-File $exifargfile -Append -Encoding UTF8
"-filename"            | Out-File $exifargfile -Append -Encoding UTF8
"-xmp-dc:Bprocessed"   | Out-File $exifargfile -Append -Encoding UTF8
...

Argfile is in UTF8 format (confirmed via notepad.exe, Save As: shows UTF8) and contains:
-m
-filename
-xmp-dc:Bprocessed
-xmp-dc:Bquality
-xmp-dc:Boriginalsize
E:\_JPGTEST2\_dossier1\map_B1\B 000 foto's\Tijdelijke map foto's\20050203\DSCN0089.JPG
-execute

Responses from the resident exiftool via StdOut:
======== E:/_JPGTEST2/_dossier1/map_B1/B 000 foto's/Tijdelijke map foto's/20050203/DSCN0089.JPG
File Name                       : DSCN0089.JPG
    1 image files read
    1 files could not be read
{ready}


The file exists at the specified location.
Not sure what is happening here.
If I create argfile in ASCII format, I get the normal sequence of responses (filename, <xmp-dc tags if present>, {ready}).

It's nog really urgent, but I had already worked out a really ugly - maybe the ugliest ever - workaround along the lines of your suggestion for the accented filenames, by piping the filename variable via DOS to a text file and then type that >> argfile. Not only awkward but additional overhead, causing slow execution, reading exif tags now takes about twice as long as before :
$filename | set-content tmpfile              # in Powershell, this outputs a Unicode variable to an ANSI/ASCII text file, retaining most of the accented chars (not all)
&$exifhelperbatch tmpfile $exifargfile       # execute $exifhelperbatch, this is a cmd batch file that contains: type %1 >> exifargfile


Thanks & have a nice weekend before all,
Patrick

[Phil Harvey]

My point is that if you can get it to work by using exiftool to pipe the filename to a text file then at least you know it is possible. All you need to do is to use that same encoding when writing to your argfile.

- Phil

[jean]

Hello
I hope that it can help: Under Windows i always use WideCharToMultiByte(CP_UTF8,....) before passing a filename to exiftool and it always works fine
(stay_open but i don't use argfiles)
jean

[PatE]

Thank you Jean.
You seem to suggest that I better convert the strings myself, instead of relying on Powershell intrinsic conversions.
I like that idea, it's better to be in control of everything.

I've googled the function WideCharToMultiByte(CP_UTF8,....) it seems to be a C++ function, but I don't program in that language.
But do you maybe have any idea/instruction/tip how to call it from Powershell ?
And another question (maybe I should study Exiftools documentation more profoundly to understand) : how do you send your exiftool commands to exiftool if not using an argfile ? Maybe I could also apply this technique from Powershell, it could possibly eliminate the character/encoding problem entirely.

[jean]

> it seems to be a C++ function,
No, it's a SDK function, you can call it from basic, c, c++, pascal etc... and certainly Powershell
A link to use the SDK from Powershell:
https://deletethis.net/dave/?q=dllimport

[PatE]

Thanks, I'll look into it once I get the meat of my application running. (JPG resizer app, now testing on 80.000 files)

Noticing strange things during my test runs, seeing ASCII chars getting translated (by exiftool ?).
E.g. sending via ASCII argfile a filename:
     xxxxx©xxxx.jpg
The char © is ASCII 169. (http://www.ascii-code.com/)
But after exiftool processing and having it return me the filename with the -filename option via StdOut, the character has changed to code 174, which is ®. Ok, they look alike, but it's a different character.

Not sure where this translation happens.
I can verify upto the argfile, which shows in notepad.exe as type ANSI (=ASCII), and in which the filename still has the correct character ©.
What happens internally within exiftool, or during its communication to StdOut, and afterwards how the exiftool response gets processed as it presumably passes through layers and layers of SDK, .NET etc. processing before it reaches my Powershell variable, I have no clue.

As a workaround I am now just filtering out filenames containing ASCII codes >127, effectively barring files containing quite common accented characters as é, à, etc. from processing. Those files still make up only about 0,01 % of all files to be processed, and my client doesn't care, but it's a loose end nevertheless.

[Phil Harvey]

You can use the -charset filename option to specify the character set you use for file names.  The -charset exiftool (or just -charset) option specifies the character set used by ExifTool for extracted values (including the FileName value).  If -charset filename is not specified, then the file names specified on the command line are not translated.

See this section of the application documentation for some pointers.

- Phil

[PatE]

Thanks Phil, I was initially using -charset filename=Latin when I started the topic, after a quick read of your extensive help text, but don't remember the logic I followed in selecting that option.
But at that time I was still concentrated on getting exiftool stay_open to work from Powershell.
Only after that was solved, I started concentrating on the character conversion problems.
Those appeared while using the charset option, so I removed it somewhere along the way, to eliminate possible reasons for the unwanted conversions, there are several levels involved. I'll try to reengineer my decision to use the charset Latin. Am now looking at how to read StdOut in different encodings, may be onto something.
Patrick

[PatE]

Tried to create a Streamreader onto the exiftool process object $p, like:
    $ASCIIstream = new-object system.IO.Streamreader($p.standardoutput.basestream,[System.Text.Encoding]::ASCII)
    $s=$asciistream.readline()

(tip from http://stackoverflow.com/questions/2855675/process-standardinput-encoding-problem)
This seems to be the way to force StdOut, StdIn etc. into a certain encoding.

However, the exiftool response on the option -filename still contains any accented characters converted into "?" or something else.
Checked argfile using notepad.exe before sending execute \n, always has the intended encoding.
Tried charset filename=Latin, and charset filename=UTF8
both :
- writing to argfile with UTF8 encoding -> getting the same 3 responses as earlier although argfile looks OK and is in the intended format :
  0 files read
  1 file could not be read
  {ready}
- writing to argfile in Unicode -> argfile looks OK and is verifiable Unicode, but exiftool does not output anything since my program hangs on the readline() from StdOut call.
(the reason for trying the charset UTF8 + argfile encoding was the following documentation excerpt, which in my lacking knowledge on the matter I may be interpreting too much on face value :
"...character set, preferably UTF8 (see the -charset option for a complete list). Setting this triggers the use of Windows wide-character i/o routines, thus providing support for all Unicode file names"

After further empirical multi-case input-output testing, I can confirm that exiftool does process the files with accented characters OK with the following settings:
- writing argfile in ASCII encoding
- charset filename=Latin (although I'm not sure if this has any effect)
- accented filenames appended to argfile via cmd prompt to retain correct Unicode->ASCII conversion.
So passing the accented filenames to exiftool is OK.

However, it is the response to the -filename option, "Filename      : xxxx", that does not pass OK from exiftool to Powershell. Accented characters get converted.
Given Phil's thorough approach, I assume this does not happen internally in exiftool but outside, in the OS/SDK/.NET etc. layers through which the result string passes from exiftool to Powershell.
Probably developers with a deeper understanding of these layers and encoding could find ways to investigate this.
I tried using the Streamreader approach, this did also return converted characters, but given my lack of experience with the system level routines involved, I will not draw the conclusion that it is exiftool sending converted characters down its StdOut.

I'm going to content myself with writing a string compare function that ignores any mismatches between input and output filename on the positions where the input filename contains an accented char.

Thanks everyone for your input.
Patrick

[PatE]

Hello Phil,

my Powershell JPG downsampling application is aimed at processing +/- 300.000 - 400.000 JPGs in one run.
For every JPG, it sends commands to exiftool (stay_open mode) via ASCII argfile.
The following command strings are appended to argfile for every file in the processing loop:
-charset
filename=Latin
-m
-filename
-xmp-dc:MyTag1
-xmp-dc:MyTag2
-xmp-dc:MyTag3
c:\test\test\testfile0000001.jpg
-execute

I am noticing in several test runs that my app hangs after +/- 84.000 files - 3:30 hours processing time.
Difficult to debug, given that it happens only after a long time.

Just to be sure: is there any upper limit on argfile filesize ? Is there any risk of exiftool internal buffer/memory overrun after processing that many commands ?
Argfile filesize is about 22KB when app hangs.

Should I better clear argfile after each command i.e. truncate to 0 bytes ?
(not sure how to do it, assuming exiftool needs the file handle to remain valid, so I guess I cannot just delete argfile and recreate 0 size)
I could also stop exiftool with a "-stay_open\nFalse\n" every say 10.000 files, and restart it with a fresh argfile.
Your thoughts ?

Thanks,
Patrick

[PatE]

Note: my assumption is that Powershell is hanging while waiting for something to appear on StdOut from exiftool, since it immediately stops/becomes responding again when I manually kill exiftool.exe.
To catch this I tried to make my reading from StdOut safer by first looking if there is anything available in the StdOut buffer, using $p.standardoutput.peek(), before actually reading it with $p.standardoutput.readline() .
However, standardoutput.peek() does not seem to work correctly, as it many times returns -1 (=buffer empty) even when there is something available in the buffer.
So am going for an exiftool stay_open restart every n files.

[PatE]

Confirmed after logging to file: Powershell app hangs on standardoutput.readline() waiting for exiftool response, now after processing 92.000 files.
In previous tests it hung at +/- 84.000.
The total input filelist is 313.000 files that I want exiftool to read.
There are some 40-50 filenames with special characters é, à, ï and more exotic that do no make it intact to exiftool.
Exiftool handles those gracefully, responding just "{ready}" on StdOut, and maybe also something on StdErr, but I'm not reading that out.

I have now manually renamed some of the accented filenames in the first 90.000 to a "valid" filename without accents.
I am now witnessing the app advancing further into the inputlist, now hangs at about 94.000.
This is after 29 filenames have been rejected by exiftool.
Will now manually correct 10 more filenames in the preceding 90.000, to verify if the app still makes it further into the list without hanging.

Forgot to mention that I am restarting exiftool every 20.000 files, so probably is not exiftool related after all (cumulative memory probs etc.)

While (command-line) PS script still hanging, tried to write a "-stay_open`nFalse`n" to argfile from another PS script, see if exiftool exits gracefully.
No response from exiftool, process stays in memory. As soon as I kill the process, PS script terminates with an error about NULL processpointer in readline() read, which is to be exspected.

A deep dive using Process Explorer and Procmon seems inevitable.

[Hayo Baan]

From your last reports it looks as if special characters in the filenames are at least part of the cause. You specify Latin as character set for the filenames, but that only holds a limited number of special characters and requires the input to be in Latin too. I'm quite sure that isn't the case on the windows command line (and certainly does not allow all special characters), so this is probably an area to have a look at. Best would be to be able to give exiftool UTF8 input, but that's not always possible on Windows (though I think with power shell this is possible).

[PatE]

Hello Hayo, thanks for chiming in. To be honest, I cannot really get my head around the encoding stuff. Have been raised on assembler and C in the '80s-'90s and then there was only ASCII, have not been programming at systems level for at least 20 yrs.

I would very much like to solve this, but as I explained earlier, when passing charset filename=UTF8 to exiftool, and writing the argfile also in UTF8 (am I correct that I need to write the argfile in the chosen charset encoding ? Or should I always write the argfile in ASCII, but use charset filename=UTF8 ?), I get the unexspected responses from exiftool (with any filename, not just ones containing special characters) :
  0 files read
  1 file could not be read
  {ready}

Have now changed my function that calls exiftool to not send the "execute\n" to the argfile, so that exiftool only reads the commands in the growing argfile but never executes them. My Powershell script has now been running happily all night and has processed 274.000 of 313.000 input files... much more than the 80.000 previously, so the problem definitely happens between sending "execute\n" to the argfile and exiftools response.
So a simplified code excerpt:
...
"-xmp-dc:MyTag1"     | Out-File $exifargfile -Append -Encoding ASCII    # ask exiftool to read the tags MyTag1 and MyTag2 from JPG file
"-xmp-dc:MyTag2"    | Out-File $exifargfile -Append -Encoding ASCII
$fullname               | Out-File $exifargfile -Append -Encoding ASCII;   # file to be read
"-execute`n"            | Out-File $exifargfile -Append -Encoding ASCII;   # do it
$s = $p.StandardOutput.Readline()    <----- here my application hangs, waiting for exiftool response
...

[Hayo Baan]

I'm not familiar with PowerSHell (Unix/Mac user myself), but I'm sure the ASCII encoding won't be good enough, if you can UTF-8 really is preferred.

But my suggested approach would be to forgo executing anything in exiftool to begin with. So only use powershell to write the arg file as a first step. Then have a good look at that file and look for anything that's off (especially file names). Actually from your experiment you should already have a good idea what file is causing the issue so have a look at how it's encoded in the arg file.

As a side note, I think you could/should make use of the -common_args option, that saves you having to pass the tag names each time in the arg file. This also allows you to easily reuse the same arg file and change the things you want exiftool to do. E.g., have it print only  the filename, etc.

[PatE]

Reduced my input filename list to only +/- 8000 names with accents.
On this input exiftool tripped up systematically after processing only 16 filenames.

Response on each "bad" filename was {ready} instead of the processed filename requested by my -filename.
My application until now simply decided: if the -filename response is not equal to the filename passed, something must have gone wrong during exiftool processing (probably file not found due to the accented chars getting converted).
My code never read out exiftools StdErr buffer.

Now I added code to read StdErr buffer (consume until empty) after each non-match between filename and exiftool response. Now exiftool does not trip up anymore, my application continues processing the 8000 accented filenames OK.

Conclusion: exiftool StdErr buffer overrun if its StdErr if it's not emptied by the calling app ?
Input filepaths were 234 chars,it tripped after the 16th file, meaning +/- 3.744 bytes in StdErr buffer.

@Phil: is this a plausible limitation ?

[Phil Harvey]

Sorry for the delay in responding.  I don't know much about the buffer limitations on Windows, so I can't say whether 3kB is plausible, but it wouldn't surprise me if it was.

- Phil