Capturing Exiftool StdOut from Powershell script (asynchroneous communication)

[PatE]

I needed to read some custom tags from a list of files, as fast as possible.
The best way seems to load Exiftool in memory with stay_open. But how to communicate with it ?
I wonder why noone had a description of how to do this.
It took me several hours to cobble it together, below is a working solution, I hope this can be useful for others.
(credits to several authors).

1. Define this function to start exiftool.exe resident in memory and allow capturing its StdOut:

function Start-Proc  {
     param (
             [string]$exe = $(Throw "An executable must be specified"),
             [string]$arguments,
             [switch]$hidden,
             [switch]$waitforexit,
             [switch]$redirectStdOut
             )   
     
     # Build Startinfo and set options according to parameters
     $startinfo = new-object System.Diagnostics.ProcessStartInfo
     $startinfo.FileName = $exe
     $startinfo.Arguments = $arguments
     if ($hidden){
         $startinfo.WindowStyle = "Hidden"
         $startinfo.CreateNoWindow = $true
     }
     if($redirectStdOut){
        $startinfo.RedirectStandardOutput=$true
        $startinfo.UseShellExecute=$false
     }
     $process = [System.Diagnostics.Process]::Start($startinfo)
     if ($waitforexit) {$process.WaitForExit()}
     $process 
}


2. pre-create empty argfile, invoke the function to startup exiftool residently, redirecting StdOut away from console

if (Test-Path exifargfile) { del exifargfile }           # delete any existing argfile
$null | Out-File exifargfile -Append -Encoding Ascii;    # and create an empty new one

$p = start-proc "exiftool.exe" "-stay_open True -@ exifargfile" -redirectStdOut -hidden      #start exiftool, make it resident monitoring exifargfile, capturing stdout


3. process your list of files in a loop.
I wanted to receive from exiftool the filename and a custom tag MyTag that had been written to my files previously.

# process files

foreach ($f in $files)
{
    # send exiftool the command to execute
    "-filename"        | Out-File exifargfile -Append -Encoding Ascii;         # return me the filename of the processed file
    "-xmp-dc:MyTag"    | Out-File exifargfile -Append -Encoding Ascii;         # return me the value of MyTag
    "$f"               | Out-File exifargfile -Append -Encoding Ascii;         # file to process
    "-execute`n"       | Out-File exifargfile -Append -Encoding Ascii;         # execute it

    # read exiftool response from stdout 
    # Readline() will wait for input if there is none (yet), effectively pausing script execution, giving exiftool time to process the command
    $resultFilename = $p.StandardOutput.Readline()      # this returns a string in the form : "Filename        : somefilename"
    $resultMyTag    = $p.StandardOutput.Readline()      # this returns a string in the form : "MyTag           : somevalue
    $dummy          = $p.StandardOutput.Readline()      # read exiftool's termination string "{ready}"
}

[Phil Harvey]

Thanks for the note.  But don't you have to create an empty exifargfile before step 2?  (and not delete it in step 3)

- Phil

[PatE]

Hello Phil, thanks for your reaction.

Yes, thanks for the remark, I corrected the code.
Note: the -append flag creates the file if it does not exist.

I may have left out another important bit: stop exiftool after processing, and make sure that no other instance is present in memory before starting processing. This to avoid the potential complication of having multiple instances in memory, all reading from the same argfile, which could produce unpredictable and unwanted results. Especially useful during the code-test-debug development cycle.
(Now that I think of it: maybe that's an idea to be exploited for implementing some kind of multithreading inside a Powershell script that has to manipulate 10.000's of files as in my case. I presume every exiftool.exe runs in it's own address space, has it's own argfile and stdout redirection and thus exif commands can be sent to several exiftool instances in parallel... interesting)

Before the body of my program, and before program exit, I stop any remaining exiftool processes as follows (there may be more elegant ways to accomplish this):
while ((get-process -erroraction silentlycontinue exiftool) -ne $null){
    stop-process -name exiftool -erroraction silentlycontinue
}

Initially I did try stopping exiftool in a controlled fashion by sending the command
"-stay_open`nFalse`n" | Out-File exifargfile -Append -Encoding Ascii;
but this not seem to work out consistently, hence the stop-process brute-force approach.

[Phil Harvey]

(Now that I think of it: maybe that's an idea to be exploited for implementing some kind of multithreading inside a Powershell script that has to manipulate 10.000's of files as in my case. I presume every exiftool.exe runs in it's own address space, has it's own argfile and stdout redirection and thus exif commands can be sent to several exiftool instances in parallel... interesting)

Yes, multiple exiftools can run simultaneously.  The C++ interface for ExifTool uses this technique.

Before the body of my program, and before program exit, I stop any remaining exiftool processes as follows (there may be more elegant ways to accomplish this):
while ((get-process -erroraction silentlycontinue exiftool) -ne $null){
    stop-process -name exiftool -erroraction silentlycontinue
}

Initially I did try stopping exiftool in a controlled fashion by sending the command
"-stay_open`nFalse`n" | Out-File exifargfile -Append -Encoding Ascii;
but this not seem to work out consistently, hence the stop-process brute-force approach.

This is worrisome.  ExifTool should exit with "-stay_open\nFalse\n".  If not, it isn't receiving commands properly.

Pulling the plug by killing the process has a chance of corrupting files that ExifTool is currently writing.

Is there any way that writes to the argfile could be buffered?  If so, be sure the buffer is flushed after your final write.

- Phil

[PatE]

I've checked again, exiftool is correctly terminating after  "-stay_open\nFalse\n". I was just not giving it enough time to react.


I have incorporated exiftool resident mode inside a larger program, it works real fluently to read tags from JPG's I need to process, rather slick actually.

But I have a problem writing custom tags to my files in stay_open mode, am at a total loss, would appreciate if you could offer your insight on this ?

When specified in a command prompt, this works OK:
exiftool -config c:\test\exifconfig.cfg -charset filename=Latin -xmp-dc:MyTag1="ABC" -xmp-dc:MyTag2="123" c:\test\test.jpg -overwrite_original

However, sending the same command piecewise to exiftool in stay_open mode does not write tags to my file.
This is the argfile that is being produced by my sequential writes to the file :
-config
c:\test\exifconfig.cfg
-charset
filename=Latin
-xmp-dc:MyTag1=ABC
-xmp-dc:MyTag1=123
c:\test\test.jpg
-overwrite_original
-execute
\n

When reading exiftool stdout response after writing these commands to the argfile, the only response I receive is {ready}, but the tags are not being written to the file.
Am I overlooking some subtle requirement ?
Can I debug exiftool in resident mode ? -v[NUM] does not seem to produce extra exiftool output o StdOut

Many thanks and greetings from Belgium-Europe,
Patrick

[Phil Harvey]

Hi Patrick,

However, sending the same command piecewise to exiftool in stay_open mode does not write tags to my file.
This is the argfile that is being produced by my sequential writes to the file :
-config
c:\test\exifconfig.cfg
...

From the documentation:

      -config CFGFILE
            Load specified configuration file instead of the default
            ".ExifTool_config".  If used, this option must come before all
            other arguments on the command line. [...]

So you can't use this option from within an argfile.

- Phil

[PatE]

Hello Phil,

thanks again.
I first tried specifying the custom config file as first option in the stay_open startup command, but it still did not work, tags were not being written :
   $p = start-proc "exiftool.exe" "-configfile c:\test\exifconfig.cfg -stay_open True -@ exifargfile" -redirectStdOut -hidden

Then I put the contents of my custom config file inside a file named .Exiftool_config file in the home directory, starting exiftool with the previous command line :
   $p = start-proc "exiftool.exe" "-stay_open True -@ exifargfile" -redirectStdOut -hidden
and then it started working, tags were written.

(note to others: in Windows, .Exiftool_config must be created by renaming it in command line. A filename with nothing in front of "." is not allowed in Explorer, as per the documentation)

So, should I conclude that it is not possible to have exiftool resident using stay_open, and at the same time specify a custom config file ? I would really like to keep the contents of my config file in a separate file, for the sake of portability and code isolation.

Patrick

[Phil Harvey]

Hi Patrick,

The problem must be that ExifTool is not getting the proper arguments.  Are you looking at the stderr output from exiftool?  Do you get a message saying "Config file not found"?

I suspect that the backslashes in your directory name may be the problem (does powershell see them as escape sequences maybe?).  Try using forward slashes instead.

- Phil

[PatE]

Hi Phil,

I'm confused, it is working now, with backslashes as originally.
Can't reconstruct why it didn't yesterday, unfortunately, lost a learning opportunity.

It also works with forward slashes as you suggested.
And when I experimented with escaped names like  \\\\server\\dir\\exiftool and `\`\server`\dir`\exiftool I saw indeed an exiftool "Config file not found" message coming in via StdErr (using $p.StandardError.ReadLine() )

My problems are solved now, thank you again for the direct feedback, a very valuable approach !

Patrick

[PatE]

Hello Phil,

I have another yet graver problem.
How to get filenames with special characters like é, à, ç etc. pass on to exiftool using argfile ?
I read a lot about it but can't seem to distill from it what I need to do, maybe you could give me a pointer.

If from Powershell I output a filename with accented characters using ASCII encoding to argfile, the accents get replaced by other characters. This is to be exspected during the translation Unicode (= Powershell internal encoding) -> ASCII. Exiftool processes the file OK, communicating the normal messages via StdOut, but of course returns me a distorted filename.

If from Powershell I output the entire argfile in Unicode, the special characters are conserved (verified with a text editor), but after the "execute\n", I get no reply from Exiftool, not on StdOut, not on StdErr.

If I output the options preceding the filename in ASCII, the filename in Unicode, and the "execute\n" back in ASCII, the argfile stays in ASCII format, but looks something like :
-charset
filename=Latin
-m
-filename
-xmp-dc:Bprocessed
-xmp-dc:Bquality
-xmp-dc:Boriginalsize
D : \ _ d i r \ s u b d i r \ f i l e n a m e w i t h l o s t a c c e n t e d c h a r a c t e r s

Again no respons from exiftool on StdOut nor StdErr.

Also tried outputting the argfile in UTF8 with option -charset filename=UTF8. This retains the accents OK in the passed argfile, but now Exiftool reponds  : "0 images read", "3 files could not be read", although only 1 file was specified in argfile. Tried finding out if exiftool was now treating the accented chars as separator, but was not able to confirm.

Thanks for any hands-on advice you could give about this,
Patrick

[Phil Harvey]

I think you may be able to get the filename in the proper format like this:

exiftool -filename -s3 DIR > out.txt

I can't test this in Windows right now, but I think that reading this back should work:

exiftool -charset filename=utf8 -@ out.txt

Sorry I don't have more time right now, but that may give you something to work with.

- Phil

[PatE]

Thanks, but in my scenario exiftool is resident (stay_open, and using argfile).

Per your suggestion, I added -charset filename=utf8 to the startup command:
exiftool -config configfile -charset filename=utf8 -stay_open True -@ exifargfile

and send the file processing commands to exiftool via argfile in UTF8, like:
...
"-m"                   | Out-File $exifargfile -Append -Encoding UTF8
"-filename"            | Out-File $exifargfile -Append -Encoding UTF8
"-xmp-dc:Bprocessed"   | Out-File $exifargfile -Append -Encoding UTF8
...

Argfile is in UTF8 format (confirmed via notepad.exe, Save As: shows UTF8) and contains:
-m
-filename
-xmp-dc:Bprocessed
-xmp-dc:Bquality
-xmp-dc:Boriginalsize
E:\_JPGTEST2\_dossier1\map_B1\B 000 foto's\Tijdelijke map foto's\20050203\DSCN0089.JPG
-execute

Responses from the resident exiftool via StdOut:
======== E:/_JPGTEST2/_dossier1/map_B1/B 000 foto's/Tijdelijke map foto's/20050203/DSCN0089.JPG
File Name                       : DSCN0089.JPG
    1 image files read
    1 files could not be read
{ready}


The file exists at the specified location.
Not sure what is happening here.
If I create argfile in ASCII format, I get the normal sequence of responses (filename, <xmp-dc tags if present>, {ready}).

It's nog really urgent, but I had already worked out a really ugly - maybe the ugliest ever - workaround along the lines of your suggestion for the accented filenames, by piping the filename variable via DOS to a text file and then type that >> argfile. Not only awkward but additional overhead, causing slow execution, reading exif tags now takes about twice as long as before :
$filename | set-content tmpfile              # in Powershell, this outputs a Unicode variable to an ANSI/ASCII text file, retaining most of the accented chars (not all)
&$exifhelperbatch tmpfile $exifargfile       # execute $exifhelperbatch, this is a cmd batch file that contains: type %1 >> exifargfile


Thanks & have a nice weekend before all,
Patrick

[Phil Harvey]

My point is that if you can get it to work by using exiftool to pipe the filename to a text file then at least you know it is possible. All you need to do is to use that same encoding when writing to your argfile.

- Phil

[jean]

Hello
I hope that it can help: Under Windows i always use WideCharToMultiByte(CP_UTF8,....) before passing a filename to exiftool and it always works fine
(stay_open but i don't use argfiles)
jean

[PatE]

Thank you Jean.
You seem to suggest that I better convert the strings myself, instead of relying on Powershell intrinsic conversions.
I like that idea, it's better to be in control of everything.

I've googled the function WideCharToMultiByte(CP_UTF8,....) it seems to be a C++ function, but I don't program in that language.
But do you maybe have any idea/instruction/tip how to call it from Powershell ?
And another question (maybe I should study Exiftools documentation more profoundly to understand) : how do you send your exiftool commands to exiftool if not using an argfile ? Maybe I could also apply this technique from Powershell, it could possibly eliminate the character/encoding problem entirely.