Help with speeding up ExifTool

[mball]

I am new to exiftool and trying to get the fastest response from within a c# application. I only want to extract the creation date and subject tags from a file. I have used the following code but I am sure there must be a way of optimizing this, I hope someone can help

This is the command line I am using "exiftool.exe -fast2 -G -t -m -q FILENAME" and I am using a class exiftoolwrapper to do the work

Then i use this code to get the tags, is there a better way of extracting just the XMP and Subject groups and if so would that increase speed

                    foreach (ExifTagItem tag in exiftool.AsEnumerable())
                    {
                        if (tag.group == "XMP" && tag.name == "Subject")
                            keytags = tag.value.Split(',');
                    }

[Phil Harvey]

The biggest gains (roughly 60x) would come if you can pre-launch exiftool.exe with the -stay_open option instead of executing exiftool for each file.

- Phil

[mball]

Yes I saw that in the documentation but I couldnt work out how to do that

[BogdanH]

Hi,
I don't know C at all, but maybe I can still give you a hint...
Now, not knowing how "exiftoolwrapper" actually works, I assume ExifTool is called for each file from which you wish to get metadata -I assume that, because otherwise you wouldn't mentioned slow speed. If I am correct on this, then in regard of speed, that's the worst scenario of using ExifTool from within your application.
It's hard to say what's the best solution for you, without knowing more about your application (and how important part ExifTool there is). But in general there are few ways to increase speed:

1. When ExifTool is called to return metadata, specify what tags values you need, i.e.:
ExifTool -Exif:CreateDate -Xmp:Subject FILENAME
-if tags aren't specified, ExifTool extracts all metadata inside each file (which takes some time).

2. Try not to call ExifTool for each file; specify files instead, i.e.:
ExifTool -Exif:CreateDate -Xmp:Subject FILENAME_1, FILENAME_2, ...
or group of files:
ExifTool -Exif:CreateDate -Xmp:Subject *.JPG
-that way, ExifTool is called only once (for all specified files), which is a big gain in speed.
Of course, you need to know how to catch metadata values of all specified files and how to process them in your app.

3. Try to use "args" file. It is a normal text file, which can contain all ExifTool options and all filenames you wish ExifTool to process. Once that file is prepared (within your app), you call ExifTool by:
ExifTool -@ MyArgs.args
Again however, you need to know how to read output given by ExifTool (I don know, maybe "exiftoolwrapper" can handle this).

4. If ExifTool is called from your app many times, then you shoud consider using ExifTool's -stay_open option. By using this option and if done properly, ExifTool will fly at speed of light -just think of this option as if ExifTool would be DLL (if you're using Windows).
There are two ways how to use -stay_open option:
A -from within args files. That's the easier way, once you figure out how to fiddle with args files.
B -you communicate with ExifTool "directly" via StdIn/StdOut pipes. That would require modifying your exiftoolwrapper I assume (or even rewrite it from scratch).
I didn't test, but I assume, speed difference between A and B is very small -anyway, I prefer B obtion.

Bogdan

[mball]

Thanks Bogdan I had dabbled with the -Exif flag but it didnt seem to help. My app builds a list of pictures from file, the pictures could be jpg but could also be a specific camera raw file, for this I use DCRAW to extract the file to a jpg but before I do that I extract the date and subject tags from the RAW file.

Its not exceedingly slow but I am sure it could be better, I will look into using the -stay_open flag and using an args file

[mball]

Thanks Bogdan I had dabbled with the -Exif flag but it didnt seem to help. My app builds a list of pictures from file, the pictures could be jpg but could also be a specific camera raw file, for this I use DCRAW to extract the file to a jpg but before I do that I extract the date and subject tags from the RAW file.

Its not exceedingly slow but I am sure it could be better, I will look into using the -stay_open flag and using an args file

Sorry, could you just give me a quick resume of how to use -stay_open, I am a bit confused as to how I can call exiftool for each of my pictures. At the moment I have a loop in my program that reads through the files one at a time and calls exiftool but presumably I wont be able to do that with -stay_open

[Phil Harvey]

Please let me know if I can clarify the exiftool documentation:

1) Execute "exiftool -stay_open True -@ ARGFILE", where ARGFILE is
            the name of an existing (possibly empty) argument file or "-" to
            pipe arguments from the standard input.

            2) Write exiftool command-line arguments to ARGFILE, one argument
            per line (see the -@ option for details).

            3) Write "-execute\n" to ARGFILE, where "\n" represents a newline
            sequence.  (Note: You may need to flush your write buffers here if
            using buffered output.)  Exiftool will then execute the command
            with the arguments received up to this point, send a "{ready}"
            message to stdout when done (unless the -q option is used), and
            continue trying to read arguments for the next command from
            ARGFILE.  To aid in command/response synchronization, any number
            appended to the "-execute" option is echoed in the "{ready}" mes-
            sage.  For example, "-execute613" results in "{ready613}".

            4) Repeat steps 2 and 3 for each command.

            5) Write "-stay_open\nFalse\n" to ARGFILE when done.  This will
            cause exiftool to process any remaining command-line arguments
            then exit normally.

- Phil

[mball]

Thanks, I had read that but I am still baffled about the arg file.
Do I create the arg file while looping through my pictures adding the args and filename to it then calling exiftool?

Sorry but not used to dealing with command line stuff

[Phil Harvey]

To take your specific example, and combine all of your common arguments into the initial command,  you could do this:

1) create an empty argument file /temp/my.args

2) execute this command from your application (in another thread so it stays running):

exiftool -stay_open True -@ /temp/my.args -common_args -fast2 -G -t -m -q

3) Write the following to /temp/my.args for each file you want processed:

Code Select Expand

FILENAME
-execute


4) Process the stdout from exiftool normally for each file up to the "{ready}" marker.

- Phil

P.S. Note that the -stay_open feature is only useful if you don't know the names of all files beforehand.  If you do know all of the names, then simply write them all to /temp/my.args before you start, and drop the  -stay_open and -execute options.

[mball]

Thanks for that, its much clearer now. I do know all the filenames beforehand so would I just create the args file with a filename on each line and then run exiftool with the arguments and point to the arg file, i.e.

exiftool -@ /temp/my.args -common_args -fast2 -G -t -m -q

[BogdanH]

Hi Phil,
Above documentation is good enough... I mean, -stay_open isn't easy to explain and it took a while before I "got it" 
How I started was, I ran Exiftool from console window. That's the best way to understand -stay_open, because results (and mistakes) can be seen immediately. Once I understood it, I only needed to transfer that knowledge into my app -and if results weren't as expected, I knew only my app was resposible for that.
If I recall corectly, it goes like this:

1. create empty txt file, for example MyArgs.txt and save it.
2. Open console window and execue:
exiftool -stay_open True -@ MyArgs.txt
..and nothing will happen.
It can't, because ExifTool was only loaded in memory and is waiting to execute some command, which is expected to be inside MyArgs.txt file.
3. While console window is open, open MyArgs.txt file and write inside:
-ver
-execute
..and re-save the file. At the momment you do this, ExifTool will execute commands in that file and result will be written into console window.... and ExifTool keeps waiting for next command(s) inside MyArgs file.
4. Open MyArgs.txt file again, delete all lines and add* (for example):
-Exif:CreateDate
MyPhoto.jpg
-execute
..and re-save file again. And again, you'll get requested result from ExifTool. Repeat that as long you wish: ExifTool is just waiting for your commands 
When you're done, open MyArgs file for the last time, delete all lines and add* inside:
-stay_open
False
..and re-save file. Now, ExifTool will exit "waiting state" and you'll get prompt in console window again.

That's it (I hope I didn't make any mistakes above). Now it's up to programmer to "transfer" all that into it's own application. That is, don't assume how -stay_open works -tryout in console first.

Bogdan
* -Corrected (see Phil's post below) to avoid confusion.

[mball]

Thanks thats is a great explanation, i will be looking into this over the next couple of days. I will let you know how I get on

[Phil Harvey]

Bogdan,

Just one correction: Don't delete the existing lines for each new command.  ExifTool continues reading the ARGFILE from the last location, so your new lines will get missed if you delete the earlier ones.

But it sounds like mball doesn't need the -stay_open option anyway.  In fact, he probably doesn't even need the -@ option if all of the file names will fit on a command line (there is a command-line length limitation in Windows).

- Phil

[BogdanH]

Thank you for correcting me, Phil -as I wrote: tryout in console window first -I realy should follow my own advice 

Bogdan

[mball]

I have got my app producing a list of file names in a text file called Exif.args

I now want to parse each file in the same loop that creates the entry in Exif.args
After opening the Exif.args file I call EXIFTOOL with "exiftool.exe Exif.args -common_args -fast2 -G -t -m -q -Exif:DateTimeOriginal -Xmp:Subject"
Then I loop through the file names and add each file name to the Exif.args file

What command do I then use to get the data for the current filename so I can parse it?

[Phil Harvey]

Your command should be:

exiftool -@ Exif.args -fast2 -G -t -m -q -Exif:DateTimeOriginal -Xmp:Subject

(you forgot the -@, and I forgot to mention that you don't need -common_args either if you're not using -execute.)

Then, parse the stdout from this command.  Lines beginning with "======== FILENAME\n" mark the start of the information from each file.

- Phil

[mball]

Thanks for that, its much clearer now. I do know all the filenames beforehand so would I just create the args file with a filename on each line and then run exiftool with the arguments and point to the arg file, i.e.

exiftool -@ /temp/my.args -common_args -fast2 -G -t -m -q

Phil
I am really struggling to get this working, I have created the arg file which contains all the pictures to process but my app just does not complete and just stays open. I am not sure if you can help on this because its probably not an exiftool issue but you may be able to point me in the right direction.

I am sure its the way I am calling exiftool, on debugging this there process is reporting errors

My code is:
            string sArguments = @" - @ Exif.args -fast2 -xmp:subject ";
                Process oP = new Process();
                oP.EnableRaisingEvents = false;
                oP.StartInfo.CreateNoWindow = true;
                oP.StartInfo.LoadUserProfile = false;
                oP.StartInfo.RedirectStandardError = false;
                oP.StartInfo.RedirectStandardOutput = true;
                oP.StartInfo.RedirectStandardInput = true;
                oP.StartInfo.StandardErrorEncoding = null;
                oP.StartInfo.StandardOutputEncoding = Encoding.UTF8;
                oP.StartInfo.UseShellExecute = false;
                oP.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
                oP.StartInfo.FileName = @"exiftool.exe";
                oP.StartInfo.Arguments = sArguments;
                /// Translate file names to 8.3
                var files8dot3 = files.Select(x => new { FileName = x, ShortName = GetShortPathName(x) }).ToArray();

                /// Pass all file names in an arg file which is piped to the process (no temporary file)
                string sArgfile = String.Join("\r\n", files8dot3.Select(x => x.ShortName).ToArray());
                FileStream fs = new FileStream(path+@"\Exif.args", FileMode.OpenOrCreate, FileAccess.Write);
           
                byte[] data = Encoding.UTF8.GetBytes(sArgfile);
                fs.Write(data, 0, data.Length);
                fs.Close();
                oP.Start();

                oP.WaitForExit();
                // this is where I would parse the output

[Phil Harvey]

I always suggest getting something simple working before trying something complicated.

Instead of your complicated command, see if you can get this working first:

exiftool -echo test

Exiftool should just print "test" and exit.

- Phil

[mball]

I have it working now thanks, just having problems parsing the results but getting there

[mball]

I have it working now thanks, just having problems parsing the results but getting there

How should I be parsing, I am calling the exiftool on a command process from windows program, then I loop through my files and add an entry into the args file, then try to parse that before moving on to next file. Its only getting the tags for the first file
Start Process
Loop
   Write filename to args file
   Wait for process to exit
   parse standardout
EndLoop

[Phil Harvey]

I thought you were writing all of the file names to the args file before calling exiftool -- processing all files in the single call:

Loop
  Write filename to args file
EndLoop
call exiftool
parse standardout

[mball]

My problem with that approach is how do I match my images with the parsed exif data.

[Phil Harvey]

As I mentioned before: Lines beginning with "======== FILENAME\n" mark the start of the information from each file.

- Phil

[mball]

Ok writing all the filenames to the arg file then calling the process result in the following

where oP is the process in c# using c# to parse the standardoutput stream
output = oP.StandardOutput.ReadLine();

The line for the first file is
XMP        Subject        Yorkshire Dales, Settle

The exiftool arguments are  -@ Exif.args -fast2 -G -t -m -q -Xmp:Subject

Using this method I can interate through the standardoutput stream line by line and get the above results but I cant match this to a filename

[Phil Harvey]

Looking back, I see you're using the -q option, which suppresses these informational messages.

- Phil

[mball]

I have been playing a bit and changed the output format to csv, this gives me the filename and the subject tags which is good but I am still having problems matching the filenames, not sure why this happens, perhaps you might know.

Arg file contains filename like this
E:\2008_04_12_Settle_Walk\2008 04 12_Settle Walk_0001.JPG

but the result from exiftool is
E:/2008_04_12_Settle_Walk/2008 04 12_Settle Walk_0001.JPG

The slashes are a different way round

[mball]

I have been playing a bit and changed the output format to csv, this gives me the filename and the subject tags which is good but I am still having problems matching the filenames, not sure why this happens, perhaps you might know.

Arg file contains filename like this
E:\2008_04_12_Settle_Walk\2008 04 12_Settle Walk_0001.JPG

but the result from exiftool is
E:/2008_04_12_Settle_Walk/2008 04 12_Settle Walk_0001.JPG

The slashes are a different way round

Ok I sorted it by doing a replace of / with \
I hope its all ok now, seems to be doing what I want. thanks all for your help

[Phil Harvey]

Yes.  ExifTool expects "/" as a directory separator, and will change "\" to "/" as you have discovered.

I wouldn't recommend -csv if you have a large number of files, since it buffers all information in memory before returning (this is mentioned in the -csv documentation and the performance tips).  Instead, look at -T, -j, -php or -X.

- Phil

[kemsky]

Hi,
is unicode supported in this mode?

[Phil Harvey]

Do you mean Unicode file names?  These are always a problem with Windows, but fine on any other system.

- Phil

[kemsky]

yep, unicode is really annoying problem on windows.

I'm going to use ExifTool to extract info from ebooks, but without unicode it is very limited solution.
i could rename file to avoid unicode input problem, but extracted info (stdout) is ansi anyway...

i really wish that some day there will be dll-wrapper (like imagemagick has) supporting unicode arguments.

as a workaround exiftool could output info to file in unicode format (or even pipes/named pipes see CreateNamedPipe)

[cuagn]

I'm not really sure I want to reply ... but I still have a question using stay_open

I've try the proposed test of BogdanH

1/ Exiftool version 9.20 installed on Windows7 Pro (French)

2/ Create a file MyExifToolArgs.txt EMPTY

3/ Start a command line windows and key in

Code Select Expand

Microsoft Windows [version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Tous droits réservés.

C:\Program Files (x86)\Exiftool>exiftool -stay_open true -@ D:\MyExifToolArgs.txt


4/ No answer. That's normal. I Chech existing processes

Code Select Expand

exiftool.exe*32 .....   860K   Read and Write meta information
exiftool.exe*32 ..... 13900K   exiftool.exe

Seems OK ...

5/ Open MyExifToolArgs.txt (using Nootepad++) and add 3 lines (to have 2 EOL)

Code Select Expand

-ver
-execute123


6/ In the command line windows I've now

Code Select Expand

Microsoft Windows [version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Tous droits réservés.

C:\Program Files (x86)\Exiftool>exiftool -stay_open true -@ D:\MyExifToolArgs.txt
9.20
{ready123}

which are the expected answers for the version information and the ready info.

However, I've a question :

How can I receive these answers in a given file or appended to the input (MyExifToolArgs.txt) file?

I'm stuck... 

Thank you for your help.

Marc

[cuagn]

Made another test with strange results...

1/ Start exiftool with outfile defined (MyExifToolResult.txt)

Code Select Expand

Microsoft Windows [version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Tous droits réservés.

C:\Program Files (x86)\Exiftool>exiftool -stay_open true -@ D:\MyExifToolArgs.txt > D:\MyExifToolResult.txt


2/ Then, add 3 lines in the MyExifToolArgs.txt EMPTY file

Code Select Expand

-ver
-execute001


3/ Open MyExifToolResult.txt

Code Select Expand

{ready001}
No version indicated ...   

4/ What about echo ? Appended in args file

Code Select Expand

-ver
-execute001
-echo HELLO
-execute002

5/ Still no result in output except the ready message

Code Select Expand

{ready001}
{ready002}

6/ However Windows command line has an output

Code Select Expand

Microsoft Windows [version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Tous droits réservés.

C:\Program Files (x86)\Exiftool>exiftool -stay_open true -@ D:\MyExifToolArgs.txt > D:\MyExifToolResult.txt
File not found: o HELLO


I know I make something wrong, but what ?

Regards

Marc

[Phil Harvey]

Hi Marc,

I just tried this under OS X, and it works fine.  Your echo command won't work as you wrote it though because each argument needs to be on a separate line, but with this fixed it works for me.  I don't understand why the -ver output didn't go to the redirected file.  This doesn't make sense to me.

- Phil

[cuagn]

Thank's for your answer.

May I make a specific test in order to give you some valuable information/feed back?

I'll try other tests this afternoon.

Thank you.

Marc

PS: in fact I want to invoke exiftool from a VB.NET application, but no being familiar with "pipe" under W7, I (try to) use a file based relation with exiftool.
A pipe guideline under .net will not solve the problem I've identified (at least in my environment), but will give me the required "knoledge" (by example).
Will go thru other topics...

[Phil Harvey]

I had a PC handy so I tried this myself.

Man.  Windows is weird.  Somehow some of the output is getting lost in the Windows pipe.

I believe that a simple redirection in Windows (ie. "> out.txt") will redirect both stdout and stderr.  Also, I think that Windows supports separate redirection of these as with a Unix Bourne shell (ie. "1> out.txt 2> err.txt"), so I tried this to redirect just stdout:

exiftool -stay_open true -@ a.args 1> out.txt

and it seems to work.  This indicates to me that there may be some sort of a bug in the Windows command shell where there is an interference that occurs when redirecting two streams to the same file.  But this is just a guess.

- Phil

[cuagn]

7 years ago I made an Home Automation based on Debian , with pipes (sorry queues...) and never had other difficulties than reading the documentation....

Unfortunately, my "family office" is under Windows and so are also the 14500 photos I would like to tag efficiently, thanks to exiftool.

I'll redo some tests and hope to find a bypass.

Thank you

Marc

[ryerman]

I won't pretend to know the intricacies of Windows re-direction, but there is good information here.

In summary:
1> redirects stdout    (> is merely an allowable short-form)
2> redirects stderr

To redirect to separate files use:
-exiftool some command line 1> out.txt 2> err.txt

To redirect to the same file use:
-exiftool some command line > both.txt 2>&1
or
-exiftool some command line 1> both.txt 2>&1
or
-exiftool some command line 2> both.txt 1>&2

Jim

[Phil Harvey]

Hi Jim,

Thanks for your input.  But now I'm even more confused because "1>" worked for me, while ">" didn't.  But I only ran a few tests, so maybe it is just inconsistent, however I hope this isn't the case.

- Phil

[ryerman]

....I know I make something wrong, but what ?...

Hi Marc.

After several hours of brute force trial & error , I can suggest this:
Every time you write to the arg file (even if it's empty), write one blank line first, before all the options.  This worked for me and I have no idea why.

Besides finally learning how -stay_open works, I also discovered this:

exiftool -stay_open true -@ Args.txt > Result.txt will append the information to the text file.
Without -stay_open, > will over-write and >> is used to append.  Both will create the file if necessary.

Why the differing actions concerning appending/over-writing?  Once again, I haven't a clue.

Jim

[Phil Harvey]

Hi Jim,

Yes, I noticed a similar effect...  Adding an extra option before -ver caused it to work.  The blank line is a good trick though.

But I'm sure you are fooling yourself with the append mode.  ">" will always overwrite, but if you don't clear your "Args.txt" before you re-run exiftool, then it will overwrite "Result.txt" with the same thing that it contained before, making it look like you are appending to the existing file when you aren't.  I tried this and verified that the output file was cleared when I used ">" with -stay_open (in Windows XP).

- Phil

[ryerman]

Hi Phil

Yes, I noticed a similar effect...  Adding an extra option before -ver caused it to work.  The blank line is a good trick though.

But I'm sure you are fooling yourself with the append mode.....

It wouldn't be the first time.

For reference: exiftool -stay_open true -@ "Args.txt" > "Result.txt"

I can't reproduce overwrite mode. When I delete the contents of the "Args.txt" and write another argument, ExifTool seems to freeze without accepting any new arguments.  I escape by killing the exiftool process.

How did you clear "Args.txt"?

However, if I write something in "Result.txt" beforehand, I can see that it is overwritten the first time I execute from "Args.txt".
But I want to try and make that happen after writing a 2nd or 3rd argument, so I do this:

1.  Execute some arguments from "Args.txt" and observe the output in "Result.txt".
2.  Change the arg file to "Newargs.txt" and delete the original "Args.txt".  Now, I assume Args.txt and all of its arguments are no longer involved.
3.  Execute an argument from "Newargs.txt".

The outcome is that "Result.txt" looks like it still contains the output produced when using "Args.txt", with the output produced with "Newargs.txt" appended.

Have I fooled myself again?

And btw, the necessity for a blank line or extra option mysteriously disappeared.

Jim

[Tarn]

exiftool -stay_open true -@ Args.txt > Result.txt will append the information to the text file.

Jim,
I'm a little confused. In this example you use only one greater than character ">". As of yet I don't understand -stay_open, or how to use -@, and arg files; but your example with the single ">" should overwrite Result.txt.

Without -stay_open, > will over-write and >> is used to append.  Both will create the file if necessary.

This makes sense.

In regards to the greater than character as a redirect, in Windows...
dir > Dir.txt
  1. Will create a file called "Dir.txt" and place in it the output that would have gone to the screen.
  2. Will overwrite a file called "Dir.txt" and place in it the output that would have gone to the screen.
dir >> Dir.txt
  1. Will add the output that would have gone to the screen to the end of the file "Dir.txt" (with no leading line).
  2. Will create a file called "Dir.txt" and place in it the output that would have gone to the screen; should the file not exist.

Maybe think of it this way, a single greater than sign will create a new file, fresh output, every time. Doubble greather than signs will add to the file. Unless it does not exist; then it will create it.

I'm not entirely sure that this was your question. But after reading this thread there does seem to be some confusion about how Windows redirects out put. So I thought I'd attempt to give a very simple clarification to you, or to anyone who might need, or want one.

For those of you who already knew and understood this, you can say to yourself something like: "I knew that", or "Tell me something I don't know", or even "Right, I thought that's how it worked."

Hope this helps someone.

P.S. At the risk of bursting people's bubbles here, I don't know everything about ExifTool. And, there is a possibility that I may have, completely, misunderstood the issue at hand. If so, feel free to enlighten me.

[Phil Harvey]

Tarn is correct.

There were 2 ways you could fool yourself.  The one I mentioned, or thinking that each -execute should clear the output file (which won't happen).

At the time you run exiftool, redirecting the output with ">" will start with a fresh output file.  The output accumulates in that file until exiftool terminates.

Similarly, the input will accumulate in args.txt.  You should not clear args.txt while ExifTool is reading from it.  If you do, then ExifTool won't read any more until the length of args.txt reaches the same size as before your cleared it.  But it doesn't make sense to do this.  If you want to clear args.txt you must first switch argument files (with -stay_open true -@ newfile.args [each argument on a separate line, of course]), or terminate the exiftool process (with -stay_open false).

- Phil

[Tarn]

So -stay_open keeps a file open. In other words to issue -stay_open on a file, my.args would prevent me from being able to delete that file with Del my.args. It simply opens the named file and keeps it that way until -stay_open is set to false, or until that process is terminated. Right?

[ryerman]

Hi Phil

......There were 2 ways you could fool yourself.  The one I mentioned, or thinking that each -execute should clear the output file (which won't happen).

At the time you run exiftool, redirecting the output with ">" will start with a fresh output file.  The output accumulates in that file until exiftool terminates........

You're correct; because of > in the command, I thought that -execute would overwrite the output file.  Silly me, considering the main goal of the -stay_open option is to avoid repeated calls to exiftool.  Now I realize that calling exiftool is when the append/overwrite choice is implemented.

But isn't "The output accumulates in that file until exiftool terminates." another way of saying that the output produced by each -execute is appended to the output file?  That's exactly what happens, even though there is > (not >>) on the command line.  It's just that ExifTool is doing the appending, not Windows.  Windows already created or wiped the output file.

If that's an accurate description, it explains my confusion.

Thanks for your help and patience.

Jim

[Phil Harvey]

So -stay_open keeps a file open. In other words to issue -stay_open on a file, my.args would prevent me from being able to delete that file with Del my.args. It simply opens the named file and keeps it that way until -stay_open is set to false, or until that process is terminated. Right?

Right.  Except the property of being able to delete an open file is system dependent.  Only Windows is susceptible to this deadlock.

But isn't "The output accumulates in that file until exiftool terminates." another way of saying that the output produced by each -execute is appended to the output file?

It is misleading to think of it this way because the file is overwritten, not appended, at the time when the command is run (the time when the output file is opened).  From a system point of view, there is nothing special about the ExifTool -execute option -- output continues as normal after this option.

- Phil

[ryerman]

Hi Phil.

True or False? "Exiftool writes to the output file after an -execute option, without destroying the existing contents."

Jim

[Phil Harvey]

Hi Jim,

This question doesn't make sense to me.  ExifTool prints output to the console.  Once something is printed to the console, it can not be un-printed.  ExifTool doesn't know that you are redirecting the console output to a file.  (This redirection is handled by the command shell, not by ExifTool.)

So the answer is true, because otherwise is not possible.

- Phil

[ryerman]

OK Phil

Let me try again.

True or False?  "Until the exiftool command terminates (eg -stay_open false), each time exiftool prints, the console output is redirected to the same output file without overwriting the existing contents."

Jim

[Phil Harvey]

Hi Jim,

Same question.  Same answer.

- Phil

[ryerman]

Hi Phil

To me, that is the definition of appending.  But appending doesn't make sense with the single >.

So how about this:
As you issue arguements using -stay_open, exiftool prints to the console. The console output continues to expand until the exiftool command is terminated.  Every time -execute is used, all the console output (not only that produced by the last -execute) is re-directed by the shell and overwrites the output file. But the old stuff is exactly the same so it looks like the new stuff was appended.

You said something similar earlier.  Have I finally understood?  I'm afraid you'll tell me that the shell has no knowledge of -execute

Jim

[Phil Harvey]

This is confusing, but it shouldn't be.  When you said "each time exiftool prints" I took it to mean "each time a -execute is issued".  But if you meant "each time the exiftool application is started", then the answer is false.  The output file is replaced when the application is started and ">" is used.

I am still not happy with your last definition.  -execute has nothing to do with what happens to the output.  So you are right.  The shell doesn't know about -execute.

- Phil

[ryerman]

I know the output file is created or overwritten when exiftool is started.  I meant as you assumed.  So I'll rephrase:

"Until the exiftool command terminates (eg -stay_open false), each time exiftool prints (and by this I mean each time an -execute is issued), the console output is redirected to the same output file without overwriting the existing contents."

If this is true, as you seem to say, why is this not called appending?

Also, when you say "-execute has nothing to do with what happens to the output", I assume you mean that once -execute causes the output to be printed to the console, it is redirected, or not, depending on the users command line and without exiftool being involved with the redirection in any way.

The output file changes when an -execute is issued.  Why?  I guess because the console received new output from exiftool and > means it should be redirected.
What is redirected?  All console output created since exiftool was first called or only the output after the previous -execute?  If it's the former, then overwriting is a possibility.  If it's the latter, then it must be appending, which seems impossible because > was used and not >>.

Here's the bottom line:  I am not picking up what you are laying down.  I'm probably out of my depth in this area.
So maybe I'll just leave you in peace to deal with ExifTool matters.

Once again, thanks for your time and patience,

Jim

[Phil Harvey]

"Until the exiftool command terminates (eg -stay_open false), each time exiftool prints (and by this I mean each time an -execute is issued), the console output is redirected to the same output file without overwriting the existing contents."

Correct.

If this is true, as you seem to say, why is this not called appending?

The term "append" is typically used to indicate that the original content of the file is preserved when the file is opened.  Here the shell opens the output file and overwrites the original content when exiftool is launched and ">" is used.  The output file stays open until exiftool terminates.  It isn't called appending if the output accumulates while the file is open.  This is the normal way that a file is written.  But perhaps this is just semantics.  I refuse to call this appending just because it has different connotations in programming.

Also, when you say "-execute has nothing to do with what happens to the output", I assume you mean that once -execute causes the output to be printed to the console, it is redirected, or not, depending on the users command line and without exiftool being involved with the redirection in any way.

The output is redirected even if you never use -execute.  If there is no output, then the output file will contain 0 bytes of data, but the shell still opens and creates the output file when you use ">".

The output file changes when an -execute is issued.  Why?  I guess because the console received new output from exiftool and > means it should be redirected.

Now you lost me again.  What you you mean by "changes"?  If you mean that the file size grows, then yes.  But the same output file stays open until the exiftool process terminates.

What is redirected?  All console output created since exiftool was first called

Yes.

- Phil

[cuagn]

Hi Phil and all,

I've been busy, but I'm back.

I redo the last test with that strange behaviour...

But I do it on my PC (W7Pro 64) after a reboot.
And the error I've indicated did not recurs.
Whatever the cause (the real one, either my mistake or a windows side effect) I must consider that it works.
Full stop.

Sorry for my alert.

Marc

[cuagn]

Hi Phil,

As it works I continue....

Out and err in the same file

Code Select Expand

Microsoft Windows [version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Tous droits réservés.

C:\Program Files (x86)\Exiftool>exiftool -stay_open true -@ D:\MyExifToolArgs.txt 1>D:\MyExifToolResult.txt 2>&1


Then a command

Code Select Expand

-iptc:all
-exif:make
-exif:Model
-ext jpg
-D
F:\Photos\2012
-execute008

With its associated result

Code Select Expand

......
======== F:/Photos/2012/20120731-125149.jpg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S
======== F:/Photos/2012/20120731-131314.jpg
  271 Make                            : Apple
  272 Camera Model Name               : Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20120803-202249.jpg
iPhone 4S
======== F:/Photos/2012/20120803-202249.jpg
======== F:/Photos/2012/20120814-124547.jpeg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S.......
======== F:/Photos/2012/20120923-182725.jpg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S
======== F:/Photos/2012/20120923-182751.jpg
  271 Make                            : Apple
  272 Camera Model Name              Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121007-201927.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121007-201943.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121007-202004.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121007-202025.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121015-090118.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121015-090132.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121015-090145.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121021-111111.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121021-111128.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-082855.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-082910.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-082924.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-082936.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-083052.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-083106.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-083118.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-083133.jpg
: iPhone 4S
======== F:/Photos/2012/20121007-201927.jpg
======== F:/Photos/2012/20121007-201943.jpg
======== F:/Photos/2012/20121007-202004.jpg
======== F:/Photos/2012/20121007-202025.jpg
======== F:/Photos/2012/20121015-090118.jpg
======== F:/Photos/2012/20121015-090132.jpg
======== F:/Photos/2012/20121015-090145.jpg
======== F:/Photos/2012/20121015-090446.jpg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S
........
    1 directories scanned
1077 image files read
{ready008}


That's fine, Iphone has some invalid tags. Assuming it's for Apple, not for Exiftool

However the two stream (StdErr and StdOut) are not synchronized.

I would have expected (first error)

Code Select Expand


======== F:/Photos/2012/20120731-131314.jpg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S
======== F:/Photos/2012/20120803-202249.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20120803-202249.jpg
======== F:/Photos/2012/20120814-124547.jpeg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S


instead of

Code Select Expand


======== F:/Photos/2012/20120731-131314.jpg
  271 Make                            : Apple
  272 Camera Model Name               : Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20120803-202249.jpg
iPhone 4S
======== F:/Photos/2012/20120803-202249.jpg
======== F:/Photos/2012/20120814-124547.jpeg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S


Error messages are just in the middle of the output of a clean file... 

Is Exiftool  a multithreaded application?
Any solution ?

Thanks,

Marc

[cuagn]

Hi Phil,

A question : how to stop ?

Theoretically

Code Select Expand

         5) Write "-stay_open\nFalse\n" to *ARGFILE* when done. This will
         cause exiftool to process any remaining command-line arguments then
         exit normally.

My command

Code Select Expand

-stay_open
false
-execute9999988888

The result

Code Select Expand

File not found: false
{ready9999988888}

Processes still active and CMD window not returning to prompt.

I'm lost ...
I've cancelled the processes directly. Now It's time to sleep (me and my PC)

Regards

Marc

[Phil Harvey]

Hi Marc,

The "-ext jpg" error is because "-ext" and "jpg" must be on separate lines in the argfile.

The warnings/errors go to stderr.  If this is the same file as stdout, the buffering on some systems may cause the outputs to be out of order in relationship to each other.

Your "-stay_open\nfalse\n" problem could be 2 things:  1) extra space somewhere in an argument, or 2) an old version of ExifTool.

- Phil

[cuagn]

Hi Phil,

Thank you for your advises.

1/The "-ext jpg" error is because "-ext" and "jpg" must be on separate lines in the argfile.
No more errors. That's OK.
However it's difficult to understand the relation between this wrong parameter for extension selection and the fact that over 1077 files analysed a little bit less than 30 have been with such warning message.  Curious and may hide something else.....
From a quick analysis done this morning, it seems that these 30 files where without any IPTC and also no Model and Make in EXIF....

2/The warnings/errors go to stderr.  If this is the same file as stdout, the buffering on some systems may cause the outputs to be out of order in relationship to each other.
Agreed. May be a flush at the end of each analysed file can solve this issue. With extra time cost.

3/Your "-stay_open\nfalse\n" problem could be 2 things:  1) extra space somewhere in an argument, or 2) an old version of ExifTool.
Agreed. It was an extra space issue. (I'm running version 9.20)

Seems that exiftool is sensible to extra space... From now I'll use only one space and only when required.

Remark : We do not receive an acknowledgement after a -stay_open false request.
Could it be possible to have a message like this

Code Select Expand

Exiftool will terminate soon...
{ready}

Marc

[Phil Harvey]

However it's difficult to understand the relation between this wrong parameter for extension selection and the fact that over 1077 files analysed a little bit less than 30 have been with such warning message.

If you request specific tags to be extracted, ExifTool will only return a warning if "Warning" was one of the tag you requested, or if none of the requested tags existed.

2/The warnings/errors go to stderr.  If this is the same file as stdout, the buffering on some systems may cause the outputs to be out of order in relationship to each other.
Agreed. May be a flush at the end of each analysed file can solve this issue. With extra time cost.

You can use -v0 to enable output flushing after each line, which will likely solve this problem, but may affect performance slightly as you mentioned.

3/Your "-stay_open\nfalse\n" problem could be 2 things:  1) extra space somewhere in an argument, or 2) an old version of ExifTool.
Agreed. It was an extra space issue. (I'm running version 9.20)

Seems that exiftool is sensible to extra space... From now I'll use only one space and only when required.

In the args file, all spaces are significant.  In this file, the newlines are the separator, not white space as on the command line.

Remark : We do not receive an acknowledgement after a -stay_open false request.
Could it be possible to have a message like this

Code Select Expand

Exiftool will terminate soon...
{ready}

You can add any terminating messages you want by using a -echo option after the -stay_open true -@ ARGFILE on the command line.

- Phil

[cuagn]

Magic! (or well analysed and delivered!)

Thank you

Marc

[jgro]

Phil (or anyone),

I'm running Windows 7 and using the command: exiftool -stay_open True -@ args.txt

Initially, my args.txt file is created, but blank.  As I append entries to the file, exiftool always seems to be behind. 

For example, when I add and save the following lines to args.txt, nothing happens:

test.JPG
-execute123


However, when I add another line to args.txt, exiftool processes the test.JPG file.  Weird!!  Can anyone explain why this is?  (Sorry for bringing up an old post, but thought it'd be better than creating a new one).

-jgro

[Phil Harvey]

It sounds like you aren't ending the last line with a newline character.

- Phil

[jgro]

Hmm, yeah it's definitely something weird with the newlines on Windows. Looks like it depends on how you write to the arguments file.  When I open the file with notepad++ or notepad, the output always seems to lag behind the latest addition to the file.  However, when I use a Python script to write to the arguments file, it works just fine! Go figure.

And of course it runs great in Linux.  Thanks for the tip, Phil.

-jgro

[Phil Harvey]

OK.  Now that sounds like a flushing problem.  For the -stay_open documentation:

            3) Write "-execute\n" to ARGFILE, where "\n" represents a newline
            sequence.  (Note: You may need to flush your write buffers here if
            using buffered output.)

- Phil

[martenzi]

Hi Phil,
Above documentation is good enough... I mean, -stay_open isn't easy to explain and it took a while before I "got it" 
How I started was, I ran Exiftool from console window. That's the best way to understand -stay_open, because results (and mistakes) can be seen immediately. Once I understood it, I only needed to transfer that knowledge into my app -and if results weren't as expected, I knew only my app was resposible for that.
If I recall corectly, it goes like this:

1. create empty txt file, for example MyArgs.txt and save it.
2. Open console window and execue:
exiftool -stay_open True -@ MyArgs.txt
..and nothing will happen.
It can't, because ExifTool was only loaded in memory and is waiting to execute some command, which is expected to be inside MyArgs.txt file.
3. While console window is open, open MyArgs.txt file and write inside:
-ver
-execute
..and re-save the file. At the momment you do this, ExifTool will execute commands in that file and result will be written into console window.... and ExifTool keeps waiting for next command(s) inside MyArgs file.
4. Open MyArgs.txt file again, delete all lines and add* (for example):
-Exif:CreateDate
MyPhoto.jpg
-execute
..and re-save file again. And again, you'll get requested result from ExifTool. Repeat that as long you wish: ExifTool is just waiting for your commands 
When you're done, open MyArgs file for the last time, delete all lines and add* inside:
-stay_open
False
..and re-save file. Now, ExifTool will exit "waiting state" and you'll get prompt in console window again.

I firmly think this practical guide should make the documentation as an example. I´m somewhere in between noob and power user (relating to cmd usage). I have some additional info to relay to other users and a problem I´m trying to debunk. I´m on High Sierra and generally using Sublime Text for editing argfiles. For some reason, saving in ST does not give any results to the console window. I tried "Textedit" but it does not allow me to save in plain "txt" files, it forces RTF. I looked briefly in ST setting/config and could not find anything that seemed related to some "automatic save" feature or "stay open" options, as I presumed ST had some feature that allowed me to automatically have a file open in ST and not save until verified. (I´m speculating here about this potential feature).

When I read Bogdans instructions I was led to believe that I can use the actual console window to write directly while stay open. This results in nothing. I now use "Atom" and the writing inside argfile.txt works and gives results in console window. Thus:

1. does bogdan mean that one can type commands directly into console (not ref to regular exiftool invokation in terminal) ? - I don´t understand where I would type out my arguments, if I choose not to use argfile.

2. I havent figured out why saving in ST does not imply direct save to argfile.txt and thus show results in console window.

[Phil Harvey]

Saving with a text editor will only work if it writes in append mode, which is definitely not guaranteed.

You can write in append mode from the Terminal like this:

echo "ARG" >> MyArgs.txt

- Phil

[Hubert]

I tried "Textedit" but it does not allow me to save in plain "txt" files, it forces RTF.

You can change Textedit's default format from RTF to Plain text in Preferences. TextWrangler (free) is a much more powerful option that also writes plain text files.

[cowwoc]

The biggest gains (roughly 60x) would come if you can pre-launch exiftool.exe with the -stay_open option instead of executing exiftool for each file.

- Phil

Hi Phil,

Out of curiosity, if one mixes -stay_open with binary output how do we identify the end of output after each execution? I've never managed to successfully switched back and forth between parsing a process' output as text and binary. Is there an easier way?

[cowwoc]

The biggest gains (roughly 60x) would come if you can pre-launch exiftool.exe with the -stay_open option instead of executing exiftool for each file.

- Phil

Hi Phil,

Out of curiosity, if one mixes -stay_open with binary output how do we identify the end of output after each execution? I've never managed to successfully switched back and forth between parsing a process' output as text and binary. Is there an easier way?

I got this working. ExifTool prints "{ready}\r\n" at the end of each command's output, regardless of whether it is outputting text or binary.

Instead of switching back and forth between text and binary mode parsing, I treat just the output as if it is always binary.

1. Convert the string "{ready}\r\n" into a byte[] once at the beginning of the program.
2. For every command invoked by the user, read the output as binary (even if it contains text).
3. Search the output for the end-of-stream byte sequence from step 1. Return all the bytes/text up until this point.

[Phil Harvey]

The output of exiftool should always be treated as binary.

- Phil