Help with speeding up ExifTool

[ryerman]

Hi Phil

......There were 2 ways you could fool yourself.  The one I mentioned, or thinking that each -execute should clear the output file (which won't happen).

At the time you run exiftool, redirecting the output with ">" will start with a fresh output file.  The output accumulates in that file until exiftool terminates........

You're correct; because of > in the command, I thought that -execute would overwrite the output file.  Silly me, considering the main goal of the -stay_open option is to avoid repeated calls to exiftool.  Now I realize that calling exiftool is when the append/overwrite choice is implemented.

But isn't "The output accumulates in that file until exiftool terminates." another way of saying that the output produced by each -execute is appended to the output file?  That's exactly what happens, even though there is > (not >>) on the command line.  It's just that ExifTool is doing the appending, not Windows.  Windows already created or wiped the output file.

If that's an accurate description, it explains my confusion.

Thanks for your help and patience.

Jim

[Phil Harvey]

So -stay_open keeps a file open. In other words to issue -stay_open on a file, my.args would prevent me from being able to delete that file with Del my.args. It simply opens the named file and keeps it that way until -stay_open is set to false, or until that process is terminated. Right?

Right.  Except the property of being able to delete an open file is system dependent.  Only Windows is susceptible to this deadlock.

But isn't "The output accumulates in that file until exiftool terminates." another way of saying that the output produced by each -execute is appended to the output file?

It is misleading to think of it this way because the file is overwritten, not appended, at the time when the command is run (the time when the output file is opened).  From a system point of view, there is nothing special about the ExifTool -execute option -- output continues as normal after this option.

- Phil

[ryerman]

Hi Phil.

True or False? "Exiftool writes to the output file after an -execute option, without destroying the existing contents."

Jim

[Phil Harvey]

Hi Jim,

This question doesn't make sense to me.  ExifTool prints output to the console.  Once something is printed to the console, it can not be un-printed.  ExifTool doesn't know that you are redirecting the console output to a file.  (This redirection is handled by the command shell, not by ExifTool.)

So the answer is true, because otherwise is not possible.

- Phil

[ryerman]

OK Phil

Let me try again.

True or False?  "Until the exiftool command terminates (eg -stay_open false), each time exiftool prints, the console output is redirected to the same output file without overwriting the existing contents."

Jim

[Phil Harvey]

Hi Jim,

Same question.  Same answer.

- Phil

[ryerman]

Hi Phil

To me, that is the definition of appending.  But appending doesn't make sense with the single >.

So how about this:
As you issue arguements using -stay_open, exiftool prints to the console. The console output continues to expand until the exiftool command is terminated.  Every time -execute is used, all the console output (not only that produced by the last -execute) is re-directed by the shell and overwrites the output file. But the old stuff is exactly the same so it looks like the new stuff was appended.

You said something similar earlier.  Have I finally understood?  I'm afraid you'll tell me that the shell has no knowledge of -execute

Jim

[Phil Harvey]

This is confusing, but it shouldn't be.  When you said "each time exiftool prints" I took it to mean "each time a -execute is issued".  But if you meant "each time the exiftool application is started", then the answer is false.  The output file is replaced when the application is started and ">" is used.

I am still not happy with your last definition.  -execute has nothing to do with what happens to the output.  So you are right.  The shell doesn't know about -execute.

- Phil

[ryerman]

I know the output file is created or overwritten when exiftool is started.  I meant as you assumed.  So I'll rephrase:

"Until the exiftool command terminates (eg -stay_open false), each time exiftool prints (and by this I mean each time an -execute is issued), the console output is redirected to the same output file without overwriting the existing contents."

If this is true, as you seem to say, why is this not called appending?

Also, when you say "-execute has nothing to do with what happens to the output", I assume you mean that once -execute causes the output to be printed to the console, it is redirected, or not, depending on the users command line and without exiftool being involved with the redirection in any way.

The output file changes when an -execute is issued.  Why?  I guess because the console received new output from exiftool and > means it should be redirected.
What is redirected?  All console output created since exiftool was first called or only the output after the previous -execute?  If it's the former, then overwriting is a possibility.  If it's the latter, then it must be appending, which seems impossible because > was used and not >>.

Here's the bottom line:  I am not picking up what you are laying down.  I'm probably out of my depth in this area.
So maybe I'll just leave you in peace to deal with ExifTool matters.

Once again, thanks for your time and patience,

Jim

[Phil Harvey]

"Until the exiftool command terminates (eg -stay_open false), each time exiftool prints (and by this I mean each time an -execute is issued), the console output is redirected to the same output file without overwriting the existing contents."

Correct.

If this is true, as you seem to say, why is this not called appending?

The term "append" is typically used to indicate that the original content of the file is preserved when the file is opened.  Here the shell opens the output file and overwrites the original content when exiftool is launched and ">" is used.  The output file stays open until exiftool terminates.  It isn't called appending if the output accumulates while the file is open.  This is the normal way that a file is written.  But perhaps this is just semantics.  I refuse to call this appending just because it has different connotations in programming.

Also, when you say "-execute has nothing to do with what happens to the output", I assume you mean that once -execute causes the output to be printed to the console, it is redirected, or not, depending on the users command line and without exiftool being involved with the redirection in any way.

The output is redirected even if you never use -execute.  If there is no output, then the output file will contain 0 bytes of data, but the shell still opens and creates the output file when you use ">".

The output file changes when an -execute is issued.  Why?  I guess because the console received new output from exiftool and > means it should be redirected.

Now you lost me again.  What you you mean by "changes"?  If you mean that the file size grows, then yes.  But the same output file stays open until the exiftool process terminates.

What is redirected?  All console output created since exiftool was first called

Yes.

- Phil

[cuagn]

Hi Phil and all,

I've been busy, but I'm back.

I redo the last test with that strange behaviour...

But I do it on my PC (W7Pro 64) after a reboot.
And the error I've indicated did not recurs.
Whatever the cause (the real one, either my mistake or a windows side effect) I must consider that it works.
Full stop.

Sorry for my alert.

Marc

[cuagn]

Hi Phil,

As it works I continue....

Out and err in the same file

Code Select Expand

Microsoft Windows [version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Tous droits réservés.

C:\Program Files (x86)\Exiftool>exiftool -stay_open true -@ D:\MyExifToolArgs.txt 1>D:\MyExifToolResult.txt 2>&1


Then a command

Code Select Expand

-iptc:all
-exif:make
-exif:Model
-ext jpg
-D
F:\Photos\2012
-execute008

With its associated result

Code Select Expand

......
======== F:/Photos/2012/20120731-125149.jpg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S
======== F:/Photos/2012/20120731-131314.jpg
  271 Make                            : Apple
  272 Camera Model Name               : Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20120803-202249.jpg
iPhone 4S
======== F:/Photos/2012/20120803-202249.jpg
======== F:/Photos/2012/20120814-124547.jpeg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S.......
======== F:/Photos/2012/20120923-182725.jpg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S
======== F:/Photos/2012/20120923-182751.jpg
  271 Make                            : Apple
  272 Camera Model Name              Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121007-201927.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121007-201943.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121007-202004.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121007-202025.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121015-090118.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121015-090132.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121015-090145.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121021-111111.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121021-111128.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-082855.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-082910.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-082924.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-082936.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-083052.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-083106.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-083118.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20121023-083133.jpg
: iPhone 4S
======== F:/Photos/2012/20121007-201927.jpg
======== F:/Photos/2012/20121007-201943.jpg
======== F:/Photos/2012/20121007-202004.jpg
======== F:/Photos/2012/20121007-202025.jpg
======== F:/Photos/2012/20121015-090118.jpg
======== F:/Photos/2012/20121015-090132.jpg
======== F:/Photos/2012/20121015-090145.jpg
======== F:/Photos/2012/20121015-090446.jpg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S
........
    1 directories scanned
1077 image files read
{ready008}


That's fine, Iphone has some invalid tags. Assuming it's for Apple, not for Exiftool

However the two stream (StdErr and StdOut) are not synchronized.

I would have expected (first error)

Code Select Expand


======== F:/Photos/2012/20120731-131314.jpg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S
======== F:/Photos/2012/20120803-202249.jpg
Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20120803-202249.jpg
======== F:/Photos/2012/20120814-124547.jpeg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S


instead of

Code Select Expand


======== F:/Photos/2012/20120731-131314.jpg
  271 Make                            : Apple
  272 Camera Model Name               : Warning: Invalid tag name 'ext jpg' - F:/Photos/2012/20120803-202249.jpg
iPhone 4S
======== F:/Photos/2012/20120803-202249.jpg
======== F:/Photos/2012/20120814-124547.jpeg
  271 Make                            : Apple
  272 Camera Model Name               : iPhone 4S


Error messages are just in the middle of the output of a clean file... 

Is Exiftool  a multithreaded application?
Any solution ?

Thanks,

Marc

[cuagn]

Hi Phil,

A question : how to stop ?

Theoretically

Code Select Expand

         5) Write "-stay_open\nFalse\n" to *ARGFILE* when done. This will
         cause exiftool to process any remaining command-line arguments then
         exit normally.

My command

Code Select Expand

-stay_open
false
-execute9999988888

The result

Code Select Expand

File not found: false
{ready9999988888}

Processes still active and CMD window not returning to prompt.

I'm lost ...
I've cancelled the processes directly. Now It's time to sleep (me and my PC)

Regards

Marc

[Phil Harvey]

Hi Marc,

The "-ext jpg" error is because "-ext" and "jpg" must be on separate lines in the argfile.

The warnings/errors go to stderr.  If this is the same file as stdout, the buffering on some systems may cause the outputs to be out of order in relationship to each other.

Your "-stay_open\nfalse\n" problem could be 2 things:  1) extra space somewhere in an argument, or 2) an old version of ExifTool.

- Phil

[cuagn]

Hi Phil,

Thank you for your advises.

1/The "-ext jpg" error is because "-ext" and "jpg" must be on separate lines in the argfile.
No more errors. That's OK.
However it's difficult to understand the relation between this wrong parameter for extension selection and the fact that over 1077 files analysed a little bit less than 30 have been with such warning message.  Curious and may hide something else.....
From a quick analysis done this morning, it seems that these 30 files where without any IPTC and also no Model and Make in EXIF....

2/The warnings/errors go to stderr.  If this is the same file as stdout, the buffering on some systems may cause the outputs to be out of order in relationship to each other.
Agreed. May be a flush at the end of each analysed file can solve this issue. With extra time cost.

3/Your "-stay_open\nfalse\n" problem could be 2 things:  1) extra space somewhere in an argument, or 2) an old version of ExifTool.
Agreed. It was an extra space issue. (I'm running version 9.20)

Seems that exiftool is sensible to extra space... From now I'll use only one space and only when required.

Remark : We do not receive an acknowledgement after a -stay_open false request.
Could it be possible to have a message like this

Code Select Expand

Exiftool will terminate soon...
{ready}

Marc