Potential bug with certain TIF files and -stay_open

[Mac2]

After digging into a specific problem for a day I wonder if this may be caused by a glitch in ExifTool. Here is what I do:

In my app I produce an ARGS file to read metadata from a file. ExifTool is started with -stay_open and reads the new arguments. My application adds an -execute99 at when all arguments have been sent and then waits for ExifTool to return {ready99}. At this time the entire output produced by ExifTool is read in and processed.

This works with all types of files, except certain TIF files.
For these TIF files (see attachment) I can do this only once. If I attempt to read metadata from the same file again, some data is missing in the output. As far as I can tell, XMP:rating and XMP:label are not contained in the output on the second or later calls. Only on the first call.

The XMP output is correct and valid for every call, but the second and later calls just don't include all tags. For the same input file and arguments! 

When I run the same arguments via an args file against the command line version of ExifTool (same executable as used in my app) the output is always right.

When I shut-down exiftool between two calls (closing with -stay_open=false, then again start it with -stay_open=true), the missing TIF metadata is again part of the output. But only on the first call. The second and later calls for the same file again fail to include the rating and label in the XMP output.

To me this looks as if something is not reset back or re-initialized in this specific case.
As said, this happens only with certain TIF files. When I use JPG or PSD files, I can run the argument file any number of times and I always get back the same output.

I have attached one of the TIF files which allow me to repro this behavior.

[Phil Harvey]

Do you invoke the MWG module between these calls?  Loading MWG is non-reversible and will affect all subsequent commands.  One effect of loading this module is to ignore some tags in non-standard locations.

- Phil

[Phil Harvey]

I just tried this argfile with your sample tiff:

Code Select Expand

-G1
a.tif
-execute99
-G1
a.tif
-execute100

(I added the 2nd set of arguments after the 1st command was executed.)

with this result:

Code Select Expand

> exiftool -stay_open true -@ a.args
[ExifTool]      ExifTool Version Number         : 8.67
[System]        File Name                       : a.tif
[System]        Directory                       : .
[System]        File Size                       : 1879 kB
[System]        File Modification Date/Time     : 2011:10:12 09:38:41-04:00
[System]        File Permissions                : rw-r--r--
[File]          File Type                       : TIFF
[File]          MIME Type                       : image/tiff
[File]          Exif Byte Order                 : Little-endian (Intel, II)
[File]          Current IPTC Digest             : bd9b30291e6f93195765d048af3317e5
[IFD0]          Subfile Type                    : Full-resolution Image
[IFD0]          Image Width                     : 800
[IFD0]          Image Height                    : 800
[IFD0]          Bits Per Sample                 : 8 8 8
[IFD0]          Compression                     : Uncompressed
[IFD0]          Photometric Interpretation      : RGB
[IFD0]          Strip Offsets                   : (Binary data 740 bytes, use -b option to extract)
[IFD0]          Samples Per Pixel               : 3
[IFD0]          Rows Per Strip                  : 8
[IFD0]          Strip Byte Counts               : (Binary data 599 bytes, use -b option to extract)
[IFD0]          X Resolution                    : 72
[IFD0]          Y Resolution                    : 72
[IFD0]          Planar Configuration            : Chunky
[IFD0]          Resolution Unit                 : inches
[IFD0]          Modify Date                     : 2011:10:12 06:30:34
[XMP-x]         XMP Toolkit                     : Image::ExifTool 8.66
[XMP-xmp]       Metadata Date                   : 2011:10:12 15:25:47+02:00
[XMP-xmp]       Rating                          : 3
[IPTC]          Coded Character Set             : UTF8
[IPTC]          Envelope Record Version         : 4
[Composite]     Base Name                       : a
[Composite]     File Extension                  : tif
[Composite]     File Type Description           : Tagged Image File Format
[Composite]     Image Size                      : 800x800
[Composite]     Physical Image Size             : 11.1x11.1 inches
{ready99}
[ExifTool]      ExifTool Version Number         : 8.67
[System]        File Name                       : a.tif
[System]        Directory                       : .
[System]        File Size                       : 1879 kB
[System]        File Modification Date/Time     : 2011:10:12 09:38:41-04:00
[System]        File Permissions                : rw-r--r--
[File]          File Type                       : TIFF
[File]          MIME Type                       : image/tiff
[File]          Exif Byte Order                 : Little-endian (Intel, II)
[File]          Current IPTC Digest             : bd9b30291e6f93195765d048af3317e5
[IFD0]          Subfile Type                    : Full-resolution Image
[IFD0]          Image Width                     : 800
[IFD0]          Image Height                    : 800
[IFD0]          Bits Per Sample                 : 8 8 8
[IFD0]          Compression                     : Uncompressed
[IFD0]          Photometric Interpretation      : RGB
[IFD0]          Strip Offsets                   : (Binary data 740 bytes, use -b option to extract)
[IFD0]          Samples Per Pixel               : 3
[IFD0]          Rows Per Strip                  : 8
[IFD0]          Strip Byte Counts               : (Binary data 599 bytes, use -b option to extract)
[IFD0]          X Resolution                    : 72
[IFD0]          Y Resolution                    : 72
[IFD0]          Planar Configuration            : Chunky
[IFD0]          Resolution Unit                 : inches
[IFD0]          Modify Date                     : 2011:10:12 06:30:34
[XMP-x]         XMP Toolkit                     : Image::ExifTool 8.66
[XMP-xmp]       Metadata Date                   : 2011:10:12 15:25:47+02:00
[XMP-xmp]       Rating                          : 3
[IPTC]          Coded Character Set             : UTF8
[IPTC]          Envelope Record Version         : 4
[Composite]     Base Name                       : a
[Composite]     File Extension                  : tif
[Composite]     File Type Description           : Tagged Image File Format
[Composite]     Image Size                      : 800x800
[Composite]     Physical Image Size             : 11.1x11.1 inches
{ready100}

The outputs are identical for me.

- Phil

[Phil Harvey]

I wonder if this is related to this thread ?

[Mac2]

Yes, I always add -use MWG as part of my args.

I cannot reproduce this behavior with other files, except these TIFF files.

Any number or calls for a JPEG or PSD files always returns the same metadata, XMP:label and Rating are always emitted in the -X output. For the TIFF, the second call and all following calls excludes the XMP:label and Composite:Rating

I've read the Java thread you pointed out and I thought about this too. But

a) using execute99 and waiting for ready99 ensures that my code is synchronized with ExifTool

b) The output is always a valid XML file, nothing is cut off  or missing. The file can be processed with an XML parser.
The missing tags for the TIFF are somewhere in the "middle" of the document, but the structure itself is valid

c) It only fails for the TIFF files. The same code path and arguments are used for all files.

d) I cannot produce the same result, even for the TIFF, when I use a "fresh" ExifTool instance for every call (do not use -stay_loaded)

The arguments I use are:
-m
-tagsfromfile
c:\images\a.tif
-all
-all:all
-use
MWG
c:\temp\output.xmp

Only the line with the source file name differs between the JPG, PSD or TIFF calls.

These segments are the only missing segments for the TIF file on the second and later runs (I've made a diff of the XMP files):

Code Select Expand

<XMP-xmp:MetadataDate>
  <rdf:Description et:id='MetadataDate' et:table='XMP::xmp'>
   <et:desc>Metadata Date</et:desc>
   <et:prt>2011:10:12 18:49:18+02:00</et:prt>
  </rdf:Description>
</XMP-xmp:MetadataDate>

<XMP-xmp:Rating>
  <rdf:Description et:id='Rating' et:table='XMP::xmp'>
   <et:desc>Rating</et:desc>
   <et:prt>4</et:prt>
  </rdf:Description>
</XMP-xmp:Rating>

<MWG:Rating>
  <rdf:Description et:id='Rating' et:table='Composite'>
   <et:desc>Rating</et:desc>
   <et:prt>4</et:prt>
  </rdf:Description>
</MWG:Rating>

everything else in the resulting XMP file is identical.
When I kill the exiftool.exe process and run the same args file again, the missing fields are back. Running it again, and the fields are missing again. Only for the TIF. Strange.

[Phil Harvey]

I get this with your argfile:

Code Select Expand

> exiftool -stay_open true -@ a.args
    1 image files created
{ready99}
Warning: Ignored non-standard IPTC at TIFF-IFD0-IFD0 - a.tif
Warning: Ignored non-standard XMP at TIFF-IFD0-IFD0 - a.tif
    1 image files created
{ready100}

The warning messages are very illuminating.  I will investigate further now that I can reproduce the problem.

- Phil

[Phil Harvey]

HOORAY!  You found a real bug in ExifTool.  Well done.  Good catch!  I really enjoy it when I can fix one of these.

This will be fixed in version 8.67, which I plan to release soon.

- Phil

[Mac2]

Oh. Ah. Yes. 

I like finding bugs. Only a dead bug is a good bug (in a programming sense, I have nothing against real bugs  )

This one puzzled me all day because it never happened before. And I tried quite many things to track it down from the database to input. Until the last resort seemed to be that this was actually an strange issue in ExifTool itself. Phew! Call me a bug magnet.

I always suspect ExifTool last because in 99% of all cases its my fault. Happy to hear that you could find it so quickly and that this did not cost you hours of debugging.