OSX visible XMP metadata in PNGs

[gdh]

Around my image processing software I use exiftool to successfully (thank you!) shuffle around exif info from Cr2, TIFF, JPG files. The added tags, such as "keywords", are all visible in OSX Finder, Preview and nicely indexed by Spotlight (OSX 10.8 Mountain Lion, exiftool v9.39).

For PNGs I need to fallback to "pure" XMP as this is the metadata container for PNG. However, exiftool-added tags do not seem to be picked up by neither Preview nor SpotLight. However, if I first add a tag in "other" software (e.g. Preview) and use exiftool to add a new tag later, this IS indexed. In this case, Exiftool appears to append/insert in the existing XMP data (xpackets). When comparing Adobes XMP SDK sample tools with exiftool inserting an XMP tag into a clean PNG, a diff shows that exiftool appends the <xpackets> at the end of the PNG file, while XMP sdk puts it at the beginning (in the header?) of the PNG. The XMP spec states that "Encoders are encouraged to place the chunk at the beginning of the file, but this is not required."

I'm afraid I'm overlooking something. Is my assumption correct that OSX and its metadata toolchain are ignoring this spec and thereby missing exiftool-generated XMP entries at the end of files? If so, any hints on shuffling the XMP around in files, or a way to have exiftool put the XMP tags "first" in a PNG?

I also posted more details on StackOverflow to keep track of this (see link) but I'll get to the point here quicker.
http://stackoverflow.com/questions/19154596/exiftool-to-create-osx-visible-xmp-metadata-in-png-images

[Phil Harvey]

Thanks for reporting this problem.

It is unfortunate if OS X ignores XMP at the end of PNG files, because, again unfortunately, the XMP specification doesn't mandate any specific position for the XMP (although it does make a recommendation, as you noted).  Since ExifTool uses a single-pass writing algorithm, it can not add new XMP to the start of the file because it has no way to know if there is existing XMP at the end until after it has already written the start of the file (and there must be no more than one XMP chunk in a PNG file).

ExifTool doesn't provide any mechanism to allow you to re-order the PNG chunks.

I suggest submitting a bug report to Apple, complaining that Spotlight doesn't index metadata in some PNG images.

- Phil

Edit:  I did some playing with Apple Preview, and not only does it not recognize XMP at the end of the file, but also it deletes this XMP when adding keywords to the image.  My guess is that Apple software ignores XMP if it comes after the IDAT chunk.  It would have been wonderful if the XMP specification had mandated that the XMP chunk come before IDAT, but it didn't, so this must be considered a bug in the Apple software.  I have added this to the list of Known problems.

[gdh]

Thanks Phil for your answer and for repeating experiments that confirm the issue! I didn't really expect to go from forum newbie to "Known problems" contributor 

Anyway, I filed a bug at Apple on OSX as you suggested. Will keep you and forum members posted if there's any follow up on that one. In the meantime I'm waiting for a OSX Mavericks update to complete to verifyin that version.

[Phil Harvey]

Just FYI, I also tried using Adobe Photoshop CS4 to add keywords to a PNG image.  The Apple utilities didn't recognize these keywords either, but for a different reason:

Adobe, in their questionable wisdom, didn't follow their own XMP specification.  CS4 writes the XMP to the file's resource fork for PNG images.  Crazy, I know.  (ExifTool will extract this, but you need to add the -ee option to get it to process the resource fork.)  I can't easily test more recent versions of Photoshop.

- Phil

[gdh]

Thanks for the heads up!

I experienced Adobe's weird way of handling metadata allright. I run the latest CS6 with the cloud subscription and it ironically sports the "powered by XMP" logo in the File Info panel. However, when it comes to PNGs all is lost --literary-- because even a "Document Title" tag is not shown after saving and re-opening inside PhotoShop. Also, the panel "Raw data" displays a re-interpreted version of some data, not the actual raw XMP data in the file. When it concerns metadata in my "image production pipeline" I guess I'm better off switching from PNG to a more elaborate and widely supported format like TIFF.

[Phil Harvey]

However, when it comes to PNGs all is lost --literary-- because even a "Document Title" tag is not shown after saving and re-opening inside PhotoShop

This is consistent with my experience that "newer is not always better", because at least CS4 will read back the metadata that it wrote to a PNG image.  (I just tested the "Document Title" to be sure.)

On a side note, and also in the "newer is not always better" category, I spent much of the time since my last post trying to install Photoshop CC on my new iMac at work.  Complete fail.  These internet installs really suck. (Even when I'm at work on my high-speed connection with a brand-new iMac.  At home I'm on a dialup connection.  Recently it seems that some software vendors won't even acknowledge the existence of people like myself.)

- Phil

Edit: I finally just got Photoshop CC to install.  The problem was some security software that I am running.  Anyway, it turns out that PS CC writes XMP inside the PNG image, before the IDAT chunk.  And these keywords are recognized by Apple Preview.  So sometimes newer is better I guess.  But overall my experience with PS CC is very negative.  Am I the only one who is really put off by the way that you have to log into adobe.com to just run the software?  It seems as if Adobe exists in some sort of Orwellian meta reality.

[gdh]

Anyway, it turns out that PS CC writes XMP inside the PNG image, before the IDAT chunk.  And these keywords are recognized by Apple Preview.

That part surprises me. I now also validated with Adobe Bridge, which luckily does seem to work correctly on all variations of XMP chunk placement (including exiftool's at-the-end of PNG). In Adobe Photoshop (version 13.1.2 20130105.r.224) however, I cannot seem to read (let alone save) any XMP info in a PNG.

Concerning Adobe's CC, I can understand the decisions they had to make from a commercial perspective. In a positive light, at least I am able to run all of their tools as long as I need them without a hefty up-front investment. Whether their technical implementation is up to par I'm not sure, neither about the implication of cloud-stored credentials waiting to be compromised (parts of which have been, already)

[Phil Harvey]

It is good that PS 13.1.2 recognizes XMP at the end of PNG images.

Yesterday I neglected to test to see if PS CC (version 14.0) will do this.  Sadly, there it seems to have the same problem as the Apple software.   I guess this means that I should change ExifTool, but doing this while maintaining compatibility with metadata written at the end of PNG images may not be possible.

- Phil

[Phil Harvey]

I have done a lot of work on this, and Exiftool 9.40 will have a new option to allow you to write XMP before the PNG IDAT chunk.  The corresponding command will look like this:

exiftool -api PNGEarlyXMP ...

"PNGEarlyXMP" is a new (case-sensitive) API option, and -api is a new command-line option that will allow direct access to the API options.  (Alternatively, you can set API options via the config file.)  Here is my current draft of the new API option documentation:

           PNGEarlyXMP
               Flag to write XMP in PNG images before the IDAT (image data)
               chunk.  By default, ExifTool adds new XMP to the end of a PNG
               file (just before IEND).  This is allowed by the PNG and XMP
               specifications, but some utilities seem to ignore XMP if it
               comes after the image data.  The PNGEarlyXMP option causes
               ExifTool to instead add new XMP before the PNG IDAT chunk.
               However, since ExifTool uses a single-pass writing algorithm,
               it has no way to tell if XMP already exists later in the file
               before writing the new XMP in this location.  If this happens,
               a minor error is issued and the file is not written.  Adding
               the "IgnoreMinorErrors" option causes the late XMP to be
               deleted, thus resolving the conflict (at the expense of
               possible metadata loss), and allowing the file to be written.
               This option is applied automatically when deleting all XMP and
               writing new XMP back in one step.  When reading, this option
               causes a warning to be issued if XMP is found after the IDAT
               chunk.

So with this new feature, you will be able to move the XMP to before IDAT using a command like this:

exiftool -xmp:all= -tagsfromfile @ -xmp:all image.png

- Phil

Edit: Added feature to automatically apply the new option when deleting all XMP and writing back again.

[StarGeek]

So with this new feature, you will be able to move the XMP to before IDAT using a command like this:

exiftool -xmp:all= -tagsfromfile @ -xmp:all image.png

Doesn't this need the new -api PNGEarlyXMP option as part of the command?

[Phil Harvey]

Doesn't this need the new -api PNGEarlyXMP option as part of the command?

No.  I was smart, and realized that there won't be any XMP after the IDAT chunk if you delete all XMP in the same command, so in this case it is possible to make this the default behaviour:

               This option is applied automatically when deleting all XMP and
               writing new XMP back in one step.

- Phil

[StarGeek]

Ah, I see.  Shame on me for not reading all of the blue text.

[Phil Harvey]

ExifTool 9.40 is now available.

Please let me know if you have any problems with this new feature.

- Phil