[Keys] tag group in QuickTime Files (.MOV)

[Mac2]

Hi, Phil

a user contacted me about some proprietary Apple tags no longer being imported by IMatch.
He has provided two files: A HEIC file and a MOV file.

The tag in question (a UUID) is reported by ExifTool as [Apple] Media Group UUID in the HEIC file. IMatch imports it just fine. I used -G1 to check.

In the MOV file, ExifTool reports the same value as [Keys] Content Identifier.
In the XML IMatch uses to import the data, the tag is <Keys:ContentIdentifier>.
IMatch does not import the value because Keys is not a ExifTool group (as reported by -listg) and hence IMatch has no "place" to store the value.

Is [Keys] something special or do I miss something? Should this not listed as an official tag group?

[Phil Harvey]

Keys is a family 1 group name.  You can list the family 1 groups with

  exiftool -listg1

- Phil

[Mac2]

My application stores ExifTool output in a database.
When a new database is created, my application imports the family 0 groups (-listg) and all tags (id, keys, names). The tags are linked to their family 0 groups.
When importing data from a file, my application uses -G -X -D -t -l ... to get the data organized by group and tag. This allows for fast database lookups.
This worked great for many years.

When I understand you correctly, ExifTool does not, for this tag and maybe other tags, output the -G group but the -G1 group, even when I explicitly request the -G group?
I have not see that before, I think. Under which conditions is a family 1 group name emitted but no family 0?

If this is a common case, I need to substantially change how IMatch stores ExifTool output, importing the family 0 and family 1 group names from now on and performing additional lookups when importing data and the group name returned for -G is not found in the G table, but might be in the -G1 table.

[Phil Harvey]

The -X output has always been just family 1 group names (although I have just recently make this more clear in the application documentation).

This has not changed.  You need to look at your usage in more detail if you say this has worked using family 0 group names for many years.  For example, you should be seeing IFD0, IFD1, ExifIFD, etc and not the family 0 EXIF group for EXIF tags.

- Phil

Edit: Ah.  I may see the difference.  When the ability to create QuickTime tags was added in version 11.39, the QuickTime family 1 group names were changed to a more specific location:

May 3, 2019 - Version 11.39 - "Create QuickTime tags"
  - Split off some QuickTime tags into different family 1 groups

[Mac2]

I apologize. I was confused. I did not have to look at this code for years and remembered it incorrectly.

IMatch actually imports the <table name=> name into the database for groups.
And when later processing tag values, it uses the et:table= for the mapping.
Everything works, the content identifier value is just in different groups / tags for the HEIC and the MOV file.
This what confused the user and finally me.

The content identifier is listed in QuickTime::ItemList\1.2\ContentIdentifier for the user's MOV files, and in Apple::Main\17\MediaGroupUUID for the user's HEIC files.
IMatch imports the data just fine (if the user has enabled the corresponding tag groups).
The user must just update everything in his workflow to deal with the different tag names for different files.

Edit: Ah.  I may see the difference.  When the ability to write QuickTime tags was added in version 11.39, the QuickTime family 1 group names were changed to a more specific location:
May 3, 2019 - Version 11.39 - "Create QuickTime tags"
  - Split off some QuickTime tags into different family 1 groups

Is there a announcement list or email list I can subscribe to, to get notice of such planned or already implemented changes?
I try to keep up with ExifTool release notes, but sometimes a change is explained with a single harmless line in the release notes, but may have dire consequences for hundreds of users...

Tag name changes, for example. Like the Composite tag name changes in a recent version, or the changed ExifTool namespace in XML.

When you have many users who build workflows and scripts based on tags (for import, export, processing, copying, ...) and then tag names change or data is moved to another tag or group due to a change in ExifTool, problems will happen. Workflows will break. Tears will flow. Mostly mine 

I usually try to handle such changes silently in IMatch, e.g. by running migration routines for the database when a new ExifTool version is integrated. Map data from the old tag names to the new names so we don't have the same data in old and new tags. I'm sure my software is not the only software or system affected by such breaking changes.

If there is a mechanism where I can subscribe to advance warnings about such planned changes, please let me know.

[Phil Harvey]

Is there a announcement list or email list I can subscribe to, to get notice of such planned or already implemented changes?
I try to keep up with ExifTool release notes, but sometimes a change is explained with a single harmless line in the release notes, but may have dire consequences for hundreds of users...

Obviously I have hit a sore point.  The revision history is where the changes are listed.  It sounds like you want a separate announcement for changes that may have a bigger impact for you, and honestly I don't know how I would determine this.  Different ExifTool users have different priorities.  In general though, I try my best to keep ExifTool backward compatible as much as possible.

Tag name changes, for example. Like the Composite tag name changes in a recent version, or the changed ExifTool namespace in XML.

The Composite tag names didn't change.  It was the ID's that changed, and these ID's are an internal detail of the ExifTool implementation that are only meant to be consistent between the -listx and the -X outputs for a given ExifTool version, allowing the user to match the tags in these outputs.  They were never meant to be cached in an external database, although for the most part this may be OK because the ExifTool internals don't change quickly/often.  I think you are probably the only one who uses the ID's like this, and are probably the only one who noticed the change in Composite tag ID's.  Having said this though, tag names do change occasionally as we gain a better understanding of the metadata, but this mainly affects only MakerNotes tags.

Similarly, the -X namespace change is self consistent with ExifTool input and output, and ExifTool does still recognize XML with the old namespaces.  This change was made to close a security hole as suggested by a security issue that was reported.  I'm sorry if it caused you problems.

- Phil

[Mac2]

The revision history is where the changes are listed.  It sounds like you want a separate announcement for changes that may have a bigger impact for you, and honestly I don't know how I would determine this.  Different ExifTool users have different priorities.  In general though, I try my best to keep ExifTool backward compatible as much as possible.

I try to keep up with the releases notes. But sometimes I'm stupid, distracted or forgetful 

You are of course free to change ExifTool to improve it. I fully support this, it what makes ExifTool so awesome.
As I have said many times, and say again, ExifTool is one of the best things since slices bread. I'm so happy that I can offer my user base access to all their metadata and produce metadata that is rich and compatible and works everywhere.

I'm just always a bit concerned about me missing something in the release notes (because it reads harmless, but isn't) and then hundreds of users lose data or days of work when they install an IMatch update which includes a new ExifTool version... 

The Composite tag names didn't change.  It was the ID's that changed, a

Yes, I remember. But when I added support for ExifTool many years back, I used the id and tag name as the "tag key", and hence my software was affected by this change. One is always smarter in hindsight

I use some sort virtualization for some tags (mapping the ExifTool tag name to an IMatch tag name) called short codes and encourage users to use them instead of the actual ExifTool tag names. This allows me to change things under the hood without anything breaking on the user's end. I only need to keep the database in sync with the tag name changes, map existing data to the new tag name etc. This is done during the migration phase when a new update is installed.

Having said this though, tag names do change occasionally as we gain a better understanding of the metadata, but this mainly affects only MakerNotes tags.

I understand this. Perfectly normal. Especially for these notorious, ever-changing, proprietary and undocumented maker notes the camera vendors bless us with. Sigh.

From a DAM perspective, if images processed before April 1. 2021 stored the data under the tag name ABC and after that date (due to a new ExifTool version) store the data in the tag DEF, several thingsa may happen:

1. Users never notice this, because the tag is an obscure maker note somewhere
2. Some users notice it, because they have used the data in this tag for display, printing, calculations, whatever
3. Many users notice that the data is no longer there

This is what I need to keep an eye on and handle for my user base. It's totally worth it, though.

The XMP namespace change was not problematic at all.
One of the community members here (an Admin or other developer, don't recall at the moment) informed me after seeing it in the release notes.
I only had to change the schema data the MS XML parser uses and that took all but a minute.

My idea of an "announcement" was merely because of me sometimes being forgetful, lazy or stupid.

Thanks for taking the time to reply and have a nice weekend.