News:

2023-03-15 Major improvements to the new Geolocation feature

Main Menu

Creating a pure, stand-alone metadata file

Started by Keith, November 27, 2019, 01:41:00 AM

Previous topic - Next topic

Keith

Hello all.  I've relied on ExifTool for many years for its robust capabilities and thoroughness.  Similarly, I appreciate the extensive documentation and helpful responses here in the Forums.  Yet even with all these resources, I have not been able to find a clear answer to a question I've been facing.  Your guidance would be most appreciated.

In short: What I'm ideally trying to achieve is a method for exporting a photo's entire collection of metadata, in as complete and unprocessed a form as possible, to a stand-alone metadata file.  Imagine as though the resulting file is a duplicate of the original with the image data simply removed.  And using this resulting stand-alone file, a current or future version of ExifTool would parse it exactly as it would the original file (including with any capabilities which may not yet exist).

Now, I understand that there are a wide variety of standard and proprietary metadata block types.  And this is why I hope the metadata can be retained in its binary or block form.  The purpose of this would be to allow the original image file to be archived in cold storage while the much smaller metadata file is kept locally for processing.

- Does such a file format already exist, and if so is it compatible with ExifTool?
- If such a file format does not exist or is not standardized, could it be constructed as such using ExifTool?
- Are there certain block types which simply cannot be exported in a raw binary form and must be parsed?
- Do arguments such as 'exif2xmp.args' and 'iptc2xmp.args' simply map EXIF and IPTC fields to XMP equivalents, or are the EXIF/IPTC blocks retained as a set?

Thank you.

StarGeek

Quote from: Keith on November 27, 2019, 01:41:00 AM
- Does such a file format already exist, and if so is it compatible with ExifTool?
- If such a file format does not exist or is not standardized, could it be constructed as such using ExifTool?

You might want to take a look at Phil's own MIE file.  See MIE Sidecar files, MIE spec PDF, and MIE tag names page.  I've never used it but Phil can provide any details.

The other option would be to create a 1 pixel square image and block copy the desired groups into that image.  That is what Phil has done on the ExifTool Meta Information Repository page.

Quote- Do arguments such as 'exif2xmp.args' and 'iptc2xmp.args' simply map EXIF and IPTC fields to XMP equivalents, or are the EXIF/IPTC blocks retained as a set?

Those arg files copy the relevant data to the matching locations in the other block types. Tags with the same names (IPTC:Headline/XMP:Headline) are directly copied.  Tags which map to different named tags are correctly assigned.   For example, iptc2xmp.args will copy IPTC:Caption-Abstract to XMP:Description, IPTC:Keywords to XMP:Subject and so forth.  Nothing is removed from the source block.  The destination tags will get overwritten unless the -wm (writemode) option is used to prevent it.  You can open the ARG files in a text editor to see what gets copied.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

Phil Harvey

You have to be more specific.  If you are talking about preserving EXIF, XMP and IPTC metadata in JPEG files, then this command would block copy this metadata to a MIE file:

exiftool -srcfile %d%f.mie -tagsfromfile @ -exif -iptc -xmp -ext jpg DIR

The disadvantage of MIE is that it is an ExifTool-specific format, but the advantage is that it can hold the other metadata types in their original form.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

StarGeek

Phil, this did get me thinking.  What about stuff like MakerNotes?  Can those be copied to an mie file?
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

Phil Harvey

Hi StarGeek,

The -exif block includes the maker notes.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

StarGeek

Cool, that's what I was thinking.  I just wanted to be sure.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

Keith

For starters, Kudos to you both for such quick and thoughtful responses!  Your engagement - even for new forum members such as myself - deserves much respect.

Quote from: StarGeek on November 27, 2019, 02:03:06 AM
You might want to take a look at Phil's own MIE file.  See MIE Sidecar files, MIE spec PDF, and MIE tag names page.  I've never used it but Phil can provide any details.
This sounds very much like what I'm looking for.  I initially wasn't sure what to make of the fact that MIE is so closely linked to a single program, but the documentation is reassuring.

Quote from: StarGeek on November 27, 2019, 02:03:06 AMThe other option would be to create a 1 pixel square image and block copy the desired groups into that image.  That is what Phil has done on the ExifTool Meta Information Repository page.
While clever, this approach seems likely to trip up my future self who may well have forgotten what exactly I was doing now.  Worst-case scenario, I could accidentally overwrite my original files by mistake.

Quote from: StarGeek on November 27, 2019, 02:03:06 AMThose arg files copy the relevant data to the matching locations in the other block types. Tags with the same names (IPTC:Headline/XMP:Headline) are directly copied.  Tags which map to different named tags are correctly assigned.   For example, iptc2xmp.args will copy IPTC:Caption-Abstract to XMP:Description, IPTC:Keywords to XMP:Subject and so forth.  Nothing is removed from the source block.  The destination tags will get overwritten unless the -wm (writemode) option is used to prevent it.  You can open the ARG files in a text editor to see what gets copied.
Ok, that's what I thought might happen.  Ideally, I'm trying not to port information around to different blocks.  Instead I'd prefer to preserve it untouched if possible.

Quote from: Phil Harvey on November 27, 2019, 07:03:19 AM
You have to be more specific.  If you are talking about preserving EXIF, XMP and IPTC metadata in JPEG files, then this command would block copy this metadata to a MIE file:

exiftool -srcfile %d%f.mie -tagsfromfile @ -exif -iptc -xmp -ext jpg DIR
Thank you, Phil.  I've gathered from reading your site that these three blocks are the most standard and useful.  In light of your other comment that MakerNotes (which I definitely need to keep) are a subset of EXIF, then if these were all I could retain I would likely be set.

But for my own understanding, can you clear up a few things?
- Are MakerNotes part of the EXIF spec, or does ExifTool just treat them as linked.
- Is XMP as well-defined as EXIF and IPTC in terms of its tags, or is the tags more user/manufacturer-defined?
- Are there additional metadata blocks beyond these three, or are additional blocks part of XMP?

*For reference, my current archive includes mostly images and video in JPG/CR2/ARW and AVI/MOV formats, although I want this solution to remain generally-applicable.
Quote from: Phil Harvey on November 27, 2019, 07:03:19 AM
The disadvantage of MIE is that it is an ExifTool-specific format, but the advantage is that it can hold the other metadata types in their original form.
While not ideal, at least if I had to select a [functionally] proprietary format to go with, ExifTool has proven itself to be a trustworthy steward.

Phil Harvey

Quote from: Keith on November 27, 2019, 12:37:44 PM
- Are MakerNotes part of the EXIF spec, or does ExifTool just treat them as linked.

The maker notes block is part of the EXIF specification, but the contents are not defined.

Quote- Is XMP as well-defined as EXIF and IPTC in terms of its tags, or is the tags more user/manufacturer-defined?

XMP is designed to be extensible, but isn't widely used by camera manufacturers yet.

Quote- Are there additional metadata blocks beyond these three, or are additional blocks part of XMP?

There are lots of other types in JPEG images.  See here.

Quote*For reference, my current archive includes mostly images and video in JPG/CR2/ARW and AVI/MOV formats, although I want this solution to remain generally-applicable.

This will require some tweaking for other formats.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Keith

Quote from: Phil Harvey on November 27, 2019, 12:54:13 PMThere are lots of other types in JPEG images.  See here.

The list of meta blocks is much more extensive than I anticipated.  I think I'll need to compromise from my original goal for the sake of consistency.
- Forgive if this question is naive, but is it possible to export entire APP* sections rather than specify each block within?
- From the list you provided, which block types are the available for block copy?  For instance, you listed -exif, -iptc, and -xmp explicitly.
Would other blocks with C and W also work this way (ex. -jfif, -ciff)?
- In your sample above: "-tagsfromfile @ -exif -iptc -xmp", are EXIF/IPTC/XMP copied as binary rather than tag-by-tag?
- Does the EXIF section contain a thumbnail image, or is that only contained in JFIF?
- I see several examples throughout the ExifTool documentation and examples without consistent casing (for instance 'All', 'tagsfromfile', and 'ThumbnailImage').  Is ExifTool case-agnostic?

Quote from: Phil Harvey on November 27, 2019, 12:54:13 PMThis will require some tweaking for other formats.
Noticing that the APP6-GoPro and APP5-Ricoh_RMETA sections are read-only, does this mean this data cannot be preserved in this manner?

Phil Harvey

The block copy metadata types are rather limited, and are all listed in the Extra tags documentation.

The advantage of a block copy is that ExifTool doesn't have to have the ability to write the individual tags, so unknown tags will be copied too.

The EXIF may contain a thumbnail.

From the OPTIONS section of the exiftool application documentation:

       Case is not significant for any command-line option (including tag and
       group names), except for single-character options when the
       corresponding upper-case option exists.


QuoteNoticing that the APP6-GoPro and APP5-Ricoh_RMETA sections are read-only, does this mean this data cannot be preserved in this manner?

Not necessarily.  But in this case, yes.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Keith

Thank you again for all the excellent support.  I will take advantage of the upcoming holiday break to experiment and refine this solution.  And if all goes well, I'll repost my findings here for anyone else who may be interested going forward (I've found 5-10yr old threads which remain perfectly helpful today).