-tagsFromFile through XML changes exposure time

[rjlittlefield]

I'm trying to prevent exposure time from changing when I use -X and -tagsFromFile to implement a split-phase copy.

Consider the following:

cp source.jpg target.jpg

exiftool -exposuretime source.jpg

Code Select Expand

Exposure Time                   : 1/200

# Copying tags from source to XML to target changes the exposure time

exiftool -X source.jpg > source_tags.xml

exiftool -tagsFromFile source_tags.xml target.jpg

Code Select Expand

    1 image files updated

exiftool -exposuretime target.jpg

Code Select Expand

Exposure Time                   : 1/197

exiftool -exposuretime source_tags.xml

Code Select Expand

Exposure Time                   : 0.0050761421319797

# But copying tags direct from source to target makes it OK again

exiftool -tagsFromFile source.jpg target.jpg

Code Select Expand

    1 image files updated

exiftool -exposuretime target.jpg

Code Select Expand

Exposure Time                   : 1/200

When I look into the original input file, I see the situation described at FAQ #4.  There are some values of 1/200, some 1/197, in different fields, but that's not a problem because whatever the tags are, they give the effect I want.

The original file works fine.  So does the target file, if I use -tagsFromFile directly from source to target.

The problem only kicks in when I do the split-phase copy, going through an XML file that contains the tags.  Somehow in that process the tags get tweaked so that the reported exposure time changes from 1/200 to 1/197.

I've tried adding -a, -b, -e, no effect.

Any ideas?

Many thanks,
--Rik

[Phil Harvey]

Hi Rik,

What is the output of this command (a la FAQ 3) for the original file?

exiftool -a -G1 -exposuretime source.jpg

- Phil

[rjlittlefield]

After doing the split-phase -tagsFromFile, it looks like this:

exiftool -a -G1 -exposuretime source.jpg

Code Select Expand

[ExifIFD]       Exposure Time                   : 1/200
[Canon]         Exposure Time                   : 1/203
[Canon]         Exposure Time                   : 1/197

exiftool -a -G1 -exposuretime source_tags.xml

Code Select Expand

[ExifIFD]       Exposure Time                   : 0.005
[Canon]         Exposure Time                   : 0.00492610837438424
[Canon]         Exposure Time                   : 0.0050761421319797

exiftool -a -G1 -exposuretime target.jpg

Code Select Expand

[ExifIFD]       Exposure Time                   : 1/197
[Canon]         Exposure Time                   : 1/203
[Canon]         Exposure Time                   : 1/197

[Phil Harvey]

There's your answer.

You copied one of the Canon:ExposureTime values into ExifIFD:ExposureTime

- Phil

[rjlittlefield]

You copied one of the Canon:ExposureTime values into ExifIFD:ExposureTime

Um, right, for some definition of "You".

But I didn't specify what tags to put where. 

I left that up to ExifTool, when I said -X and -tagsFromFile, and it was ExifTool that decided to copy one of the Canon:ExposureTime values into ExifIFD:ExposureTime. 

Further, ExifTool only made that particular decision when I asked it to copy the tags in two phases, one phase using -X to extract them from the source file and a second phase using -tagsFromFile to write them into a target file.

I would really like to run split-phase.

So, phrasing my question another way, how is that I make source --> XML --> target give the same result as source --> target ?

Many thanks,
--Rik

[Phil Harvey]

Hi Rik,

Please let me know if the -tagsFromFile documentation can be improved:

            By default, this option will update any existing and writable
            same-named tags in the output FILE, but will create new tags only
            in their preferred groups.  This allows some information to be
            automatically transferred to the appropriate group when copying
            between images of different formats. However, if a group name is
            specified for a tag then the information is written only to this
            group (unless redirected to another group, see below).  If "All"
            is used as a group name, then each tag is written to the same
            family 1 group it had in the source file (ie. the same specific
            location in the metadata), but a different family may be specified
            by adding a leading family number to the group name (eg.
            "-0All:all").

- Phil

Edit:  Perhaps if I change the documentation to this it will be more clear?:

            By default, this option will update any existing and writable
            same-named tags in the output FILE, but will create new tags only
            in their preferred groups.  This allows some information to be
            automatically transferred to the appropriate group when copying
            between images of different formats. However, if a group name is
            specified for a tag then the information is written only to this
            group (unless redirected to another group, see below).  If "All"
            is used as a group name, then the specified tag(s) are written to
            the same family 1 group they had in the source file (eg.
            "-all:all" copies all writable tags, preserving the same specific
            metadata location for each, like ExifIFD or XMP-dc).  A different
            family may be specified by adding a leading family number to the
            group name (eg. "-0all:all" preserves the same general location,
            like EXIF or XMP).

[rjlittlefield]

Phil,

(eg. "-all:all" copies all writable tags, preserving the same specific metadata location for each, like ExifIFD or XMP-dc).

That would help a lot.  But as I now understand the operation I would phrase the documentation even a little stronger, something like

(e.g. "-all:all" is required to copy all writable tags while preserving the same specific metadata location for each, like ExifIFD or XMP-dc).

Even so, that doesn't address my problems with split-phase operation.

In the example I used to start this thread, I began by simply copying source.jpg to target.jpg.  I thought that would make a good way to illustrate the problem.  In retrospect, probably not the best choice.

What I'm really interested in is the case where the target starts with no EXIF at all, and I want to propagate all the source's tags into the target.

In that case, it works fine to -tagsFromFile from .jpg to .jpg, with or without -all:all.  Here's an example without -all:all:

cp NoExif.jpg target.jpg

exiftool -a -G1 -exposuretime target.jpg
(no output)

exiftool -tagsFromFile source.jpg target.jpg

Code Select Expand

    1 image files updated

exiftool -a -G1 -exposuretime target.jpg

Code Select Expand

[ExifIFD]       Exposure Time                   : 1/200
[Canon]         Exposure Time                   : 1/203
[Canon]         Exposure Time                   : 1/197

But adding -all:all to the split copy still does not transfer a lot of the tags:

cp NoExif.jpg target.jpg

exiftool -all:all -X source.jpg > source_tags.xml

exiftool -tagsFromFile source_tags.xml -all:all target.jpg

Code Select Expand

    1 image files updated

exiftool -a -G1 -exposuretime target.jpg

Code Select Expand

[ExifIFD]       Exposure Time                   : 1/200

So, I'm still stuck.  I now see why the exposuretime was getting altered, but I still don't see how to do a two-phase copy and get what I want, which is all the tags transferred, keeping their original groups.

It's now looking to me like going .jpg --> XML --> .jpg may be causing exiftool to treat this as "copying between images of different formats", even though the ultimate source and target do happen to be the same format. 

If that's what's happening, then I guess I can split phase by using a little placeholder .jpg instead of the XML file, but it sure would be a lot more convenient if there's a way to do it using XML.

Thoughts?

--Rik

[Phil Harvey]

OK, I think I've tortured you for long enough.

Try this:

1. exiftool -all -makernotes -X -b source.jpg > source_tags.xml

2. exiftool -tagsfromfile source_tags.xml -all:all target.jpg

The other problem we hadn't addressed yet is that makernote tags may not be created individually, so they must be copied as a block.

The only problem with this is that the makernote offsets won't be adjusted properly coming out of the XML file, so you will get a warning when reading this file, but touching it by writing any tag afterward will fix this.

- Phil

P.S. The inconsistency you noticed between the 1-step and 2-step process was simply due to the fact that the tags are in a different order in the two source files, and if there are 3 tags with the same name being written to the same location, the last one takes precedence.

Edit: Added missing -X option.

[rjlittlefield]

Phil,

Thanks for sticking with me on this!

OK, I think I've tortured you for long enough.

Torture?  I've been programming for 45 years; this kind of stuff counts as "titillation".   

That said, I do appreciate the reprieve.  Even the finest of titillation can wear thin with enough confusion.

Thanks especially for the comments about makernote tags.  I'm sure I don't completely understand either the tags or the comments, but at least my sensitivities are raised.

You wrote:

Try this:

1. exiftool -all -makernotes -b source.jpg > source_tags.xml

2. exiftool -tagsfromfile source_tags.xml -all:all target.jpg

On copy/pasting the commands, I was momentarily surprised to get this result:

Code Select Expand

Error: Unknown file type - source_tags.xml
Warning: No writable tags set from source_tags.xml
    0 image files updated
    1 image files unchanged

Then when I fired up a text editor on source_tags.xml, I was relieved to discover that the first command was just missing a -X .

With the -X added, your commands seem to give what I wanted.

In my real application, there are also some tags that I don't want copied because they will be misleading or flat-out wrong, given the way the app has manipulated the image.   

So, what I have right now is a -@ script for phase 1 that looks like this:

Code Select Expand


-a
-e
-all:all
-makernotes
--ExifTool:all
--File:all
--all:icc_profile
--icc_profile:all
--icc_header:all
--icc_view:all
--icc_meas:all
--all:colorspace
--all:previewimage
--all:thumbnailimage
--IFD0:Orientation
-b
-X
-w!
%0fD:\Testing\source_tags.xml
D:\Testing\source.jpg


I'm not sure I need the -a given that -X seems to handle multiple occurrences of Canon:ExposureTime without any problems. 
Likewise I'm not sure I need the -e but I wanted to make sure that computed composite tags didn't get injected as ordinary explicit tags. 

Recognizing that I'm horribly naive about TIFF tags in general, as well as being an ExifTool newbie...

Do you see anything about this script that looks like I'm headed down the wrong path?

Many thanks,
--Rik

[Phil Harvey]

Hi Rik,

Then when I fired up a text editor on source_tags.xml, I was relieved to discover that the first command was just missing a -X .

Ooops. Right.  Sorry about that.  I'll edit my original post to avoid confusing others.

I'm not sure I need the -a

Right, you don't.  This is mentioned in the -X (and -a) documentation.

Do you see anything about this script that looks like I'm headed down the wrong path?

No, looks good.

- Phil

Edit:  I've rewritten the previously mentioned section of the documentation many times trying to get a good balance while trying to keep it succinct.  Here is my current version:

            By default, this option will update any existing and writable
            same-named tags in the output FILE, but will create new tags only
            in their preferred groups.  This allows some information to be
            automatically transferred to the appropriate group when copying
            between images of different formats. However, if a group name is
            specified for a tag then the information is written only to this
            group (unless redirected to another group, see below).  If "All"
            is used as a group name, then the specified tag(s) are written to
            the same family 1 group they had in the source file (ie. the same
            specific location, like ExifIFD or XMP-dc).  For example, the
            common operation of copying all writable tags to the same specific
            locations in the output FILE is achieved by adding "-all:all".  A
            different family may be specified by adding a leading family
            number to the group name (eg. "-0all:all" preserves the same
            general location, like EXIF or XMP).

[rjlittlefield]

Edit:  I've rewritten the previously mentioned section of the documentation many times trying to get a good balance while trying to keep it succinct.  Here is my current version:

The edit certainly helps.

However, I don't think that one change would have been nearly enough to let me figure out for myself how to do what I wanted.

In further experiments, it seems clear that in order to get all the tags copied properly, going through the XML file, I really do need to have all of the parameters -- including -makernotes.  But now searching http://www.exiftool.org/exiftool_pod.html for "makernotes", I can't see anything that would give me a hint about that.

Given all the time you've spent working me through this, I wonder if it would be a good idea to add the result as an example, something like:

1. exiftool -all -makernotes -X -b source.jpg > source_tags.xml
2. exiftool -tagsfromfile source_tags.xml -all:all target.jpg

Two-phase copy: extract all tags from source.jpg to source_tags.xml, then insert them into target.jpg, placing each tag in its original group.

Or is there something about this example that would make it not a good thing to show?

As always, many thanks!

--Rik

[Phil Harvey]

Hi Rik,

However, I don't think that one change would have been nearly enough to let me figure out for myself how to do what I wanted.

No, I didn't expect it would.  Particularly because I can't expect anyone to read the docs...

In further experiments, it seems clear that in order to get all the tags copied properly, going through the XML file, I really do need to have all of the parameters -- including -makernotes.  But now searching http://www.exiftool.org/exiftool_pod.html for "makernotes", I can't see anything that would give me a hint about that.

Right.  Tag-specific details like this are usually relegated to the Tag Name documentation.  Specifically here, MakerNotes is a Shortcut tag (and although it mentions about block copying, I don't think it was specific enough to have helped here).  Sadly, I isn't realistic to expect people to find the information buried here (how would you know for example that MakerNotes was a Shortcut tag, or that it even existed?).  But I can't include all these details in the application documentation because then it would be completely unreadable.

Given all the time you've spent working me through this, I wonder if it would be a good idea to add the result as an example, [snip]
Or is there something about this example that would make it not a good thing to show?

This suggestion makes me realize that the place to document this would be the XML section of the Metadata Sidecar Files page.  But I don't know that this would have helped -- did you know this page existed?

- Phil

[rjlittlefield]

No, I didn't expect it would.  Particularly because I can't expect anyone to read the docs...

It's a perpetual problem -- one in which I have sympathy for everybody.  For my own software, a PDF of the web documentation now runs to several hundred pages, and still there's a lot of details that aren't in there, and some useful but nonobvious ways to put the features together.  So I do what I perceive you doing: make myself available to answer questions.  Some of those are easily answered by pointing to some place in the documentation that the user might have found by browsing, or by searching, if they knew exactly what to ask. But a fair number need some personalized attention from an expert.  (I thank you for providing that!)

This suggestion makes me realize that the place to document this would be the XML section of the Metadata Sidecar Files page.  But I don't know that this would have helped -- did you know this page existed?

I did not.  As background, I got to where I am by first posting a question last summer about how to do split-phase copy at all (https://exiftool.org/forum/index.php?topic=5954.0).  It seemed pretty straightforward at that time, but life got in the way and I had to work on other things until pretty recently.  When I started putting the stuff together for real, one of my users pointed out that his 1/200 second exposure time had somehow turned into 1/197 second, and then the challenge became "Why did that happen, and more important, what can I do about it?"  After trying to figure that out myself for some hours and getting nowhere, I finally gave up and called for help.

On the assumption that nobody else has ever wanted to do exactly what I've been asking about, perhaps there never was a viable path except this conversation we've been having.

Many thanks,
--Rik

[Phil Harvey]

Hi Rik,

Thanks for the sympathy.

Since I have learned something here too, and don't want to forget about it, I think it makes sense for me to add a description of this to the Metadata Sidecar Files XML section.  I'll do that when I get a chance.

As with your software, complete documentation would easily run several hundred pages.  I would be very happy to produce it if I thought that it would be worth the effort.

- Phil

Edit:  I have updated the Metadata Sidecar Files page to add this example.