Extracting xmp from jpegs - sometimes it works, sometimes it doesn't...

[areohbee]

Extracting xmp from jpegs - sometimes it works, sometimes it doesn't... - not sure what else to say, except there is an example of one which doesn't work, attached.

[Alan Clifford]

Extracting xmp from jpegs - sometimes it works, sometimes it doesn't... - not sure what else to say, except there is an example of one which doesn't work, attached.

Using

exiftool -G -a -xmp:all 20131024RC113633NB4506.jpg

I see

Code Select Expand


[XMP]           XMP Toolkit                     : Adobe XMP Core 5.5-c002 1.148022, 2012/07/15-18:06:45
[XMP]           Creator Tool                    : Adobe Photoshop Lightroom 5.3 (Windows)
[XMP]           Modify Date                     : 2013:12:14 02:31:10-08:00
[XMP]           Create Date                     : 2013:10:24 11:36:33.56
[XMP]           Metadata Date                   : 2013:12:14 02:31:10-08:00
[XMP]           Label                           : Final
[XMP]           Serial Number                   : 3141286
[XMP]           Lens Info                       : 105mm f/2.8
[XMP]           Lens                            : 105.0 mm f/2.8
[XMP]           Lens ID                         : 138
[XMP]           Image Number                    : 24462
[XMP]           Approximate Focus Distance      : 1.12
[XMP]           Date Created                    : 2013:10:24 11:36:33.56
[XMP]           Authors Position                : Software Developer
[XMP]           Document ID                     : xmp.did:920b0b4a-089a-a643-a537-6517eed39e9c
[XMP]           Original Document ID            : 798712531BC82BA9A9576912A801ACD7
[XMP]           Instance ID                     : xmp.iid:920b0b4a-089a-a643-a537-6517eed39e9c
[XMP]           Format                          : image/jpeg
[XMP]           Has Extended XMP                : 68994A9EC031E5F5ABFB9CBF7E10882C
[XMP]           History Action                  : saved, saved, derived, saved
[XMP]           History Instance ID             : xmp.iid:565472cd-2352-6544-8148-4d64d773abbb, xmp.iid:ec3ae5ed-6624-134c-8077-088616062633, xmp.iid:920b0b4a-089a-a643-a537-6517eed39e9c
[XMP]           History When                    : 2013:11:05 18:12:38-08:00, 2013:11:30 21:02:05-08:00, 2013:12:14 02:31:10-08:00
[XMP]           History Software Agent          : Adobe Photoshop Lightroom 5.0 (Windows), Adobe Photoshop Lightroom 5.0 (Windows), Adobe Photoshop Lightroom 5.3 (Windows)
[XMP]           History Changed                 : /metadata, /metadata, /
[XMP]           History Parameters              : converted from image/x-nikon-nef to image/jpeg, saved to new location
[XMP]           Derived From Instance ID        : xmp.iid:ec3ae5ed-6624-134c-8077-088616062633
[XMP]           Derived From Document ID        : 798712531BC82BA9A9576912A801ACD7
[XMP]           Derived From Original Document ID: 798712531BC82BA9A9576912A801ACD7
[XMP]           Title                           : Fleur de Praying Mantis attack
[XMP]           Creator                         : Rob Cole
[XMP]           Subject                         : LittleWhiteFlowers, ObjectDeArt, PrayingMantis, RobColesWebsitePersonal, Websites
[XMP]           Creator Work URL                : www.robcole.com
[XMP]           Version                         : 8.3
[XMP]           Process Version                 : 6.7
[XMP]           White Balance                   : Custom
[XMP]           Auto White Version              : 134348800
[XMP]           Color Temperature               : 4300
[XMP]           Tint                            : +7
[XMP]           Saturation                      : +1
[XMP]           Sharpness                       : 50
[XMP]           Luminance Smoothing             : 10
[XMP]           Color Noise Reduction           : 25
[XMP]           Vignette Amount                 : 0
[XMP]           Shadow Tint                     : 0
[XMP]           Red Hue                         : +5
[XMP]           Red Saturation                  : -2
[XMP]           Green Hue                       : -2
[XMP]           Green Saturation                : +3
[XMP]           Blue Hue                        : -5
[XMP]           Blue Saturation                 : +15
[XMP]           Vibrance                        : +2
[XMP]           Hue Adjustment Red              : +5
[XMP]           Hue Adjustment Orange           : 0
[XMP]           Hue Adjustment Yellow           : +10
[XMP]           Hue Adjustment Green            : +5
[XMP]           Hue Adjustment Aqua             : 0
[XMP]           Hue Adjustment Blue             : 0
[XMP]           Hue Adjustment Purple           : 0
[XMP]           Hue Adjustment Magenta          : +5
[XMP]           Saturation Adjustment Red       : +5
[XMP]           Saturation Adjustment Orange    : +5
[XMP]           Saturation Adjustment Yellow    : +10
[XMP]           Saturation Adjustment Green     : +3
[XMP]           Saturation Adjustment Aqua      : -2
[XMP]           Saturation Adjustment Blue      : 0
[XMP]           Saturation Adjustment Purple    : 0
[XMP]           Saturation Adjustment Magenta   : 0
[XMP]           Luminance Adjustment Red        : 0
[XMP]           Luminance Adjustment Orange     : 0
[XMP]           Luminance Adjustment Yellow     : 0
[XMP]           Luminance Adjustment Green      : 0
[XMP]           Luminance Adjustment Aqua       : 0
[XMP]           Luminance Adjustment Blue       : 0
[XMP]           Luminance Adjustment Purple     : 0
[XMP]           Luminance Adjustment Magenta    : 0
[XMP]           Split Toning Shadow Hue         : 0
[XMP]           Split Toning Shadow Saturation  : 0
[XMP]           Split Toning Highlight Hue      : 0
[XMP]           Split Toning Highlight Saturation: 0
[XMP]           Split Toning Balance            : 0
[XMP]           Parametric Shadows              : 0
[XMP]           Parametric Darks                : 0
[XMP]           Parametric Lights               : +10
[XMP]           Parametric Highlights           : -20
[XMP]           Parametric Shadow Split         : 25
[XMP]           Parametric Midtone Split        : 50
[XMP]           Parametric Highlight Split      : 90
[XMP]           Sharpen Radius                  : +1.0
[XMP]           Sharpen Detail                  : 0
[XMP]           Sharpen Edge Masking            : 0
[XMP]           Post Crop Vignette Amount       : 0
[XMP]           Grain Amount                    : 0
[XMP]           Luminance Noise Reduction Detail: 85
[XMP]           Color Noise Reduction Detail    : 65
[XMP]           Luminance Noise Reduction Contrast: 30
[XMP]           Color Noise Reduction Smoothness: 65
[XMP]           Lens Profile Enable             : 1
[XMP]           Lens Manual Distortion Amount   : 0
[XMP]           Perspective Vertical            : 0
[XMP]           Perspective Horizontal          : 0
[XMP]           Perspective Rotate              : 0.0
[XMP]           Perspective Scale               : 100
[XMP]           Perspective Aspect              : 0
[XMP]           Perspective Upright             : 0
[XMP]           Auto Lateral CA                 : 1
[XMP]           Exposure 2012                   : +0.70
[XMP]           Contrast 2012                   : +25
[XMP]           Highlights 2012                 : -30
[XMP]           Shadows 2012                    : +30
[XMP]           Whites 2012                     : +20
[XMP]           Blacks 2012                     : -20
[XMP]           Clarity 2012                    : +15
[XMP]           Defringe Purple Amount          : 0
[XMP]           Defringe Purple Hue Lo          : 30
[XMP]           Defringe Purple Hue Hi          : 70
[XMP]           Defringe Green Amount           : 0
[XMP]           Defringe Green Hue Lo           : 40
[XMP]           Defringe Green Hue Hi           : 60
[XMP]           Convert To Grayscale            : False
[XMP]           Tone Curve Name 2012            : Linear
[XMP]           Camera Profile                  : Nikon D300 Adobe Standard (RC Debright +CC)
[XMP]           Lens Profile Setup              : Custom
[XMP]           Lens Profile Name               : Adobe (Nikon AF-S VR Micro-Nikkor 105mm f/2.8G IF-ED)
[XMP]           Lens Profile Filename           : NIKON D3 (Nikon AF-S VR Micro-Nikkor 105mm f2.8G IF-ED) - RAW.lcp
[XMP]           Lens Profile Digest             : 34DC555F627CAEE8EB3D7602D6DC28BB
[XMP]           Lens Profile Distortion Scale   : 0
[XMP]           Lens Profile Chromatic Aberration Scale: 100
[XMP]           Lens Profile Vignetting Scale   : 80
[XMP]           Has Settings                    : True
[XMP]           Has Crop                        : False
[XMP]           Already Applied                 : True
[XMP]           Tone Curve PV2012               : 0, 0, 255, 255
[XMP]           Tone Curve PV2012 Red           : 0, 0, 255, 255
[XMP]           Tone Curve PV2012 Green         : 0, 0, 255, 255
[XMP]           Tone Curve PV2012 Blue          : 0, 0, 255, 255
[XMP]           Paint Correction What           : Correction
[XMP]           Paint Correction Amount         : 1.000000
[XMP]           Paint Correction Active         : true
[XMP]           Paint Correction Saturation     : 0.050000
[XMP]           Paint Correction Sharpness      : 0.100000
[XMP]           Paint Correction Brightness     : 0.000000
[XMP]           Paint Correction Hue            : 164.000000
[XMP]           Paint Correction Saturation     : 0.000000
[XMP]           Paint Correction Exposure 2012  : -0.025000
[XMP]           Paint Correction Contrast 2012  : -0.250000
[XMP]           Paint Correction Highlights 2012: -0.150000
[XMP]           Paint Correction Shadows 2012   : 0.000000
[XMP]           Paint Correction Clarity 2012   : 0.100000
[XMP]           Paint Correction Luminance Noise: 0.000000
[XMP]           Paint Correction Moire          : 0.000000
[XMP]           Paint Correction Defringe       : 0.000000
[XMP]           Paint Correction Temperature    : 0.000000
[XMP]           Paint Correction Tint           : 0.000000
[XMP]           Paint Correction Mask What      : Mask/Paint
[XMP]           Paint Correction Mask Value     : 1.000000
[XMP]           Paint Correction Mask Radius    : 0.006330
[XMP]           Paint Correction Mask Flow      : 0.700000
[XMP]           Paint Correction Mask Center Weight: 0.065044
[XMP]           Paint Correction Mask Dabs      : d 0.656950 0.555126
[XMP]           Paint Correction Mask Dabs      : d 0.656369 0.552683
[XMP]           Paint Correction Mask Dabs      : d 0.656269 0.549846
[XMP]           Paint Correction Mask Dabs      : d 0.656414 0.546995
[XMP]           Paint Correction Mask Dabs      : d 0.657057 0.544457
[XMP]           Paint Correction Mask Dabs      : d 0.657669 0.541754
[XMP]           Paint Correction Mask Dabs      : d 0.658508 0.539223
[XMP]           Paint Correction Mask Dabs      : d 0.659200 0.536560



Actually, I couldn't post all I could see as there was too much for the forum to accept.

[areohbee]

Thanks Alan,

I was using the command

exiftool -xmp -b asdf.jpg

which should work as well, no?

does that command also work for you?

Thanks again,
Rob

[Phil Harvey]

Hi Rob,

Alan is right, ExifTool reads the XMP just fine.  But I think you are probably trying to extract the XMP as a block.  In this file, the XMP in this file is split into 2 segments:

Code Select Expand

> exiftool -a -xmp ~/Desktop/20131024RC113633NB4506.jpg
XMP                             : (Binary data 5083 bytes, use -b option to extract)
XMP                             : (Binary data 101313 bytes, use -b option to extract)

ExifTool will extract both into your output file if you add the -a option. (Assuming you are using -b and redirecting the output to a file.)  ExifTool will read the combined output file without any difficulty, but some other utilities may not.  To fix this, use ExifTool to write anything to this file.  This will restructure the XMP into a single record that other utilities should be OK with.

Alternatively you could copy out the XMP as individual tags.  I would need more details about how you are extracting the information to make a recommendation.

- Phil

[areohbee]

Thanks guys,

So, problem was due to my misunderstanding of the -xmp option, which I assumed would round up *all* xmp segments, but only gets the first.

I've since switched to

exiftool -o asdf.xmp asdf.jpg

which works.

Apparently, exiftool is smart enough to realize by extension of output filename that the whole kit-n-kaboodle (all xmp) is desired.

I guess the -a option was the missing ingredient in my original recipe.

PS - the use case is this:
I'm extracting xmp from jpeg exported by Lightroom (with All metadata included), and re-applying it (as sidecar file) to the original raw, so I follow up the xmp extraction with:

exiftool -alreadyApplied=false asdf.xmp

so asdf.xmp can be read (by Lightroom) into asdf.raw to recover develop settings and other (xmp) metadata.

It seems to be working, so I guess the extra command (setting already-applied tag) straightens things out for Lr or something.

PPS - I think it would help to clarify the documentation - it's not so clear that -xmp will only get the first segment, and it's not clear that -a should be used to get all segments (I wouldn't consider additional xmp segments to be equivalent to duplicate tags).

Thanks again,
Rob

[Phil Harvey]

Hi Rob,

I've since switched to

exiftool -o asdf.xmp asdf.jpg

which works.

Apparently, exiftool is smart enough to realize by extension of output filename that the whole kit-n-kaboodle (all xmp) is desired.

This command is nearly identical to exiftool -tagsfromfile asdf.jpg asdf.cmp, but you should add -all:all to both (after the -o or -tagsFromFile option) to keep EXIF from commuting into XMP.  With this, the command will do exactly what you want, provided that all XMP tags are writable by ExifTool.

PPS - I think it would help to clarify the documentation - it's not so clear that -xmp will only get the first segment, and it's not clear that -a should be used to get all segments (I wouldn't consider additional xmp segments to be equivalent to duplicate tags).

There is nothing special about the -xmp tag.  Duplicate tags are suppressed by default unless -a is used.  This is not an exception.  I agree that one might not think about this in relation to the segmented XMP in a JPEG file, but the place to remind someone about this would be the Extra tags documentation (were the XMP block tag is documented).  If I did mention it there, would you have found it?

- Phil

[areohbee]

Hi Phil,

you should add -all:all to keep EXIF from commuting into XMP

Thanks for the tip. I'm pretty sure I'd not have got that one on my own
(how could I have figured that out on my own??)

Regarding -xmp tag and -a, I probably wouldn't have found the reminder in extra tags documentation, although I might next time 

Perhaps a good place to comment would be in the xmp examples (http://www.exiftool.org/metafiles.html) - in there could be a reference to extra tags documentation..

ExifTool is an amazingly powerful piece of software, but figuring out which options/tags to use can be extremely challenging. Is there an exiftool book you could recommend? Or some way to get a better feel for things? Too much of the time I am just guessing and trial+error'ing...

Rob

[Phil Harvey]

Hi Rob,

you should add -all:all to keep EXIF from commuting into XMP

Thanks for the tip. I'm pretty sure I'd not have got that one on my own
(how could I have figured that out on my own??)

By reading (and understanding) the first 2 paragraphs in the description of the -tagsFromFile option in the application documentation.

Regarding -xmp tag and -a, I probably wouldn't have found the reminder in extra tags documentation, although I might next time 

Perhaps a good place to comment would be in the xmp examples (http://www.exiftool.org/metafiles.html)

Good idea.  I've added a note there.  This extended XMP is a bit of a pain.  However, the specification states that the important information should go in the first segment.  Additional segments normally carry only Photoshop editing information, which really shouldn't be stored in the XMP to begin with.

ExifTool is an amazingly powerful piece of software, but figuring out which options/tags to use can be extremely challenging.

I agree. But exiftool is still simpler than the metadata it manages.

Is there an exiftool book you could recommend? Or some way to get a better feel for things? Too much of the time I am just guessing and trial+error'ing...

The application documentation is complete.  I admit it isn't easy reading, but it is the best source if you want to understand the exiftool application in detail.  This documentation is available in html, PDF, by running exiftool with no arguments, and by typing "man exiftool" or "perldoc exiftool" if you have done the full Perl installation.

- Phil

[herb]

Hello Rob, hello Phil,

this discussion is very interesting and also helpful for me, because I came to know "extended XMP" and how to handle it.

So please allow that I interrupt with some short technical questions:
- is there any technical reason that XMP must be split into a base-part and an extended-part?
- is there any situation for Exiftool to split the XMP section?
- is there any technical possibility to copy both XMP parts from one file into another file using the extra-tag -XMP?
  Both parts should be copied unchanged.
  Here I think about something like: exiftool -tagsfromfile source <file>  -a  -b  -XMP  "-copy1_of_XMP"  target

Thanks for your answers/comments in advance
Best regards
Herb

[areohbee]

Hi Herb,

is there any technical reason that XMP must be split into a base-part and an extended-part

Well, there are certainly reasons to split it into multiple physical blocks:
1. xmp size may exceed 65k, which is the max block size for some containers, thus xmp may be spread over multiple blocks.
2. initial xmp can be kept in the front of the file where it is quickest to access when not reading the whole file, with "extended" xmp coming after the image, thus one does not need to re-write the entire file (including image) in order to append xmp.

Why these segments would be distinct from an exiftool tag point of view, I'm not so sure.

is there any situation for Exiftool to split the XMP section?

Phil would be the best person to answer this question, but I believe the answer is contained by his previous statement "there is nothing special about the xmp tag, from the point of view of exif-tool", i.e. how it is is how it turns out when there is no special handling...

That said, I agree, it would be better from a usability standpoint if, by default, *all* xmp was returned, thus even those of us without a complete understanding would be successful with our initial attempts. A non-default option could then be used to get partial xmp, if such was desired. It's kinda like when you extract an embedded image - ya kinda expect to get the whole thing, even if stored in multiple segments...

Rob

[Phil Harvey]

Rob is correct about the 64k JPEG segment-size limit.  ExifTool will split the XMP if it exceeds 64kB.

I agree that it would be nice if ExifTool combined these back together when extracting with -xmp -b, but just concatenating them produces non-standard XMP, and doing anything else wouldn't be returning the original binary data, which is what -xmp -b does.

Life would be much simpler if Adobe didn't use XMP for its nefarious (and space-wasting) editing information.  I have talked with the head of XMP at Adobe about this and he agrees that it was a bad idea.

Let me think about this some more.

- Phil

[areohbee]

Thanks Phil - you are great! 

In case y'all aren't aware, whether camera-raw settings are included in the xmp of exported jpegs is now an option in Lr5 (maybe @Lr4). And if you are still using Lr3, then Jeffrey Friedl's Metadata Wrangler can suppress them.

Merry Ho-Ho and happy everything...

~R.

[areohbee]

By reading (and understanding) the first 2 paragraphs in the description of the -tagsFromFile option in the application documentation.

OK, I've re-read those two paragraphs about a 100 times, but I'm not sure I'll ever understand them without some pre-requisite clarifications:

"same-named tags in the preferred location"

Question 1: what is meant by "same-named". I mean, the output file does not exist yet, so "same" as what?

Question 2: what are the preferred locations. I *think* I may understand this one, sort-of, but let me check: it means tags that could be in xmp, but also could be in exif, for example, won't be written to xmp, but will be (or would be) written to exif instead (exif being "preferred" over xmp).

Thus the reason '-all:all' needs to be specified when writing an xmp sidecar (assuming *all* tags are desired in the xmp).

Note: it sounds like '-xmp:all>all:all' is "always" a good idea, if the objective is a complete/standard xmp sidecar file, e.g. as Adobe Lightroom would write it, and read it..

The explanation being, '-xmp:all' says "get all tags relevant to xmp from the source file" (and ignore the rest), and the '>all:all' says "map them to xmp, since we don't want exif tags being dropped or mis-mapped" ("commuted" was the word you used previously), as example.

How'd I do?

PS - I think it would help to make the documentation more understandable if there was a "concepts and definitions" section, i.e. what one needs to understand up front in order to make sense of the documentation. Sometimes when reading it I find myself going around and around and over and over it, and still never really understanding, because I'm lacking some key/critical concept and/or definition.

R

[Phil Harvey]

Question 1: what is meant by "same-named". I mean, the output file does not exist yet, so "same" as what?

Same name as another writable tag in a different group.  See the Tag Name documentation for a complete list.

[/quote]Question 2: what are the preferred locations. I *think* I may understand this one, sort-of, but let me check: it means tags that could be in xmp, but also could be in exif, for example, won't be written to xmp, but will be (or would be) written to exif instead (exif being "preferred" over xmp).[/quote]

From the exiftool documentation:

            TAG may contain one or more leading family 0, 1 or 2 group names,
            prefixed by optional family numbers, and separated colons.  If no
            group name is specified, the tag is created in the preferred
            group, and updated in any other location where a same-named tag
            already exists.  The preferred group is the first group in the
            following list where TAG is valid: 1) EXIF, 2) IPTC, 3) XMP.

The explanation being, '-xmp:all' says "get all tags relevant to xmp from the source file" (and ignore the rest), and the '>all:all' says "map them to xmp, since we don't want exif tags being dropped or mis-mapped" ("commuted" was the word you used previously), as example.

How'd I do?

Perfect.

PS - I think it would help to make the documentation more understandable if there was a "concepts and definitions" section, i.e. what one needs to understand up front in order to make sense of the documentation. Sometimes when reading it I find myself going around and around and over and over it, and still never really understanding, because I'm lacking some key/critical concept and/or definition.

This would definitely be useful, and I have tried writing similar documentation, but I haven't had any luck so far coming up with anything I think would be used/useful.

- Phil

[areohbee]

OK - I think it's starting to sink in.

Same name as another writable tag in a different group

Of course, in hind sight - sorta obvious..

Regarding this statement in -tagsFromFile app doc:

all possible tags from the source file are copied to same-named tags in the preferred location of the output file

If I may take a crack at an elaboration:

Not all tags necessarily have a samed-named tag and therefore not a preferred location either. In such cases tags will be "mirrored as-is" (copied verbatim). In cases where there is a same-named tag, what happens depends on whether one is creating a new xmp file (or in any case, tag does not exist in output file), versus updating an existing xmp file (and same-named tags exist in the output file). In the former case (new file or tag not existing in output file...), the tag will be written only once - regardless of how it's found in the source file, it will, by default, be mapped to a preferred location. In the latter case (same-named tags exist in output file, e.g. in non-preferred locations too), all existing tags of same name will be updated based on (set equal to) source tag value.

That sounds right to me, more or less (?).

And again, the purpose of the '-all:all' or '-xmp:all>all:all' is to prevent the default behavior of mapping to preferred location (and creating only once), in case one just wants things done as close as possible to how they were in the source file, right or wrong, preferred or not, duplication and all...

Still pretty-much on track?

PS - My current homework is trying to get a better handle on the distinction between "location" and "namespace" and/or "family" / "group".

PPS - I get that it's very hard to describe key concepts and define things out of context - you've done a great job Phil, which I appreciate immensely.

Thanks again for helping me/us get a handle on this stuff,
Rob