Problems updating lists using -tagsFromFile and ARGFILE

[davidh]

I have a short workflow to update RAW files with data from tags in an XMP sidecar file written from Lightroom. Typically the XMP file will contain Title, Caption (Description), User Comment and Keywords entered in Lightroom and not in the RAW file. It will also typically contain GPS data that might not be in the RAW file. Some of the Lightroom Keywords are imported Picasa Faces data. Aside: Phil, very interested in picasa_faces.config: great work by StarGeek and you!  .

My intention was to overwrite most of a small set of RAW file tags with information from the corresponding XMP file tags, but to add new Lightroom keywords from XMP-dc:Subject to IPTC:Keywords. That was because IPTC:Keywords might have been updated by another program.

The problem is that, after processing a file that already has been processed one or more times previously, there are two copies of all keywords in the RAW file's XMP-dc:Subject and XMP-lr:HierarchicalSubject. It seems that rather than overwriting these tags, i.e. processing them once, the command I'm using is processing them twice. It's not repeatedly adding one more copy on each execution. There is one copy of keywords after processing a RAW file that had none before and two copies after processing a RAW file that had one or two copies before. All other tags, including IPTC:Keywords, process correctly. (Hardly surprising for the value tags!)

I thought I understood FAQ17 and I also thought I had this working. I'm probably wrong on both counts as, in testing the workflow because of a completely unrelated issue of Lightroom Titles being truncated and international characters altered, my diffs showed up these issues. I've since tried dumping verbose output (-v2) to see what's going on, re-arranging my command line parameters and changing the arguments in the ARGFILE and still can't get it to do what I want in a single command.

I did most of my testing on OS X Yosemite with Exiftool 9.76, then re-did it and continued with more testing with Exiftool 9.90. FWIW, if it ever was working, it was on 9.69. Still, I think this is a user issue, not an Exiftool issue. I think my problem is in understanding the interaction of -tagsFromFile and the -execute in my ARGFILE.

The original command I used was
exiftool -@ ~/Documents/src/exiftool/LightroomXMPtoRAW.args \
-common_args -P -tagsFromFile /tmp/IMG_test.xmp \
/tmp/IMG_test.CR2

The file LightroomXMPtoRAW.args, with all comments and most blank lines removed, contains

Code Select Expand

-IPTC:Keywords -< XMP-dc:Subject

-execute

-GPS:all < XMP-exif:all
-GPS:GPSDateStamp < XMP-exif:GPSDateTime
-GPS:GPSTimeStamp < XMP-exif:GPSDateTime
-XMP-exif:GPSDateTime < XMP-exif:GPSDateTime

-XMP-dc:Subject < XMP-dc:Subject
-XMP-lr:HierarchicalSubject < XMP-lr:HierarchicalSubject
-IPTC:Keywords +< XMP-dc:Subject

-Exif:UserComment < XMP-exif:UserComment
-Exif:ImageDescription < XMP-dc:Description
-IPTC:Caption-Abstract < XMP-dc:Description
-XMP-dc:Description < XMP-dc:Description
-XMP-dc:Title < XMP-dc:Title
-IPTC:ObjectName < XMP-dc:Title
-IPTCDigest=new


After some testing I re-read the documentation on tagsFromFile, particularly Notes 4 and 5, and decided that my command ordering might be incorrect. I moved it out of the common arguments and placed it before -@ like this
exiftool -tagsFromFile /tmp/IMG_test.xmp \
-@ ~/Documents/src/exiftool/LightroomXMPtoRAW.args \
-common_args -P \
/tmp/IMG_test.CR2

That looked promising until I found that it appears to add the first tag in the RAW file IPTC:Keywords back into IPTC:Keywords on each execution.

This is already a long post and I hope the problem is already obvious from the information above, but I've included below some output to demonstrate the behaviour. I've extracted the tags being used from the XMP file and those being modified in the RAW file before and after executing the first command above, i.e.
exiftool -@ ~/Documents/src/exiftool/LightroomXMPtoRAW.args \
-common_args -P -tagsFromFile /tmp/IMG_test.xmp \
/tmp/IMG_test.CR2
and also included a diff of the tags from the RAW file.

Tags in XMP file before running command:

Code Select Expand

$ cat IMG_test_xmp_1.txt
---- XMP-dc ----
Subject                         : First Person, Location1, Picasa Faces
Title                           : First on Location1
---- XMP-exif ----
GPS Latitude                    : 48 deg 52' 9.30" N
GPS Longitude                   : 2 deg 18' 34.90" E
---- XMP-lr ----
Hierarchical Subject            : Location1, Picasa Faces|First Person
---- Composite ----
GPS Latitude Ref                : North
GPS Longitude Ref               : East


Tags in RAW file before running command:

Code Select Expand

$ cat IMG_test_CR2_1.txt
---- XMP-dc ----
Subject                         : First Person, Location1, Picasa Faces
Title                           : First on Location1
---- XMP-lr ----
Hierarchical Subject            : Location1, Picasa Faces|First Person
---- IPTC ----
Object Name                     : First on Location1
Keywords                        : First Person, Location1, Picasa Faces
---- ExifIFD ----
User Comment                    :
---- GPS ----
GPS Version ID                  : 2.2.0.0
GPS Latitude Ref                : North
GPS Latitude                    : 48 deg 52' 9.30"
GPS Longitude Ref               : East
GPS Longitude                   : 2 deg 18' 34.90"


Tags in RAW file after running command:

Code Select Expand

$ cat IMG_test_CR2_2.txt
---- XMP-dc ----
Subject                         : First Person, Location1, Picasa Faces, First Person, Location1, Picasa Faces
Title                           : First on Location1
---- XMP-lr ----
Hierarchical Subject            : Location1, Picasa Faces|First Person, Location1, Picasa Faces|First Person
---- IPTC ----
Object Name                     : First on Location1
Keywords                        : First Person, Location1, Picasa Faces
---- ExifIFD ----
User Comment                    :
---- GPS ----
GPS Version ID                  : 2.2.0.0
GPS Latitude Ref                : North
GPS Latitude                    : 48 deg 52' 9.30"
GPS Longitude Ref               : East
GPS Longitude                   : 2 deg 18' 34.90"


diff of tags in RAW files before and after running command:

Code Select Expand

$ diff IMG_test_CR2_[12].txt
2c2
< Subject                         : First Person, Location1, Picasa Faces
---
> Subject                         : First Person, Location1, Picasa Faces, First Person, Location1, Picasa Faces
5c5
< Hierarchical Subject            : Location1, Picasa Faces|First Person
---
> Hierarchical Subject            : Location1, Picasa Faces|First Person, Location1, Picasa Faces|First Person


One minor thing: I tried searching for the meaning of the + and - in the -v2 verbose but couldn't find a definition. It seems to me to indicate tags that are in the SRC and DST files respectively. Is that right? For example this is snipped from the log output from the first execution of the command above:

Code Select Expand

    - XMP-lr:HierarchicalSubject = 'Location1'
    - XMP-lr:HierarchicalSubject = 'Picasa Faces|First Person'
    + XMP-lr:HierarchicalSubject = 'Location1'
    + XMP-lr:HierarchicalSubject = 'Picasa Faces|First Person'


and this is snipped from the second execution:

Code Select Expand

    - XMP-lr:HierarchicalSubject = 'Location1'
    - XMP-lr:HierarchicalSubject = 'Picasa Faces|First Person'
    - XMP-lr:HierarchicalSubject = 'Location1'
    - XMP-lr:HierarchicalSubject = 'Picasa Faces|First Person'
    + XMP-lr:HierarchicalSubject = 'Location1'
    + XMP-lr:HierarchicalSubject = 'Picasa Faces|First Person'


Thanks in advance for any help on this.

[Phil Harvey]

Hi David,

Sorry for the delay in responding.  I didn't have time to answer such a long post until now.

The + and - in the -v output indicate tags that are added and removed from the target file.

I think your problem may be the -tagsFromFile ordering.  I have a couple of comments/questions:

1) I don't think you need the -execute at all, because I think you can do what you want in a single execution.

2) Your second command should remove XMP-dc:Subject values found in the XMP file, and copy the rest of the tags internally, which I don't think is what you want.  Instead, isn't this what you want?:

exiftool -tagsFromFile /tmp/IMG_test.xmp \
-@ ~/Documents/src/exiftool/LightroomXMPtoRAW.args \
-common_args -P \
/tmp/IMG_test.CR2

Code Select Expand

-GPS:all < XMP-exif:all
-GPS:GPSDateStamp < XMP-exif:GPSDateTime
-GPS:GPSTimeStamp < XMP-exif:GPSDateTime
-XMP-exif:GPSDateTime < XMP-exif:GPSDateTime

-XMP-dc:Subject < XMP-dc:Subject
-XMP-lr:HierarchicalSubject < XMP-lr:HierarchicalSubject
-IPTC:Keywords -< XMP-dc:Subject
-IPTC:Keywords +< XMP-dc:Subject

-Exif:UserComment < XMP-exif:UserComment
-Exif:ImageDescription < XMP-dc:Description
-IPTC:Caption-Abstract < XMP-dc:Description
-XMP-dc:Description < XMP-dc:Description
-XMP-dc:Title < XMP-dc:Title
-IPTC:ObjectName < XMP-dc:Title
-IPTCDigest=new

I'm not 100% sure about this because I don't understand why your list items were duplicated by either command, and I'm not sure about exactly what you are trying to do (and why you wanted to use -execute in the first place).

- Phil