Using Exiftool to work around Apple Photos bug over many files with sidecars

[tmichael]

I have a situation where Apple Photos (Big Sur) will not export JPEG with keywords (known bug). Additionally, when exporting in TIFF format, while it does export keywords, it is dropping the GPS info. What I have learned is that if I export the unedited original, but select the "Export IPTC as XMP", then I get an XMP sidecar file containing both the keyword info and the GPS info.

Exiftool does a phenomenal job of allowing me to then add this info back into the edited file using a command like this:
exiftool -tagsfromfile img0001.xmp -all:all img0001.tiff
or
exiftool -tagsfromfile img0001.xmp -all:all img0001.jpeg

My question is this:
Assuming I have a directory with a thousand of images and a separate xmp sidecar for each, that will effectively be the equivalent of running these 1000 commands:
exiftool -tagsfromfile img0001.xmp -all:all img0001.tiff
exiftool -tagsfromfile img0002.xmp -all:all img0002.tiff
exiftool -tagsfromfile img0003.xmp -all:all img0003.tiff
...
exiftool -tagsfromfile img1000.xmp -all:all img1000.tiff
?

I can imagine using some automation tool to generate the 1000 individual commands, but would prefer to use exiftool's capability if it has it. I have read this:
15. Copy XMP from sidecar files back to the same locations in the source files:

(the inverse of the previous command)

exiftool -ext EXT -tagsfromfile %d%f.xmp -all:all -r DIR
Here, -all:all copies all metadata (in this case only XMP, since the sidecar XMP file contains no other types) to the same specific locations in the target files (extension EXT).

But this writes the info from 1 xmp file to all of the TIFF files, whereas I am trying to pull img0001.xmp into img0001.tiff, img0002.xmp into img0002.tiff, etc.

[StarGeek]

15. Copy XMP from sidecar files back to the same locations in the source files:

(the inverse of the previous command)

exiftool -ext EXT -tagsfromfile %d%f.xmp -all:all -r DIR
Here, -all:all copies all metadata (in this case only XMP, since the sidecar XMP file contains no other types) to the same specific locations in the target files (extension EXT).

But this writes the info from 1 xmp file to all of the TIFF files, whereas I am trying to pull img0001.xmp into img0001.tiff, img0002.xmp into img0002.tiff, etc.

Something else must be happening, as that is the basic command to recursively write the XMP tags into the matching image file. It won't write img0001.xmp into img0002.tiff because the %f stands for the base filename, which would be img0001 for img0001.tiff, img0002 for img0002.tiff, etc.

Just to clarify, you are changing -ext (-extension) option to match the extensions you want to copy?  Alternatively, you could --ext xmp to match all files except XMP sidecars, as that would copy the XMP files back onto themselves.

You are using a Mac and not running this as a bat file from Windows, correct?

Here's an example.  The directory has 8 image files and 8 matching xmp sidecars.  The image files do not have a Subject tag and the XMP files do have one containing the base name of the file.  After running the command, all the image files now have the Subject tag with the data from the sidecar file.

Code Select Expand

C:\>exiftool -fileorder filename -G1 -a -s -subject Y:\!temp\ccccc\e
======== Y:/!temp/ccccc/e/img0001.jpg
======== Y:/!temp/ccccc/e/img0001.xmp
[XMP-dc]        Subject                         : img0001
======== Y:/!temp/ccccc/e/img0002.jpg
======== Y:/!temp/ccccc/e/img0002.xmp
[XMP-dc]        Subject                         : img0002
======== Y:/!temp/ccccc/e/img0003.jpg
======== Y:/!temp/ccccc/e/img0003.xmp
[XMP-dc]        Subject                         : img0003
======== Y:/!temp/ccccc/e/img0004.jpg
======== Y:/!temp/ccccc/e/img0004.xmp
[XMP-dc]        Subject                         : img0004
======== Y:/!temp/ccccc/e/img0005.jpg
======== Y:/!temp/ccccc/e/img0005.xmp
[XMP-dc]        Subject                         : img0005
======== Y:/!temp/ccccc/e/img0006.jpg
======== Y:/!temp/ccccc/e/img0006.xmp
[XMP-dc]        Subject                         : img0006
======== Y:/!temp/ccccc/e/img0007.jpg
======== Y:/!temp/ccccc/e/img0007.xmp
[XMP-dc]        Subject                         : img0007
======== Y:/!temp/ccccc/e/img0008.jpg
======== Y:/!temp/ccccc/e/img0008.xmp
[XMP-dc]        Subject                         : img0008
    1 directories scanned
   16 image files read

C:\>exiftool --ext xmp -tagsfromfile %d%f.xmp -all:all Y:\!temp\ccccc\e
    1 directories scanned
    8 image files updated

C:\>exiftool -fileorder filename -G1 -a -s -subject Y:\!temp\ccccc\e
======== Y:/!temp/ccccc/e/img0001.jpg
[XMP-dc]        Subject                         : img0001
======== Y:/!temp/ccccc/e/img0001.xmp
[XMP-dc]        Subject                         : img0001
======== Y:/!temp/ccccc/e/img0002.jpg
[XMP-dc]        Subject                         : img0002
======== Y:/!temp/ccccc/e/img0002.xmp
[XMP-dc]        Subject                         : img0002
======== Y:/!temp/ccccc/e/img0003.jpg
[XMP-dc]        Subject                         : img0003
======== Y:/!temp/ccccc/e/img0003.xmp
[XMP-dc]        Subject                         : img0003
======== Y:/!temp/ccccc/e/img0004.jpg
[XMP-dc]        Subject                         : img0004
======== Y:/!temp/ccccc/e/img0004.xmp
[XMP-dc]        Subject                         : img0004
======== Y:/!temp/ccccc/e/img0005.jpg
[XMP-dc]        Subject                         : img0005
======== Y:/!temp/ccccc/e/img0005.xmp
[XMP-dc]        Subject                         : img0005
======== Y:/!temp/ccccc/e/img0006.jpg
[XMP-dc]        Subject                         : img0006
======== Y:/!temp/ccccc/e/img0006.xmp
[XMP-dc]        Subject                         : img0006
======== Y:/!temp/ccccc/e/img0007.jpg
[XMP-dc]        Subject                         : img0007
======== Y:/!temp/ccccc/e/img0007.xmp
[XMP-dc]        Subject                         : img0007
======== Y:/!temp/ccccc/e/img0008.jpg
[XMP-dc]        Subject                         : img0008
======== Y:/!temp/ccccc/e/img0008.xmp
[XMP-dc]        Subject                         : img0008
    1 directories scanned
   16 image files read

[tmichael]

Thank you for the informative reply. Because I am running exiftool in the directory I turned it into this command:

Code Select Expand

exiftool --ext xmp -tagsfromfile %d%f.xmp -all:all .

Of course with one problem solved, then I come to the next. Apple photos puts out the GPS info into XPS sidecar file like this:
Western Hemisphere location

Code Select Expand

<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 6.0.0">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:exif="http://ns.adobe.com/exif/1.0/"
            xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/"
            xmlns:dc="http://purl.org/dc/elements/1.1/">
         <exif:GPSLongitude>97.747501600000007</exif:GPSLongitude>
         <exif:GPSLongitudeRef>W</exif:GPSLongitudeRef>
         <exif:GPSHPositioningError>1</exif:GPSHPositioningError>
         <exif:GPSLatitudeRef>N</exif:GPSLatitudeRef>
         <exif:GPSLatitude>30.264264300000001</exif:GPSLatitude>
         <exif:GPSTimeStamp>2021-06-21T21:28:16Z</exif:GPSTimeStamp>
         <photoshop:DateCreated>2010-12-31T23:00:00-06:00</photoshop:DateCreated>
         <dc:subject>
            <rdf:Seq>
               <rdf:li>Image</rdf:li>
               <rdf:li>Scanned 6/17/21</rdf:li>
            </rdf:Seq>
         </dc:subject>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>

Eastern Hemisphere location

Code Select Expand

<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 6.0.0">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:exif="http://ns.adobe.com/exif/1.0/"
            xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/"
            xmlns:dc="http://purl.org/dc/elements/1.1/">
         <exif:GPSLongitude>116.4008194</exif:GPSLongitude>
         <exif:GPSLongitudeRef>E</exif:GPSLongitudeRef>
         <exif:GPSHPositioningError>1</exif:GPSHPositioningError>
         <exif:GPSLatitudeRef>N</exif:GPSLatitudeRef>
         <exif:GPSLatitude>39.9027484</exif:GPSLatitude>
         <exif:GPSTimeStamp>2021-06-21T21:28:31Z</exif:GPSTimeStamp>
         <photoshop:DateCreated>2010-12-31T23:00:00-06:00</photoshop:DateCreated>
         <dc:subject>
            <rdf:Seq>
               <rdf:li>Image</rdf:li>
               <rdf:li>Scanned 6/17/21</rdf:li>
            </rdf:Seq>
         </dc:subject>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>

Exif Tool interprets this as follows:
Western Hemisphere location

Code Select Expand

======== ./East West - 1.xmp
[ExifTool]      ExifToolVersion                 : 12.27
[System]        FileName                        : East West - 1.xmp
[System]        Directory                       : .
[System]        FileSize                        : 1031 bytes
[System]        FileModifyDate                  : 2021:06:21 16:41:41-05:00
[System]        FileAccessDate                  : 2021:06:21 16:41:42-05:00
[System]        FileInodeChangeDate             : 2021:06:21 16:41:41-05:00
[System]        FilePermissions                 : -rw-r--r--
[File]          FileType                        : XMP
[File]          FileTypeExtension               : xmp
[File]          MIMEType                        : application/rdf+xml
[XMP-x]         XMPToolkit                      : XMP Core 6.0.0
[XMP-exif]      GPSLongitude                    : 97 deg 44' 51.01" E
[XMP-exif]      GPSLongitudeRef                 : W
[XMP-exif]      GPSHPositioningError            : 1 m
[XMP-exif]      GPSLatitudeRef                  : N
[XMP-exif]      GPSLatitude                     : 30 deg 15' 51.35" N
[XMP-exif]      GPSDateTime                     : 2021:06:21 21:28:16Z
[XMP-photoshop] DateCreated                     : 2010:12:31 23:00:00-06:00
[XMP-dc]        Subject                         : Image, Scanned 6/17/21
[Composite]     GPSLatitudeRef                  : North
[Composite]     GPSLongitudeRef                 : East
[Composite]     GPSPosition                     : 30 deg 15' 51.35" N, 97 deg 44' 51.01" E

Eastern Hemisphere location

Code Select Expand

[ExifTool]      ExifToolVersion                 : 12.27
[System]        FileName                        : East West - 2.xmp
[System]        Directory                       : .
[System]        FileSize                        : 1016 bytes
[System]        FileModifyDate                  : 2021:06:21 16:41:41-05:00
[System]        FileAccessDate                  : 2021:06:21 16:41:42-05:00
[System]        FileInodeChangeDate             : 2021:06:21 16:41:41-05:00
[System]        FilePermissions                 : -rw-r--r--
[File]          FileType                        : XMP
[File]          FileTypeExtension               : xmp
[File]          MIMEType                        : application/rdf+xml
[XMP-x]         XMPToolkit                      : XMP Core 6.0.0
[XMP-exif]      GPSLongitude                    : 116 deg 24' 2.95" E
[XMP-exif]      GPSLongitudeRef                 : E
[XMP-exif]      GPSHPositioningError            : 1 m
[XMP-exif]      GPSLatitudeRef                  : N
[XMP-exif]      GPSLatitude                     : 39 deg 54' 9.89" N
[XMP-exif]      GPSDateTime                     : 2021:06:21 21:28:31Z
[XMP-photoshop] DateCreated                     : 2010:12:31 23:00:00-06:00
[XMP-dc]        Subject                         : Image, Scanned 6/17/21
[Composite]     GPSLatitudeRef                  : North
[Composite]     GPSLongitudeRef                 : East
[Composite]     GPSPosition                     : 39 deg 54' 9.89" N, 116 deg 24' 2.95" E


So despite the XMP file containing differences:
         <exif:GPSLongitudeRef>W</exif:GPSLongitudeRef>
         <exif:GPSLongitudeRef>E</exif:GPSLongitudeRef>
Both longitudes get interpreted as E (Eastern Hemisphere).

Any suggestion on what I should do to correct this?

[Phil Harvey]

This is a bug in the way Apple writes the XMP.  According to the specification, the XMP GPSLatitude/Longitude tags have the reference directions included instead of in a separate tag.

For this XMP, you should ignore the Composite GPS tags.

I suppose maybe I should add yet another ExifTool patch due to stupid programmers who can't read the specifications.  But I can't blame them entirely -- the guys who wrote the specification also need a few more brain cells amongst them.

- Phil

[StarGeek]

(a little slow, but posting anyway)

Oh yeah, that's right.  Apple ignores the standard for GPS data.  The reference direction in an XMP GPS tag is supposed to be included in the data, not held separately.  It's EXIF data that separates the coordinates and the reference.

So that means extra assignments need to be added to fix the problem.  Try this
exiftool --ext xmp -tagsfromfile %d%f.xmp -all:all '-XMP:GPSLatitude<$xmp:GPSLatitude# $xmp:GPSLatitudeRef' '-XMP:GPSLongitude<$XMP:GPSLongitude# $xmp:GPSLongitudeRef' .

Because subsequent assignments will overwrite previous assignments, we can overwrite the GPS coordinates that would have been assigned by All:All.  Because the XMP ref tags are non-standard, exiftool won't copy those over, so you don't have to worry about problems with them later.

I would also suggest adding this to the command to copy the GPS coordinates into the more universal EXIF tags.  Place it after all previous assignments so it will take precedence
'-GPSLatitude*<$xmp:GPSLatitude# $xmp:GPSLatitudeRef' '-GPSLongitude<$XMP:GPSLongitude# $xmp:GPSLongitudeRef'

So a complete command would be
exiftool --ext xmp -tagsfromfile %d%f.xmp -all:all '-XMP:GPSLatitude<$xmp:GPSLatitude# $xmp:GPSLatitudeRef' '-XMP:GPSLongitude<$XMP:GPSLongitude# $xmp:GPSLongitudeRef' '-GPSLatitude*<$xmp:GPSLatitude# $xmp:GPSLatitudeRef' '-GPSLongitude<$XMP:GPSLongitude# $xmp:GPSLongitudeRef' .

[Phil Harvey]

For reference, here is the Exif 2.32 for XMP specification:

A.2.4.4 GPSCoodinate
Value type of GPSCoodinate is a Text value in the form "DDD,MM,SSk" or "DDD,MM.mmk", where:
  - DDD is a number of degrees
  - MM is a number of minutes
  - SS is a number of seconds
  - mm is a fraction of minutes
  - k is a single character N, S, E, or W indicating a direction (north, south, east, west)
Leading zeros are not necessary for the for DDD, MM, and SS values. The DDD,MM.mmk form should be used when any of the native Exif component rational values has a denominator other than 1. There can be any number of fractional digits.

So not only did they get the "k" wrong, they wrote the value as DDD.ddddddd, which is also incorrect.  However, ExifTool is lenient about this one.

- Phil

[tmichael]

Thank you both. There is no reason that you all and Exiftool should have to account for Apple's failures, but I guess that is the advantage of being a mega-corporation. You get to do what you want most of the time and make others adapt to you.

[wywh]

So a complete command would be
exiftool --ext xmp -tagsfromfile %d%f.xmp -all:all '-XMP:GPSLatitude<$xmp:GPSLatitude# $xmp:GPSLatitudeRef' '-XMP:GPSLongitude<$XMP:GPSLongitude# $xmp:GPSLongitudeRef' '-GPSLatitude*<$xmp:GPSLatitude# $xmp:GPSLatitudeRef' '-GPSLongitude<$XMP:GPSLongitude# $xmp:GPSLongitudeRef' .

One * was missing. FTFY:

exiftool --ext xmp -tagsfromfile %d%f.xmp -all:all '-XMP:GPSLatitude<$xmp:GPSLatitude# $xmp:GPSLatitudeRef' '-XMP:GPSLongitude<$XMP:GPSLongitude# $xmp:GPSLongitudeRef' '-GPSLatitude*<$xmp:GPSLatitude# $xmp:GPSLatitudeRef' '-GPSLongitude*<$XMP:GPSLongitude# $xmp:GPSLongitudeRef' .

- Matti