How to insert every sidecar metadata created in Photos.app into the natives

[Grouaaah]

Hello,

I would like some help please, I am trying to export my Photos.app library keeping every metadata created in the app. I heard Photos created a XMP Sidecar => "Export IPTC as XPM" option

I would like to merge this metadata into the original file. (Not for the faces or keywords, but for the GPS info). Would Exiftool do this easy for ~7,000 pictures or it would be better to use something like GraphicConverter to do it?

Is there any important information I need to know before doing this large operation on my full library without losing anything please?

Thank you very much

Kind regards

[StarGeek]

To copy GPS data from an XMP sidecar back into a file, you would use the XMP2GPS.args[/url] file that can be found on GitHub.

Your command would be something like
exiftool --ext xmp -TagsFromFile %d%f.xmp -@ /path/to/xmp2gps.args /path/to/files/

There are other ARGS files there to copy data between different groups.

You can find example commands for dealing with sidecar files on the Metadata Sidecar Files page.

[wywh]

Photos.app can export original pixel data with updated metadata in .xmp sidecar files via Photos > [select images and movies] > File > Export > Export Unmodified Originals > Export IPTC as XMP.

You can join the updated metadata in the .xmp to the original images and movies with GraphicConverter 12 (or exiftool): GraphicConverter > File > Browse... > [select folder] > [select files] > IPTC/XMP menu > XMP Sidecar File... > Copy into JPEGs, TIFFs, PNGs, webp, HEICs, MOVs, MP4s, M4Vs. Notice that GraphicConverter deletes movie .xmp files so you might want to make a backup before that command. exiftool commands in the links below.

Also notice that macOS 14 Sonoma and earlier Photos exports non-standard .xmp GPS but GraphicConverter 12 and exiftool have a workaround for that (otherwise western and southern locations are put to the other side of the globe!).

Also notice that Photos can not properly import .xmp sidecars: it ignores GPS in images and silently fails to import movies with sidecar .xmp files.

So it is best to use those 3rd party tools to join .xmp to the originals and then delete .xmp files before importing to Photos.

https://exiftool.org/forum/index.php?topic=12634.msg68327#msg68327
https://exiftool.org/forum/index.php?topic=14943.msg80546#msg80546
https://discussions.apple.com/docs/DOC-250002750

- Matti

[Grouaaah]

Hello Matti,

Thank you so much for your answer, it works pretty well with GraphicConverter for photos AND videos. Your support is always pretty much appreciated!

I would have a few more questions before running my workflow please? Thank you again for your help


May I ask what is not working for videos, despite the fact that GC12 deletes the Sidecar file when merging?

And is there any difference between Exiftool workflow functions, and GC12 worflow you showed me, when trying to merge every data in every file? Maybe exiftool xmp2gps args are to use if you need a specific metadata at a specific place (which is not my case)?
Because GC12 seems to do the work perfectly (it even adds additional XMP tags!)

But I did not test the pixel loss. I trust you if you tell me there is no loss

About this...

Also notice that Photos can not properly import .xmp sidecars: it ignores GPS in images and silently fails to import movies with sidecar .xmp files.

I guess it is not a big deal since every XMP sidecar file can be safely backed up (or thrown away) after the final import in Photos.app?

And to answer to Stargeek, also thank you for your support,

There are other ARGS files there to copy data between different groups.

if achieving the same outcome with GC12 takes less time than executing multiple commands to merge data (and generating new tags if needed), wouldn't it be better to utilize the GC12 workflow, as it aligns with my requirements? Or should I still consider learning to run the commands regardless, as it provides insight into the actual edits made to my files?
Thank you

... To conclude

Also notice that macOS 14 Sonoma and earlier Photos exports non-standard .xmp GPS but GraphicConverter 12 and exiftool have a workaround for that (otherwise western and southern locations are put to the other side of the globe!).

Do you mean that Photos.app (still) exports the data in the sidecar XMP with a certain standard which is not convenient, and as long as the GC12 workflow you shared me bypasses this, I do not have to worry about that problem for now?

Again, thank you very much.

[wywh]

That flawed Photos .xmp can be fixed with exiftool:

Code Select Expand

exiftool -P -overwrite_original -ext xmp -XMP-exif:All= -tagsFromFile @ -XMP-exif:All '-XMP-exif:GPSLongitude<${XMP-exif:GPSLongitude#}${XMP-exif:GPSLongitudeRef#}' '-XMP-exif:GPSLatitude<${XMP-exif:GPSLatitude#}${XMP-exif:GPSLatitudeRef#}' .
Then the following command can be used to copy .xmp to .jpg and .heic images (here to both XMP and IPTC, File dates use XMP-photoshop offset so they might be off):

Code Select Expand

exiftool -m -overwrite_original_in_place -ext jpg -ext heic '-IPTC:CodedCharacterSet=UTF8' -tagsFromFile %d%f.xmp '-AllDates<XMP-photoshop:DateCreated' '-ExifIFD:OffsetTime*<XMP-photoshop:DateCreated' '-XMP-photoshop:DateCreated<XMP-photoshop:DateCreated' '-EXIF:All<XMP-exif:All' '-GPS:GPSLatitudeRef<Composite:GPSLatitudeRef' '-GPS:GPSLongitudeRef<Composite:GPSLongitudeRef' '-XMP:All<XMP:All' '-IPTC:ObjectName<XMP-dc:Title' '-IPTC:Caption-Abstract<XMP-dc:Description' '-EXIF:ImageDescription<XMP-dc:Description' '-IPTC:Keywords<XMP-dc:Subject' '-FileCreateDate<XMP-photoshop:DateCreated' '-FileModifyDate<XMP-photoshop:DateCreated' .
...and to .mp4, .m4v and .mov movies:

Code Select Expand

exiftool -m -overwrite_original -api QuickTimeUTC=1 -ext mp4 -ext m4v -ext mov -tagsFromFile %d%f.xmp -api QuickTimeUTC=1 '-QuickTime:CreateDate<XMP-photoshop:DateCreated' '-QuickTime:ModifyDate<XMP-photoshop:DateCreated' '-Track*Date<XMP-photoshop:DateCreated' '-Media*Date<XMP-photoshop:DateCreated' '-Keys:CreationDate<XMP-photoshop:DateCreated' '-Keys:GPSCoordinates<$XMP:GPSLatitude#, $XMP:GPSLongitude#' '-Keys:Title<XMP-dc:Title' '-Keys:Description<XMP-dc:Description' '-Keys:Keywords<XMP-dc:Subject' '-FileCreateDate<XMP-photoshop:DateCreated' '-FileModifyDate<XMP-photoshop:DateCreated' .
So GraphicConverter 12 and exiftool can workaround the flawed .xmp that Photos.app exports essentially the same. Minor differences (to exiftool commands above) are that GraphicConverter does not write OffsetTime*, file dates and by default does not insert Keys:CreationDate. GraphicConverter copies the nonstandard XMP-exif:GPS*Ref tags (but all my apps ignore it and use GPS:GPS* tags where it is OK).

- Matti

[wywh]

edit: QuickTimeUTC option was slipped twice in my previous post's command. That does not really matter but it should be:

...and to .mp4, .m4v and .mov movies:

Code Select Expand

exiftool -m -overwrite_original -api QuickTimeUTC=1 -ext mp4 -ext m4v -ext mov -tagsFromFile %d%f.xmp '-QuickTime:CreateDate<XMP-photoshop:DateCreated' '-QuickTime:ModifyDate<XMP-photoshop:DateCreated' '-Track*Date<XMP-photoshop:DateCreated' '-Media*Date<XMP-photoshop:DateCreated' '-Keys:CreationDate<XMP-photoshop:DateCreated' '-Keys:GPSCoordinates<$XMP:GPSLatitude#, $XMP:GPSLongitude#' '-Keys:Title<XMP-dc:Title' '-Keys:Description<XMP-dc:Description' '-Keys:Keywords<XMP-dc:Subject' '-FileCreateDate<XMP-photoshop:DateCreated' '-FileModifyDate<XMP-photoshop:DateCreated' .
edit 2: GraphicConverter copies the nonstandard XMP-exif:GPS*Ref tags but also correctly writes references to XMP-exif:GPSLatitude and GPSLongitude to images so all apps read that OK (GPS:GPS* tags are OK and none of my apps use XMP locations anyway).

Code Select Expand

[XMP-exif]      GPSLatitude                     : -33.451855
[XMP-exif]      GPSLatitudeRef                  : S
[XMP-exif]      GPSLongitude                    : -70.6504633333333
[XMP-exif]      GPSLongitudeRef                 : W

[GPS]           GPSLatitude                     : 33.4518555555556
[GPS]           GPSLatitudeRef                  : S
[GPS]           GPSLongitude                    : 70.6504638888889
[GPS]           GPSLongitudeRef                 : W
GraphicConverter does not write OffsetTime*, file dates and does not insert Keys:CreationDate (those tags then might need another GraphicConverter command).

edit 3: On the other hand GraphicConverter copies that nonstandard .xmp location to movies (Keys:GPSCoordinates are OK). So XMP in movies should be disabled via GC > Settings... > Browser > Edit Metadata 1 > Movie (MOV, MP4, M4V and MTS) editing: XMP data: Do nothing.

None of my apps use XMP locations anyway but Adobe apps might do so. For example, Adobe Bridge 2024 puts the following .mp4 and .m4v to the other side of the globe while .mov is OK:

Code Select Expand

[XMP-exif]      GPSLatitude                     : 33.4518558
[XMP-exif]      GPSLatitudeRef                  : S
[XMP-exif]      GPSLongitude                    : 70.6504661
[XMP-exif]      GPSLongitudeRef                 : W

[Keys]          GPSCoordinates                  : -33.45186 -70.65047
-> So in summary both GC and exiftool can be used to copy .xmp to the originals. Just remember to disable XMP in movies in GC settings (I let someone else to nag about that to GC author).

- Matti

[Grouaaah]

Hello,

Thank you for these precisions and sorry for the late reply,

I tried to do some cleanup on the metadata while inserting every GPS Sidecar XMP data into the original file with GC12/ExifTool, but I ran into some problems. Here is my workflow:


1. XMP
Sidecar XMP Files > Copy to RAW Files
Sidecar XMP Files > Copy to JPEG Files, ..., ... => here, I got an error on a jpg with an indication "please rebuild metadata": Error: Bad format (0) for ExifIFD entry 24 - /User/.../20170402_125618.jpg
0 image files updated
1 files weren't updated due to errors
(it solved the problem after a rebuild)

2. EXIF
Exif Tags Orientation > Delete and Reset Duplicate Tags (JPEG only)
Exif Preview > Rebuild Preview (JPEG only)

3. IPTC/XMP
Keywords Field > Remove Duplicates
People Field > Remove Duplicates
Specific XMP Metadata > Check and Rewrite XMP GPS Tags

4. General Metadata
General Metadata > Rebuild (was is it doing actually?)
JPEG > Correct Restored JPEG File Size

Remove Trailing Zeros from JPEG Files
Specific Metadata > Remove Keywords, Rating, Label, and Finder Tags... (is this removing the tags I wanted to keep?)

5. Date
Finder Tags > Remove Finder Tags... (or maybe this is removing the tags I wanted to keep)
Extended Attributes (Xattr) > Remove All (those ones also?)
File Name Extension > Check and Correct





6. Date > Set Modification Date to Creation Date


7. Finish: manually open all files at a specific time to standardize "last opened" date before import into Photos.app




- After this workflow, I had some missing tags XMP tags, and XMP version switched from Core vX.XX.X to Exiftool vX.XX.X

- I wonder if some other data, above all, dates, have been touched during the process.
Maybe the modification dates, or some OffsetTime tags wouldn't be consistent anymore because of the fact that my computer was not set to the right timezone during the process, for pictures taken abroad?

- I read your guide entirely about Movie dates and Photos.app, and your articles about the "nonstandard XMP GPS info" but I am sorry, I still have trouble understanding why this command given by Phil Harvey himself solved your problem:

Code Select Expand

exiftool -xmp-exif:all= -tagsfromfile @ -xmp-exif:all \
  '-xmp-exif:gpslongitude<${xmp-exif:gpslongitude#}${xmp-exif:gpslongituderef#}' \
  '-xmp-exif:gpslatitude<${xmp-exif:gpslatitude#}${xmp-exif:gpslatituderef#}' \
  *.xmp
if the GC12 option "insert GPS data from Sidecar file" is enough to get the right data?

Because you recently said it worked for you to fix the GPS showing a position moved to the other side of the globe, and that GC12 has this workaround

— is this command included in the "insert GPS from Sidecar file" that GC12 offers?
... from what I understand, Photos is not affected by this sign inversion on the GPS metadata but if I want to have good data (i.e for Adobe Bridge) I must deactivate the insertion of GPS in the XMP for videos: but if we start from principle that I already have non-standard GPS in my videos, would placing the command provided by Phil solve the problem?

Because if I deactivate the option for videos about XMP data, how am I supposed to get my GPS data retrieved from Photos.app sidecars if the UserData contains no GPS info like this?

Code Select Expand

---- UserData ----
Date of creation of original data:
2024:01:27 09:58:46+03:00
Rental Information:
(none) Role=shooting Lat=-2.32469 Lon=34.73878 Alt=1420.50 Body=earth Notes=
User Rating:
0
Maker:
Canon

Sorry again, maybe my English is not good enough to understand the subtleties of this point

Thank you very much!!

[wywh]

I am very conservative with removing or adding any metadata to images and movies unless it is really needed.

None of my apps read XMP from movies so I try not to write XMP to movies. I don't use Adobe viewers (i.e. apps that read metadata) but AFAIK Adobe Bridge can read my non-XMP movie metadata like Keys:GPSCoordinates anyway. There might be XMP snippets in my movies, but I do not actively try to nuke it unless some app happens to read it causing unexpected behavior in my apps. If that happens, I check how widespread that problem is what is the best way to fix the offending files. I am cautious doing any massive global fixes to hundreds of files because it might produce unexpected issues that I might notice only after the originals have been deleted.

GC12 can handle .xmp sidecars just fine. The exiftool command that fixes Photos .xmp GPS takes care of that issue that might introduce issues with apps like Adobe's that read XMP GPS in movies.

Oh, maybe I should again ask GC author to fix that sidecar command to handle also movie XMP GPS the same way as he already fixed it to work with images. A year ago I was a PITA when I repeatedly nagged about this and that when he added more movie metadata support to GC (this and .m4v Spotlight metadata and a minor .m4v set time zone are the only remaining minor issues). Then I could make that explanation shorter and more clear.

> how am I supposed to get my GPS data retrieved from Photos.app sidecars if the UserData contains no GPS info like this

There is UserData "Rental Information" -- I have never seen that before. Some of my apps insert similar "LocationInformation" (AFAIR it is ffmpeg when trying to copy metadata). I try to ignore it but sometimes it makes Composite:GPSPosition unusable (the test movie below has almost all GPS tags deliberately set different):

Code Select Expand

exiftool -a -G1 -s -n -Location:All movie.mp4
[UserData]      GPSCoordinates                  : 36.6101 -66.91515 119.9
[ItemList]      GPSCoordinates                  : -36.6101 66.91515 119.9
[UserData]      LocationInformation             : 36.6101, 66.91515, 119.9 Role=shooting Lat=999.00000 Lon=999.00000 Alt=0.00 Body= Notes=
[Keys]          GPSCoordinates                  : -36.6101 -66.91515 119.9
[Keys]          LocationName                    : Keys:LocationName
[XMP-iptcCore]  Location                        : XMP-iptcCore:Location
[XMP-exif]      GPSAltitude                     : 119.9
[XMP-exif]      GPSAltitudeRef                  : 0
[XMP-exif]      GPSLatitude                     : 18.6101
[XMP-exif]      GPSLongitude                    : 33.91515
[XMP-exif]      GPSProcessingMethod             : MANUAL
[XMP-mediapro]  Location                        : XMP-mediapro:Location
[Composite]     GPSAltitude                     : 119.9
[Composite]     GPSAltitude                     : 119.9
[Composite]     GPSAltitude                     : 0
[Composite]     GPSAltitudeRef                  : 0
[Composite]     GPSAltitudeRef                  : 0
[Composite]     GPSLatitude                     : -36.6101
[Composite]     GPSLatitude                     : 999.00000
[Composite]     GPSLongitude                    : -66.91515
[Composite]     GPSLongitude                    : 999.00000
[Composite]     GPSLatitudeRef                  : N
[Composite]     GPSLongitudeRef                 : E
[Composite]     GPSPosition                     : 999.00000 999.00000
- Matti

[Grouaaah]

Hello Matti,

Thank you very much for your answer

I finally had time to test this on a copy of my full library and it doesn't work. I have made some feedback to GC authors.

I forgot about the other commands and tried:
1.XMP
Additional XMP files > Copy to RAW files
Additional XMP files > Copy to JPEG files, ..., ...

2. Specific XMP metadata > Check and rewrite XMP GPS tags

but I encountered an inconsistency in metadata;

The GPS Atomic time (UTC) written in XMP data from the Sidecar provided by Photos is totally random. Sometimes good, sometimes shift, sometimes even totally different from the creation date. It may come from GC algorithm, but I'm not sure. I cancelled my whole workflow from now, waiting for a durable solution to include every Sidecar file into my pictures without getting a random Atomic time in XMP. If maybe I could disable the insertion of this data... it could solve the problem, but I don't think so.

You were right about the other options I mentioned, I don't know why they're part of GraphicConverter solutions (there should be a reason) but they always alter the metadata in a way that is undesirable. For example, "File name extension > Check and fix" wrote a Green Finder tag on random JPEG and converted the extension to something like "MOP" format. Even though it didn't work on purpose it was interesting to see how my whole library reacts to this type of actions

To conclude, I'll say that I didn't notice any shifting in terms of GPS data (nonstandard data...) for videos like you mentioned, (maybe I didn't check enough) but the option that GC12 provides is not reliable because of this XMP GPS atomic time.

If only I could find an Exiftool option to do it correctly without encountering this...?

Or maybe find a way to delete this Atomic GPS XMP time which is random, after the insertion of the Sidecar with a GC/Exiftool command, but it would probably alter the consistency of the data...





(edit: Oh, and I first thought that it was the "Date/time creation uses UTC reference" option in "Browser > Metadata Edition (1)" that was unchecked (because my camera saves time with the timezone included, right?) but in fact it wasn't. It is another problem coming from somewhere else)

Thank you anyway,
if you have some Exiftool command to achieve this perfectly please feel free to tell me about

Kind regards,

[wywh]

> The GPS Atomic time (UTC) written in XMP data from the Sidecar provided by Photos is totally random.

If you mean XMP-exif:GPSDateTime, it seems to be the time when a location in Photos.app was edited. I usually pay no attention to GPSDateTime because it varies depending on when the device has managed to get GPS signal so maybe I'd just ignore or delete it. If that bothers you, you could delete that line from the exported .xmp before joining it to the image with GC. Or use a fine-tuned exiftool command to not copy it (with --XMP-exif:GPSDateTime, the following command needs fixed .xmp at least with western or southern GPS):

Code Select Expand

exiftool -m -overwrite_original -ext jpg -ext heic '-IPTC:CodedCharacterSet=UTF8' -tagsFromFile %d%f.xmp '-AllDates<XMP-photoshop:DateCreated' '-ExifIFD:OffsetTime*<XMP-photoshop:DateCreated' '-XMP-photoshop:DateCreated<XMP-photoshop:DateCreated' '-EXIF:All<XMP-exif:All' '-GPS:GPSLatitudeRef<Composite:GPSLatitudeRef' '-GPS:GPSLongitudeRef<Composite:GPSLongitudeRef' '-XMP:All<XMP:All' --XMP-exif:GPSDateTime '-IPTC:ObjectName<XMP-dc:Title' '-IPTC:Caption-Abstract<XMP-dc:Description' '-EXIF:ImageDescription<XMP-dc:Description' '-IPTC:Keywords<XMP-dc:Subject' '-FileCreateDate<XMP-photoshop:DateCreated' '-FileModifyDate<XMP-photoshop:DateCreated' .
...hmm if the location was edited in Photos.app, then the old GPSSpeed*, GPSImgDirection*, GPSDestBearing*, GPSHPositioningError, and GPSAltitude tags might be incorrect because .xmp does not include them...

> "File name extension > Check and fix" wrote a Green Finder tag on random JPEG and converted the extension to something like "MOP" format.

Yes, GraphicConverter adds a green Finder tag to the files where it has corrected the file suffix. Have you checked if the '.mop' file is somehow corrupted? For example, a .jpg is something like:

Code Select Expand

xxd -l 4 -p image.jpg
ffd8ffe0 (JFIF JPEG)
ffd8ffe1 (EXIF JPEG)
ffd8ffee (?)
ffd8ffdb (?)
https://en.wikipedia.org/wiki/List_of_file_signatures

> I didn't notice any shifting in terms of GPS data (nonstandard data...)

It happens only in western and southern locations because Photos.app writes those references incorrectly. GC has built-in workaround for that.

- Matti

[Grouaaah]

Hello and thank you for your answer,

I contacted GC support and told them about the XMP-Exif metadata deletion issue when using the "Copy into JPEG, MOV, ..." option. It is likely to be fixed in the next beta version with a Merge option.

Thank you for your precisions about everything. I don't worry anymore about using GC for this type of merging actions, but I'll wait until the version is released so the XMP-Exif data is not deleted when inserting GPS info.

But, IPTC data is deleted when merging, you can try by your own if you want
(You can download the beta on the website http://www.lemkesoft.org/beta.html)

I think I will wait until the release to merge my GPS data into the originals


Last question,

If you mean XMP-exif:GPSDateTime, it seems to be the time when a location in Photos.app was edited.

This bothers me a lot more because I hate having those useless info in my original file, that time it is not about the device having managed to get GPS signal at a moment or another... so I need to find a way to delete this specific data here for all my photos in once

Would you know how I could do this please?

Thank you very much,

Have a nice day,

[Grouaaah]

Do you think

Code Select Expand

exiftool -XMP-exif:GPSDateTime= .
will work? Thanks,

[Phil Harvey]

Yes

[Grouaaah]

Thank you all for your answer

So GraphicConverter 12 and exiftool can workaround the flawed .xmp that Photos.app exports essentially the same. Minor differences (to exiftool commands above) are that GraphicConverter does not write OffsetTime*, file dates and by default does not insert Keys:CreationDate. GraphicConverter copies the nonstandard XMP-exif:GPS*Ref tags (but all my apps ignore it and use GPS:GPS* tags where it is OK).

Hello, and sorry to relaunch this topic, I understand better the situation now and I had a question remaining before running the commands on my full library...

you said that the nonstandard references generated by Photos.app were about western and southern locations

and that GraphicConverter couldn't put the correct Xmp-Exif GPS data right into the videos (although it works for pictures only because of the embedded fix in "Merge into JPEG, RAW, ..." function).

But you provided me with this command:

That flawed Photos .xmp can be fixed with exiftool:

Code Select Expand

Code Select Expand

exiftool -P -overwrite_original -ext xmp -XMP-exif:All= -tagsFromFile @ -XMP-exif:All '-XMP-exif:GPSLongitude<${XMP-exif:GPSLongitude#}${XMP-exif:GPSLongitudeRef#}' '-XMP-exif:GPSLatitude<${XMP-exif:GPSLatitude#}${XMP-exif:GPSLatitudeRef#}' .

which corrects the format, and fixes the nonstandard files in any case, right?

My question is...

As the nonstandard fix is only included in the "Merge into JPEG, RAW, ..." option for PICTURES... could I just run the previous command on my full library, pictures and videos included, and then check the box in GC12: "insert XMP in videos" now that the .xmp files are all valid in order to get my correct GPS data in all tags for every file format?
instead of just deactivating XMP insertion in videos?


Thank you very much for your help

I told GC12 author about the XMP-exif:GPSDateTime issue too

Regards

[wywh]

>> flawed Photos .xmp can be fixed with exiftool:
> which corrects the format, and fixes the nonstandard files in any case, right?

Yes, using that .xmp fix does no harm but might very well be needed in some cases.

> could I just run the previous command on my full library, pictures and videos included, and then check the box in GC12: "insert XMP in videos" now that the .xmp files are all valid in order to get my correct GPS data in all tags for every file format? instead of just deactivating XMP insertion in videos?

Yes, but I don't insert XMP to movies because none of my apps read it and I try to avoid creating unnecessary tags that carry the same information.

> I told GC12 author about the XMP-exif:GPSDateTime issue too

Great. Was he willing to fix that also for movies when two users insist doing that

- Matti