Scan for Metadata Corruption - Apple photos issue

[vthokie]

Hi ExifTool Forum,

Important Background of my issue:  I've been working with senior iOS/iCloud support at Apple for the last 3 months trying to resolve an issue with my Photos Library (~ 17000 photos and 400 videos), whereby the photo library images/videos upload to iCloud, but will NOT download to a Windows machine (iCloud for Windows).  After countless hours of troubleshooting with Apple, deleting photos, libraries, install/uninstall software etc. their position is that I have photos with potentially corrupted metadata in them, which in turn is causing corruption to carry over to my account in their iCloud servers.   They are able to see corruption in my account on the server, but are not able to pin point which photo(s) are causing it.

Fast Forward to today, they had me export (drag/drop) all my photos from Photos.app (macOS) to a temp folder, and re-import it into a NEW photo library.  I now have over 1500+ photos with bare minimum metadata left, and all dates are replaced in them with a recent Date Modified stamp.  I've also lost all location info, tags etc. since it was a direct export vs. export with location, details.  They specifically don't want me to File > export all metadata to avoid carrying over any corruption.

So after hours of searching the net for solutions, i came across ExifTool and this looks promising! 

My Question - Is there a way to use ExifTool to run a scan across all my photos to validate their metadata for any corruption? I'm not even sure what "corruption" Apple thinks is in my Photos... special characters maybe?

Thanks much!

[Phil Harvey]

ExifTool does have an experimental -validate option that you could try.  The command-line in Terminal would look like this:

exiftool -validate -error -warning -a -r DIR

Also, you could use ExifTool to copy metadata from the "bad" library to the good one (if they have the same file names).

- Phil

[vthokie]

Thanks Phil! Very helpful.   I am running the experimental -validate option right now.  Make take some time to finish across 17,000 photos.   I'm see Warnings across several photos already.  Here are a couple samples... anything to be concerned about?   Is there an argument I could add to the -validate to have this export to a text file?  Might be easier to look for Errors that way...

Code: [Select]

======== /Users/G/Pictures/DSC_0001.jpg
Validate                        : 9 Warnings (3 minor)
Warning                         : Entries in IFD0 are out of order
Warning                         : Tag ID 0x010f out of sequence in IFD0
Warning                         : Tag ID 0x0128 out of sequence in IFD0
Warning                         : Tag ID 0x8769 out of sequence in IFD0
Warning                         : Entries in ExifIFD are out of order
Warning                         : Tag ID 0xa403 out of sequence in ExifIFD
Warning                         : [minor] Non-standard ExifIFD tag 0xea1c Padding
Warning                         : [minor] Non-standard ExifIFD tag 0xea1d OffsetSchema
Warning                         : [minor] Non-standard IFD0 tag 0xea1c Padding


Code: [Select]

======== /Users/G/Pictures/DSC00895.jpg
Validate                        : 5 Warnings (2 minor)
Warning                         : Entries in IFD0 are out of order
Warning                         : Tag ID 0x011b out of sequence in IFD0
Warning                         : Tag ID 0x8769 out of sequence in IFD0
Warning                         : [minor] Possibly incorrect maker notes offsets (fix by 48?)
Warning                         : [minor] Suspicious MakerNotes offset for ShotInfo


Code: [Select]

======== /Users/G/Pictures/CIMG2757.jpg
Validate                        : 10 Warnings
Warning                         : Wrong IFD for 0xa401 CustomRendered (should be ExifIFD not IFD0)
Warning                         : Wrong IFD for 0xa402 ExposureMode (should be ExifIFD not IFD0)
Warning                         : Wrong IFD for 0xa403 WhiteBalance (should be ExifIFD not IFD0)
Warning                         : Wrong IFD for 0xa404 DigitalZoomRatio (should be ExifIFD not IFD0)
Warning                         : Wrong IFD for 0xa405 FocalLengthIn35mmFormat (should be ExifIFD not IFD0)
Warning                         : Wrong IFD for 0xa406 SceneCaptureType (should be ExifIFD not IFD0)
Warning                         : Wrong IFD for 0xa407 GainControl (should be ExifIFD not IFD0)
Warning                         : Wrong IFD for 0xa408 Contrast (should be ExifIFD not IFD0)
Warning                         : Wrong IFD for 0xa409 Saturation (should be ExifIFD not IFD0)
Warning                         : Wrong IFD for 0xa40a Sharpness (should be ExifIFD not IFD0)


[Phil Harvey]

You can export to a single text file by adding "> out.txt" at the end of the command (without the quotes).

So far, the errors don't look very serious, and I don't expect any of them should cause problems for Photos.  The out-of-sequence tags aren't too uncommon, and the non-stanard tags mentioned are just junk that Microsoft has inserted.

- Phil

[vthokie]

It completed the scan.  Note: these were the photos that Apple had me drag/drop to a Temp folder and then reimport into a new photo library.  So most of the metadata (per Apple) is removed to begin with.  I'm going to File > Export Originals next into another temp folder and run ExifTool validate against those originals.  That would be a better test of the photo integrity.

Here are some other Warnings that came up.  Several hundred photos threw warnings, but they fall into category of the 3 examples I shared earlier, or the 2 below.

Code: [Select]

======== /Users/G/Pictures/Temp Photos/LR4-IMG_7938-.jpg
Validate                        : 1 Warning (minor)
Warning                         : [Minor] Extracted only 1000 crs:PaintBasedCorrectionsCorrectionMasksDabs items. Ignore minor errors to extract all


Code: [Select]

======== /Users/G/Pictures/Temp Photos/DSC03912 - 2007-11-10 at 03-58-25.jpg
Validate                        : 3 Warnings (2 minor)
Warning                         : [minor] Possibly incorrect maker notes offsets (fix by -76?)
Warning                         : [minor] Bad offset for MakerNotes tag 0xa200
Warning                         : JPEG format error


[Phil Harvey]

Hmm.  The JPEG format error should probably be an error, not a warning.  Could you email me this file so I can see? (philharvey66 at gmail.com)

The PaintBasedCorrectionsCorrectionMasksDabs is just a heavily edited file.  ExifTool limits the number of these that are extracted by default for performance reasons.

- Phil

[Phil Harvey]

I got the file, thanks.  The JPEG image data is definitely corrupted.  I'll have to think about whether or not I should make this an error in ExifTool, but it is definitely a big problem because the image isn't displayed properly.

- Phil

[vthokie]

Thanks for reviewing it Phil.  I found another file that had the same issue and deleted it from my library.   The issue i'm facing now is that Apple had me create a new Photo library a few weeks back and somewhere in the process ~ 2500 out of 17,000 photos had all their dates (create, modified) changed to 'Feb 19, 2018',  which if i recall was the date they had me do this troubleshooting step.   When I import these photos into a new .photoslibrary container, it makes all of these photos out of order.  I guess there is no way to go back to the original create date stamp on them ... since the files in the source are also modified to Feb 19, 2018

I ran a exiftool -s FILE against one of the source files, and all the exif data shows Feb 19, 2018 as the date stamps.  Ugh! 

And to add to it, there is a difference of 59 photos between the source library and the new library.  And Apple's Photos app won't tell me which files it fails to import.  Going to have to dig into this more using some kind of folder comparison.

[vthokie]

Hey Phil... I found a method to export those Feb 19, 2018 modified files out of Apple Photos where the new files actually preserves the 'DateCreated' information!   Now when I run a exiftool -s FILE it shows me the information i was expecting.   Here is an output...

Is this the right command to fix the modified date stamp to match the DateCreated?
exiftool "-filemodifydate<datetimeoriginal" DIR

Output of exiftool after Export as JPEG (vs. Unmodified Original)

Code: [Select]

MacBook:~ Gaurav$ exiftool -s /Users/G/Desktop/Temp/VT\ -\ 1999-11-18\ at\ 17-02-30\ \(1\).jpg
ExifToolVersion                 : 10.86
FileName                        : VT - 1999-11-18 at 17-02-30 (1).jpg
Directory                       : /Users/G/Desktop/Temp
FileSize                        : 624 kB
FileModifyDate                  : 2018:03:18 09:11:59-04:00
FileAccessDate                  : 2018:03:18 09:12:00-04:00
FileInodeChangeDate             : 2018:03:18 09:11:59-04:00
FilePermissions                 : rw-r--r--
FileType                        : JPEG
FileTypeExtension               : jpg
MIMEType                        : image/jpeg
JFIFVersion                     : 1.01
ResolutionUnit                  : None
ExifByteOrder                   : Big-endian (Motorola, MM)
Orientation                     : Horizontal (normal)
XResolution                     : 72
YResolution                     : 72
Software                        : Photos 3.0
ModifyDate                      : 1999:11:18 17:02:30
DateTimeOriginal                : 1999:11:18 17:02:30
CreateDate                      : 1999:11:18 17:02:30
ColorSpace                      : sRGB
ExifImageWidth                  : 963
ExifImageHeight                 : 619
XMPToolkit                      : XMP Core 5.4.0
CreatorTool                     : Photos 3.0
DateCreated                     : 1999:11:18 17:02:30
Subject                         : VT
Title                           : VT
CurrentIPTCDigest               : a783685429d66029cbc15ce76f1d4a3a
CodedCharacterSet               : UTF8
ApplicationRecordVersion        : 2
ObjectName                      : VT
Keywords                        : VT
IPTCDigest                      : a783685429d66029cbc15ce76f1d4a3a
ImageWidth                      : 963
ImageHeight                     : 619
EncodingProcess                 : Baseline DCT, Huffman coding
BitsPerSample                   : 8
ColorComponents                 : 3
YCbCrSubSampling                : YCbCr4:4:4 (1 1)
ImageSize                       : 963x619
Megapixels                      : 0.596


[Phil Harvey]

I don't know what you mean by "DateCreated".  On OSX, you can access the filesystem creation date/time via the MDItemFSCreationDate tag, but you need to have the "setfile" utility for writing (see here for details).

- Phil

[vthokie]

Ah, my bad.  It's "CreateDate"... see field in the output i provided earlier.

I'm going to try two scenarios today/tomorrow -

1) Scenario 1 - using setfile / MDItemFSCreationDate tag  per your recommendation

2) Scenario 2 - going to my source photolibrary, and exporting only the items modified on Feb 19, 2018 to JPEG and re-import them into the NEW photolibrary.  I found that I can create a Smart Album to filter on just these erroneously modified files, so I don't have to sift through 1000's of photos for them.  Interestingly all of these photos are JPEG format to begin with, but there must be something funky in how Apple exports them out as 'Unmodified Originals' that is stripping their metadata - especially the 'CreateDate' field.

Will share results once I get through these tests.   Thank you thank you for taking the time to share guidance.

[Phil Harvey]

1) Scenario 1 - using setfile / MDItemFSCreationDate tag  per your recommendation

Just to be clear, you only need to have setfile available on your system.  You don't have to use the setfile utility yourself.  ExifTool will execute setfile when you write anything to MDItemFSCreationDate.

- Phil

[vthokie]

Hi Phil,
I was finally able to use a different third-party software (not naming here in case it violates public forum rules), to get to where I needed to be.   Neither MDItemFSCreationDate nor the File > Export options worked for me.  In MDItemFSCreationDate it picked Feb 19, 2018 itself as the CreationDate.  Not sure if this is related to my upgrade to High Sierra, AFPS or any of the recent Photos app updates, or a troubleshooting step I took with Apple last month.  For the 2nd option, the available filters in Photos app could not get me the correct sample set to export out. 

I'm going to revert to Apple now with the new Photo library and see if this resolves the metadata corruption they are seeing server side when I upload my photos.  Without Exiftool, I would not have been able to identify the corrupt JPEG's as well as confirm what metadata is being copied through the various Apple export options.

My thanks to you!  Your software is truly amazing in what it does.   If I run into more issues after working with Apple, I will post back here, or start a new thread depending on the problem.   Happy to send you a direct PM, if you want to know the software I used to get all my photos over into a new library with min. loss of metadata.

[Phil Harvey]

(not naming here in case it violates public forum rules)

No such rules here.  I would be interested to know what the software was, and what it did to solve your problem.  It may also be helpful for others who arrived here due to the same problem.

Glad you sorted things out.

- Phil

[vthokie]

It's PowerPhotos ... from FatCatSoftware.   I used to use it with iPhoto many years ago, and found out that their new version PowerPhotos is a free upgrade if you owned a previous license key.

Here is a link to the copying limitations - https://www.fatcatsoftware.com/powerphotos/Help/copying%20limitations.html