Image data corruption when update large number of raw files with exiftool

[sidneyd]

Hi Phil,

interestingly minor errors are sometimes generated when using "-XMP:Identifier<$ImageDataMD5"

When writing XMP:Identifier in a few files using the following command, minor error occur:
exiftool.exe -overwrite_original -preserve -F "-FileModifyDate=Now" "-XMP:Identifier<$ImageDataMD5" -ext nef .
Warning: [minor] Error 3 placing :filterId in structure or list - ./_DSC2769.nef
Warning: [minor] Error 3 placing :filterParametersIntegerName in structure or list - ./_DSC2772.nef
Warning: [minor] Error 3 placing :filterActive in structure or list - ./_DSC2773.nef

If I made the command to store a static value such as hello world or one of the calculated MD5 values, there were no minor errors:
exiftool.exe -overwrite_original -preserve -F "-FileModifyDate=Now" "-XMP:Identifier="518fef2958a1e1866f6391a699ec3fd5" -ext nef .

However as soon as it revers to "-XMP:Identifier<$ImageDataMD5" there are these minor errors poping up.

[Phil Harvey]

This can happen when copying tags if the XMP is structure incorrectly in the source file.  What software wrote this XMP, and could you post the raw XMP so I can take a look?  (the output of exiftool -b -xmp FILE > out.xmp)

- Phil

Note: I have split off MichaelKnight's thread concerning corruption on MacOS into a separate topic because I think this was a different problem.

[Phil Harvey]

In the next ExifTool release I will add an XMP-et:OriginalImageMD5 tag as a better place to store the ImageDataMD5 value (as opposed to repurposing XMP:Identifier).

- Phil

[sidneyd]

These files had previously been updated with GPS and location data via GeoSetter, which used exiftool to write the data.  I believe the data had been written using ExifTool 12.21.

I have copied the nef files as well as the xmp outputs to the previously supplied Google Drive.

[Phil Harvey]

Thanks.  These warnings are coming from the XMP embedded in the Nikon trailer from editing with some Nikon software.  I'll have to look into this in more detail to see exactly why these warnings are generated, but it could be just because ExifTool doesn't have definitions for these (in which case the warnings should probably be suppressed).

- Phil

Edit: I've looked in more detail and it isn't actually true XMP in the Nikon trailer, hence the warnings.  I'll add a patch to suppress these in ExifTool 12.59.

[sidneyd]

I do remember there was one folder where I had tried Nikon StudioNX to look at GPS tags sometime in the past, so perhaps it was Nikon StudioNX which mangled the tags in some of these files. 

I will use the following command as it is my understanding this should rewrite any damaged tag structures in the files.
exiftool -m -r -overwrite_original -all= -tagsfromfile @ -all:all -ext nef .

[Phil Harvey]

Just to be clear:  The Nikon software didn't mangle any XMP tags -- it was XML-format metadata that ExifTool was processing (for convenience) as XMP.  The warnings shouldn't have been generated.

I've got the ImageDataMD5 feature working for RW2, CR3, MRW, MOV and MP4 files now too.  I've been working on this since last week -- it has actually required a fair bit of effort since these different formats store the image data differently.

- Phil

[Phil Harvey]

OUCH!  I just read the command in your last post.

DO NOT APPLY -all= TO ANY RAW FILE!  This may remove proprietary data that can't be added back in.

The command you gave should only be used for JPEG-format files.

- Phil

[sidneyd]

So would exiftool -m -r -overwrite_original -tagsfromfile @ -all:all -ext nef . be OK to use?

[Phil Harvey]

I wouldn't use any blunt instrument like this on a raw file.  In theory it should be OK, but it is very risky to change so much in a raw file.  Also, what are you trying to accomplish with this command?

- Phil

[sidneyd]

I wanted to fix or cleanup the XMP embedded stuff which was left behind, after digging through the various tags left by Nikon StudioNX, I will try the following
exiftool.exe -r -overwrite_original -NikonApp:all= -ext nef .

[Phil Harvey]

This is a safe way to do what you want.

- Phil

[sidneyd]

Hi Phil,

I have been occupied with some other projects recently, so only now have had a chance to test all of the new MD5 functionality with 12.60 in a number of tests.
MD5 generation for heaps of different image types now works great.
It seems that EXIFTOOL "-XMP-et:OriginalImageMD5<$ImageDataMD5" abc.nef works,

However I can not seem to be able to read the value of -XMP-et:OriginalImageMD5 back from the file as per following examples. 


For example
exiftool.exe -v2 -overwrite_original "-XMP-et:OriginalImageMD5=1234"  -ext nef .
WWriting XMP-et:OriginalImageMD5
======== ./_DSC0653.nef
Rewriting ./_DSC0653.nef...
  Editing tags in: File IFD0 TIFF XMP
  Creating tags in: File IFD0 TIFF XMP
  FileType = NEF
  FileTypeExtension = NEF
  MIMEType = image/x-nikon-nef
  Rewriting IFD0
  Rewriting SubIFD
  Rewriting SubIFD1
  Rewriting SubIFD2
  Rewriting XMP
    - XMP-et:OriginalImageMD5 = '4333b1a3541e03dee7d6a26cef86ee90'
    + XMP-et:OriginalImageMD5 = '1234'
  Rewriting IPTC


But the command to read or extract the stored MD5 checksum
exiftool.exe -table "-Filename" "-XMP-et:OriginalImageMD5"  -ext nef .
_DSC0653.nef    -

exiftool.exe -p "$Filename,$ImageDataMD5,$XMP-et:OriginalImageMD5"  -ext nef .
Warning: [Minor] Tag 'XMP-et:OriginalImageMD5' not defined - ./_DSC0653.nef
    1 directories scanned
    1 image files read


exiftool.exe" -a  "-XMP:all" "-XMP-et:all" -ext nef .
======== ./_DSC0653.nef
XMP Toolkit                    : Image::ExifTool 12.60
Rights                          : (c) Copyright 123
Date/Time Digitized            : 2023:03:12 08:15:10+01:00
Date/Time Original              : 2023:03:12 08:15:10+01:00
Date Created                    : 2023:03:12 08:15:10+01:00
Copyright Status                : Protected
Create Date                    : 2023:03:12 08:15:10.55
Creator Tool                    : NIKON D810 Ver.1.14
Modify Date                    : 2023:03:12 08:15:10+01:00
Marked                          : True
    1 directories scanned
    1 image files read


Am I missing somethign here

[Phil Harvey]

Oops.  You're right. Sorry, I should have tested this.

The value is written, but ExifTool is ignoring the "et" namespace when reading for reasons that I don't remember related to reading the -X option RDF/XML.

I'll fix this in the next release.

- Phil

[sidneyd]

Hey Phil, thanks for confirming that I was not doing something silly and as always your fast responses.

If you want to drop me a pre-releae version to test, let me know, otherwise I will keep an eye out for the next release.

Regards
Sidney