Image data corruption when update large number of raw files with exiftool

[sidneyd]

Hi Phil,

I just tried reading and displaying the $ImageDataMD5 value with 12.61 release and it still comes up as not defined whether I use $XMP-et:OriginalImageMD5 or $OriginalImageMD5.

As example:
exiftool.exe -ver
12.61

Code Select Expand

exiftool.exe -p "$Filename,$ImageDataMD5,$XMP-et:OriginalImageMD5"  -ext nef .
Warning: [Minor] Tag 'XMP-et:OriginalImageMD5' not defined - ./_DSC0653.nef
Warning: [Minor] Tag 'XMP-et:OriginalImageMD5' not defined - ./_DSC0654.nef
Warning: [Minor] Tag 'XMP-et:OriginalImageMD5' not defined - ./_DSC0655.nef
Warning: [Minor] Tag 'XMP-et:OriginalImageMD5' not defined - ./_DSC7632.nef
Warning: [Minor] Tag 'XMP-et:OriginalImageMD5' not defined - ./_DSC7634.nef
Warning: [Minor] Tag 'XMP-et:OriginalImageMD5' not defined - ./_DSC7635.nef
    1 directories scanned
    6 image files read

Code Select Expand

exiftool.exe -p "$Filename,$ImageDataMD5,$OriginalImageMD5"  -ext nef .
Warning: [Minor] Tag 'OriginalImageMD5' not defined - ./_DSC0653.nef
Warning: [Minor] Tag 'OriginalImageMD5' not defined - ./_DSC0654.nef
Warning: [Minor] Tag 'OriginalImageMD5' not defined - ./_DSC0655.nef
Warning: [Minor] Tag 'OriginalImageMD5' not defined - ./_DSC7632.nef
Warning: [Minor] Tag 'OriginalImageMD5' not defined - ./_DSC7634.nef
Warning: [Minor] Tag 'OriginalImageMD5' not defined - ./_DSC7635.nef
    1 directories scanned
    6 image files read

Code Select Expand

exiftool.exe "-Filename" "-ImageDataMD5" "-$XMP-et:OriginalImageMD5" "-OriginalImageMD5"  -ext nef .
Invalid TAG name: "$XMP-et:OriginalImageMD5"
======== ./_DSC0653.nef
File Name                      : _DSC0653.nef
Image Data MD5                  : 4333b1a3541e03dee7d6a26cef86ee90
======== ./_DSC0654.nef
File Name                      : _DSC0654.nef
Image Data MD5                  : 80e452f8a0adb3b8725acc365f4b2e56
======== ./_DSC0655.nef
File Name                      : _DSC0655.nef
Image Data MD5                  : 9b11a58338dd7ea11acfc78e3bd9ef33
======== ./_DSC7632.nef
File Name                      : _DSC7632.nef
Image Data MD5                  : 87a28d422a3c61c13ebb47b3e5b410c8
======== ./_DSC7634.nef
File Name                      : _DSC7634.nef
Image Data MD5                  : c4151d2a2b199e0b97c29873369b563c
======== ./_DSC7635.nef
File Name                      : _DSC7635.nef
Image Data MD5                  : cb3182c5151ed2a44cffb5454e97ee0d
    1 directories scanned
    6 image files read

Regards
Sidney

[StarGeek]

Have you written a value to XMP-et:OriginalImageMD5 yet?  This warning comes up when using the -p (-printFormat) option for any tag when the tag does not exist.

Code Select Expand

C:\>exiftool -G1 -a -s -Description y:\!temp\Test4.jpg

C:\>exiftool -G1 -a -s -p "$Description" y:\!temp\Test4.jpg
Warning: [Minor] Tag 'Description' not defined - y:/!temp/Test4.jpg

C:\>exiftool -P -overwrite_original -Description=test y:\!temp\Test4.jpg
    1 image files updated

C:\>exiftool -G1 -a -s -p "$Description" y:\!temp\Test4.jpg
test
From the docs
     If a specified tag does not exist, a minor warning is issued and the line with the missing tag is not printed

[sidneyd]

Hi Stargeek,

yes the value had been written to the files as indicated and Phil had indicated the big was present in the previous code, and fix in 12.61.  I had written the value into the NEF files using a command similar to:
exiftool.exe -v2 -overwrite_original "-XMP-et:OriginalImageMD5<$ImageDataMD5"  -ext nef .

As the display issue was present, I set the value to 1234 and used the -v2 option so that exiftool would show that the value was changed, which it seems to he been, as below.

Code Select Expand

exiftool.exe -v2 -overwrite_original "-XMP-et:OriginalImageMD5=1234" _DSC0653.nef
Writing XMP-et:OriginalImageMD5
======== ./_DSC0653.nef
Rewriting ./_DSC0653.nef...
  Editing tags in: File IFD0 TIFF XMP
  Creating tags in: File IFD0 TIFF XMP
  FileType = NEF
  FileTypeExtension = NEF
  MIMEType = image/x-nikon-nef
  Rewriting IFD0
  Rewriting SubIFD
  Rewriting SubIFD1
  Rewriting SubIFD2
  Rewriting XMP
    - XMP-et:OriginalImageMD5 = '4333b1a3541e03dee7d6a26cef86ee90'
    + XMP-et:OriginalImageMD5 = '1234'
  Rewriting IPTC


[sidneyd]

Hi StarGeek & Phil.

After I had responded to Stargeek, I did some additional tests.

The value of XMP-et:OriginalImageMD5 had "seemingly" been written to the files with the 12.60 version, but could never be displayed (as earlier in the thread) and I had assumed that the values was correctly written to the file.

So out of curiosity, I used the latest version 12.61 to write $XMP-et:OriginalImageMD5 again into the file.
If I used 12.61 to write the value, into the NEF file, then it can now be correctly displayed.

[olegos]

This ImageDataMD5 tag is very useful, thanks for adding it. Could you support it for HEIC images too, please.

By the way, I've used this command with JPEGs before for this purpose, usually to verify that two files are actually the same image with only metadata changes:

Code Select Expand

jpegtran -copy none image.jpg | md5sum
Interesting that ImageDataMD5 gives a different value than the above.

[StarGeek]

Jpegtran is probably optimizing the image data.  Cameras don't always use the most optimized routines to write a jpeg, as speed is more essential.

As an example, I processed a file with
exiftool -all= file.jpg
processed another with the default jpegtran command and -copy none.  Then finally once more with jpegtran with -copy none and -optimize.  The resulting image will still have identical image data but it will be formatted differently.

Code Select Expand

C:\>exiftool -G1 -a -s -fileorder filename -filesize# Y:\!temp\aaaa
======== Y:/!temp/aaaa/2023-03-25_12.33.53-Exiftool.JPG
[System]        FileSize                        : 7184204
======== Y:/!temp/aaaa/2023-03-25_12.33.53-JpegTranDefault.JPG
[System]        FileSize                        : 7184238
======== Y:/!temp/aaaa/2023-03-25_12.33.53-JpegTranOptimized.JPG
[System]        FileSize                        : 7088213
======== Y:/!temp/aaaa/2023-03-25_12.33.53.JPG
[System]        FileSize                        : 7230200
    1 directories scanned
    4 image files read
One thing that jumps out is that the JpegTran default is larger than the exiftool result.  Looking closer, jpegtran is added a JFIF block, which exiftool does not.  But with the optimize option, the file is smaller.

So my first assumption was wrong.  The difference would be the JFIF block that jpegtran is adding.

Taking it a step further, checking with exiftool's ImageDataMD5

Code Select Expand

C:\>exiftool -G1 -a -s -fileorder filename -ImageDataMD5 Y:\!temp\aaaa
======== Y:/!temp/aaaa/2023-03-25_12.33.53-Exiftool.JPG
[File]          ImageDataMD5                    : 4a583b879c8b2a2f645f2870b9fff172
======== Y:/!temp/aaaa/2023-03-25_12.33.53-JpegTranDefault.JPG
[File]          ImageDataMD5                    : 4a583b879c8b2a2f645f2870b9fff172
======== Y:/!temp/aaaa/2023-03-25_12.33.53-JpegTranOptimized.JPG
[File]          ImageDataMD5                    : 17934ab470e00f176f36e9c5345415a1
======== Y:/!temp/aaaa/2023-03-25_12.33.53.JPG
[File]          ImageDataMD5                    : 4a583b879c8b2a2f645f2870b9fff172
    1 directories scanned
    4 image files read
The original, the one stripped by exiftool, and the one stripped by jpegtran all have the same hash.  So by default jpegtran isn't changing the image stream from the original.  The optimized one, of course, has a different hash.  But this was a lossless optimization and if you compare the images, for example, with ImageMagick's Compare option, you will find the image data to be identical.

[Phil Harvey]

The value of XMP-et:OriginalImageMD5 had "seemingly" been written to the files with the 12.60 version, but could never be displayed (as earlier in the thread) and I had assumed that the values was correctly written to the file.

Code Select Expand

> Image-ExifTool-12.60/exiftool a.nef "-originalimagemd5<imagedatamd5"
    1 image files updated
> Image-ExifTool-12.60/exiftool a.nef -originalimagemd5
> Image-ExifTool-12.61/exiftool a.nef -originalimagemd5
Original Image MD5              : 518fef2958a1e1866f6391a699ec3fd5
- Phil

[Phil Harvey]

Could you support it for HEIC images too, please.

Done, and version 12.62 is now available with this feature.

- Phil

[olegos]

Could you support it for HEIC images too, please.

Done, and version 12.62 is now available with this feature.

- Phil

Thank you!