How to analyze CR3 file structure before and after use of Exiftool

[bekesizl]

Hello,

some time ago I run into an issue with DxO Photolab not being able to read my CR3 files after they were added some metadata by Exiftool.
As DxO is not urging to repair their possible bug and as others have also run into it, I thought I would ask some help to get nearer to the solution.
I don't know the exact commands that were executed to the file, but the software doing the changes was using "ExifTool version 11.62, released on August 15. 2019".
DxO Photolab is picky about the file. Canon DPP, Lightroom, Faststone Image Viewer, Fast Raw Viewer are able to open the file.

The file size has clearly changed:
before: 21.031.607 bytes
after: 21.029.293 bytes

Could you suggest me some commands to analyze the structure of the files before and after the changes?
I attached a -htmldump from both files here.
I don't have many corrupted files left and the one I have has people on it, I don't want to publicate it.

Thanks,
Zoltán

[Phil Harvey]

Hi Zoltán,

The -htmlDump option dumps only EXIF/TIFF format information.  You are better off using -v3 to analyze the structure of a CR3.

What camera is this from?  I will take a look at some samples here to see if I can come up with any theories.

- Phil

[bekesizl]

Hi Phil,

thank you for the quick reply.
I ran Exiftool 11.80 through the files with the -v3 option.
The attached results show, that there are size changes in many "directories" from the start of the file.

The files come from a Canon EOS M50. I have used the C-RAW compressed version.
I have many files, but only kept these ones from the "corrupted" as an example.
The file sizes range from 16-25 MB. I have files with both firmware version 1.0.1 and 1.0.3, although according to the release notes, there were no changes in the file format.

Thanks,
Zoltán

[Phil Harvey]

I can't reproduce this problem.  The steps I took were:

1. exiftool -artist=me FILE

Where FILE is a Canon M50 CRAW CR3 file, and ExifTool is version 11.88.

2. Open the new file in DxO 3.13 build 48 for MacOS.

3. Edit exposure compensation and export as JPEG.

All worked fine.

- Phil

[bekesizl]

That sounds good. Maybe the problem was solved in the background or the problem is within the paramter passed to Exiftool.

I don't know the exact commands, because the DAM tool IMatch was making changes, added probably a keyword and face tags too.
But the version of Exiftool IMatch uses is definitely 11.62.

The "bug" still exists in DxO Photolab as other users were having the same problem last week.
Thread at IMatch forum: https://www.photools.com/community/index.php?topic=9487.msg68415#msg68415
Thread at DxO Forum: https://feedback.dxo.com/t/cr3-files-shown-as-exif-corrupter-not-editable-in-dxo-after-imatch-metadata-writeback/10399

[Phil Harvey]

If you have some steps to allow me to reproduce this problem then I'll look into it further.

But be aware:  In the past DxO has taken shortcuts in their reader routines (ie. instead of properly parsing the file format, it seems they they were assuming a fixed offset for some data), which would explain their policy of not supporting edited raw files.

- Phil

[bekesizl]

Hi Phil,

I don't have the exact commands used by IMatch and Mario won't tell it to us.
I have attached two very small RAW files.
One is from the camera, unchanged. Can be opened in Photolab.
The other was treated by IMatch. Can't be opened by Photolab.
When I run exiftool -a -G1 RAWFILE>export.txt, I can see the byte order changed in one Exif "part". (attached picture)
I have run attached argfiles (one is mimicking the values changed by IMatch, one is from a website) on the original files and all were readable by Photolab.
I used the Exiftool version from the IMatch directory, so it is probably not having to do with the version.

So the question would be: what command can cause the change in the byte order or even better, is there a command line for Exiftool that can "revive" these files to make them readable by Photolab again.
This might help the users of this combination CR3+IMatch+Photolab to be able to use all these tools, until DxO solves the problem.

These tests showed me, how wonderful your tool is, if correctly used.

Thanks,
Zoltán

[Phil Harvey]

Hi Zoltán,

The only way to change the byte order should be to delete the EXIF then add it back again.  That would be a very dangerous thing to do to a RAW file, so I would be surprised if iMatch was doing this.  But I'll take a look at the files on Monday and see if I can answer some questions then.

- Phil

[bekesizl]

Hi Phil,

I was also thinking about such possibilities as in my older example files, there are nearly no changes to the data itself, but very much was written (see my uploaded exports in the previous posts).
I tried one more version: Copy the XMP tags from a Mylio (another DAM) created XMP sidecar to a CR3 with following command.
exiftool -tagsfromfile 2019_12_30_9999_98.xmp -xmp:all 2019_12_30_9999_98.CR3
The result was readable by DxO, no byte order changes, even the keywords added were visible.

So I am looking forward and wait patiently if you find something.
Have a good weekend.

Zoltán

[Phil Harvey]

Hi Zoltán,

It is only the GPS information in the edited file that has a different byte order.  If you change it back to little-endian with the following command then DxO can read the file:

exiftool 2019_07_14_6059_edited.CR3 -o fixed.cr3 -gps:all= -tagsfromfile @ -gps:all -exifbyteorder=little

(note that without the -o fixed.cr3, DxO somehow won't recognize the original file after it is fixed, even though it is binary equivalent to the newly created file fixed.cr3 -- it must be caching some information about the file, and I don't know how to clear the cache)

The DxO programmers are cutting corners that will make more work for them in the long run.  They must be using a dedicated code to read the GPS from this file instead of a general EXIF/TIFF reader which would have handled the byte order difference with no problem.

In the mean time, you should maybe tell Mario at iMatch that he should use -exifbyteorder=little when adding GPS to these files.

- Phil

[bekesizl]

Hi Phil,

thank you very much for this workaround to the problem.
I posted it at both the  DxO and IMatch forums.

Zoltán

PH Edit: Added forum links.

[Phil Harvey]

I have just released ExifTool 11.89 which is patched to create new GPS information using a default byte order that is the same as the existing EXIF boxes.  This should hopefully solve the DxO incompatibility problem.  Mario plans to include this new version in his upcoming IMatch release.

- Phil

[bekesizl]

Thank you very much for the solution and also for the explanation at the IMatch forum.
I think, the CR3 and DxO user minority of IMatch will be glad to hear this.

Thank you for giving us this wonderful tool.

Zoltán