Inability of exiftool to parse Auto Focus info from Canon EOS-R DNG

[ralfbergs]

Hi there.

I use the old Lightroom 6.14, and the Show Focus Point plugin that under the hood uses exiftool to extract the focus point info.

My new camera, a Canon EOS-R, creates CR3 files, that LR can't natively read. So I converted them to DNG, using Adobe's DNG Converter 12.4.0.555, running under Windows.

However, exiftool 12.07 is either unable to parse the focus points info from the DNG, or this info is not even included in the DNG.

Code Select Expand

C:\Users\rabe\Pictures\2020\202010\New folder (2)>"C:\Program Files (x86)\Tools\exiftool.exe" -*Focus* -*AF* 0N8A0074.dng
Approximate Focus Distance      : 0.58

Here's the info in the CR3 (massively trimmed...):

Code Select Expand

C:\Users\rabe\Pictures\2020\202010\New folder (2)>"C:\Program Files (x86)\Tools\exiftool.exe" -*Focus* -*AF* 0N8A0074.cr3
[...]
AF Area Widths                  : 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362 362
AF Area Heights                 : 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334 334
AF Area X Positions             : -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736 -2736 -2280 -1824 -1368 -912 -456 0 456 912 1368 1824 2280 2736
AF Area Y Positions             : 2006 2006 2006 2006 2006 2006 2006 2006 2006 2006 2006 2006 2006 1605 1605 1605 1605 1605 1605 1605 1605 1605 1605 1605 1605 1605 1204 1204 1204 1204 1204 1204 1204 1204 1204 1204 1204 1204 1204 803 803 803 803 803 803 803 803 803 803 803 803 803 402 402 402 402 402 402 402 402 402 402 402 402 402 0 0 0 0 0 0 0 0 0 0 0 0 0 -401 -401 -401 -401 -401 -401 -401 -401 -401 -401 -401 -401 -401 -802 -802 -802 -802 -802 -802 -802 -802 -802 -802 -802 -802 -802 -1203 -1203 -1203 -1203 -1203 -1203 -1203 -1203 -1203 -1203 -1203 -1203 -1203 -1604 -1604 -1604 -1604 -1604 -1604 -1604 -1604 -1604 -1604 -1604 -1604 -1604 -2005 -2005 -2005 -2005 -2005 -2005 -2005 -2005 -2005 -2005 -2005 -2005 -2005
AF Points In Focus              : 55,68
AF Points Selected              : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142
[...]


(The JPG version ooc seems to contain the very same info... Yes, I know that the Show Focus Point plugin can read the points from the JPG, but I only show a single photo in Lightroom for both the DNG and JPG versions, and that's always the DNG...)

I know from my own research, that at least for some Canon cameras (5D Mark 2), and with some DNG Converter versions, the focus point info IS included.

How can I (we?) found out whether the info is at all included in the DNG?

If it's not included, then there's nothing we can do (apart from getting in touch with Adobe, with I would then try...). But if it's included, but exiftool fails to properly parse it, then there's obviously hope...

So how can we further investigate it? Let me know what I could do to help...

I also noticed the following weird things:

For the DNG file, the following info is shown, but for the CR3 and JPG files the info is not shown:

Code Select Expand

Approximate Focus Distance : 0.58
I'm wondering where this info even comes from, because the EOS-R's manual mentions that the focus distance would be displayed only for RF lenses, and I use EF lenses, so my guess was that these lenses do not even deliver the focus distance (or why would the cam not display it, if the lense delivers it?). And why is it only shown for the DNG, and not for the other flavors?

Many thanks in advance.

Kind regards,

Ralf

[Phil Harvey]

Hi Ralf,

I didn't read your whole post, but it is well known that the DNG converter doesn't preserve the Canon makernote information when converting from CR3.  I have it on my list of things to do to add an ExifTool feature to allow you to copy maker notes from CR3 to DNG.

- Phil

[ralfbergs]

Hi Phil.

it is well known that the DNG converter doesn't preserve the Canon makernote information when converting from CR3.  I have it on my list of things to do to add an ExifTool feature to allow you to copy maker notes from CR3 to DNG.

Sorry, I'm completely new to this, and I was using the DNG Converter for the first time, so to me it was not well-known. ;-)

It would be a great feature to be able to copy the missing info with exiftool (so many thanks in advance for your efforts on this!), and it would be definitely better than having nothing at all, but I feel the "best" solution would be if the Adobe DNG Converter was copying that data right away. Do we know why Adobe chose not to implement it? Would it make sense to get in touch with them and ask to add the missing info?

Kind regards,

Ralf

[greybeard]

Its not just Canon and its not just the DNG Raw Converter.

The most frustrating thing is that the maker notes aren't passed across during an image export from Lightroom.

People post images on forums thinking they are showing the "Straight out of Camera" image because they haven't explicitly made a change but Adobe will have stripped out the most useful metadata.

[ralfbergs]

Its not just Canon and its not just the DNG Raw Converter.

That doesn't make it any better... ;-)

[greybeard]

Its not just Canon and its not just the DNG Raw Converter.

That doesn't make it any better... ;-)

I know - sorry - I was just venting

[Phil Harvey]

Hi Ralf,

Do we know why Adobe chose not to implement it? Would it make sense to get in touch with them and ask to add the missing info?

Yes.

- Phil

Edit: I forgot to answer the first question:  Lazy, incompetent, and/or underpaid programmers would be my guess.

[Phil Harvey]

Its not just Canon and its not just the DNG Raw Converter.

The most frustrating thing is that the maker notes aren't passed across during an image export from Lightroom.

I am currently using the DNG converter for my D500 NEF files and it does preserve the maker notes (although it changes them to a different Adobe-specific format).  And when I checked this years ago, it also preserved the maker notes for every other type of raw file I tested (including CR2).

- Phil

[Phil Harvey]

OK.  I've taken a closer look at the CR3 maker notes.  The maker notes are split into 4 sections in a CR3.  The leading static section has most of the relevant information.  The other 3 parts are in the time metadata, and the only records here that aren't in the static section are:

ColorData9
Flavor
CanonCameraInfoUnknown
VignettingCorrUnknown1
Canon_0x4033
Canon_0x402e

The ColorData9 is fairly significant, but not a deal breaker if it is missing.  So it seems that just extracting the static section may be sufficient, and this would save me a lot of work.  This section does contain the auto focus info you want.  I'll look into adding the feature to copy this in ExifTool 12.09.

- Phil

[greybeard]

Its not just Canon and its not just the DNG Raw Converter.

The most frustrating thing is that the maker notes aren't passed across during an image export from Lightroom.

I am currently using the DNG converter for my D500 NEF files and it does preserve the maker notes (although it changes them to a different Adobe-specific format).  And when I checked this years ago, it also preserved the maker notes for every other type of raw file I tested (including CR2).

- Phil

You're right - I got carried away - its only the Lightroom export that removes the maker notes (for FujiFilm anyway) - not the DNG Converter

[ralfbergs]

I'll look into adding the feature to copy this in ExifTool 12.09

That would be really great, thank you so much in advance for your efforts!

And if you ever come close to Düsseldorf, I'll buy you a beer.😁

PS: I've posted into Adobe's support forum and raised a feature request.

[Phil Harvey]

ExifTool 12.09 (just released) has the ability to copy the static makernotes from a CR3 file to other file types (including DNG, JPEG and TIFF).

Note that I am writing them using the standard Canon makernote convention, and not trying to implement Adobe's butchery of the makernote structure in DNG's.  (Their engineers are idiots for doing this in the first place.)

- Phil

[ralfbergs]

ExifTool 12.09 (just released) has the ability to copy the static makernotes from a CR3 file to other file types (including DNG, JPEG and TIFF).

Amazing service... Thank you so much...

I will try it out tomorrow... I suppose if it works well (i. e. the Lightroom plugin will be able to extract the focus point info from the DNG) I will have to write myself a wrapper that runs over all DNG files to copy the data from the corresponding CR3 files, but that will not be a big problem...

Note that I am writing them using the standard Canon makernote convention, and not trying to implement Adobe's butchery of the makernote structure in DNG's.  (Their engineers are idiots for doing this in the first place.)

Cannot really comment on this, as I'm clueless about EXIF, but I strongly believe in standards, and if Adobe really violate them they cannot be blamed enough for doing so...

Many thanks again. I'll let you know how it goes...

[ralfbergs]

I will try it out tomorrow...

I'm sorry that it took much longer than intended, but I've finally found the time to try it out.

And it seems to work!!!

I checked only a single file, by using the following invocation:

Code Select Expand

"\Program Files (x86)\Tools\exiftool(-k).exe" -tagsFromFile 0N8A0994.cr3 0N8A0994.dng


Then I checked in Lightroom, and presto -- the plugin showed the focus point. For plausibility, I checked some other DNGs, and they were lacking the focus point info. So the new functionality you kindly added seems to work well.

Could you please confirm that this is the right option? I have absolutely no clue about the different kinds of meta information, so I just picked the first option that would likely do something similar to what I want, and it worked... :-)

I also noticed that exiftool seems to be able to work on whole directories, as least for select options. So maybe instead of writing a "wrapper" for it, it can also do something like

Code Select Expand

"\Program Files (x86)\Tools\exiftool(-k).exe" -tagsFromFile *.cr3 *.dng


(That's for sure the wrong way to do it, but you get the idea... I would like to copy the tags from all CR3 files to the DNG file with the same base name...)

Many thanks again.

And remember: If you ever come to Germany and will stay close to Düsseldorf, you'll get the beer I promised... :-)

[Phil Harvey]

That looks good.  It will probably copy more than you want though.  To copy just the focus points, you could do -tagsfromfile %d%f.cr3 -makernotes ...

To apply to a whole directory:

exiftool -tagsfromfile %d%f.cr3 -makernotes -ext dng DIR

- Phil

[ralfbergs]

That looks good.  It will probably copy more than you want though.  To copy just the focus points, you could do -tagsfromfile %d%f.cr3 -makernotes ...

To apply to a whole directory:

exiftool -tagsfromfile %d%f.cr3 -makernotes -ext dng DIR

The last command did exactly what I needed -- thanks a bunch.

[ralfbergs]

That looks good.  It will probably copy more than you want though.  To copy just the focus points, you could do -tagsfromfile %d%f.cr3 -makernotes ...

Hi Phil, I need to ask a stupid question again, maybe I misunderstood the above?

Does the above only copy the auto focus info, and drop (overwrite) "vital" info that the Adobe DNG Converter put into the DNG file?

Reason I'm asking is that I suddenly have a very strange problem in Lightroom. Normally as soon as I switched to the "Develop" tab in LightRoom, the photo I selected would visibly change, due to the lens profile corrections being applied (among others, it would be distorted a bit, which is very visible). That suddenly doesn't happen anymore. :-(

"Enable Profile Corrections" is still enabled when I switch to "Develop", but no corrections are applied. Under "Lens Profile" I see a make of "None," and I have to manually change "Make" to "Canon" to then have the lense profile corrections applied -- the photo immediately changes as I was used to it.

I'm pretty sure I didn't change anything in Lightroom...

Thanks in advance for your time.

[Phil Harvey]

That command copies all of the maker notes, which includes much more than just the auto focus information.

- Phil

[ralfbergs]

That command copies all of the maker notes, which includes much more than just the auto focus information.

Many thanks for confirming.

Then I need to search somewhere else...

[ralfbergs]

Reason I'm asking is that I suddenly have a very strange problem in Lightroom. Normally as soon as I switched to the "Develop" tab in LightRoom, the photo I selected would visibly change, due to the lens profile corrections being applied (among others, it would be distorted a bit, which is very visible). That suddenly doesn't happen anymore. :-(

Hey Phil,

sorry to bother you again with this, but I somehow have the impression that the procedure to copy the "makernotes" from the CR3 into the DNG somehow "corrupts" (or at least modifies) this makernotes, so that Lightroom is unable to properly parse it anymore.

I use a development profile during import to automatically apply lens corrections. This worked always so far (with the CR2 for my 550D and 7D), even with the DNGs I had to create now that I have a new cam with CR3 that LR6 no longer understands. But as soon as I use exiftool to copy the auto focus info (or rather the full "makerdata"), the lens corrections are no longer applied automatically anymore.

Can you please guide me what to do so that I can help you to find out what's wrong? As I suggested above already, it seems that somehow the makernotes are modified in a way that LR6 no longer properly reads the lens model from it... ;-(

Many thanks in advance for your time.

PS: Two screenshots from LR6 attached. One when importing unmodified DNG, one when importing a DNG modified with exiftool.

Kind regards,

Ralf

[Phil Harvey]

I tried this with LR5 and a sample Canon EOS R sample CR3 I had handy, but LR5 didn't have a lens profile for the RF lens of my sample.  Could you send me the CR3 you are using with the EF 70-200mm so I can test this myself?  My email is philharvey66 at gmail.com

- Phil

[ralfbergs]

I tried this with LR5 and a sample Canon EOS R sample CR3 I had handy, but LR5 didn't have a lens profile for the RF lens of my sample.  Could you send me the CR3 you are using with the EF 70-200mm so I can test this myself?

Many thanks, Phil.

I just sent you one sample file to your email. Please let me know if I can help in any other way.

Many thanks again for your kind help and efforts.

[Phil Harvey]

Hi Ralf,

I've taken a look at the samples you sent.

as soon as I use exiftool to copy the auto focus info (or rather the full "makerdata"), the lens corrections are no longer applied automatically anymore.

First I verified that I could apply the lens profile for the 70-200 to the DNG file you sent using LR5.

Then I ran this command:

exiftool -tagsfromfile 0N8A1407.CR3 -makernotes 0N8A1407.dng -o 0N8A1407-2.dng

Then I opened 0N8A1407-2.dng in LR5 and found that I could still apply the lens profile.

So I am not able to reproduce the problem that I think you are indicating.  If these are the steps you took, the only other difference is that I am using LR5 and you are using LR6, which makes it difficult for me to look into this issue any further since I don't have LR6.

- Phil

[ralfbergs]

So I am not able to reproduce the problem that I think you are indicating.  If these are the steps you took, the only other difference is that I am using LR5 and you are using LR6, which makes it difficult for me to look into this issue any further since I don't have LR6.

Hi Phil.

Many thanks for testing.

Maybe the way you did it is slightly different to how I did it?

When I import photos, I apply a development profile that automatically applies lens profile corrections. So as soon as I switch to the "Develop" tab, the profile corrections will be applied automatically, without having to click anything in that area.

After I have modified the DNG with exiftool, this no longer works. In the "lens profile" UI cluster, make, model, profile show "None." I have to manually switch the "Make" to "Canon," so that it will then show the (correct) model and profile. That means that the profile corrections are no longer applied automatically, I would have to manually apply them to each and every photo individually after I have imported a bunch of new photos I shot.

I hope this was not too confusing, and I made my point clear.

What I could offer is to meet online (e. g. Skype or Teams or Zoom) and present to you my desktop and how Lightroom (mis-) behaves as I tried to describe above.

Let me know if this is interesting for you, and then I'll get in touch with you via email so that we can schedule a meeting.

Again, many thanks for your time and dedication.

Best regards,

Ralf

[Phil Harvey]

I didn't have to manually select anything.  The make and lens model were already indicated when I switched to the Development tab.  I'll do it again and upload a screen capture...

- Phil

[Phil Harvey]

This is what I see after just switching to the Development tab for the exiftool-edited DNG file:



(whoa.  That screen grab turns out twice the size of what I see in LR.  I guess because of my monitor resolution settings)