Some Canon Raw DPP VRD tags not found

[juhai]

I've used ExifTool for date and time changes so far. I am using Canon DPP 3.15.0 for the workflow with my Canon EOS 7D. I have files that have been processed with many different versions of DPP.
Recently, I started looking into removing some RAW files that have never been processed. The problem is that it's very tricky to figure out when a CR2 file has a recipe that changes from the standard picture provided by the camera. I think ExifTool could give me the details that I need to detect the files that have never been seen modifications in DPP. Note that setting a mark and removing it leaves all default DPP data in the image. The task is to detect if the data is default or not.

My problem is that when Picture Style is Standard (with the camera logo next to it), any change to tone curves, contrast, highlight, shadow, don't trigger a change in exiftool output. There are changes in the RAW file but none of these are seen by ExifTool, except modification date.

My env is as follows.
1. Windows 7 64bit, through the windows executable and also cygwin source install, both give same result
2. using exiftool 10.09 on both
3. exiftool IMG_XXXX.CR2 -canonvrd -* to print all tags. I do this before and after changing color saturation in the raw controls of the image in DPP. The only change in ExifTool is this:

Code Select Expand


$ diff f1 f2
6c6
< File Modification Date/Time     : 2016:01:08 21:55:52+00:00
---
> File Modification Date/Time     : 2016:01:08 21:56:23+00:00
8c8
< File Inode Change Date/Time     : 2016:01:08 21:55:52+00:00
---
> File Inode Change Date/Time     : 2016:01:08 21:56:23+00:00

4. See attachment
5. I will email a sample if needed but I think this should be reproducible on any Canon CR2 format RAW file. I tried the ExifTool test pictures but DPP crashes with them.

I did some digging into this. Unfortunately, I've become more rusty in Perl than I thought. If I change Picture Style in DPP to Standard (without camera logo) and modify e.g. Contrast. The value gets changed in Standard Raw Contrast. So, it seems that if tone/shadow/highlight changes are done with the default picture style, these can't be found by ExifTool. I might try going deeper still, but if you had any ideas where to look at, that'd be much appreciated. I am getting better with CanonVRD.pm day by day.

[TorC]

Sounds like it may be a case of tags that Exiftool doesn't yet understand.  If that's the case, take a look at this thread.  I'm working ond the Nikon D610 there, but the principles are probably very similar.

[Phil Harvey]

Yes.  See if anything changes with the -u (or -U) option.  Let me know what you discover and I'll add it to ExifTool.

- Phil

[juhai]

Thanks for the encouragement and help. Using -U already revealed the shadow value being there.
< Canon VRD Ver 2 0x00ec          : 30
---
> Canon VRD Ver 2 0x00ec          : -20
The former corresponds to +3 and latter to -2
I am good to continue this study and report back once I have something more conclusive.

[Phil Harvey]

OK.  This new, and will be tricky because the digital lens optimizer data is stored at this offset if DLO is enabled.  With DPP 3.11.2, the Ver2 data ended at offset 0xe1 unless DLO was enabled.

Try enabling DLO then changing this setting again to see what byte changes.  I may have to get you to send me some samples, or see if I can download DPP 3.15.0 myself because I think this may require some special logic.

- Phil

[juhai]

Great, you are right. Until I do DLO, the Camera Raw settings can be found as:
    0xe1 => 'CameraRawColorTone',
    0xe2 => 'CameraRawSaturation',
    0xe3 => 'CameraRawContrast',
    0xe4 => { Name => 'CameraRawLinear', %noYes },
    0xe5 => 'CameraRawSharpness',
    0xe6 => 'CameraRawHighlightPoint',
    0xe7 => 'CameraRawShadowPoint',
    0xe8 => 'CameraRawOutputHighlightPoint',
    0xe9 => 'CameraRawOutputShadowPoint',
    0xeb => 'CameraRawHighlight',
    0xec => 'CameraRawShadow',
    0xee => 'CameraUnsharpMaskStrength',
    0xf0 => 'CameraUnsharpMaskFineness',
    0xf2 => 'CameraUnsharpMaskThreshold',

When I add the DLO, these move by 12 bytes. I will prepare a few files for you tomorrow. Based on what I've seen so far, this needs a group that moves around based on the preceding data, right?

In camera, you have fixed and three custom settings. Each pre-defined and custom can be tuned to your liking. I never use these styles, but would be nice to nail them. I tried a few custom but can't really make them match on what ExifTool tells after modification. There may be some issues there as well but let's work on this first.

[Phil Harvey]

Sounds good.  Send me the VRD data for these files for testing and I'll see if I can add support for this.  I want to make sure I get the logic right to find them when the DLO is added.  You can extract the VRD like this: 

exiftool -w %d%f.vrd -canonvrd -b DIR

Send them to phil@owl.phy.queensu.ca

- Phil

[juhai]

I shared the files with you through Google Drive as DLO adds ~30MB of data to the VRD binary.  You should get a notification.

[Phil Harvey]

I got the samples, thanks!

I have implemented a mechanism to allow the tag offsets to be shifted due to the presence of DLO data.  I've managed to install DPP 3.15, and will try to test this out myself to be sure it works.  I'm a bit worried about possible padding if the DLO data is an odd length (it happens to be a multiple of 4 in the sample you sent).  Attached is my working version of CanonVRD.pm.

You have also added the tags 0xc9-0xcd which I didn't add because I didn't know what settings these referred to.  Should they possibly be "CameraRawSharpness", etc?  Or is just "RawSharpness" correct?

- Phil

Edit:  I've played around a bit with DPP 3.15.0, and if you turn off DLO the DLODataLength doesn't go to zero even though the data isn't stored.  So I have updated the attached CanonVRD.pm to check DLOOn instead.  Also, I've tested it with an odd DLODataLength, and there doesn't seem to be any padding to worry about.

[juhai]

Cool, I will check this on some more samples. Now that you asked about 0xc9-0xcd, I can't find where they came from aside from your comments on the code. It might be something I turned on to see if they are triggered. I'll get back hopefully tomorrow with latest updates. Thanks for doing this, the tool is just so good.