Bad MakerNotes offset for NEFBitDepth

[majo44]

Hello

- Os: Win 7 (64bit)
- ExifTool version: 8.25
- Command :
exiftool -AllDates+=1 .

-Output:

Code Select Expand

Warning: Can't read MakerNotes data. Ignored. - ./AiE20100619_103030(9).dng
Error: [minor] Bad MakerNotes offset for NEFBitDepth - ./AiE20100619_103030(9).d
ng
Warning: Can't read MakerNotes data. Ignored. - ./AiE20100619_104839(9).dng
Error: [minor] Bad MakerNotes offset for NEFBitDepth - ./AiE20100619_104839(9).d
ng
....


The same for all files in directory.
Files was created by Adobe DNG Converter v. 5.6.0.148 from Nikon D700 NEF files.

I can't attach the example file, is to big (9MB) probably, I have always timeout error.

Regards
Pawel

P.S.
This is really nice application, good job, thanks a lot. 

[Phil Harvey]

Hi Pawel,

Thanks for this report, however I am not able to replicate this problem.

When I convert one of my D700 NEF images using DNG converter 5.6.0.148 I get no warnings or errors with this command.

Did you use any other utility to modify the DNG after it was written?

You can try mailing me a sample (philharvey66 at gmail.com) to see if that works.

- Phil

[majo44]

Hello

Thanks for your quick answer.
Sorry, I forgot to add that the files were probably modified in Adobe Camera Raw. In the evening I see if everything is ok for the files that were not modified. Meanwhile, I sent you the file to your email.

Regards
Pawel

[Phil Harvey]

Thanks for the sample.

The makernotes have been truncated by some Adobe utility.  I'm not sure exactly how this happened because I haven't been able to reproduce the problem myself.  One thing I find hard to understand is that although your D700 has the same firmware version as the sample I tested, the tags stored in the maker notes are quite different.  I would like to also see the original NEF if you could send it.

If you can determine exactly the steps you used to cause this problem, you should send a bug report to Adobe.

BTW, the DNG converter has had this problem in the past for other image types.

ExifTool can still be used to write standard metadata to the corrupted files with the -m option.  In this case exiftool will not attempt to fix the makernote problems, and will just copy the truncated makernotes byte-for-byte.

- Phil

[majo44]

Thank you for your reply. I just sent you NEFfile. In the evening I will do some tests and write in the forum what are their results.

Another observation is that it may cause is in Adobe Bridge, I use it to marrking and labeling the images. Check it out.

--
Pawel

[Phil Harvey]

Thanks for the NEF sample, but this image has already been modified by some software, which may be causing your problem.

If you use ExifTool to modify the NEF before converting to DNG, the problem will go away.  So maybe the bug report should be sent to the utility used to modify your NEF.  You will have to do some investigating here.

- Phil

[Hayo Baan]

Hi Phil,

I have recently switched my workflow to .dng and have been bit by this problem too. For instance when I am trying to simply delete the Urgency field (I'm not editing any tag in the makernotes). Some but not all of my dng files suffer from this problem by the way.

Is there a solution/work around for this problem? Is there a way to still update the rest of the file?

If you need a sample files, I can provide them.

Thanks,
Hayo

[Phil Harvey]

Hi Hayo,

You can use the -m option to ignore makernote errors, which should allow you to write to these files.  But beware that doing so may cause further corruption of the maker note information.

- Phil

[Hayo Baan]

Hello Phil,

Thank you for quickly answering this question. While I was actually working with the perl library to modify the files, the command-line option was helpful too (for my investigation below)

I did some more research on this error though, and I have found a cure (sort of).
To begin, I wanted to be absolute certain what caused the dng error in the first place; I want my image files to be as reliable as possible and keep all info as correct as possible (that is also one of the reasons I'm using the exiftool perl library; I am building my own metadata check scripts).

Anyway, I am also a heavy user of PhotoMechanic so my first object was to make sure both it and the DNG converter/Adobe Camera Raw were not the cause. Here's what I did:

• For a problematic dng, I went back to the original .nef file and converted it to a fresh .dng. Checking with exiftool showed no issues.
• I then applied the same IPTC/XMP settings as were in the bad file, using PhotoMechanic. This should rule out PM as culprit. The updated dng still showed no issue with exiftool.
• I then ran the file through ACR to update its develop settings as with the original (bad) dng. Still no problems according to exiftool. This also meant tha, if all else failed, I always had a way to recreate the dng without makernote coding errors from scratch (albeit slow and timeconsuming).

So now I had two versions of basically the same dng. One with makernote coding errors, and one without. But still no way to reproduce the problem...
Some more experimenting with the files in PhotoMechanic, however, showed a way to get rid of the problem in the bad dng file, and how to recreate it!

File A is the newly created dng, B is the "bad" one.

Code Select Expand

$ exiftool -a -G -s -m -Urgency 20140209_09310308_D3_[AB].dng
======== 20140209_09310308_D3_A.dng
======== 20140209_09310308_D3_B.dng
Warning: Bad NikonScanIFD SubDirectory start - /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_B.dng
Warning: Bad MakerNotes offset for NEFBitDepth - /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_B.dng

Right, no Urgency in the file, ok. So how about setting it (in PM)?

Code Select Expand

======== /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_A.dng
[XMP]           Urgency                         : 1 (most urgent)
[IPTC]          Urgency                         : 1 (most urgent)
======== /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_B.dng
[XMP]           Urgency                         : 1 (most urgent)
[IPTC]          Urgency                         : 1 (most urgent)
Ah, interesting: now there are no errors...
Resetting the urgency in PM to undefined brought back the error as in the first output of exiftool.
I then remembered, that updating the ColorClass (or Rating) of a file in PM somehow set the Urgency to 0 (this time only in the IPTC record though), so I tried that. I actually think this is a minor bug in PM, but hey, I wanted to see what this would do.
As you probably would not have guessed: this also resulted in a file that according to exiftool has no issues

Code Select Expand

======== 20140209_09310308_D3_A.dng
[IPTC]          Urgency                         : 0 (reserved)
======== 20140209_09310308_D3_B.dng
[IPTC]          Urgency                         : 0 (reserved)

So in the whole process of manipulating the file with PM, there sometimes is a problem introduced – which luckily can be corrected.
Perhaps this would allow you to write code to fix the problem too? I am of course happy to help test this

Thanks,
Hayo

[Phil Harvey]

Hi Hayo,

Right, no Urgency in the file, ok. So how about setting it (in PM)?

Code Select Expand

======== /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_A.dng
[XMP]           Urgency                         : 1 (most urgent)
[IPTC]          Urgency                         : 1 (most urgent)
======== /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_B.dng
[XMP]           Urgency                         : 1 (most urgent)
[IPTC]          Urgency                         : 1 (most urgent)
Ah, interesting: now there are no errors...

I think you are fooling yourself.  The exiftool application only prints warnings when extracting if the requested tag wasn't found.  Try requesting -warning instead.

- Phil

[Hayo Baan]

Hi Phil,

Darn. You're right
With -warning I still do get the error. So I'm back to square one and have no work-around. Now to find what's causing this behaviour; clearly not all my dng files suffer from the issue and when I recreate the dng fresh and then apply all metadata and edits again (thereby largely mimicking what I had done prior), the problem does not return either. So what could be causing this???

Cheers,
Hayo

[Phil Harvey]

Hi Hayo,

I can't say what is causing the problem in this specific case, but I do know that Adobe's DNG converter is really bad at handling maker notes.  It restructures the proprietary camera maker notes into a proprietary Adobe format (stupid, I know), sometimes losing information in the process.

- Phil

[Hayo Baan]

I can't say what is causing the problem in this specific case, but I do know that Adobe's DNG converter is really bad at handling maker notes.  It restructures the proprietary camera maker notes into a proprietary Adobe format (stupid, I know), sometimes losing information in the process.

Hmm, doesn't sound very good. As I said, it doesn't seem to make the same mistake twice though; when I redo the whole thing afresh, the error is completely gone???
Do you think the culprit really is the Adobe DNG (converter) or could it also be Lightroom or ACR later in the process too?

I am running exiftool -warning on all my dng files now and indeed, so far there are lots with errors and also lots without. So far I have not found any consistency...

How safe do you think will ignoring the warning and writing the file anyway be?

[Phil Harvey]

Ignoring warnings like this will almost certainly cause a loss of the makernote information indicated in the warning, if it wasn't lost already.

I can't speculate about the culprit.  I do use LR occasionally, but I avoid using DNG specifically because LR tampers with DNG files without asking.  If I keep them as proprietary RAW files, LR leaves them alone.

- Phil

[Hayo Baan]

Hi Phil,

Thank you for your time and trying to help out with this weird (Adobe) issue. So far of the 2259 files I have processed just now, there were 122 errors in 2481 files (it's still processing though). Pity as 122 is a bit much to go through the whole process of manually recreating the dng afresh to get everything back again.

I'll dig deeper into this in the coming future and if I ever find the cause (or better yet: the solution/fix), I'll sure let you know

Now back to my other priorities: processing the files from yesterday's event 

Many thanks so far,
Hayo

[Hayo Baan]

Hi Phil,

I did some more research and while I still have not found the exact cause, the fact that all those 122 bad files where processed early on in my switch to DNG may indicate it was an older version of the Adobe DNG converter or ACR/Lightroom software (or even PhotoMechanic, though I doubt that) that may have caused this.

I have now manually reconverted these images (I still have the original raw files for all my images!)and reapplied the metadata and edits (simply synchronising them is PM and ACR), so my DNG files are all without errors now 

Ill keep watching this and when I do find a likely cause, I'll report it.
(note: this whole exercise has triggered me to have a look at the warnings more closely and have found files with other minor metadata issues, perhaps time to fix those too )

[Phil Harvey]

Hi Hayo,

Well, it is reassuring if indeed the problem was fixed in the software that was causing it.

- Phil

[Hayo Baan]

Hello Phil,

Good news (in a way). Thinking about the files that suffered from the issue a bit longer, I thought, "hey weren't those the ones I opened from the (already edited) .nef and then saved as .dng within ACR?"

So I just tried recreating the issue by opening a .nef and then saving it as .dng. And, lo and behold!, the thus created .dng indeed suffered from the bad offset problem. Time to report a bug to the Adobe team (perhaps they fix it as quickly as they did with the support for some (e.g. D200) lossless compressed .nef files )

Additional info: I have reported this as a bug on the Adobe forums.

Cheers,
Hayo

[Phil Harvey]

Hi Hayo,

I can't check your link from home (the web page it far to heavy for dialup).  When you say "edited NEF", what did you edit it with?

- Phil

[Hayo Baan]

Hi Phil,

Wow, hadn't realised you could still be on dialup. Dialup is something of the past for us here (in the Netherlands); everyone is at least at multi Mbs cable or ADSL, and have been for years now. I do still remember dialup and the happiness I felt when I could combine the two ISDN lines we had then to form one 128kbs connection. Ah, those were the days  Anyway, some of the websites (Adobe's amongst them) do not really consider dialup any longer and bloat their site with heavy graphics.

Anyway, here's the text of the problem I posted:

DNG files created from raw files previously edited in Nikon Capture NX2 results in corrupted (makernote) metadata
When creating a DNG file from a file that has already been edited with Nikon Capture NX2, the rsulting DNG file will contain corrupted maker note data.
(this can easily be verified by running ExifTool on the file: you'll get a message stating that the file has a "Bad NikonScanIFD SubDirectory start")

Hopefully this can be easily fixed as this hampers switching to DNG (and ACR) for my older files quite considerably...

To answer your question, I meant editing them with NX2.
Which leads me to another question, is it possible to remove just the MakerNotes:NikonCapture portion of the MakerNotes?
When I tried this with -MakerNotes:NikonCapture= , I got Warning: Tag 'NikonCapture' does not exist and when I tried -MakerNotes:NikonCapture:*=, it told me Warning: Can't delete PhotoEffectHistoryXML

Thanks,
Hayo

[Phil Harvey]

Hi Hayo,

Wow, hadn't realised you could still be on dialup. Dialup is something of the past for us here (in the Netherlands);

Yeah.  Canada is too big.  Where I live in the country, there are a few options (wireless, cell, satellite), but they are all too expensive for me and/or would require cutting down of trees to get reception.

Which leads me to another question, is it possible to remove just the MakerNotes:NikonCapture portion of the MakerNotes?
When I tried this with -MakerNotes:NikonCapture= , I got Warning: Tag 'NikonCapture' does not exist and when I tried -MakerNotes:NikonCapture:*=, it told me Warning: Can't delete PhotoEffectHistoryXML

Did you forget to put quotes around the argument containing the '*' ?

The group delete syntax supports only a single group name.  So the syntax would be -nikoncapture:all=.  But I don't know why you are deleting the NikonCapture information.  The bad NikonScanIFD will be removed when you write anything to the file with ExifTool.  This likely won't cause any problems.  However, you also got an error "Bad MakerNotes offset for NEFBitDepth", which is more serious.  Corrupting this information may cause problems.

- Phil

[Hayo Baan]

Yeah.  Canada is too big.  Where I live in the country, there are a few options (wireless, cell, satellite), but they are all too expensive for me and/or would require cutting down of trees to get reception.

I see, dialup is the best solution then. Hopefully you get at least some decent speed out of it.
(Note: even in the Netherlands, though very uncommon, there are some places with problems similar to yours)

Which leads me to another question, is it possible to remove just the MakerNotes:NikonCapture portion of the MakerNotes?
When I tried this with -MakerNotes:NikonCapture= , I got Warning: Tag 'NikonCapture' does not exist and when I tried -MakerNotes:NikonCapture:*=, it told me Warning: Can't delete PhotoEffectHistoryXML

Did you forget to put quotes around the argument containing the '*' ?

Actually as I don't have any files that match the *, the shell wouldn't expand the * anyway, so it doesn't matter (though safer would have been to use them, I know)

The group delete syntax supports only a single group name.  So the syntax would be -nikoncapture:all=.

Excellent, the -nikoncapture:all= seems to work just fine.

But I don't know why you are deleting the NikonCapture information.  The bad NikonScanIFD will be removed when you write anything to the file with ExifTool.  This likely won't cause any problems.  However, you also got an error "Bad MakerNotes offset for NEFBitDepth", which is more serious.  Corrupting this information may cause problems.

Well, I was looking for a way to safely remove the Nikon Capture NX2 edits so I would prevent the corruption by the conversion to DNG.

Now I was able to delete the NikonCapture information, I found out something interesting: not only did it work as I hoped (the DNG from the file without the Capture info now contains no corruption), the DNG converter was in fact able to properly convert all the intermediate test files where exiftool had regenerated (and reordered) the metadata! Awesome

So basically I have a good approach available for when I want to edit some of my older nefs (i.e., the ones with NX2 edits) and convert them to DNG. I will simply delete the NikonCapture info (as that is totally unnecessary metadata for ACR) right before I open the file in ACR or convert it to DNG. Sure, not as convenient as being able to open the file in ACR directly and do a save-as DNG there, but still workable for me 
Of course, if Adobe feels it is worth it to fix their conversion problem, I wouldn't even need this work-around (but then I may still want to delete the NikonCapture info as it is unnecessary ballast.

Thanks for helping me find this solution (and giving me even more insight in ExifTool's workings).

Cheers,
Hayo

[Phil Harvey]

Hi Hayo,

Actually as I don't have any files that match the *, the shell wouldn't expand the * anyway, so it doesn't matter (though safer would have been to use them, I know)

Just checking, because I didn't remember a tag called PhotoEffectHistoryXML.  But searching for it now, I do find it in the NikonCapture information. So I understand what was happening.  This note in the app doc explains the situation:

            4) Specifying "-GROUP:all=" deletes the entire group as a block
            only if a single family 0 or 1 group is specified.  Otherwise all
            deletable tags in the specified group(s) are removed individually
            [...]

... but maker note tags like NikonCapture may not be deleted individually.  So this doesn't work.

not only did it work as I hoped (the DNG from the file without the Capture info now contains no corruption), the DNG converter was in fact able to properly convert all the intermediate test files where exiftool had regenerated (and reordered) the metadata! Awesome

Interesting.  I was thinking that just touching the NEF with ExifTool (without deleting the NikonCapture info) may solve the problem.  This makes sense since it could be something screwy that NikonCapture is doing, unrelated to the fact that the file contains NikonCapture information.

I will simply delete the NikonCapture info (as that is totally unnecessary metadata for ACR)

If you don't need it anyway, then this is a good solution.

- Phil