Bad MakerNotes offset for NEFBitDepth

[majo44]

Hello

- Os: Win 7 (64bit)
- ExifTool version: 8.25
- Command :
exiftool -AllDates+=1 .

-Output:

Code Select Expand

Warning: Can't read MakerNotes data. Ignored. - ./AiE20100619_103030(9).dng
Error: [minor] Bad MakerNotes offset for NEFBitDepth - ./AiE20100619_103030(9).d
ng
Warning: Can't read MakerNotes data. Ignored. - ./AiE20100619_104839(9).dng
Error: [minor] Bad MakerNotes offset for NEFBitDepth - ./AiE20100619_104839(9).d
ng
....


The same for all files in directory.
Files was created by Adobe DNG Converter v. 5.6.0.148 from Nikon D700 NEF files.

I can't attach the example file, is to big (9MB) probably, I have always timeout error.

Regards
Pawel

P.S.
This is really nice application, good job, thanks a lot. 

[Phil Harvey]

Hi Pawel,

Thanks for this report, however I am not able to replicate this problem.

When I convert one of my D700 NEF images using DNG converter 5.6.0.148 I get no warnings or errors with this command.

Did you use any other utility to modify the DNG after it was written?

You can try mailing me a sample (philharvey66 at gmail.com) to see if that works.

- Phil

[majo44]

Hello

Thanks for your quick answer.
Sorry, I forgot to add that the files were probably modified in Adobe Camera Raw. In the evening I see if everything is ok for the files that were not modified. Meanwhile, I sent you the file to your email.

Regards
Pawel

[Phil Harvey]

Thanks for the sample.

The makernotes have been truncated by some Adobe utility.  I'm not sure exactly how this happened because I haven't been able to reproduce the problem myself.  One thing I find hard to understand is that although your D700 has the same firmware version as the sample I tested, the tags stored in the maker notes are quite different.  I would like to also see the original NEF if you could send it.

If you can determine exactly the steps you used to cause this problem, you should send a bug report to Adobe.

BTW, the DNG converter has had this problem in the past for other image types.

ExifTool can still be used to write standard metadata to the corrupted files with the -m option.  In this case exiftool will not attempt to fix the makernote problems, and will just copy the truncated makernotes byte-for-byte.

- Phil

[majo44]

Thank you for your reply. I just sent you NEFfile. In the evening I will do some tests and write in the forum what are their results.

Another observation is that it may cause is in Adobe Bridge, I use it to marrking and labeling the images. Check it out.

--
Pawel

[Phil Harvey]

Thanks for the NEF sample, but this image has already been modified by some software, which may be causing your problem.

If you use ExifTool to modify the NEF before converting to DNG, the problem will go away.  So maybe the bug report should be sent to the utility used to modify your NEF.  You will have to do some investigating here.

- Phil

[Hayo Baan]

Hi Phil,

I have recently switched my workflow to .dng and have been bit by this problem too. For instance when I am trying to simply delete the Urgency field (I'm not editing any tag in the makernotes). Some but not all of my dng files suffer from this problem by the way.

Is there a solution/work around for this problem? Is there a way to still update the rest of the file?

If you need a sample files, I can provide them.

Thanks,
Hayo

[Phil Harvey]

Hi Hayo,

You can use the -m option to ignore makernote errors, which should allow you to write to these files.  But beware that doing so may cause further corruption of the maker note information.

- Phil

[Hayo Baan]

Hello Phil,

Thank you for quickly answering this question. While I was actually working with the perl library to modify the files, the command-line option was helpful too (for my investigation below)

I did some more research on this error though, and I have found a cure (sort of).
To begin, I wanted to be absolute certain what caused the dng error in the first place; I want my image files to be as reliable as possible and keep all info as correct as possible (that is also one of the reasons I'm using the exiftool perl library; I am building my own metadata check scripts).

Anyway, I am also a heavy user of PhotoMechanic so my first object was to make sure both it and the DNG converter/Adobe Camera Raw were not the cause. Here's what I did:

• For a problematic dng, I went back to the original .nef file and converted it to a fresh .dng. Checking with exiftool showed no issues.
• I then applied the same IPTC/XMP settings as were in the bad file, using PhotoMechanic. This should rule out PM as culprit. The updated dng still showed no issue with exiftool.
• I then ran the file through ACR to update its develop settings as with the original (bad) dng. Still no problems according to exiftool. This also meant tha, if all else failed, I always had a way to recreate the dng without makernote coding errors from scratch (albeit slow and timeconsuming).

So now I had two versions of basically the same dng. One with makernote coding errors, and one without. But still no way to reproduce the problem...
Some more experimenting with the files in PhotoMechanic, however, showed a way to get rid of the problem in the bad dng file, and how to recreate it!

File A is the newly created dng, B is the "bad" one.

Code Select Expand

$ exiftool -a -G -s -m -Urgency 20140209_09310308_D3_[AB].dng
======== 20140209_09310308_D3_A.dng
======== 20140209_09310308_D3_B.dng
Warning: Bad NikonScanIFD SubDirectory start - /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_B.dng
Warning: Bad MakerNotes offset for NEFBitDepth - /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_B.dng

Right, no Urgency in the file, ok. So how about setting it (in PM)?

Code Select Expand

======== /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_A.dng
[XMP]           Urgency                         : 1 (most urgent)
[IPTC]          Urgency                         : 1 (most urgent)
======== /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_B.dng
[XMP]           Urgency                         : 1 (most urgent)
[IPTC]          Urgency                         : 1 (most urgent)
Ah, interesting: now there are no errors...
Resetting the urgency in PM to undefined brought back the error as in the first output of exiftool.
I then remembered, that updating the ColorClass (or Rating) of a file in PM somehow set the Urgency to 0 (this time only in the IPTC record though), so I tried that. I actually think this is a minor bug in PM, but hey, I wanted to see what this would do.
As you probably would not have guessed: this also resulted in a file that according to exiftool has no issues

Code Select Expand

======== 20140209_09310308_D3_A.dng
[IPTC]          Urgency                         : 0 (reserved)
======== 20140209_09310308_D3_B.dng
[IPTC]          Urgency                         : 0 (reserved)

So in the whole process of manipulating the file with PM, there sometimes is a problem introduced – which luckily can be corrected.
Perhaps this would allow you to write code to fix the problem too? I am of course happy to help test this

Thanks,
Hayo

[Phil Harvey]

Hi Hayo,

Right, no Urgency in the file, ok. So how about setting it (in PM)?

Code Select Expand

======== /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_A.dng
[XMP]           Urgency                         : 1 (most urgent)
[IPTC]          Urgency                         : 1 (most urgent)
======== /Volumes/Data-2/Temp/DNG/20140209_09310308_D3_B.dng
[XMP]           Urgency                         : 1 (most urgent)
[IPTC]          Urgency                         : 1 (most urgent)
Ah, interesting: now there are no errors...

I think you are fooling yourself.  The exiftool application only prints warnings when extracting if the requested tag wasn't found.  Try requesting -warning instead.

- Phil

[Hayo Baan]

Hi Phil,

Darn. You're right
With -warning I still do get the error. So I'm back to square one and have no work-around. Now to find what's causing this behaviour; clearly not all my dng files suffer from the issue and when I recreate the dng fresh and then apply all metadata and edits again (thereby largely mimicking what I had done prior), the problem does not return either. So what could be causing this???

Cheers,
Hayo

[Phil Harvey]

Hi Hayo,

I can't say what is causing the problem in this specific case, but I do know that Adobe's DNG converter is really bad at handling maker notes.  It restructures the proprietary camera maker notes into a proprietary Adobe format (stupid, I know), sometimes losing information in the process.

- Phil

[Hayo Baan]

I can't say what is causing the problem in this specific case, but I do know that Adobe's DNG converter is really bad at handling maker notes.  It restructures the proprietary camera maker notes into a proprietary Adobe format (stupid, I know), sometimes losing information in the process.

Hmm, doesn't sound very good. As I said, it doesn't seem to make the same mistake twice though; when I redo the whole thing afresh, the error is completely gone???
Do you think the culprit really is the Adobe DNG (converter) or could it also be Lightroom or ACR later in the process too?

I am running exiftool -warning on all my dng files now and indeed, so far there are lots with errors and also lots without. So far I have not found any consistency...

How safe do you think will ignoring the warning and writing the file anyway be?

[Phil Harvey]

Ignoring warnings like this will almost certainly cause a loss of the makernote information indicated in the warning, if it wasn't lost already.

I can't speculate about the culprit.  I do use LR occasionally, but I avoid using DNG specifically because LR tampers with DNG files without asking.  If I keep them as proprietary RAW files, LR leaves them alone.

- Phil

[Hayo Baan]

Hi Phil,

Thank you for your time and trying to help out with this weird (Adobe) issue. So far of the 2259 files I have processed just now, there were 122 errors in 2481 files (it's still processing though). Pity as 122 is a bit much to go through the whole process of manually recreating the dng afresh to get everything back again.

I'll dig deeper into this in the coming future and if I ever find the cause (or better yet: the solution/fix), I'll sure let you know

Now back to my other priorities: processing the files from yesterday's event 

Many thanks so far,
Hayo