Nikon D800 NEFs ==> SubIFD Tags corrupted ==> Rewrite possible?

[spark]

Hi Phil,

I have stumbled upon a problem in which D800 NEFs SubIFD Tags got corrupted, when transfered with older Nikon-Software (Nikon Transfer 1).

Do you see any possibility to rewrite these tags and replace the ones missing, for instance on the basis of a not corrupted NEF?

For comparission I've attached the Metadata in TXT-form, of the same image, in both it's corrupted and uncorrupted form. nkt1.txt represents the corrupted NEF, nkt2 the one that isn't.

Thanks a lot for looking into this and a good weekend to you!
-Sascha

[Phil Harvey]

Hi Sascha,

Wow, it really made a mess of the RAW image IFD.

It may be possible to recover the data, but I can't tell from this output.  It would be tricky though, because you would have to somehow re-generate the list of strip offsets/sizes, but since the raw image is uncompressed, I think this may be possible.

- Phil

[spark]

Hi Phil,
NkTransfer sure did a great job messing up the IFD didn't it ^^

Ufff... yeah, noticed the missing strip offset/sizes
My friend who did this mess is helpless... absence minded as he was, he transfered the NEFs with the first best application that came up... in his case Nikon Transfer 1, which ain't supposed to be used with D800. It was only days later that he noticed the mess. Having used the memory card the NEFs were stored on extensively since, there is no chance of retrieving the originals again The pictures of course were for a big costumer... so he prety much has putten his relations with them at risk.

I'd really like to help him out but haven't managed with my limited knowledge yet. If there's any direction and/or good resources for self-study you could point us at, we'd really appreciate it!
I wouldn't want to ask you for any more of your time but any help is welcome.

Best regards,
Sascha

[Phil Harvey]

Hi Sascha,

Hi Phil,
NkTransfer sure did a great job messing up the IFD didn't it ^^

Certainly a big embarrassment for their software department, that's for sure!  Software like this really should be forward compatible, and should abort with a warning instead of corrupting the file if it sees anything it doesn't recognize.

If you send me one of the corrupted files I'll take a look at it (philharvey66 at gmail.com).  Your only other chance is if someone has already run into the problem and come up with a solution.  But if Google doesn't turn up anything, your only chance may be some custom programming if it is possible at all to recover the file.

- Phil

[spark]

I can only agree come to Nikon Software... Despite the many steps forward they have made, these things clearly shouldn't happen...

As to others having run into the same problem and having found a solution... so far I haven't stumbled upon these people, which is why I turn to you Phil aka my Metadata / RAW filestructure god ^^

I can't find the words to express, how your and the colaborative effort of other exif tool followers/users, has helped me in the past. Donation is on it's way!

[spark]

and done... 44L9667....

I <3 EXIF TOOL 

[Phil Harvey]

Hi Sascha,

I got the images, and the donation, thanks!

I'll take a look at the images (probably tomorrow), and let you know what I find.

- Phil

[Phil Harvey]

I failed in my first attempt to recover the image:  I rebuilt the entries in SubIFD1 by adding the missing tags and using the "OtherImage" as the raw strip data.  Unfortunately, this did not produce a readable image.

I notice that the OtherImageLength is exactly 73111552 bytes in all of the samples you sent.  However, in good D800 NEF images the StripByteCounts varies and is about 3 or 4 MB.  This isn't a good sign, but there is a small chance that the original strip data still exists somewhere in the OtherImage data.  I can test for this if you can provide me with 2 more samples:

1) A good original NEF from the D800.

2) The same NEF as corrupted by Nikon Transfer 1.

You would of course need to shoot a new image, then download it using the 2 different techniques (by copying directly, and by using Nikon Transfer 1).

- Phil

[spark]

Good man yourself.
Actually have exactly the files you need.

Will upload them and send you a link via email!

Thanks again for your efforts!

-Sascha

[spark]

Having troubles with the upload... will send you the email later this day...

[Phil Harvey]

Hi Sascha,

I got the images, thanks.

I think I can recover the original raw image, but it will take some work.  The corrupted file does contain the original raw data.  Give me a bit more time to work on this.

- Phil

[spark]

Hi Phil,

I'm happy enough that somebody seems to be able to help me out on that matter, so thanks again and please take your time

In case you're able to recover the images, I'd be happy if you could let me know how.
The more detail, the better but any info would do.

Cheers and best regards,
Sascha

[Phil Harvey]

I did it!

I have a created a patched version of ExifTool that is able to fix all of the corrupted NEF images you sent.

It currently works on uncompressed NEF images only.  I would need samples of compressed NEF images to be able to fix these too.

The command looks like this:

exiftool -@ fix_corrupted_nef.args SRCDIR -o OUTDIR

With this command, exiftool will repair all of the corrupted uncompressed NEF images in directory SRCDIR and write the fixed images to directory OUTDIR.

Here is the patched version of ExifTool (including the "fix_corrupted_nef.args" argument file).  You must have Perl installed to run this.  (you already have this installed unless you are running Windows)

The patched version effectively regenerates the necessary StripOffsets and StripByteCounts in SubIFD1 of the NEF, and the .args file writes the other tags required in SubIFD1.

Let me know if you have any questions or problems.

- Phil

Update: This procedure has been improved.  Click here for complete details.

[spark]

Hi Phil,

Tried your suggested fix but the NEFs created this way, still can't be opened (at least not in Capture NX 2).
Will try if they'll open in CS5 and will let you know what happens.

You said it worked for you? Have you confirmed that the NEFs can be opened again and if yes, in which software?
If everything works fine for you, I'd be happy if you could upload them 6 D800 NEFs (Netto_xxx.nef) somewhere, where I could download them.


Checking the changes in Metadate between original and "fixed" NEFs I've noticed, that compared to the last samples I sent you (Nkt1.nef and NKT2.nef), there were a few differences still in SubIFD, SubIFD1, SubIFD2 and PreviewIFD.
I edited the "fix_corrupted_nef.args" to include those changes, which worked in all cases except for PreviewIFD, which I don't think is so relevant anyway, unfortunately to no avail...

Come to the StripOffsets in SubIFD1 I noticed a difference of 1 byte too (according to Exif Tool) but didn't get to check it further.

Thanks for everything you did so far, especially the bit about regenerating the StripOffsets and StripBytecounts!

I'll probably get back to you on the matter of compressed NEFs...

-Sascha

[Phil Harvey]

Hi Sascha,

Yes, it worked for me using Apple Preview on Mac OS X Lion to open the fixed NEF image.  Just to be sure, the fixed NEF should look like this:

> exiftool -a -subifd1:all fixed.nef
Subfile Type                    : Full-resolution Image
Image Width                     : 7424
Image Height                    : 4924
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : (Binary data 10949 bytes, use -b option to extract)
Samples Per Pixel               : 1
Rows Per Strip                  : 4924
Strip Byte Counts               : (Binary data 7385 bytes, use -b option to extract)
X Resolution                    : 300
Y Resolution                    : 300
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Sensing Method                  : One-chip color area

The StripOffsets size may be different, but everything else should be identical for all images.

I don't have any other software I can test with here, but I have LR4 at home that I can check.

- Phil