How to extract the lens model from Nikon's maker notes

[explicat]

Hi,

first of all, I wanted to say a huge thanks to Phil for your very nice tool and your documentation which is the most helpful I can find about metadata on the internet 

For a small application, I'm trying to figure out the lens model from images taken by the Nikon D800. Unfortunately, it seems like the camera doesn't write the provided exif field, but stores basic information about the lens in the maker notes. The output of exitool shows correctly that 70-200 f/2.8 lens was used.

---- MakerNotes ----
Maker Note Version              : 2.10
Quality                         : Basic
White Balance                   : Auto1
Focus Mode                      : AF-C
Flash Setting                   :
Flash Type                      :
White Balance Fine Tune         : 0 0
WB RB Levels                    : 1.375 1.8828125 1 1
Program Shift                   : 0
Exposure Difference             : 0
Compression                     : JPEG (old-style)
Preview Image Start             : 14272
Preview Image Length            : 25316
Flash Exposure Compensation     : 0
ISO Setting                     : 1250
Image Boundary                  : 0 0 7360 4912
External Flash Exposure Comp    : 0
Flash Exposure Bracket Value    : 0.0
Exposure Bracket Value          : 0
Crop Hi Speed                   : Off (7424x4924 cropped to 7424x4924 at pixel 0,0)
Exposure Tuning                 : 0
Serial Number                   : 6129418
VR Info Version                 : 0100
Vibration Reduction             : On
VR Mode                         : Normal
Active D-Lighting               : Off
Picture Control Version         : 0100
Picture Control Name            : Standard
Picture Control Base            : Standard
Picture Control Adjust          : Default Settings
Picture Control Quick Adjust    : Normal
Brightness                      : Normal
Hue Adjustment                  : None
Filter Effect                   : n/a
Toning Effect                   : n/a
Toning Saturation               : n/a
Timezone                        : +01:00
Daylight Savings                : Yes
Date Display Format             : D/M/Y
ISO Expansion                   : Off
ISO2                            : 1270
ISO Expansion 2                 : Off
Vignette Control                : Off
Auto Distortion Control         : Off
HDR Info Version                : 0100
HDR                             : Off
HDR Level                       : Auto
HDR Smoothing                   : Off
HDR Level 2                     : n/a
Lens Type                       : G VR
Lens                            : 70-200mm f/2.8
Flash Mode                      : Did Not Fire
Shooting Mode                   : Continuous
Shot Info Version               : 0222
Firmware Version                : 1.02a
Repeating Flash Output External : 1
Flash Exposure Comp 2           : 0
Sequence Number                 : 0
Auto Bracketing Set             : AE & Flash
Auto Bracket Order              : 0,-,+
Auto Bracketing Mode            : Flash/Speed
Flash Sync Speed                : 1/320 s (auto FP)
Flash Shutter Speed             : 1/60 s
Flash Control Built-in          : TTL
Commander Channel               : 1
Commander Internal Flash        : Off
Commander Internal Manual Output: 1/10
Commander Group A Mode          : Manual
Commander Group A Manual Output : 1/16
Commander Group B Mode          : Manual
Commander Group B Manual Output : 1/102
Modeling Flash                  : On
Commander Internal TTL Comp     : 0
Commander Group A TTL-AA Comp   : 0
Commander Group B TTL-AA Comp   : 0
Noise Reduction                 : Off
WB GRBG Levels                  : 256 352 482 256
Lens Data Version               : 0204
Exit Pupil Position             : 97.5 mm
AF Aperture                     : 2.8
Focus Position                  : 0x11
Focus Distance                  : 1.58 m
Lens ID Number                  : 162
Lens F Stops                    : 6.00
Min Focal Length                : 71.3 mm
Max Focal Length                : 201.6 mm
Max Aperture At Min Focal       : 2.8
Max Aperture At Max Focal       : 2.8
MCU Version                     : 164
Effective Max Aperture          : 2.8
Retouch History                 : None
Image Data Size                 : 5416953
Shutter Count                   : 11897
Flash Info Version              : 0105
Flash Source                    : None
External Flash Firmware         : n/a
External Flash Flags            : (none)
Flash Commander Mode            : Off
Flash Control Mode              : Off
Flash Compensation              : 0
Flash GN Distance               : 0
Flash Color Filter              : None
Flash Group A Control Mode      : Off
Flash Group B Control Mode      : Off
Flash Group C Control Mode      : Off
Flash Group A Compensation      : 0
Flash Group B Compensation      : 0
Flash Group C Compensation      : 0
External Flash Compensation     : 0
Flash Exposure Comp 3           : 0
Flash Exposure Comp 4           : 0
Multi Exposure Version          : 0100
Multi Exposure Mode             : Off
Multi Exposure Shots            : 0
Multi Exposure Auto Gain        : Off
High ISO Noise Reduction        : Normal
Power Up Time                   : 0000:00:00 00:00:00
AF Info 2 Version               : 0100
Contrast Detect AF              : Off
AF Area Mode                    : Dynamic Area
Phase Detect AF                 : On (51-point)
Primary AF Point                : C9
AF Points Used                  : C9
Contrast Detect AF In Focus     : No
File Info Version               : 0100
Directory Number                : 103
File Number                     : 8973
AF Fine Tune                    : Off
AF Fine Tune Index              : n/a
AF Fine Tune Adj                : 0


In my application I was able to extract the maker notes as byte code (see attachment). They start with the sequence Nikon.....MM.*.....6 as stated here.
By looking at the ASCII output of a bytecode viewer, I can detect sequences like BASIC, AUTO1 and AF-C which correspond to the first tags Quality, White Balance and Focus Mode as shown in exiftool's output. The further ASCII output is not any more human-readable.

The table of Nikon Tags says that lens type and lens can be found at the tags 0x0083 and 0x0084. But how can I find these tags in the byte code and extract the corresponding values?

Do I understand correctly that I should be able to find a particular hex value in the byte code which maps to a certain lens model? Taken from this table, In this case it would be
'A2 48 5C 80 24 24 A4 0E'   = AF-S Nikkor 70-200mm f/2.8G ED VR II
When I convert this hex code to binary and search for that binary sequence in the byte code (as binary), no matches can be found.


Would be great if anyone could help.
Thanks in advance!
explicat

[Phil Harvey]

The Nikon maker notes contain a TIFF-formatted IFD that must be parsed to extract this information.  See the Nikon LensID table for details about what tags are used in determining the exact lens model.  (They are not contiguous in memory.)  I recommend using the exiftool -v3 and/or -htmldump options (in different commands) to see details about how the Nikon maker notes are decoded.

- Phil

[explicat]

Thanks a lot, that is exactly was I was looking for! Especially the htmldump is nice.

With the htmldump I figured out that the maker notes start with
- Header: 18 bytes
- Number of entries: 2 bytes
- For each entry:
   * Tag number: 2 bytes
   * Format: 2 bytes
   * Length: 4 bytes
   * Value offset: 4 bytes

Then you can look up the tag number and jump to the value offset (somehow you need to add 0xa to that offset) where you can find the desired information encoded as given in the format and length field.

[Phil Harvey]

I'm not sure what 0xa you are talking about, but there is often an 0xc that needs to be added to get an absolute offset because the EXIF data often starts after a 0xc bytes in the JPEG file.  The offsets in the Nikon maker notes should represent offsets from the start of the EXIF data.

- Phil

[aaduadok]

According to https://exiftool.org/TagNames/Nikon.html#LensID one needs to extract 6 bytes of "lens data". Question - do you parse that data from tag 0x98?

https://exiftool.org/TagNames/Nikon.html#LensData00

Regards,
Aadu Adok

[aaduadok]

I'll rephrase my question.

After extracting tag 0x98 (from maker-notes) is get lensdata_value=30 32 30 31 28 09 c1 f7 e1 14  e9 1f 0c 02 6f 6e a2 b7 71 e1 d4 79 1c d1 7a bb  19 94 2c e1 b3

Question: from that sequence - how do I know into which of the lensdata "sections" it maps to in https://exiftool.org/TagNames/Nikon.html#LensData00 ?

There are several: LensData00, LensData01, LensData0204, LensData0400, LensData0402, LensData0403, LensData0800

Looking at the first 4 bytes, it kind of hints that my guy should be "LensData0201"? If so - where can I find mapping for other bytes?

Thanks!

[Phil Harvey]

You have to go to the ExifTool Nikon.pm source code to see which LensData table to choose based on the first 4 bytes.  The tables are all found in this code.  Also note that this data in encrypted, but you can find the decryption routine in this code too.

- Phil

[aaduadok]

Phil,

I really appreciate your help! I'm trying to parse Nikon lens data (encrypted) in my C++ code. I have zero experience with perl and it makes reading your code a bit hard

So, this is what I have:

a) tag_0x98 encrypted = 30 32 30 31 28 09 c1 f7 e1 14  e9 1f 0c 02 6f 6e a2 b7 71 e1 d4 79 1c d1 7a bb 19 94 2c e1 b3  (31 bytes)
b) judging by first 4 bytes it falls into https://exiftool.org/TagNames/Nikon.html#LensData01
c) now I'm decrypting the remining 27 bytes (31 - 4) from 'a'
d) according to 'b', LensData01 size should be 13 bytes (after 0201, ending with EffectiveMaxAperture, int8u)

So, the thing I dont get is: decrypting 27 bytes in 'c' should result in 27 bytes of decrypted data. how do I "scale" (or map) those 27 bytes into 13 bytes of LensData01?

I'm not competing with your library, just want to have Nikon lens data to my home site https://aadu.eu  that shows my travel pictures as a browser startup page

[Phil Harvey]

EffectiveMaxAperture is at offset 18, so there are 19 bytes decoded.  15 not including the 4-byte header.  Not all of the lens data may be decoded.  There may be some unknown information after this.  It would help if you told me the camera model and firmware version we are dealing with.  You should be using the exiftool -v3 output to help with your understanding.  It will show the raw and decrypted bytes.

I don't mind if you are developing a competing library/application.  I'm very willing to help for whatever reason.  Competition is great, particularly if you can share some new discoveries of your own.

- Phil

[aaduadok]

Works now! Thank you for your help and have a wonderful 2019!

Aadu from Tallinn, Estonia