Discovering Nikon D610 FlashInfo0106/ShotInfo0232

[TorC]

I have the D610, an external third-party speedlite (Flashpoint zoom Li-on), and am interested in helping to decode these segments.  Are there any tips on what information is needed and how to best gather it?  From the reading I have done, I take it the first thing to do is gather metadata diffs of images with nearly identical settings with records.  Any more specific tips?

Edit to add:  I also have a D70 if comparisons between metadata are useful anywhere.

[TorC]

Well, I took a first try and gathered a set of images adjusting the main and flash exposure compensation with the on-board flash.

Here is what I got from diffs of the flash force on at flash EC of 0,+1,-3 (for Flash Info V0106):

exiftool -U -a '-*flash*' $file

Code Select Expand


Flash Exposure Compensation     : 0=0,   +1=+1,   -3=-3
Nikon Flash Info Unknown 0x001d : 0=0,   +1=250,  -3=18
Nikon Flash Info Unknown 0x0027 : 0=0,   +1=250,  -3=18
Nikon Flash Info Unknown 0x002b : 0=46,  +1=58,   -3=10


Am I on the right track with this, and are there other tools to speed data collection I should be using?  Am I right to use basic small jpgs to minimize space required for most testing?

Thank you,

Tor

[Phil Harvey]

Hi Tor,

Thanks for the offer.  There is a lot of flash information in the Nikon custom settings that takes some work to decode.  Take a look at the Nikon Tags documentation to see what has been decoded for other models.  You are on the right track with what you are doing.  A simple diff of the -U output is what you want.  There will be a lot of noise (bytes that change randomly), but if you know where to look you should be able to figure out where the settings are stored.  It would be great if you could also go through the full set of custom settings.  If you want, I can help with decoding... shooting small JPEG's is the thing to do. Be sure to change each setting back to the default before moving on to the next one.

- Phil

[Hayo Baan]

Phil already confirmed you are on the right track

Using small jpgs is a smart choice as it will contain all metadata, while still staying small. To minimise file size even more, consider taking images with the lens cap (as long as that doesn't interfere with your flash tests, of course).

[TorC]

Thanks for the confirmation.  I'll start work as I have time available.  Can "default" setting mean "my preferred setting", or is there a reason it needs to be "factory default"?

[Phil Harvey]

Your preferred setting is fine.  The point is to have only one different setting change for each image.  Then you can compare any two images and you should see only the two settings that changed.

- Phil

[TorC]

Hmmm... tried one from a test set of three.  I'm not entirely sure how to start reading it - or perhaps I just don't have enough data.  I've attached the exiftool -U output from them.  Setting is custom setting A1, AF-C priority selection - release, and A2, AF-S priority selection - release.  Base is focus priority on both.

Looking at this, I'm going to also set a fixed WB (I see some unknown colour balance that I suspect is the Auto-WB) and turn VR off in addition to manual fixed shutter/aperture/focus.  Any other settings you can name that I can fix in the base settings to make most of the diffs easier to parse?

It'll probably be a few days before I can attack this in earnest, but I can use the time to get my workflow planned out to make the parsing as easy as possible.

P.S.  I've attached the 'exiftool -U' output to be sure we're looking at the same thing.  Turns out it 3x the size of the original (dark) jpgs.

[Phil Harvey]

With such a small set it is very difficult to find the location of the settings.  Try changing a setting back and forth a number of times and mail me the JPEG images (philharvey66 at gmail.com).  Use a pseudo-random pattern when changing the setting, something like this:  off-on-on-off-off-off-on

Once we determine the location for one setting, the others will all be nearby.

- Phil

[TorC]

Sent you the files.  I spent too much time on it this morning, and think Custom setting A1, AF-C priority selection, is in:

-   Nikon Shot Info 0x07cf          : 3
+   Nikon Shot Info 0x07cf          : 131

Found this by doing 'diff -U1000 | grep -v [+-]' to get what stayed the same between same setting files and then a diff -u of the result.  This is the only one with a neat change with setting change.

Edit to note: this is the only change that showed up without other changes within 3 lines.  The final diff was still rather longer than I expected.

[Phil Harvey]

Excellent!  Good work!!  It looks like you got it.  So the A1 AF-CPrioritySelection setting is offset 0x07cf bitmask 0x80.

Nice.  All of the other settings should be nearby 0x07cf.  And you may find that the structure of the settings block is similar to other cameras.

Let me know and I'll add anything you find.  It would be easiest for me if you present your findings like this:

0x07cf mask 0x80 - A1: AF-CPrioritySelection, 0x00 = Release, 0x80 = Focus

- Phil

[TorC]

I think I mostly understand.  I've sent you a set of JPGs from the next setting to confirm I'm translating this correctly.  If I'm right, then:

0x07cf mask 0x20 - A2: AF-SPrioritySelection, 0x00 = Release, 0x20 = Focus

I'm still not sure where the +3 comes from - or is the assumption for now that a different setting is also on 0x07cf and causing that?

[TorC]

Thinking a bit more about trying to automate the process.  Can I safely assume a colon is only a field separator in exiftool output?

I'm thinking something like the following.  basefile is the condensed exiftool ouput from a set of files in base configuration with changing lines stripped out.  testfile is the exiftool output for a setting to check.

Code Select Expand


cat testfile|while read setting
do
  if [[ grep $setting_field1 $basefile ]]
  then
    setting >>$settingsfile
  fi
done

diff -u $basefile $settings


Undoubtedly not working code, and surely "should" be optimized, but it should give the idea.  Probably will miss some shot info that changes, but I think it will be reasonably robust for the custom settings, at least.

[Phil Harvey]

Great, thanks.  Yes, this is exactly correct for AF-SPrioritySelection.  Excellent!!

The remaining unknown value of 3 will be due to some other custom setting that you'll discover later.  So far, the decoding is similar to the D810, so I predict that the 3 may be FocusTrackingLockOn (if the D610 has this feature).

I find my attached script very useful.  Here is a sample usage:

Code Select Expand

> exiftool ~/Desktop/D610\ AF-S/ -a -U -G1 -w tmp/%f.txt
    1 directories scanned
    9 image files read
    9 output files created
> showtest -h -s "Nikon Shot Info" tmp
Looking for 'Nikon Shot Info 0x0000'
Enter Tag ID (RTN=0x0000, p=previous, q=quit): 0x7c8
Enter Tag ID (RTN=0x07c9, p=previous, q=quit):
Enter Tag ID (RTN=0x07ca, p=previous, q=quit):
Enter Tag ID (RTN=0x07cb, p=previous, q=quit):
Enter Tag ID (RTN=0x07cc, p=previous, q=quit):
Enter Tag ID (RTN=0x07cd, p=previous, q=quit):
Enter Tag ID (RTN=0x07ce, p=previous, q=quit):
Enter Tag ID (RTN=0x07cf, p=previous, q=quit):
tmp/DSC_5232.txt:[Nikon]         Nikon Shot Info 0x07cf          : 0x83
tmp/DSC_5233.txt:[Nikon]         Nikon Shot Info 0x07cf          : 0xa3
tmp/DSC_5234.txt:[Nikon]         Nikon Shot Info 0x07cf          : 0xa3
tmp/DSC_5235.txt:[Nikon]         Nikon Shot Info 0x07cf          : 0x83
tmp/DSC_5236.txt:[Nikon]         Nikon Shot Info 0x07cf          : 0xa3
tmp/DSC_5237.txt:[Nikon]         Nikon Shot Info 0x07cf          : 0xa3
tmp/DSC_5238.txt:[Nikon]         Nikon Shot Info 0x07cf          : 0x83
tmp/DSC_5239.txt:[Nikon]         Nikon Shot Info 0x07cf          : 0x83
tmp/DSC_5240.txt:[Nikon]         Nikon Shot Info 0x07cf          : 0x23
Enter Tag ID (RTN=0x07d0, p=previous, q=quit): q

I've just added the showtest -h option to make decoding these bitmasks easier.  I'm glad you're a programmer and already understand hexadecimal.

- Phil

[TorC]

On checking the custom settings page, I would agree that is likely, and I believe the timing is about right for them to be similar.  I can't verify, but the D600 is likely to be very close to the D610 - perhaps identical +/- $tagoffset.  Any chance of a table of contents for the custom settings page to make it easier to compare and see which cameras have known info?

I'll look at the script later.

I'm glad you're a programmer and already understand hexadecimal.

At least I know enough Bash and Python to get some useful work done with effort and know the principles of hex.  Whether that qualifies me as a programmer may still be an open question.

[Phil Harvey]

I can't verify, but the D600 is likely to be very close to the D610 - perhaps identical +/- $tagoffset.

Yes.  Too bad the D600 settings haven't been decoded yet.

Any chance of a table of contents for the custom settings page to make it easier to compare and see which cameras have known info?

Not likely I'm afraid.

- Phil

[TorC]

By TOC, I was hoping for links at the top of the page to the anchors already in the page for each camera to make it easy to A) know which cameras have at least some settings decoded and B) easy to get to the list of settings for each of those cameras.   More than that I understand would be more work than it's worth.

Your guess of:

Code Select Expand


0x07cf mask 0x07 - A3: FocusTrackingLockOn, 0x0 = Off, 0x1 = 1 (Short), 0x2 = 2, 0x3 = 3 (Normal), 0x4 = 4, 0x5 = 5 (Long)

is spot on.

Time to get the spreadsheet properly laid out.  Would a DSV like below be good?

Code Select Expand


$offset, $bitmask, $setting_menu_position, $setting_name, 0xNN, $option_name


Is there by chance an easy list for the D810 in particular and perhaps Nikon cameras in general of which settings are in which byte?
Now I see how the list is ordered - by byte_number.setting_within_byte.  That works.  Should go a lot easier now.

Do you by chance have a DSV of the D810 settings to work from?

I did note that D810 setting 2.3 AFPointBrightness is in my (U.S. English) camera and downloaded D810 manual as AF Point Illumination.

[Phil Harvey]

By TOC, I was hoping for links at the top of the page to the anchors already in the page

Look at the list of links for the ShotInfo tag on the main Nikon page.

Your guess of:

Code Select Expand


0x07cf mask 0x07 - A3: FocusTrackingLockOn, 0x0 = Off, 0x1 = 1 (Short), 0x2 = 2, 0x3 = 3 (Normal), 0x4 = 4, 0x5 = 5 (Long)

is spot on.

Great.  You could maybe consult the D810 decoding for reference then.

Time to get the spreadsheet properly laid out.  Would a DSV like below be good?

Sure.

Do you by chance have a DSV of the D810 settings to work from?

No.  The format I work from is a Perl structure (see the source code).  ExifTool will output XML format (with the -listx option), but not CSV.

I did note that D810 setting 2.3 AFPointBrightness is in my (U.S. English) camera and downloaded D810 manual as AF Point Illumination.

Yes.  Nikon sometimes changes the names of the settings, but the tag names in ExifTool generally correspond to their original name (for the earliest model with the setting).

- Phil

[TorC]

Looking just fine.  Could showtest sort filenames when displaying tags?  Right now with the workflow you gave me I end up with my test files all out of order, which makes it hard to keep track of which setting is on which line.  I made a (failed) try, but it's a bit of a challenge with zero knowledge of perl.

[Phil Harvey]

Very odd.  The files should already be in alphabetical order on Unix systems.  They were in the output I provided.  I could see this happening on Windows though.  To sort the result manually, change this line of showtest:

Code Select Expand

    my @lines = `$cmd`;

to this

Code Select Expand

    my @lines = sort `$cmd`;

- Phil

[TorC]

Well, I am most definitely not on windows and the output was definitely not in alphabetical order.  I'm using GNU bash, version 4.3.30(1)-release (x86_64-pc-linux-gnu) on Debian Jessie.

Actually, I've found that Linux systems will often feed files in some form of disk order which may or may not be alphabetical when not explicitly sorted.  I've seen files processed out of ASCII order before with the (dangerous) 'ls|while read' combination (or at least I seem to recall such).  Probably does provide a performance increase for cases where humans only care that the job gets done.

In any case, the change fixes the problem.  Thanks.

Edit to add:  After staying up too late, I've gotten through the Bs in the custom settings.  I also figured out the fine-tune Optimal Exposure (8.2-9.2 on both D610 and D810), which values should be safe to add to the D810.  Any notes like that I'll add with a sharp at the start of a field after all real settings on the line.  Specifically, it represents +/-1EV in 1/6th EV steps as a 4-bit twos complement signed integer.

[Phil Harvey]

Actually, I've found that Linux systems will often feed files in some form of disk order which may or may not be alphabetical when not explicitly sorted.

Interesting.  I've never see this myself, but I mainly use OS X (FreeBSD).

Edit to add:  After staying up too late, I've gotten through the Bs in the custom settings.  I also figured out the fine-tune Optimal Exposure (8.2-9.2 on both D610 and D810), which values should be safe to add to the D810.  Any notes like that I'll add with a sharp at the start of a field after all real settings on the line.  Specifically, it represents +/-1EV in 1/6th EV steps as a 4-bit twos complement signed integer.

Excellent!  I appreciate all of the work you are doing.

- Phil

[TorC]

If the setting is simply the decimal number in hex, do you need the values laid out?  For example, the analogue to the D801 20.3 SelfTimerShotCount is just a 4-bit unsigned number (and conveniently has a 0x0f mask).

I wrote out the Fine-tuneOptimalExposure, but that wants a bit of translation (print sign and add "/6 EV"), which I figure you need.

[Phil Harvey]

If the setting is simply the decimal number in hex, do you need the values laid out?  For example, the analogue to the D801 20.3 SelfTimerShotCount is just a 4-bit unsigned number (and conveniently has a 0x0f mask).

No.  For numbers that aren't converted I don't need a conversion.  ExifTool will mask the byte.

I wrote out the Fine-tuneOptimalExposure, but that wants a bit of translation (print sign and add "/6 EV"), which I figure you need.

Yes.  If a math formula should be applied, just give the formula (eg. val/6 EV).

Thanks!!

- Phil

[TorC]

More specifically, in the case the formula should be printed to match the menu/manual.  The following is a 4-bit signed integer, caring only for the unmasked bits, including one with a 0xf0 mask.  Is this a case where I can give you something like "$val/6 EV"?  Or is exiftool unprepared for this particular case, and I can only do this if it is a uint?

Code Select Expand


0x07d6  Mask  0x0f  B5A  FineTuneOptMatrixMetering  0x0=0 (Off), 0x1=+1/6 EV, 0xf=-1/6 EV (binary -1), ...


[Phil Harvey]

This case is a bit tricky since it is signed and you may want the EV values to be formatting in a specific way.  In this case I would probably just do sprintf("%+.1f", ($val < 8 ? $val : $val - 16) / 6)), and not worry about displaying it as a fraction.  Here I have handled the negative values manually.

- Phil

[TorC]

Actually, not so difficult because it is displayed as an unreduced fraction where the integer is the numerator.  The "/6 EV" can be treated as a string.  Perhaps then something like '0x0=0 (Off), sprintf("%+d/6 EV", ($val < 8 ? $val : $val - 16))', though I've probably mangled the syntax.

[Phil Harvey]

Looks good!

- Phil

[TorC]

Great.  I've gotten through the "C" settings.  Attached is the CSV of what I have so far.  Field 1 is a derived field I added to help keep track of where to find a setting in existing settings, and can be safely discarded.  Let me know if there are any problems with the way I am going about this.

Tor

[Phil Harvey]

Hi Tor,

I haven't gone through this in detail, so I may have some questions later, but this looks fantastic so far!

- Phil

[TorC]

I've been busy with other things, but I'm thinking I might get back to this soon.  Fortunately I posted what I did, because in the intervening time I had a hard drive failure that destroyed everything I'd done on this.

[Phil Harvey]

Hi Tor,

I've just added your discoveries so far to ExifTool and they will appear in the next version (10.26) when released.

It would be great to have the rest of the settings if you can decode them.

Thanks again for all you have done so far.

- Phil