Nikon D800E EXIF NEF-> JPG Score: 65 tags Good, 1 mangled, 22 verified, 43 MIA

Started by BrianP, March 09, 2017, 03:53:37 PM

Previous topic - Next topic

BrianP

I wrote a Perl script to whittle down the ~230 lines of EXIF data on a Nikon D800E .NEF file and generate a summary report. I also extract the JpgFromRaw for weeding but Nikon only provides 3 of the 65 interesting NEF tags in the JpgFromRaw.

My goal is to have the same EXIF data in the convenient JPG extract as in the original NEF.

I wrote a script to transplant the missing items, adding 9 custom tags to the .ExifTool_config for tags which were "not writable" or had other errors in SetNewValue().
The results are only 34% successful with exiftool, latest version 10.46:

  • 65 tags with SetNewValue($tag, $val)  -> TRUE
  • 1 tag mangled (FlashMode::  JPG tag value  !==  NEF value)
  • 22 tags copied VERBATIM
  • 42 tags successfully set to new value but NOT subsequently found

1 tag mangled:   
JPG tag 'FlashMode' -> 'On' != NEF_tag 'Fired, TTL Mode'!

22 tags copied VERBATIM:
Aperture ColorSpace CreateDate ExposureCompensation ExposureMode ExposureTime Flash FlashCompensation FocalLength ISO LensID LightValue MeteringMode Model Orientation SequenceNumber SerialNumber ShutterCount ShutterSpeed SubSecCreateDate UserComment WB_GRBGLevels

42 tags successfully set, but missing in JPG:
AFAperture AFAreaMode AFPointsUsed CommanderChannel CommanderGroupAManualOutput CommanderGroupAMode CommanderGroupA_TTL-AAComp CommanderInternalFlash CommanderInternalManualOutput CommanderInternalTTLComp ContrastDetectAF ContrastDetectAFInFocus DOF ExposureBracketValue ExposureDifference ExternalFlashCompensation ExternalFlashExposureComp ExternalFlashFlags FOV FlashCommanderMode FlashControlBuilt-in FlashControlMode FlashExposureComp4 FlashGNDistance FlashGroupACompensation FlashGroupAControlMode FlashSetting FlashShutterSpeed FlashSource FlashSyncSpeed FlashType FocusDistance FocusMode FocusPosition HyperfocalDistance MinFocalLength PhaseDetectAF PrimaryAFPoint RepeatingFlashOutputExternal ShootingMode VibrationReduction WhiteBalance

UserDefined tags:
ShutterCount, Aperture, LightValue, DOF, FOV, HyperfocalDistance, LensID, ShutterSpeed, WB_GRBGLevels

Of the 9 attempted UserDefines, 6 work, 3 fail (%Image::ExifTool::UserDefined)
OK:  ShutterCount, Aperture, LightValue, LensID, ShutterSpeed, WB_GRBGLevels
BAD: DOF, FOV, HyperfocalDistance,

------------------------------------------------------
/home/brianp/bin/.ExifTool_config  excerpt:

%Image::ExifTool::UserDefined = (
    # All EXIF tags are added to the Main table, and WriteGroup is used to
    # specify where the tag is written (default is ExifIFD if not specified):
    'Image::ExifTool::Exif::Main' => {
        # Example 1.  EXIF:NewEXIFTag
        0xd000 => {
            Name => 'ShutterCount',
            Writable => 'int32u',
            WriteGroup => 'IFD0',
        },
        0xd002 => {
            Name => 'Aperture',
            Writable => 'string',
            WriteGroup => 'IFD0',
        },
        0xd003 => {
            Name => 'LightValue',
            Writable => 'rational64s',
            WriteGroup => 'IFD0',
        },
        0xd004 => {
            Name => 'DOF',                             <<  FAILS >>
            Writable => 'string',
            WriteGroup => 'IFD0',
        },
        0xd005 => {
            Name => 'FOV',                            <<  FAILS >>
            Writable => 'string',
            WriteGroup => 'IFD0',
        },
        0xd006 => {
            Name => 'HyperfocalDistance',     <<  FAILS >>
            Writable => 'string',
            WriteGroup => 'IFD0',
        },
        0xd007 => {
            Name => 'LensID',
            Writable => 'string',
            WriteGroup => 'IFD0',
        },
        0xd008 => {
            Name => 'ShutterSpeed',
            Writable => 'string',
            WriteGroup => 'IFD0',
        },
        0xd009 => {
            Name => 'WB_GRBGLevels',
            Writable => 'string',
            WriteGroup => 'IFD0',
        },
    },

------------------------------------

Full source of transplant function:

%ET_OPT=(FastScan=>1, FixBase=>1, MakerNotes=>1, Duplicates=>0, Binary=>1,
   IgnoreMinorErrors=>1, List=>1, Sort=>'Input');

sub nef_exif_to_jfn()  {
    my(@etag, %t2v, $tag, $val, $nfn, $jfn, $ftag, $et, $jt, %ii, $verbose, $ii);
    my($success, $estr, $good, $bad, $no_exif_backup, $debug, %good, %ji, @good);
    my($ntag);
    use Image::ExifTool qw(:Public);
   $nfn='za-2017.0308-263694.nef';      # Nikon NEF
   $jfn='za-2017.0308-263694.jfn.jpg';  # exiftool -b -JpgFromRaw
   $debug=1;  $verbose=1;  $good=0;  $bad=0;  $no_exif_backup=0; %good=();
   @etag=qw(AFAperture AFAreaMode AFPointsUsed Aperture ApproximateFocusDistance Caption Caption-Abstract Category ColorSpace ColorTemperature CommanderChannel CommanderGroupA_TTL-AAComp CommanderGroupAManualOutput CommanderGroupAMode CommanderInternalFlash CommanderInternalManualOutput CommanderInternalTTLComp ContrastDetectAF ContrastDetectAFInFocus CreateDate DOF ExposureBracketValue ExposureCompensation ExposureDifference ExposureMode ExposureTime ExternalFlashCompensation ExternalFlashExposureComp ExternalFlashFlags Flash FlashCommanderMode FlashCompensation FlashControlBuilt-in FlashControlMode FlashExposureComp4 FlashFocalLength FlashGNDistance FlashGroupACompensation FlashGroupAControlMode FlashMode FlashSetting FlashShutterSpeed FlashSource FlashSyncSpeed FlashType FocalLength FocusDistance FocusMode FocusPosition FOV HyperfocalDistance ImageAltText ImageNumber IntellectualGenre ISO Keywords LensID LightValue MeteringMode MinFocalLength Model Orientation PhaseDetectAF PrimaryAFPoint RepeatingFlashCountExternal RepeatingFlashOutputExternal Scene SequenceNumber SerialNumber ShootingMode ShutterCount ShutterSpeed SubjectCode SubSecCreateDate SupplementalCategories Tint Title UserComment VibrationReduction WB_GRBGLevels WhiteBalance);


   $ftag=clone(\@etag);  # Clones array, returns ref. ImageInfo mangles @
   $et = new Image::ExifTool;
   $jt = new Image::ExifTool;
   %ii = %{$et->ImageInfo($nfn, \%ET_OPT, $ftag)};  $ii=0;
   printf("M: $ii) Found: %s\n\n  Searched for [%d]:\n%s\n\n",
      &dumphash(\%ii, 'II hash', 0, 1), scalar @etag, join(" ", @etag))
      if $verbose;
   $ii=-1;
   foreach $tag (sort keys %ii)  {
      $ii++;
      $val = $ii{$tag};
      ($success, $estr) = $jt->SetNewValue($tag, $val);
      if($success)  { 
         print("GOOD tag $tag -> '$val'\n");
         $good{$tag} = $val;  next;  }

      print("$ii) ERROR on tag $tag -> $estr\n"); 
      $tag = "EXIF:$tag";  # Prepend 'EXIF:' to raw tag
      ($success, $estr) = $jt->SetNewValue($tag, $val);
      if($success)  {
         print("$ii) FIXED tag $tag\n");  $good{$tag} = $val;  next;  }
      print("$ii) Failed again on tag $tag\n");
      $bad++;  next; 

   }
   $good = scalar keys %good;
   $ii++;  # Convert from 0 based index to 1 based COUNT
   print("NETJ: Tried $ii new values; $good good, $bad bad in jpg $jfn\n\n");

>>NETJ: Tried 65 new values; 65 good, 0 bad in jpg za-2017.0308-263694.jfn.jpg

   if($ii == $good)  {
      &write_exif($jt, $jfn, $no_exif_backup, $debug);  }
   else  {  printf("Good $good != tried $ii, NOT writing JPG\n\n");  }


   print("\n=============================\nChecking:\n");
   undef $et;  undef $jt;  @good=();
   $ftag=clone(\@etag);  # Clones array, returns ref. ImageInfo mangles @
   $jt = new Image::ExifTool;
   %ji = %{$jt->ImageInfo($jfn, \%ET_OPT, $ftag)};  $ii=0;
   printf("M: $ii) Found: %s\n\n  Searched for [%d]:\n%s\n\n",
      &dumphash(\%ji, 'II hash', 0, 1), scalar @etag, join(" ", @etag))
      if $verbose;
   $ii=-1;  $good=0;  $bad=0;
   foreach $tag (sort keys %ji)  {
      $val = $ji{$tag};
      if(defined(($ntag=$ii{$tag}))  &&  ($ntag eq $val))  {
         $good++;
         push @good, $tag;
         delete $ii{$tag};  # Tag -> val duplicated on II & JI
         next;  }
      printf("ERROR %d! JPG tag '$tag' -> '$val' != NEF_tag '$ntag' !\n",
         $bad++);
   }

   printf("M: $good good, $bad bad;\n%d tags copied VERBATIM: %s\n\n%s\n%s\n",
      $good, join(" ", @good), &dumphash(\%ii, 'NEF Orphan tags', 0, 1),
      join(" ", sort keys %ii));
}

=====================================
NEF EXIF extract Report (with XX indicating ERROR):

XX    AFArea                =>  Dynamic, 9 points, A7
Aperture              =>  8.0
XX    DOF_FOV               =>  0.18m (1.69-1.87m), 23.7°
ExposureCompensation  =>  -0.67
XX    ExposureDifference    =>  -1.40 -> Error=-0.7 EV Dark
XX    ExposureTime          =>  1/250 sec, ExpMode=Manual, VibRed=On
FileName              =>  za-2017.0308-263694.nef, 73 MB  << Expected difference
XX    Flash                 =>  Fired Rear, iTTL-BL +0.3EV, Commander On (-2.0EV)
XX    FlashShutterSync      =>  1/8 - 1/250 sec
FocalLength           =>  82.0 mm
XX    FocusDistance         =>  1.78 m
XX    FocusMode             =>  AF-C
XX    HyperfocalDistance    =>  34.00 m
ISO                   =>  640
Lens                  =>  AF-S Nikkor 28-300mm f/3.5-5.6G ED VR
LightValue            =>  11.3
MeteringMode          =>  Multi-segment
Model                 =>  Nikon D800E 3000955
ShutterCount          =>  263694
SubSecCreateDate      =>  2017-03-08 14:13:04.90
UserComment           =>  BPB_3943.nef
WB_GRBGLevels         =>  256 526 338 256
XX    WhiteBalance          =>  Auto1

-----------------------------------------------
EXIF values Nikon copies to JpgFromRaw

CreateDate    =>  2017-03-08 15:09:21
FileName      =>  za-2017.0308-263835.jfn.o.jpg, 1497 kB
ShutterCount  =>  263835

----------------------------------------
Result of NEF EXIF transplanted into the JPG

Aperture              =>  8.0
ExposureCompensation  =>  -0.67
ExposureTime          =>  1/250 sec, ExpMode=Manual
FileName              =>  za-2017.0308-263694.jfn.jpg, 2.2 MB
Flash                 =>  On, Return not detected
FlashCompensation     =>  -2
FlashMode             =>  On
FocalLength           =>  82.0 mm
ISO                   =>  640
Lens                  =>  AF-S Nikkor 28-300mm f/3.5-5.6G ED VR
LightValue            =>  11.3
MeteringMode          =>  Multi-segment
Model                 =>  Nikon D800E 3000955
ShutterCount          =>  263694
ShutterSpeed          =>  1/250
SubSecCreateDate      =>  2017-03-08 14:13:04.90
UserComment           =>  BPB_3943.nef
WB_GRBGLevels         =>  256 526 338 256


==================================================
Net result from log file:

ERROR 0! JPG tag 'FlashMode' -> 'On' != NEF_tag 'Fired, TTL Mode' !
M: 22 good, 1 bad;
22 tags copied VERBATIM: Aperture ColorSpace CreateDate ExposureCompensation ExposureMode ExposureTime Flash FlashCompensation FocalLength ISO LensID LightValue MeteringMode Model Orientation SequenceNumber SerialNumber ShutterCount ShutterSpeed SubSecCreateDate UserComment WB_GRBGLevels

{43}, NEF Orphan tags
=================================================

And, the command line version confirms the Perl script results exactly:

brianp@raptor:~/tmp/ltza$ /usr/local/bin/exiftool -ver -*AFAperture -*AFAreaMode -*AFPointsUsed -*CommanderChannel -*CommanderGroupAManualOutput -*CommanderGroupAMode -*CommanderGroupA_TTL-AAComp -*CommanderInternalFlash -*CommanderInternalManualOutput -*CommanderInternalTTLComp -*ContrastDetectAF -*ContrastDetectAFInFocus -*DOF -*ExposureBracketValue -*ExposureDifference -*ExternalFlashCompensation -*ExternalFlashExposureComp -*ExternalFlashFlags -*FOV -*FlashCommanderMode -*FlashControlBuilt-in -*FlashControlMode -*FlashExposureComp4 -*FlashGNDistance -*FlashGroupACompensation -*FlashGroupAControlMode -*FlashMode -*FlashSetting -*FlashShutterSpeed -*FlashSource -*FlashSyncSpeed -*FlashType -*FocusDistance -*FocusMode -*FocusPosition -*HyperfocalDistance -*MinFocalLength -*PhaseDetectAF -*PrimaryAFPoint -*RepeatingFlashOutputExternal -*ShootingMode -*VibrationReduction -*WhiteBalance za-2017.0308-263694.jfn.jpg
10.46
Flash Mode                      : On

Latest ExifTool version=10.46, 1 tag (FlashMode) mangled, everything else MIA.


brianp@raptor:~/tmp/ltza$ /usr/local/bin/exiftool -*FlashMode* za-2017.0308-263694.jfn.o.jpg za-2017.0308-263694.jfn.jpg za-2017.0308-263694.nef
======== za-2017.0308-263694.jfn.o.jpg  << extracted from NEF
======== za-2017.0308-263694.jfn.jpg     << After transplant
Flash Mode                      : On
======== za-2017.0308-263694.nef          << Raw_file
Flash Mode                      : Fired, TTL Mode

FlashMode scrambled




Phil Harvey

TLDR;  I suspect most of the problems you are having are due to an attempt to create individual makernote tags, which can't be done.  Am I correct?  If so, did you consider copying the makernotes as a block?: $et->SetNewValuesFromFile($nefFile, "MakerNotes",...);

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

BrianP

Phil,
I will try your suggestion.

The root problem here appears to be that Nikon provides only 3 of the 65 interesting EXIF NEF items in JPGs. I would like to know the FocusDistance whether I am looking at a JPG or the NEF. Why _should_ the WhiteBalance and Field_Of_View angle be hidden from JPG users? 

Rather than having each user patch engineering inconsistencies for each vendor/model/bios, how hard would it be to format this vacuous, Goldbergian morass, start from scratch and add a few, simple key -> value, XML-like hash items? 

There comes a point when scrapping makes more sense than fixing. There is ~nothing to lose and much to gain by starting with a clean slate. How hard could it be to take a bitmap, add a few XML key->value pairs, add a CRC and call it a JPG or a PNG?

The other question is how can SetNewValue($tag, $val) return SUCCESS only to have the same tool fail to extract the data just added?  Black Hole Storage Feature? The data are actually in there, I just can't get them out (easily). STRINGS() shows that some items did get written:

brianp@raptor:~/tmp/ltza$ strings -n 8  za-2017.0308-263694.jfn.jpg
NIKON D800E
0.18 m (1.69 - 1.87 m)
23.7 deg (0.74 m)
AF-S Nikkor 28-300mm f/3.5-5.6G ED VR
256 526 338 256
2017:03:08 14:13:04
BPB_3943.nef3000955
<snip>

How about:
$et = new Image::ExifTool($Mal_Engineered_JPG_with_nothing_of_value_but_bitmap.jpg);
$et->Destroy_Goldbergian_Junk_EXIF();  # Purge Daedalean rat's nest of degraded EXIF residuum
$et->Add_Key_Val_Hash(\%my_keys_vals);  # Add key->val pairs without IRS Schedule_ZZ complexity

6 of the 9 UserDefined items worked, but 3 failed. We need a PHIL_Format,  that just works...

Thx 1E6,
    Brian

BrianP

Phil,

Results of copying makernotes to JPG:
$jt->SetNewValuesFromFile($nef_filename, "MakerNotes"); 


brianp@raptor:~/tmp/ltza$ dif.pl -et za-2017.0308-263694.nef.exf za-2017.0308-263694.jfn.jpg.exf
  03) Aperture                       =>  8.0
  10) DOF_FOV                        =>  0.18m (1.69-1.87m), 23.7¦
  11) ExposureCompensation           =>  -0.67
  13) ExposureTime                   =>  1/250 sec, ExpMode=Manual
  25) FocalLength                    =>  82.0 mm  << Different?
  29) HyperfocalDistance             =>  34.00 m
  32) LightValue                     =>  11.3
  33) MeteringMode                   =>  Multi-segment
  35) Model                          =>  Nikon D800E

  21) FocalLength                    =>  82.3 mm    << Different?

NOTE on DIF.pl:
-e -> Ephemeral. Filter random, changeable junk like dates and digests.
-t -> Trim. Zap all leading and trailing whitespace before comparison.

Still missing 10 items and the FocalLength has mutated from 82.0 mm -> 82.3 mm.   Hmmm...

Confirmed:
brianp@raptor:~/tmp/ltza$ exiftool -s  -FocalLength  za-2017.0308-263694.nef za-2017.0308-263694.jfn.jpg
======== za-2017.0308-263694.nef
FocalLength                     : 82.0 mm
======== za-2017.0308-263694.jfn.jpg
FocalLength                     : 82.3 mm
----------------------------------------

Comparing FREQuency counts (-m -> Multiple vs -s -> Singular), there are 140 identical EXIF items between the NEF and its JPG (enhanced with MakerNotes), but still 127 Singular items.

brianp@raptor:~/tmp/ltza$ exiftool -s  -all  za-2017.0308-263694.nef za-2017.0308-263694.jfn.jpg | freq -m | wc
    140     613    6191
brianp@raptor:~/tmp/ltza$ exiftool -s  -all  za-2017.0308-263694.nef za-2017.0308-263694.jfn.jpg | freq -s | wc
    127     523    6086

I have some cheesy HDR software which mindlessly thrashes EXIF info ("We don't use that data"). I just load the original, EXIF-correct image into ImageMagick, superimpose the faulty, HDR image over it and execute:
    $imagea->Append(stack=>'true');  # Flatten stack down to 1 level
to merge the correct EXIF layer with the desired bitmap layer.

Is it possible to create a Frankenstein NEF-JPG monstrosity with the  correct EXIF data and a tiny, compressed image bitmap?

I feel like Moses before the Nikon_Pharoah, "Let my EXIF GO so that it may serve ME!" :)

Making progress,

    Brian

Hayo Baan

So it actually looks like you want ALL information that is in the NEF to be (also) available in the JPG you extracted?

Why don't you simply copy everything over then? Have a look at the copying examples on how this could be achieved.

Note: The embedded JPG you extract does not contain ANY metadata at all! It's just image data so any metadata you find is the metadata that is mandatory to make it a valid jpg, nothing else...
Hayo Baan – Photography
Web: www.hayobaan.nl