Decoding CameraSettings and other similar makernote tags

[mvadu]

Hi Phil,
I am back again with another question.

In my program once I am done with all standard exif tags I get OffsetSchema (59933) and Makernote tags, and start decoding maker notes. I am able to read the individual tags inside makernote tag. Now I have to decode encoded tags like CameraSettings, and have a question.

As I understand these tags (http://www.exiftool.org/TagNames/Canon.html#CameraSettings) can have differnt meaning depending on the placement (index). But what I don't understand is how to determine if an index has an value or just not used by the camera at all.

I went through %Image::ExifTool::Canon::CameraSettings function, and I could not grasp it as I am not familiar with Perl. I need to know if a byte/bytes are used by camera with value of zero, or camera did not use the tag at all and it's just zero padding.

I have this CanonCameraSettings value block from Canon T1i,

Code Select Expand

62 00 02 00  00 00 03 00 10 00 01 00
00 00 02 00 00 00 01 00  00 00 01 00 00 00 00 00
01 00 ff 7f ff 7f 03 00  02 00 00 00 03 00 ff ff
b2 00 87 00 1c 00 01 00  78 00 20 01 00 00 08 20
00 00 00 00 ff ff ff ff  ff ff 00 00 00 00 00 00
00 00 ff ff ff ff 00 00  00 00 ff 7f ff ff ff ff
ff ff 00 00 ff ff
after 2 tags (98, 2) I have a 0, and I am not sure what the case with it is.

I have a linked list implementation which I build with an entry per tag. This list will be used by UI class to parse and presentation.
I want to create a linked list entry for valid tags, and skip to next entry if it's not used.

[Phil Harvey]

I need to know if a byte/bytes are used by camera with value of zero, or camera did not use the tag at all and it's just zero padding.

There is no way to tell, but to be safe one should assume that all bytes are significant (although the meaning of many is still unknown to us).

after 2 tags (98, 2) I have a 0, and I am not sure what the case with it is.

I don't understand what you are asking.

- Phil

[mvadu]

Thanks for the clarification Phil.

after 2 tags (98, 2) I have a 0, and I am not sure what the case with it is.

I don't understand what you are asking.

I was not sure about third member of that array. In the same context of byte being used as padding, and used to store some value.
So going by your answer I think I have to have a lookup table with all index's which my code can decode, and skip rest..

[Phil Harvey]

after 2 tags (98, 2) I have a 0, and I am not sure what the case with it is.

I was not sure about third member of that array. In the same context of byte being used as padding, and used to store some value.

Ah, I see now.  98 = 0x62,0x00.  Right.

So in this case the 3rd entry (index=2) in the Canon CameraSettings is SelfTimer, which is "Off" when the value is 0.  See the PrintConv in Canon.pm for the decoding of this.

But there are many unknown entries, like the 7th (index=6) for example.

- Phil

[mvadu]

gotcha.. I am feeling guilty for copy pasting your look table from your page (http://www.exiftool.org/TagNames/Canon.html) to my code. I am really grateful for your guidance.

PH Edit: fixed link

[Phil Harvey]

I am feeling guilty for copy pasting your look table from your page

Don't feel guilty.  I made the code public so other people could make use of it.

It was a lot of work to figure out all of this stuff, and it doesn't make sense that other programmers have to repeat this work for their own software.  Many other people have already borrowed from my work.  What also makes sense to me is that if people borrow from my work, then they give back when they make discoveries of their own.  This way, everyone benefits.

- Phil

[mvadu]

Thanks Phil, I will surely do.

by the way you are really lucky to have started ExifTool in Perl, which is a typeless language. I am working on VC++, and each stage having to worry about types, type conversion, string, memory allocation and deallocation etc, I am not sure when will start my next module (to store decoded tag values in an indexed file based database )

I was going through Canon.pm, and noticed these two lines:
line 1326: Sharpness:   RawConv => '$val == 0x7fff ? undef : $val',
line 1356: CameraISO:  RawConv => '$val != 0x7fff ? $val : undef',

Both are same comparison with 0x7FFF (32767) but the if/else are interchanged. Any specific reasons?
Wouldn't it be better to have same comparison for the sake easy readability?

[Phil Harvey]

by the way you are really lucky to have started ExifTool in Perl, which is a typeless language.

Yes, this does come in very handy.

memory allocation and deallocation etc,

and this is a godsend!

Wouldn't it be better to have same comparison for the sake easy readability?

Yes, thanks.  I'll change this.

- Phil

[mvadu]

Hi Phil, sorry for keeping this old thread alive..

This time for LensType tag inside CameraSettings. Your lookup (which is also used for few camera specific decoding too). As I understand this is a int16s type value, but look up has few rational values(for few Sigma and other lenses like 10.1). How is that possible?

[Phil Harvey]

Code Select Expand

     Notes => q{
        Decimal values differentiate lenses which would otherwise have the same
        LensType, and are used by the Composite LensID tag when attempting to
        identify the specific lens model.
     },


The stored lens ID's are all integer, but I needed some way to represent multiple lenses which have the same ID number, so I use decimal values for this.

- Phil

[mvadu]

gotcha.. makes sense..

One more question.. In your code what does ValueConvInv represent? ValueConv is for decoded value, ValueConvInv is for encoding it? (say while rebuilding maker notes, or writing it back)?

Nice profile picture by the way

[Phil Harvey]

Yes, ValueConvInv is the opposite of ValueConv, and is used when writing.

All of these tag information members are documented in lib/Image/ExifTool/README of the full distribution.

Nice profile picture by the way

Thanks.  I just discovered this feature of the forum.

- Phil

[mvadu]

Hi Phil, looks like I am not leaving you for quite sometime.

I was working on a file by Canon SX110, and when I use -h -v switch i see a warning

Code Select Expand

Warning: [minor] Adjusted MakerNotes base by 66 - SX110.jpg

I understand that value 66 is from microsoft's ill tag OffsetSchema tag (59933). But i searched the codebase for "Adjusted MakerNotes base" word to see how you handle taht tag. Only one hit is from \lib\Image\ExifTool\Panasonic.pm. SO I am confused, how do you use this tag? why panasonic procedure output is coming for Canon camera?

I appreciate your patience and timely answers. You are not only a great coder, but great mentor too.

[Phil Harvey]

I understand that value 66 is from microsoft's ill tag OffsetSchema tag (59933).

No, no, NO!!!!  Ignore Microsoft's OffsetSchema tag!  (Sorry, I get upset just thinking about this stupid tag.)  OffsetSchema is unreliable because it gives a relative shift which is easily invalidated by an EXIF editor which doesn't recalculate and update the value when rewriting the image.

The value I calculate is from the original makernote offset stored in the Canon makernote footer.  (This is what Microsoft should have stored if they had any brains amongst their programmers, but alas, this is not the case.)

But i searched the codebase for "Adjusted MakerNotes base" word to see how you handle taht tag.

Look at MakerNotes::FixBase().  The print statement is [qq]$exifTool->Warn("Adjusted $dirName base by $fix",1);[/qq]

- Phil

[mvadu]

Great!! as usual thanks a ton.
Could you please explain what is the logic behind Get32u(\$footer, 4); (Get32u in general, I could not find the function body).

I think I will search for a PERL debugger so that I can run your code and check the values myself (instead of troubling you)
Any suggestions (on Windows)

[Phil Harvey]

Could you please explain what is the logic behind Get32u(\$footer, 4); (Get32u in general, I could not find the function body).

This function is defined in lib/Image/Exiftool.pm, but isn't very easy to read.  Basically all these functions just extract a single value at the specified offset in the data.  Get32u() extracts a 32-bit unsigned integer (at offset 4 in $footer of your example).  The following lookup table from ExifTool.pm defines the functions that are used to read values of each data type:

Code Select Expand

my %readValueProc = (
   int8s => \&Get8s,
   int8u => \&Get8u,
   int16s => \&Get16s,
   int16u => \&Get16u,
   int16uRev => \&Get16uRev,
   int32s => \&Get32s,
   int32u => \&Get32u,
   int64s => \&Get64s,
   int64u => \&Get64u,
   rational32s => \&GetRational32s,
   rational32u => \&GetRational32u,
   rational64s => \&GetRational64s,
   rational64u => \&GetRational64u,
   fixed16s => \&GetFixed16s,
   fixed16u => \&GetFixed16u,
   fixed32s => \&GetFixed32s,
   fixed32u => \&GetFixed32u,
   float => \&GetFloat,
   double => \&GetDouble,
   extended => \&GetExtended,
   ifd => \&Get32u,
   ifd64 => \&Get64u,
);


I think I will search for a PERL debugger so that I can run your code and check the values myself (instead of troubling you)
Any suggestions (on Windows)

Perl has a built-in debugger (invoked with the perl -d option), which I have used on occasion, but I do most of my debugging by inserting print statements (which isn't as bad as it sounds because there is no compile step in the debugging cycle).

- Phil

[mvadu]

Thanks for the response Phil.
I was looking at http://www.epic-ide.org/index.php (I already have Eclipse for AVR developemtn work).

[Phil Harvey]

You do AVR development?  Cool.  I've done some 8-bit AVR development on this system, but we're upgrading to AVR32 and I need to figure out how to get the new compiler working on a Mac so I can get going on this.

- Phil

[mvadu]

AVR development in Eclipse is fun! you need
1. Eclipse for C/C++ (http://www.eclipse.org/cdt/)
2. AVR LibC (http://www.nongnu.org/avr-libc/)
3. AVR GCC Toolchain (http://avr-eclipse.sourceforge.net/wiki/index.php/The_AVR_GCC_Toolchain).

Debugging is easier in AVR Studio (I have never used hardware debugger).

[mvadu]

but I do most of my debugging by inserting print statements (which isn't as bad as it sounds because there is no compile step in the debugging cycle).

- Phil

Phil, today I found an easy way.. When I execute the standalone exiftool.exe in windows, it extracts (i don't know what kind of packaging you used) all source files, and editing them works

Now I need your help in figuring out where to add print statements.

Code Select Expand

1 => { #PH
       Name => 'AutoISO',
       Notes => 'actual ISO used = BaseISO * AutoISO / 100',
       ValueConv => 'exp($val/32*log(2))*100',
       ValueConvInv => '32*log($val/100)/log(2)',
       PrintConv => 'sprintf("%.0f",$val)',
       PrintConvInv => '$val',
   },
Say this is the code I want to debug (shotinfo look up table). Where should I add the print statement?
say I want to print the original tag value, converted value, the PROCESS_PROC for this tag is Image::ExifTool::ProcessBinaryData, so I went and added

Code Select Expand

    foreach (keys %{$$tagTablePtr{GROUPS}}) {
            print $tagTablePtr->{GROUPS}{"Name"} . "\n";
        }

But I got few blank line in the output.. removing {"Name"} gave me HASH(0x2a2d8ac), HASH(0x2a2d8ac) etc.

[Phil Harvey]

ExifTool is a PAR package, which is basically a bundle of Perl and ExifTool with all the necessary libraries.  They are all unpacked in your temporary directory when running.  Since ExifTool is pure Perl, the ExifTool script and libraries are Perl source code and may be edited as you have discovered.

I'm not sure what you are trying to do, but the GROUPS hash contains key values of 1, 2 and 3, which are the default group names for families 1-3 for the tags in the table.

If you just want to print the original values of all tags, you could use the -U -v3 options.

- Phil

[mvadu]

I'm not sure what you are trying to do, but the GROUPS hash contains key values of 1, 2 and 3, which are the default group names for families 1-3 for the tag in the table.

If you just want to print the original values of all tags, you could use the -U -v3 options.

- Phil

Thanks for the information Phil, Yes, -U -v3 gives me the information I was looking for.

I wanted to see the value transformation during runtime, so I was looking for a way to alter your code to print values/tag names etc. from within your code. (like dumping whole GROUP hash)
I tried

Code Select Expand


while ( ($key,$val) = each %{$$tagTablePtr{GROUPS}} ) {
print "$key => $val\n";
}

But I got

Code Select Expand

1 => Canon
0 => MakerNotes
2 => Camera
1 => Canon
0 => MakerNotes
....so on


So how do I go about printing values assigned in %Image::ExifTool::Canon::ShotInfo function?

P.S> sorry for pulling you in to Perl teaching, but please bear with me

[Phil Harvey]

Code Select Expand

sub dumpHash($)
{
    my $hash = shift;
    foreach my $key (sort keys %$hash) {
        print "$key => $$hash{$key}\n";
    }
}

dumpHash($$tagTablePtr{GROUPS});


I can get a lot fancier, but that is the basic idea.

- Phil

[mvadu]

Yeah.. this is lot easier.. thank you "Guru"