Debugging MakerNotes warnings

[Leonard WHYTE]

I'm getting the message "[ExifTool]      Warning                         : [minor] Possibly incorrect maker notes offsets (fix by -182?)" in an image of mine.  How can I get additional explanatory material to help debug this?  I've tried "-v5" which produces lots of additional data, but it doesn't appear to shed any more light on the cause of the problem.

[Phil Harvey]

Hi Leonard,

The cause of this problem is often that the maker notes get moved by some software that doesn't properly update the embedded offsets.  The -htmlDump output (viewed in a web browser) is the best way to see what is happening here, but it will require some knowledge of the EXIF structure to understand what is going on.

- Phil

[Leonard WHYTE]

Indeed, it is my own software that is doing the move and adjusting the offsets.  It usually works (in that your exiftool doesn't produce any warnings about the results), but this particular image, from a Sony HDR PJ970VE video camera, has a much more complex Maker Note, and your exiftool complains by producing this warning.  The strange thing is, that all the values appear correctly in the exiftool output from the adjusted image.  I see that the html dump produces voluminous output which I'll have to take away and study!!!

[Leonard WHYTE]

Phil, I've tracked down the issue.  My code rebuilds the Sony MN for the image.  The MN comprises the "SONY DSC" string, some zero bytes, the MN IFD Header, and the MN IFD data.  For the first time, in rewriting Maker Notes, my code introduced unused bytes between the end of the MN IFD Header and the beginning of the MN IFD Data for this image.  This would appear to be perfectly valid to me, but appears to upset your exiftool parser which complains about an incorrect offset the size of which is the separation between the end of the MN IFD Header and the start of the MN IFD Data.  I don't believe that a separation is invalid. Do you?  If not, perhaps you have a bug in exiftool which could be ironed out.  I'm happy to send you the offending image for you to work with, if you wish.

[Phil Harvey]

Extra space between the MN IFD and its data is problematic because this space is one of the key indicators in determining whether or not the offsets are valid.  For well-formed maker notes, this is usually either 0 or 4 bytes, and this fact is used by ExifTool in the logic to fix incorrect offsets.

- Phil

[Leonard WHYTE]

Phil, I can live with that I guess.  When I re-write an IFD, be it IFD0, IFD1, or the IFD within Maker Notes, I always provide some unused space at the end of the HDR and at the end of the Data so that I can, if I need, patch the IFD.  Space at the end of the IFD HDR allows me to add additional tags without having to move the associated data, which would require that all the offsets need to be recalculated.  It is curious that exiftool only complains about this space between the HDR and Data in the MN IFD, and not in the TIFF or EXIF IFDs in the same image.  Len
PS:  The htmpdump is wonderful.  I hadn't noticed it before.  The donation I made earlier was well invested.  If I continue to get more and more from exiftool, I'll have to consider donating again!

[Phil Harvey]

Hi Len,

It is curious that exiftool only complains about this space between the HDR and Data in the MN IFD, and not in the TIFF or EXIF IFDs in the same image.

This should make sense to you.  The maker notes are not part of the EXIF specification, so it is common for editing software to mess up these offsets.  But this isn't a problem for standard IFD's, so the logic to fix offsets is not required for these.

PS:  The htmpdump is wonderful.  I hadn't noticed it before.

I'm glad you mentioned this.  It is certainly an under-appreciated feature, and it would be wonderful if all TIFF/EXIF software developers knew about this.  If they did, then TIFF formatting errors would be a thing of the past.  But sadly they don't, and the current rate of TIFF programming bugs in other software (including camera firmware) is very high.

I worked very hard on this feature.  I rely heavily on it for ExifTool development, and to debug problems in images that people send me.

If you haven't already found the HtmlDump documentation page, it may be useful for you to read this.

- Phil

[Leonard WHYTE]

Thanks Phil.  All understood.  I'll undo my arrangements to pad the MN and make the HDR and Data contiguous.   As I said, I can't praise your exiftool highly enough.  It helps me identify so many problems which, as you say, often exist within the original camera data, let alone other software.  I still can't get over the damage iPhoto does to the metadata, after it rotates portrait images (in the image data) in order to ensure that the orientation tag is always 1.  Unbelievable.  It's a pleasure to talk with someone such as yourself who so clearly knows their stuff.  Thanks again.  Len.  Over and out from me.

[Phil Harvey]

Hi Len,

I know you said "Over and out", but I had to check the to see what iPhoto (9.4.1) does when it rotates an image.  Basically, it did the same thing as Photoshop, and deleted the maker notes (in the exported image -- the original was left intact).  Other than this, the remaining EXIF seems to be formatted properly.  Oddly though, when I do the same thing in Preview 6.0.1 it adds an incorrect ExifIFD:MaxApertureValue.  (I was using a sample JPEG from a Pentax K-5.)

- Phil

[Leonard WHYTE]

Phil,  A bit rash on my part.  The behaviour I described was certainly a much earlier version of iPhoto and, to be honest, I haven't checked for quite a while, having chosen to work around the problem.  My wife still uses iPhoto for building albums and posting to Facebook, but we never export from iPhoto for archival purposes.  I've been using Media Pro 1 (now with PhaseOne, who seem to have made it much more robust and ironed out many bugs, after having gone through various previous owners including Microsoft) for cataloguing, and I always import the original camera images to avoid any issues with possible damage to images enroute.  I'm actually on a short vacation at the moment, and don't have access to my MacPro and all the software and images it hosts, so I'll get back to you on this thread, next week when I'm back home, after I've had a chance to re-check how the latest version of iPhoto behaves with some of my images.  Of course, I've no idea why iPhoto doesn't just leave the image data and the orientation tag well alone, rather than choosing to unwind any rotations!  Then it could always leave the Maker Note intact, as I believe it does for images which start out with an Orientation tag value of 1! PS: I also think that iPhoto used to do endian changes in the course of the orientation conversion, and then got into extra hot water as a result.   Len

[Leonard WHYTE]

Phil,  I checked the latest v9.5.1 of iPhoto, and it is much better behaved.  It even retains and exports the exact original file for portrait images.  I take back all the mean and nasty things I said!!!  Len

[Phil Harvey]

Hi Len,

That's good to hear.  At least they learn from their mistakes.

- Phil