Unexpected changes in metadata

[panicnow]

I'm not sure if this is the best sub-forum to ask my question but as it is unexpected changes to the metadata which I would like to understand, maybe this is the best place.

I've just used exiftool for the first time to change some date/time tags on a set of photos where the camera was set to the wrong time. Specifically the DateTime, DateTimeOriginal and DateTimeDigitized values. The script seemed to do the job but checking the results I discovered a number of other changes in the jpegs which I was not expecting.

Firstly the filesize increases, only by a few bytes but I don't understand why it should change. It seems to grow by the same amount as the increase in the value of the tag "length" in "Marker: APP1". After that marker, all of the offsets change by the same value. That change is logical given the change in the value of "length".

There are two tags in the "EXIF MakerIFD" section which had their values changed. "Canon.0x000D" (which contains a list of integer values) and "Canon.0x0018" (which contains a list of hex values).

Whether those changes are logical or not I don't know since I have no idea what those values represent.

There also seems to be a whole new section: "Embedded JPEG Thumbnail". I assume this must have been there before as it is much bigger than the growth in the size of the file, but it was not extracted by JPEGsnoop which is what I had been using to check display the contents of the EXIF.

As long as the image itself has not changed (which I haven't been able to establish yet and that may be the subject of a later post) this may be an academic question but I'd like to understand the changes.

In case my script is relevant here is the section which makes the changes.

   my $info = ImageInfo($image);
   my $exiftool    = Image::ExifTool->new();
   $exiftool      -> Options(FixBase => '');
   $exiftool      -> ExtractInfo($image);
   $exiftool      -> SetNewValue(DateTimeOriginal => $datetime);
   $exiftool      -> SetNewValue(ModifyDate => $datetime);
   $exiftool      -> SetNewValue(CreateDate => $datetime);
   my $success = $exiftool->WriteInfo($image);
   
I must admit, I don't really understand FixBase and maybe that made the changes but the script didn't work without it so I can't test by pulling it out.

[Phil Harvey]

Try setting $exifTool->Options(Verbose => 2) to see what ExifTool is doing when it is writing.

If you want to be sure only the EXIF tags get changed, you should specify a group name. (ie. 'EXIF:DateTimeOriginal')

It seems as if your ExtractInfo() call is doing nothing but burning CPU cycles.  I'd remove it unless you are using the information it returns.  Calling WriteInfo() will cause this information to be freed from memory anyway.

You say the script doesn't work without FixBase set.  Is this because a minor error is issued?  You can also use IgnoreMinorErrors for this.  If WriteInfo() returns false, you should call GetValue('Error') to see what went wrong.

Also, I suggest you read FAQ's 13 and 15, since they may be relevant.

- Phil

[panicnow]

Thanks for the tips.

The ExtractInfo() call was just a remnant from a previous script. I'd been changing it bit by bit to test the effect of changes and ended up with a bit of a mess ;-) I obviously didn't tidy it up well enough.

I removed the FixBase option to remind myself what the error was but I must have been using a different set of test files when I put it in as I'm not getting an error at all on the files I really want to change (well copies of them).

I'm afraid I don't understand where to specify the group name. Including it in SetNewValue seems to make no difference to what is being written and I haven't been able to find any documented way to specify it as an option to WriteInfo. The verbose output from the point a which writing starts is:

Writing ExifIFD:DateTimeOriginal
Writing IFD0:ModifyDate
Writing ExifIFD:CreateDate
Rewriting /Tests/Pics/IMG_0055.JPG...
  Editing tags in: APP0 APP1 ExifIFD IFD0 JFIF
  Creating tags in: APP1 ExifIFD IFD0
JPEG APP1 (5850 bytes):
  Rewriting IFD0
  ExifByteOrder = II
    - IFD0:ModifyDate = '2006:01:01 23:45:00'
    + IFD0:ModifyDate = '2006:01:01 23:45:00'
  Rewriting ExifIFD
    - ExifIFD:DateTimeOriginal = '2006:01:01 23:45:00'
    + ExifIFD:DateTimeOriginal = '2006:01:01 23:45:00'
    - ExifIFD:CreateDate = '2006:01:01 23:45:00'
    + ExifIFD:CreateDate = '2006:01:01 23:45:00'
  Rewriting MakerNoteCanon
  Rewriting CanonCameraSettings
  Rewriting CanonFocalLength
  Rewriting CanonShotInfo
  Rewriting CanonCameraInfoPowerShot2
  Rewriting MyColors
  Rewriting IFD1
JPEG DQT (65 bytes):
JPEG DQT (65 bytes):
JPEG SOF0:
JPEG DHT (29 bytes):
JPEG DHT (179 bytes):
JPEG DHT (29 bytes):
JPEG DHT (179 bytes):
JPEG SOS

The fact the times haven't changed here is not a problem, I ran the script several times without changing the time so it has just overwritten with a value which happens to be the same.

I assume the "writing" is the point at which the new values are being set and the "rewriting" is the WriteInfo() call.

[Phil Harvey]

I'm afraid I don't understand where to specify the group name. Including it in SetNewValue seems to make no difference to what is being written and I haven't been able to find any documented way to specify it as an option to WriteInfo.

You do it in SetNewInfo(), using something like 'EXIF:DateTimeOriginal' instead of 'DateTimeOriginal'.

I assume the "writing" is the point at which the new values are being set and the "rewriting" is the WriteInfo() call.

Correct.

So I'm a bit lost now.  What is unexpected about the results here?

- Phil

[panicnow]

What is unexpected about the results here?

Sorry for causing confusion.

The unexpected result was the change in values of tags which I hadn't changed.

The FAQ answered the part about the offsets changing but not about the values of the tags in the Maker notes. Of course now I can prevent those changing the question is academic and doesn't really need to be answered.

I'm afraid I don't understand where to specify the group name. Including it in SetNewValue seems to make no difference to what is being written and I haven't been able to find any documented way to specify it as an option to WriteInfo.

You do it in SetNewInfo(), using something like 'EXIF:DateTimeOriginal' instead of 'DateTimeOriginal'.

I'm afraid setting the verbose option led to some laziness in my testing. The output did not seem to change after inserting the group name in SetNewValue() (other than that being included in the log). It still had the lines about rewriting other groups and I mistakenly assumed that meant it was still making changes. Having checked the contents it seems I was wrong and that implementing your suggestion of including the group in SetNewValue did work.

Thanks for your help

[panicnow]

Sorry to post to a thread that looks as if it was finished but I seem to be taking two steps forward and one step back. At the weekend I was sure I had a script which changed the values I wanted to change and nothing else. Somehow in tidying up the commented out lines I may have removed something important because I am back to square one with other values changing.

Now I either need to get back to the stage of what I thought was a working script (if only I was running backup on my test directories) or understand what the values are which are being changed so I can decide whether I may ever need them in the future.

I've confirmed that my script as it stands (trimmed down to just change one value to simplify comparisons) gives identical results to exiftool -"ExifIFD:DateTimeOriginal"='datetime' filename

I've also established that running exiftool −a −u −g filename on the original and modified files gives identical output (other than some system information, the value I modified and the thumbnail offset). However, as I mentioned, I have been using JPEGsnoop to check for differences and it seem that it outputs a larger number of tags and it's in those tags that exiftool is changing values.

I'm attaching the JPEGsnoop output of the original and modified JPEGs for reference. I'll be approaching its author as well to see if he has information on what those values are.

[Phil Harvey]

All of these except the makernotes changes are FAQ 13.

The changes to Canon.0x000D and Canon.0x0018 I can't explain.  ExifTool would not do this.

- Phil

[panicnow]

Thanks Phil

Yes, the changes to Canon.0x000D and Canon.0x0018 are the only ones which I'm concerned about.

Logically there are only 2 possible explanations to the change: one is that JPEGsnoop is misreading the information and that the change in offsets (or some other factor) somehow affects the reading of those values, the other is that ExifTool has somehow made the change. Checking the values using ExifTool that would help confirm which it is so I have just tried to extract more information. I have managed to get Canon.0x0018 to display by using exiftool −a −u -u -f −g but not Canon.0x000D. Interestingly ExifTool reports "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0[...]" for both original and modified versions which does suggest no change has been made.

I'll post information I get back from JPEGsnoop's author here.

I've also asked Canon for that information but I think the chances of getting a useful response from them are fairly low.

[panicnow]

I've also asked Canon for that information but I think the chances of getting a useful response from them are fairly low.

In line with my expectations, the reply from Canon wasn't very enlightening. Basically they just said they "do not disclose information on makernotes contents to customers". This makes the question even more academic. If Canon keep the information confidential then there is no way I could ever use it anyway.

[Phil Harvey]

Could you send me the original image you are using, and tell me the exact command?  I would like to run some tests myself.  My email is philharvey66 at gmail.com

- Phil

Edit:  You don't need to bother with the sample.  After studying the JpegSnoop output it is clear that JpegSnoop is displaying the wrong data for these tags.  For example, for Canon.0x0018 it gives:

    [Canon.0x0018                        ] = 0x[08080000
    19000300 01000000 01000000
    1C000300 01000000 00000000
    1D000300 10000000 08090000
    1E000400 01000000 00010001
    1F000300 45000000 28090000...]

JpegSnoop is displaying the Canon MakerNotes IFD itself, beginning at the offset value for tag 0x18, instead of the actual data (notice the tag ID's 0x19, 0x1c, 0x1d, 0x1e, 0x1f -- compare this to the ExifTool -htmldump output).  The numbers that are changing when you edit with ExifTool are the offsets within this IFD.

P.S. I really don't like debugging other softwares' problems, but hey, it's the only way to be sure it isn't a problem with ExifTool.

[panicnow]

Thanks Phil

I know what you mean out debugging other people's software but I appreciate your help.

[impulse]

I realize this was ages ago, but there was indeed a bug in the way JPEGsnoop was decoding the multi-component unsigned byte arrays which was causing it to display data in the IFD rather than following the pointer. I did fix it in a later version, but unfortunately didn't see this bug report until recently.

So, my apologies to you for any time spent debugging this or causing any potential for doubt in exiftool's excellent operation

PS> The htmldump is truly the most fantastic feature!