"ExifIFD pointer references previous IFD0 directory" ??

Started by pb, August 16, 2011, 05:04:08 PM

Previous topic - Next topic

pb

I am continuing to try to set copyright fields using exiftoolgui.  This time, out of thousands of jpeg files, I get the exiftool error in the subject for 2 files.

Question 1:  What does this mean?  (Sorry I could not find it anywhere.)

Question 2:  These are files that have previously been successfully changed by exiftool, via exiftoolgui.  Now, exiftool is apparently refusing to change them.

I note that one of the following is true:

a.  I am trying to change the copyright field to the value that is already in it, or

b.  Despite claiming that it didn't process the file, exiftool actually did change the field.

(I don't know which of those is the case, because I am not certain what the exact state of the metadata in those files was before exiftool was run.)

exiftool 8.61, exiftoolgui 4.18, win xp sp3

Phil Harvey

Quote from: pb on August 16, 2011, 05:04:08 PM
Question 1:  What does this mean?  (Sorry I could not find it anywhere.)

It means that the EXIF information is corrupted, and some may have been lost.

QuoteQuestion 2:  These are files that have previously been successfully changed by exiftool, via exiftoolgui.

Something must have happened to them after they were written by ExifTool.  Did you use another image editor on them maybe?  ExifTool will not write a file like this.

Quotea.  I am trying to change the copyright field to the value that is already in it, or

b.  Despite claiming that it didn't process the file, exiftool actually did change the field.

If ExifTool issued an error when writing, the file is not updated.  However, the file may be updated if only warnings are issued.  But an invalid ExifIFD pointer is a serious error.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

pb

Quote from: Phil Harvey on August 16, 2011, 07:21:53 PM
Quote from: pb on August 16, 2011, 05:04:08 PM
Question 1:  What does this mean?  (Sorry I could not find it anywhere.)

It means that the EXIF information is corrupted, and some may have been lost.

Aha, well, that's certainly very useful information, thanks.  Could I have found that answer in the exiftool documentation anywhere, so I would know where to look the next time?

Quote

QuoteQuestion 2:  These are files that have previously been successfully changed by exiftool, via exiftoolgui.

Something must have happened to them after they were written by ExifTool.  Did you use another image editor on them maybe?  ExifTool will not write a file like this.

I haven't edited those files with anything.  I haven't even looked at those images in recent times, and the file timestamp is still the same as the metadata create timestamp.  The only thing I have been doing to those files is changing copyright fields with exiftoolgui.  I do have backups from various earlier times, so I can try to find out roughly when corruption may have occurred.   Also, I have not had any crashes.  However, note that the problem I have been working on is that exiftool/exiftoolgui would hang when processing this directory.  The only way out of that deadlock was to kill the exiftool processes.  I believe that when killed, exiftool was blocked in writing to stdout (or maybe stderr).  However, it did not have any files open as I recall.  Nevertheless, it may be that exiftool got killed when it had only partially updated metadata, for example it was blocked writing a diagnostic, and for some reason the file was closed, but would have been reopened to continue processing.  (Sounds unlikely, but it's the best I can come up with.)

For my purposes, I can restore the files in question from backup and start over on those files.  The only reasons to pursue this further would be in case there is a bug in exiftool, or just out of curiosity to figure out how the corruption could have occurred.

Quote

Quotea.  I am trying to change the copyright field to the value that is already in it, or

b.  Despite claiming that it didn't process the file, exiftool actually did change the field.

If ExifTool issued an error when writing, the file is not updated.  However, the file may be updated if only warnings are issued.  But an invalid ExifIFD pointer is a serious error.

The Exiftool message displayed by exiftoolgui says in its entirety:
"Error [minor] ExifIFD pointer references previous IFD0 directory <filename>
0 image files updated
files weren't updated due to errors". 
(0 image files updated because I just now ran it on only one of the offending files.)
I'm willing to believe that I had already previously set the field (thus giving the appearance that it was updated but actually was already that way), but now I am confused why you say it's a "serious" error, but the exiftool output says "minor"?

AHA!!  I had exiftoolgui set to tell exiftool to NOT update on "minor" errors.  (I don't recall offhand what the command line switch is for that, but it doesn't matter.)  Changing it to allow update, yields a slightly different error message:

"Warning:  ExifIFD pointer references previous IFD0 directory <filename>
1 image files updated"

So, it occurred to me that this corruption has existed for a long time, but was simply not noticed because of multiple other warnings.  However, I checked saved output from exiftool that I ran directly from the command line on this directory several days ago when working on the gui hang problem, and there was no warning for either of the offending files.

My only remaining question about this is the discrepancy between "minor" and your saying it's "serious"?

Sorry for prolonging the agony of this post.

Quote

- Phil

pb

Here's a related question.  I would like to run this whole directory through exiftool to detect whether there are any other metadata errors/warnings, without actually writing any metadata or dumping it somewhere, i.e. to do a purely diagnostic run.  (This would help avoid future surprises.)  I don't see any obvious way to do this.  How would you recommend accomplishing that?

Phil Harvey

Quote from: pb on August 16, 2011, 09:46:55 PM
Aha, well, that's certainly very useful information, thanks.  Could I have found that answer in the exiftool documentation anywhere, so I would know where to look the next time?

No, I don't have detailed explanations of all the error messages.  But in general, any error message indicates that the metadata is corrupted or was written improperly somehow.  Whenever this happens there is a chance of losing some information.

QuoteI haven't edited those files with anything. [...]

Could you send me one of the files so I can take a look?  My email is philharvey66 at gmail.com.

If exiftool crashes while writing a file it will leave a temporary file on your hard disk and the original file will be left untouched.  The only exception is if you use -overwrite_original_in_place, which may result in a corrupted file if it crashes during the final copy stage (unlikely).

Quotebut now I am confused why you say it's a "serious" error, but the exiftool output says "minor"?

You're right, sorry.  I have made this a minor.  So files like this may be written with the -m option, but if this is done the offending IFD will be dropped.

I can see why I made this a minor error: 1) If the IFD pointer is correct, then the information is duplicated anyway and dropping the IFD won't result in a loss of information.

However, 2) if the IFD pointer is incorrect and was meant to point to some other IFD somewhere, then the information in this other IFD will be lost.

I would have to go back through the exiftool revision histories to try to figure out why I made this minor, but it is likely I was dealing with images of type 1 above.

There are many ways that metadata may be corrupted, and there is no way that exiftool can ever be comprehensive enough to anticipate them all.  This is one reason why I don't give more detailed explanation of the error messages -- exiftool may be observing just one symptom of a different problem for example.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

pb

Quote from: Phil Harvey on August 17, 2011, 07:17:00 AM
Quote from: pb on August 16, 2011, 09:46:55 PM
Aha, well, that's certainly very useful information, thanks.  Could I have found that answer in the exiftool documentation anywhere, so I would know where to look the next time?

No, I don't have detailed explanations of all the error messages.  But in general, any error message indicates that the metadata is corrupted or was written improperly somehow.  Whenever this happens there is a chance of losing some information.

QuoteI haven't edited those files with anything. [...]

Could you send me one of the files so I can take a look?  My email is philharvey66 at gmail.com.

Ok, will do.  I'll also send the original version of the file, as it came from the camera, or at least as I think it came from the camera.  What I can't do is with certainty send the exact file at the last state before it was corrupted.  These files have been copied from the originals multiple times using the OS (i.e. Windows Explorer), in order to create test directories to work on, so some corruption in that process is always a possibility, though a very remote one.
Quote

If exiftool crashes while writing a file it will leave a temporary file on your hard disk and the original file will be left untouched.  The only exception is if you use -overwrite_original_in_place, which may result in a corrupted file if it crashes during the final copy stage (unlikely).

Yes, I realized that while tossing and turning last night, but was too lazy to get up and modify my post.  And, I'm pretty sure that exiftoolgui used -overwrite_original not -overwrite_original_in_place, though not 100% certain since a few different gui versions have been involved.

Quote

Quotebut now I am confused why you say it's a "serious" error, but the exiftool output says "minor"?

You're right, sorry.  I have made this a minor.  So files like this may be written with the -m option, but if this is done the offending IFD will be dropped.

I can see why I made this a minor error: 1) If the IFD pointer is correct, then the information is duplicated anyway and dropping the IFD won't result in a loss of information.

However, 2) if the IFD pointer is incorrect and was meant to point to some other IFD somewhere, then the information in this other IFD will be lost.

I would have to go back through the exiftool revision histories to try to figure out why I made this minor, but it is likely I was dealing with images of type 1 above.

There are many ways that metadata may be corrupted, and there is no way that exiftool can ever be comprehensive enough to anticipate them all.  This is one reason why I don't give more detailed explanation of the error messages -- exiftool may be observing just one symptom of a different problem for example.

- Phil
Thanks.

In all likelihood I did something stupid that changed those files, and am sending you on a wild goose chase, but I can't imagine what, because the timestamps do not reflect any change.

pb

I have emailed (to Phil) 3 versions of one of the offending files:  (1) offender, (2) last version I believe to be prior to first use of exiftool, which does NOT trigger an exiftool error/warning, and (3) oldest backup of that file.

All 3 have stuff in the header (IPTC?) that was apparently put there by Picasa, though I do not believe I actually did anything other than look at it with Picasa.  Maybe I tried to add a caption and decided not to.

I also verified that #3 (oldest version) also does NOT trigger an exiftool error/warning.

Phil Harvey

Thanks, I got the files.

I had a chance to take a quick look.

Do NOT write these with the -m (ignore minor warnings) option.  You will lose metadata.

But there is something funny going on, and I need to study this in more detail when I get a chance.  I'll get back when I know more (probably tomorrow morning).

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#8
Wow.  Sometimes it pays to track down every problem.

You are absolutely correct:  ExifTool wrote that file.  The file is fine, but ExifTool is wrong when it reports the error.  There is a bug in the code that calculates the offset of the IFD, and only in very rare and special situations could it result in a conflict with a previous IFD like it does here.  The odd thing about your files that enabled this problem is that Picassa had previously written a Photoshop segment containing IPTC before the EXIF segment in the JPEG image (which is contrary to the EXIF specification, I might add).

But this is a definite ExifTool bug.  I will fix the problem and ExifTool 8.62 will be able to write this image without complaining.

Thanks for mentioning this problem, and for the sample image.

- Phil

Edit: Fixed typo.  It was a late night. :P
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

pb

Quote from: Phil Harvey on August 17, 2011, 10:56:45 PM
Wow.  Sometimes it pays to track down every problem.

You are absolutely correct:  ExifTool wrote that file.  The file is fine, but ExifTool is wrong when it reports the error.  There is a bug in the code that calculates the offset of the error, and only in very rare and special situations could it result in a conflict with a previous IFD like it does here.  The odd thing about your files that enabled this problem is that Picassa had previously written a Photoshop segment containing IPTC before the EXIF segment in the JPEG image (which is contrary to the EXIF specification, I might add).

But this is a definite ExifTool bug.  I will fix the problem and ExifTool 8.62 will be able to write this image without complaining.

Thanks for mentioning this problem, and for the sample image.

- Phil

Great, thanks for nailing this down.  I'm also impressed at the elapsed clock time of 3 hours!

pb

I now see from another post how you were able to do this in 3 hours:  you had to, you were on vacation!