Strange problem in case of timestamp shift

[mahikeulbody]

If I do exiftool -AllDates+=1:05 <directory> on ORF files (i.e. Olympus raw files) which have an orientation tag not "Horizontal" ("Rotate 90 CW for example), the modified files cannot be opened anymore by Darktable (Error reading TIFF structure. Unknown Type 0x3043 encountered).

There is no problem when shifting time of ORF files which have an orientation tag "Horizontal" !

Of course, I suppose that is a Darktable problem (which uses Exiv2 if I am not wrong) but I don't understand how this problem can raise only changing values of existing fields. May be Exiftool (ver 9.93) changes a little bit the metadata structure doing that ?

[Phil Harvey]

Thanks for this report.

I have a few questions that may help me understand what is happening:

Which camera model is this?

Do you have any other utilities that can open ORF files?  Do they have the same problem as Darktable?

- Phil

[mahikeulbody]

The camera is an Olympus OM-D E-M10.

RawTherapee, XnViewMP and Shotwell can open the files time shifted without problem.

[Phil Harvey]

Thanks.

I've looked into this.  I downloaded a rotated E-M1 ORF and applied your command, and tried opening the ORF in various applications (Olympus Viewer 3, Apple Preview, Adobe Lightroom 5, dcraw), and none of them had any problems with the ExifTool-edited ORF.

Also, I have analyzed the file structure, and the only significant change I can find is that ExifTool fixes an overlapping tag value in the Olympus ImageProcessingIFD.  But this fix shouldn't affect Darktable.

I suspect a problem in Darktable, and recommend sending them a sample problem ORF.

- Phil

Edit: Fixed typo

[mahikeulbody]

Thanks !

[pedrocr]

Also, I have analyzed the file structure, and the only significant change I can find is that ExifTool fixes an overlapping tag value in the Olympus ImageProcessingIFD.  But this fix shouldn't affect Darktable.

Hi, I'm the darktable developer that works most with raw formats. I've added a bunch of formats to rawspeed and we have since stopped using libraw (dcraw based) completely. We would really appreciate if people would stop editing raw files, it's a very bad idea that can only lead to issues like these. Because of how poorly documented and absolute-offset based these formats are there's just no sane way to edit them and be sure they didn't break.

As for this case in particular the issue seems to be with that "fix" exiftool did. I haven't looked at it in detail but the end result is that the Tiff parsing of the Olympus Image Processing IFD now fails by finding a tag whose type is 0x3e9d. rawspeed will throw an exception at that point and abort reading the IFD. So even if WB has been read before it won't be used. I suspect other tools are reading up to that value and failing silently after that.

Ideally exiftool (and we've had the same issue with digikam) would stop editing raw files completely. It's never a good idea and users will keep shooting themselves in the foot with it.

Cheers,

Pedro

[Phil Harvey]

Hi Pedro,

We would really appreciate if people would stop editing raw files, it's a very bad idea that can only lead to issues like these. Because of how poorly documented and absolute-offset based these formats are there's just no sane way to edit them and be sure they didn't break.

Thanks for your opinion.  There are some who would agree.  However, editing RAW files is not going to go away.  (Food for thought: Adobe will always edit DNG images, and they have all the same pitfalls as other RAW formats, and nobody is going to stop Nikon from shipping software that modifies their NEF images.)

I haven't looked at it in detail but the end result is that the Tiff parsing of the Olympus Image Processing IFD now fails by finding a tag whose type is 0x3e9d.

Thanks for looking into this, but some more details would be useful.

@mahikeulbody, it would be useful if you could send me the same sample so I can try to figure out what Darktable is doing by looking for 0x3e9d.  Thanks.

- Phil

Edit: Added some food for thought.

Edit2: I thought I would try installing Darktable and debugging this myself (on OS X 10.10.3), but after wasting about an hour trying to get all of the dependencies installed, I finally gave up because the Darktable build couldn't find gtk+-3.0 (which I had installed with "brew install gtk+3").

[mahikeulbody]

File link : https://drive.google.com/file/d/0BzRds7xWx26TOS1La3VvQmx3Unc/view?usp=sharing

Please note that for another image I get "Unknown Type 0x188e encountered" so it is not ever 0x3e9d (both images modified by Exiftool with the same command, but one is 90cw rotated, the other one is 270cw rotated).

[Phil Harvey]

Thanks for the file.  Unfortunately, a quick look for 0x3e9d (little endian, so 0x9d 0x3e) didn't yield any interesting results (certainly nothing in the ImageProcessingIFD).

- Phil

[pedrocr]

Unfortunately, a quick look for 0x3e9d (little endian, so 0x9d 0x3e) didn't yield any interesting results (certainly nothing in the ImageProcessingIFD).

The offset at which rawspeed is trying to read the TiffEntry is 0xeca6 which does indeed have those values in there. Don't know what exiftool did to the file that we are now looking in the wrong place.

[Phil Harvey]

Hi Pedro,

The offset at which rawspeed is trying to read the TiffEntry is 0xeca6 which does indeed have those values in there. Don't know what exiftool did to the file that we are now looking in the wrong place.

Thanks.  That offset is inside the PreviewImage.  I can't see how rawspeed could be landing there.

BTW: The maker note offsets are relative to the start of the maker notes, not absolute as you mentioned in your earlier post.

- Phil

[pedrocr]

Thanks.  That offset is inside the PreviewImage.  I can't see how rawspeed could be landing there.

This is the relevant rawspeed code:

https://github.com/darktable-org/darktable/blob/master/src/external/rawspeed/RawSpeed/OrfDecoder.cpp#L289-L290

My guess is that exiftool has somehow messed with those offsets.

BTW: The maker note offsets are relative to the start of the maker notes, not absolute as you mentioned in your earlier post.

I don't think that's true of all formats, and I wasn't referring to just makernotes, but raw formats in general. They are only partially understood and Tiff is absolute addressed. Editing raw files is always a bad idea. Even if you don't change the structure (just replace N bytes with N bytes) there are sometimes some fields that depend on others (e.g., the "encrypted" stuff in Sony files) so editing fields may mean losing the key to "decrypt" other fields.

[Phil Harvey]

This is the relevant rawspeed code:

Thanks for the reference.  I'm still trying to compile darktable so I can debug this for you.

My guess is that exiftool has somehow messed with those offsets.

Of course, ExifTool has messed with all of the offsets.

I don't think that's true of all formats

Correct.

there are sometimes some fields that depend on others (e.g., the "encrypted" stuff in Sony files) so editing fields may mean losing the key to "decrypt" other fields.

Yes.  And ExifTool handles these dependencies.  It won't currently let you change the decryption key in Sony files.

- Phil

[pedrocr]

Thanks for the reference.  I'm still trying to compile darktable so I can debug this for you.

Unless there's actually something wrong with the code (as in something that doesn't work with ORF files out of the camera) I won't be changing it to accommodate files that have been altered. Our resources are spread thin enough just trying to support the formats as the manufacturers generate them. You're welcome to submit a patch to rawspeed or darktable if you like. The rawspeed repo is here:

https://github.com/klauspost/rawspeed

Yes.  And ExifTool handles these dependencies.  It won't currently let you change the decryption key in Sony files.

You handle those you know about, but there is plenty about these formats we just don't know.

[Phil Harvey]

Unless there's actually something wrong with the code (as in something that doesn't work with ORF files out of the camera) I won't be changing it to accommodate files that have been altered.

Yes.  I've seen this sentiment before, but in the past problems like this have pointed out weaknesses in code, and fixing them has resulted in more robust software.  (And I'm not talking about ExifTool.)

- Phil