If I do exiftool -AllDates+=1:05 <directory> on ORF files (i.e. Olympus raw files) which have an orientation tag not "Horizontal" ("Rotate 90 CW for example), the modified files cannot be opened anymore by Darktable (Error reading TIFF structure. Unknown Type 0x3043 encountered).
There is no problem when shifting time of ORF files which have an orientation tag "Horizontal" !
Of course, I suppose that is a Darktable problem (which uses Exiv2 if I am not wrong) but I don't understand how this problem can raise only changing values of existing fields. May be Exiftool (ver 9.93) changes a little bit the metadata structure doing that ?
Thanks for this report.
I have a few questions that may help me understand what is happening:
Which camera model is this?
Do you have any other utilities that can open ORF files? Do they have the same problem as Darktable?
- Phil
The camera is an Olympus OM-D E-M10.
RawTherapee, XnViewMP and Shotwell can open the files time shifted without problem.
Thanks.
I've looked into this. I downloaded a rotated E-M1 ORF and applied your command, and tried opening the ORF in various applications (Olympus Viewer 3, Apple Preview, Adobe Lightroom 5, dcraw), and none of them had any problems with the ExifTool-edited ORF.
Also, I have analyzed the file structure, and the only significant change I can find is that ExifTool fixes an overlapping tag value in the Olympus ImageProcessingIFD. But this fix shouldn't affect Darktable.
I suspect a problem in Darktable, and recommend sending them a sample problem ORF.
- Phil
Edit: Fixed typo
Thanks !
Quote from: Phil Harvey on April 30, 2015, 10:51:26 AM
Also, I have analyzed the file structure, and the only significant change I can find is that ExifTool fixes an overlapping tag value in the Olympus ImageProcessingIFD. But this fix shouldn't affect Darktable.
Hi, I'm the darktable developer that works most with raw formats. I've added a bunch of formats to rawspeed and we have since stopped using libraw (dcraw based) completely. We would really appreciate if people would stop editing raw files, it's a very bad idea that can only lead to issues like these. Because of how poorly documented and absolute-offset based these formats are there's just no sane way to edit them and be sure they didn't break.
As for this case in particular the issue seems to be with that "fix" exiftool did. I haven't looked at it in detail but the end result is that the Tiff parsing of the Olympus Image Processing IFD now fails by finding a tag whose type is 0x3e9d. rawspeed will throw an exception at that point and abort reading the IFD. So even if WB has been read before it won't be used. I suspect other tools are reading up to that value and failing silently after that.
Ideally exiftool (and we've had the same issue with digikam) would stop editing raw files completely. It's never a good idea and users will keep shooting themselves in the foot with it.
Cheers,
Pedro
Hi Pedro,
Quote from: pedrocr on May 04, 2015, 06:50:47 AM
We would really appreciate if people would stop editing raw files, it's a very bad idea that can only lead to issues like these. Because of how poorly documented and absolute-offset based these formats are there's just no sane way to edit them and be sure they didn't break.
Thanks for your opinion. There are some who would agree. However, editing RAW files is not going to go away. (Food for thought: Adobe will always edit DNG images, and they have all the same pitfalls as other RAW formats, and nobody is going to stop Nikon from shipping software that modifies their NEF images.)
QuoteI haven't looked at it in detail but the end result is that the Tiff parsing of the Olympus Image Processing IFD now fails by finding a tag whose type is 0x3e9d.
Thanks for looking into this, but some more details would be useful.
@mahikeulbody, it would be useful if you could send me the same sample so I can try to figure out what Darktable is doing by looking for 0x3e9d. Thanks.
- Phil
Edit: Added some food for thought.Edit2: I thought I would try installing Darktable and debugging this myself (on OS X 10.10.3), but after wasting about an hour trying to get all of the dependencies installed, I finally gave up because the Darktable build couldn't find gtk+-3.0 (which I had installed with "brew install gtk+3").
File link : https://drive.google.com/file/d/0BzRds7xWx26TOS1La3VvQmx3Unc/view?usp=sharing
Please note that for another image I get "Unknown Type 0x188e encountered" so it is not ever 0x3e9d (both images modified by Exiftool with the same command, but one is 90cw rotated, the other one is 270cw rotated).
Thanks for the file. Unfortunately, a quick look for 0x3e9d (little endian, so 0x9d 0x3e) didn't yield any interesting results (certainly nothing in the ImageProcessingIFD).
- Phil
Quote from: Phil Harvey on May 04, 2015, 08:13:54 AM
Unfortunately, a quick look for 0x3e9d (little endian, so 0x9d 0x3e) didn't yield any interesting results (certainly nothing in the ImageProcessingIFD).
The offset at which rawspeed is trying to read the TiffEntry is 0xeca6 which does indeed have those values in there. Don't know what exiftool did to the file that we are now looking in the wrong place.
Hi Pedro,
Quote from: pedrocr on May 04, 2015, 09:16:47 AM
The offset at which rawspeed is trying to read the TiffEntry is 0xeca6 which does indeed have those values in there. Don't know what exiftool did to the file that we are now looking in the wrong place.
Thanks. That offset is inside the PreviewImage. I can't see how rawspeed could be landing there. :(
BTW: The maker note offsets are relative to the start of the maker notes, not absolute as you mentioned in your earlier post.
- Phil
Quote from: Phil Harvey on May 04, 2015, 09:28:02 AM
Thanks. That offset is inside the PreviewImage. I can't see how rawspeed could be landing there. :(
This is the relevant rawspeed code:
https://github.com/darktable-org/darktable/blob/master/src/external/rawspeed/RawSpeed/OrfDecoder.cpp#L289-L290
My guess is that exiftool has somehow messed with those offsets.
Quote from: Phil Harvey on May 04, 2015, 09:28:02 AM
BTW: The maker note offsets are relative to the start of the maker notes, not absolute as you mentioned in your earlier post.
I don't think that's true of all formats, and I wasn't referring to just makernotes, but raw formats in general. They are only partially understood and Tiff is absolute addressed. Editing raw files is always a bad idea. Even if you don't change the structure (just replace N bytes with N bytes) there are sometimes some fields that depend on others (e.g., the "encrypted" stuff in Sony files) so editing fields may mean losing the key to "decrypt" other fields.
Quote from: pedrocr on May 04, 2015, 12:12:02 PM
This is the relevant rawspeed code:
Thanks for the reference. I'm still trying to compile darktable so I can debug this for you.
QuoteMy guess is that exiftool has somehow messed with those offsets.
Of course, ExifTool has messed with all of the offsets.
QuoteI don't think that's true of all formats
Correct.
Quotethere are sometimes some fields that depend on others (e.g., the "encrypted" stuff in Sony files) so editing fields may mean losing the key to "decrypt" other fields.
Yes. And ExifTool handles these dependencies. It won't currently let you change the decryption key in Sony files.
- Phil
Quote from: Phil Harvey on May 04, 2015, 12:30:34 PM
Thanks for the reference. I'm still trying to compile darktable so I can debug this for you.
Unless there's actually something wrong with the code (as in something that doesn't work with ORF files out of the camera) I won't be changing it to accommodate files that have been altered. Our resources are spread thin enough just trying to support the formats as the manufacturers generate them. You're welcome to submit a patch to rawspeed or darktable if you like. The rawspeed repo is here:
https://github.com/klauspost/rawspeed
Quote from: Phil Harvey on May 04, 2015, 12:30:34 PM
Yes. And ExifTool handles these dependencies. It won't currently let you change the decryption key in Sony files.
You handle those you know about, but there is plenty about these formats we just don't know.
Quote from: pedrocr on May 04, 2015, 12:37:01 PM
Unless there's actually something wrong with the code (as in something that doesn't work with ORF files out of the camera) I won't be changing it to accommodate files that have been altered.
Yes. I've seen this sentiment before, but in the past problems like this have pointed out weaknesses in code, and fixing them has resulted in more robust software. (And I'm not talking about ExifTool.)
- Phil
Quote from: Phil Harvey on May 04, 2015, 01:53:40 PM
Yes. I've seen this sentiment before, but in the past problems like this have pointed out weaknesses in code, and fixing them has resulted in more robust software. (And I'm not talking about ExifTool.)
I'm happy to fix weaknesses in code. This issue already highlighted that we weren't recovering properly from exceptions in white balance parsing and that's been fixed. All I'm saying is that since we consider editing raw files to be a bad idea we won't be spending time debugging weird issues caused by it. If someone else provides a patch or a description of what we are doing wrong I'm perfectly willing to have a look.
OK, thanks. I'll let you know if I discover anything, provided I can get Darktable to build.
I'm now past the gtk+3 problem, but have run into this Darktable build problem (http://darktable.org/redmine/issues/10288) which was reported back in January, but apparently not fixed yet.
- Phil
Quote from: Phil Harvey on May 04, 2015, 07:33:48 PM
I'm now past the gtk+3 problem, but have run into this Darktable build problem (http://darktable.org/redmine/issues/10288) which was reported back in January, but apparently not fixed yet.
I suggest building with gcc. We've been testing with clang to use some of it's correctness features but it's definitely not a very well tested configuration.
Actually according to that bug report the darktable-1.6.x branch (current stable) should build and that one has the same rawspeed code.
Yes, thanks. I was going to try that next. (version 1.6.x)
- Phil
Wasted another hour or so trying to get darktable-1.6.x compiling. I'm now stumped with a different undefined symbol problem:
Linking C shared library libdarktable.dylib
clang: warning: argument unused during compilation: '-pthread'
Undefined symbols for architecture x86_64:
"_dt_cairo_image_surface_create", referenced from:
_dt_bauhaus_popup_expose in bauhaus.c.o
_dt_bauhaus_expose in bauhaus.c.o
_dt_control_expose in control.c.o
_dt_control_expose_endmarker in control.c.o
_expose_borders in gtk.c.o
_configure in gtk.c.o
_dt_gui_gtk_run in gtk.c.o
...
"_dt_cairo_image_surface_get_height", referenced from:
_dt_control_expose in control.c.o
"_dt_cairo_image_surface_get_width", referenced from:
_dt_control_expose in control.c.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [src/libdarktable.dylib] Error 1
make[1]: *** [src/CMakeFiles/lib_darktable.dir/all] Error 2
make: *** [all] Error 2
- Phil
Quote from: Phil Harvey on May 05, 2015, 07:18:32 AM
Wasted another hour or so trying to get darktable-1.6.x compiling. I'm now stumped with a different undefined symbol problem:
You're probably trying to compile this on osx, no? We do support it but there's only one developer that compiles it there. No idea what the issue could be. gcc is probably a better bet.
Quote from: pedrocr on May 05, 2015, 07:46:40 AM
You're probably trying to compile this on osx, no?
Yes. OS X 10.3.3
QuoteNo idea what the issue could be. gcc is probably a better bet.
I have gcc installed, but don't know how to get the Darktable build to use it.
- Phil
No clue on the OSX build. I pinged our OSX developer on IRC though. Here's a hint to what could be the problem though. This is how dcraw does it:
https://github.com/pedrocr/dcraw/blob/412d49a2cb0d73400dee90c660d09e464a77834a/dcraw.c#L5376-L5377
It doesn't actually use the tag value to figure out where the IFD is. It just assumes it comes right after the tag in the file. Rawspeed actually uses the number. My guess is that exiftool is changing the IFD location and not writing the proper offset into the tag.
The solution of course would be not to mess with the file at all, who knows what else could be broken... :)
Quote from: pedrocr on May 05, 2015, 08:23:20 AM
This is how dcraw does it:
https://github.com/pedrocr/dcraw/blob/412d49a2cb0d73400dee90c660d09e464a77834a/dcraw.c#L5376-L5377
It doesn't actually use the tag value to figure out where the IFD is. It just assumes it comes right after the tag in the file.
No. tiff_get() sets the current file position to the tag offset.
QuoteMy guess is that exiftool is changing the IFD location and not writing the proper offset into the tag.
No again. ExifTool correctly updates the tag offset (the ImageProcessingIFD pointer). However, the offset base changes because the makernotes have been repositioned and the offset is relative to their start. But rawspeed must handle this correctly because it (apparently) works for non-rotated edited ORF's.
Quote
The solution of course would be not to mess with the file at all, who knows what else could be broken... :)
Likely nothing is broken currently. I'm just trying to prove that.
- Phil
Quote from: Phil Harvey on May 05, 2015, 08:42:37 AMBut rawspeed must handle this correctly because it (apparently) works for non-rotated edited ORF's.
Well, now I am discovering that non rotated files modified by Exiftool have also problem into DT (but later when open it into darkroom)... I have to check it a little bit more deeply.
Edit: I posted an update to pedrocr. Since the non-rotated files seem correctly imported I did'nt see that they have also a problem in DT later, when trying to open them into the darkroom. Sorry if gave a false track to diagnose the problem :(
Quote from: Phil Harvey on May 05, 2015, 08:42:37 AM
No. tiff_get() sets the current file position to the tag offset.
It doesn't. The tag type is 13 and those are inline there's no fseek:
https://github.com/pedrocr/dcraw/blob/412d49a2cb0d73400dee90c660d09e464a77834a/dcraw.c#L5075-L5076
Quote from: Phil Harvey on May 05, 2015, 08:42:37 AM
No again. ExifTool correctly updates the tag offset (the ImageProcessingIFD pointer). However, the offset base changes because the makernotes have been repositioned and the offset is relative to their start. But rawspeed must handle this correctly because it (apparently) works for non-rotated edited ORF's.
That's not the issue as we calculate the offset based on the parent. But apparently you rewrite the file so much you now require 4 bytes instead of 2 for that relative offset which is what triggers the wrong calculation. I'm not sure 4 bytes is correct but is doesn't screw up any other sample files I have so I've changed it:
https://github.com/darktable-org/darktable/commit/824dc130b1aa43c796910ac3f5be387b106eea92
Quote from: pedrocr on May 05, 2015, 09:16:27 AM
It doesn't. The tag type is 13 and those are inline there's no fseek:
(Is is just me, or is the dcraw code difficult to follow?)
QuoteQuote from: Phil Harvey on May 05, 2015, 08:42:37 AM
No again. ExifTool correctly updates the tag offset (the ImageProcessingIFD pointer). However, the offset base changes because the makernotes have been repositioned and the offset is relative to their start. But rawspeed must handle this correctly because it (apparently) works for non-rotated edited ORF's.
That's not the issue as we calculate the offset based on the parent. But apparently you rewrite the file so much you now require 4 bytes instead of 2 for that relative offset which is what triggers the wrong calculation. I'm not sure 4 bytes is correct but is doesn't screw up any other sample files I have so I've changed it:
Ah, nice! Yes. TIFF offsets are 32 bits. (Format type 13 is an unsigned 32-bit IFD offset)
So with this change Darktable can now read the file correctly?
- Phil
Quote from: Phil Harvey on May 05, 2015, 09:29:22 AM
(Is is just me, or is the dcraw code difficult to follow?)
Very much so, which is one of the reasons we've moved to 100% rawspeed but it does have the most accumulated knowledge of all the raw formats out there. This is actually standard though. Only >4 byte values are stored at the offset, all the rest are stored inline, which makes sense (as offsets use 4 bytes anyway).
Quote from: Phil Harvey on May 05, 2015, 09:29:22 AM
Ah, nice! Yes. TIFF offsets are 32 bits. (Format type 13 is an unsigned 32-bit IFD offset)
Do you have a reference for that type 13? I couldn't find it in the standard and as far as I can tell SubIFDs are actually stored as LONG (format 4).
Quote from: Phil Harvey on May 05, 2015, 09:29:22 AM
So with this change Darktable can now read the file correctly?
Seems to.
Hi Pedro,
Quote from: pedrocr on May 05, 2015, 09:40:46 AM
Do you have a reference for that type 13? I couldn't find it in the standard and as far as I can tell SubIFDs are actually stored as LONG (format 4).
Yes. See the reference in my solution to TIFF problems (https://exiftool.org/standards.html#TIFF). Olympus has actually implemented this solution, which is fantastic, and makes ExifTool much more forward compatible with future ORF versions.
QuoteQuote from: Phil Harvey on May 05, 2015, 09:29:22 AM
So with this change Darktable can now read the file correctly?
Seems to.
Excellent!! Thanks for your prompt response and work on this!
- Phil
Pedro and Phil : thanks you !
Hi Pedro,
Just being picky here:
You called TIFF format type 13 "TIFF_CRAZY" in your patch (https://github.com/darktable-org/darktable/commit/c84f69b7a7db8d768f32b8160f48a4a8e510008a).
It should be called "TIFF_IFD" according to the reference I gave (http://partners.adobe.com/public/developer/en/tiff/TIFFPM6.pdf).
(It is not crazy at all.)
- Phil
Quote from: Phil Harvey on May 05, 2015, 11:56:44 AM
You called TIFF format type 13 "TIFF_CRAZY" in your patch (https://github.com/darktable-org/darktable/commit/c84f69b7a7db8d768f32b8160f48a4a8e510008a).
It should be called "TIFF_IFD" according to the reference I gave (http://partners.adobe.com/public/developer/en/tiff/TIFFPM6.pdf).
(It is not crazy at all.)
It's pretty crazy... Trying to mutate a TIFF consistently doesn't seem really feasible, let alone one of these very ill specified raw formats. I've called it TIFF_OFFSET to make it more descriptive. It's too bad CIFF didn't win or DNG isn't used more.
Hi Pedro,
I agree that CIFF would have been much better.
But DNG isn't any better than any other RAW format from a metadata point of view. In fact it is worse: The Adobe DNG converter takes proprietary maker notes, breaks them apart, and buries the pieces inside another proprietary Adobe layer. Also, the writers of the DNG specification need some serious reprimanding (did you see the DNG 1.3 problem (https://exiftool.org/standards.html#DNG) on the page I referenced earlier?). Just another crazy TIFF mutation for sure. ;)
- Phil