Strange problem in case of timestamp shift

[pedrocr]

Yes.  I've seen this sentiment before, but in the past problems like this have pointed out weaknesses in code, and fixing them has resulted in more robust software.  (And I'm not talking about ExifTool.)

I'm happy to fix weaknesses in code. This issue already highlighted that we weren't recovering properly from exceptions in white balance parsing and that's been fixed. All I'm saying is that since we consider editing raw files to be a bad idea we won't be spending time debugging weird issues caused by it. If someone else provides a patch or a description of what we are doing wrong I'm perfectly willing to have a look.

[Phil Harvey]

OK, thanks.  I'll let you know if I discover anything, provided I can get Darktable to build.

I'm now past the gtk+3 problem, but have run into this Darktable build problem which was reported back in January, but apparently not fixed yet.

- Phil

[pedrocr]

I'm now past the gtk+3 problem, but have run into this Darktable build problem which was reported back in January, but apparently not fixed yet.

I suggest building with gcc. We've been testing with clang to use some of it's correctness features but it's definitely not a very well tested configuration.

[pedrocr]

Actually according to that bug report the darktable-1.6.x branch (current stable) should build and that one has the same rawspeed code.

[Phil Harvey]

Yes, thanks.  I was going to try that next. (version 1.6.x)

- Phil

[Phil Harvey]

Wasted another hour or so trying to get darktable-1.6.x compiling.  I'm now stumped with a different undefined symbol problem:

Code Select Expand

Linking C shared library libdarktable.dylib
clang: warning: argument unused during compilation: '-pthread'
Undefined symbols for architecture x86_64:
  "_dt_cairo_image_surface_create", referenced from:
      _dt_bauhaus_popup_expose in bauhaus.c.o
      _dt_bauhaus_expose in bauhaus.c.o
      _dt_control_expose in control.c.o
      _dt_control_expose_endmarker in control.c.o
      _expose_borders in gtk.c.o
      _configure in gtk.c.o
      _dt_gui_gtk_run in gtk.c.o
      ...
  "_dt_cairo_image_surface_get_height", referenced from:
      _dt_control_expose in control.c.o
  "_dt_cairo_image_surface_get_width", referenced from:
      _dt_control_expose in control.c.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [src/libdarktable.dylib] Error 1
make[1]: *** [src/CMakeFiles/lib_darktable.dir/all] Error 2
make: *** [all] Error 2

- Phil

[pedrocr]

Wasted another hour or so trying to get darktable-1.6.x compiling.  I'm now stumped with a different undefined symbol problem:

You're probably trying to compile this on osx, no? We do support it but there's only one developer that compiles it there. No idea what the issue could be. gcc is probably a better bet.

[Phil Harvey]

You're probably trying to compile this on osx, no?

Yes.  OS X 10.3.3

No idea what the issue could be. gcc is probably a better bet.

I have gcc installed, but don't know how to get the Darktable build to use it.

- Phil

[pedrocr]

No clue on the OSX build. I pinged our OSX developer on IRC though. Here's a hint to what could be the problem though. This is how dcraw does it:

https://github.com/pedrocr/dcraw/blob/412d49a2cb0d73400dee90c660d09e464a77834a/dcraw.c#L5376-L5377

It doesn't actually use the tag value to figure out where the IFD is. It just assumes it comes right after the tag in the file. Rawspeed actually uses the number. My guess is that exiftool is changing the IFD location and not writing the proper offset into the tag.

The solution of course would be not to mess with the file at all, who knows what else could be broken...

[Phil Harvey]

This is how dcraw does it:

https://github.com/pedrocr/dcraw/blob/412d49a2cb0d73400dee90c660d09e464a77834a/dcraw.c#L5376-L5377

It doesn't actually use the tag value to figure out where the IFD is.  It just assumes it comes right after the tag in the file.

No.  tiff_get() sets the current file position to the tag offset.

My guess is that exiftool is changing the IFD location and not writing the proper offset into the tag.

No again.  ExifTool correctly updates the tag offset (the ImageProcessingIFD pointer).  However, the offset base changes because the makernotes have been repositioned and the offset is relative to their start.  But rawspeed must handle this correctly because it (apparently) works for non-rotated edited ORF's.

The solution of course would be not to mess with the file at all, who knows what else could be broken...

Likely nothing is broken currently.  I'm just trying to prove that.

- Phil

[mahikeulbody]

But rawspeed must handle this correctly because it (apparently) works for non-rotated edited ORF's.

Well, now I am discovering that non rotated files modified by Exiftool have also problem into DT (but later when open it into darkroom)... I have to check it a little bit more deeply.

Edit: I posted an update to pedrocr. Since the non-rotated files seem correctly imported I did'nt see that they have also a problem in DT later, when trying to open them into the darkroom. Sorry if gave a false track to diagnose the problem

[pedrocr]

No.  tiff_get() sets the current file position to the tag offset.

It doesn't. The tag type is 13 and those are inline there's no fseek:

https://github.com/pedrocr/dcraw/blob/412d49a2cb0d73400dee90c660d09e464a77834a/dcraw.c#L5075-L5076

No again.  ExifTool correctly updates the tag offset (the ImageProcessingIFD pointer).  However, the offset base changes because the makernotes have been repositioned and the offset is relative to their start.  But rawspeed must handle this correctly because it (apparently) works for non-rotated edited ORF's.

That's not the issue as we calculate the offset based on the parent. But apparently you rewrite the file so much you now require 4 bytes instead of 2 for that relative offset which is what triggers the wrong calculation. I'm not sure 4 bytes is correct but is doesn't screw up any other sample files I have so I've changed it:

https://github.com/darktable-org/darktable/commit/824dc130b1aa43c796910ac3f5be387b106eea92

[Phil Harvey]

It doesn't. The tag type is 13 and those are inline there's no fseek:

(Is is just me, or is the dcraw code difficult to follow?)

No again.  ExifTool correctly updates the tag offset (the ImageProcessingIFD pointer).  However, the offset base changes because the makernotes have been repositioned and the offset is relative to their start.  But rawspeed must handle this correctly because it (apparently) works for non-rotated edited ORF's.

That's not the issue as we calculate the offset based on the parent. But apparently you rewrite the file so much you now require 4 bytes instead of 2 for that relative offset which is what triggers the wrong calculation. I'm not sure 4 bytes is correct but is doesn't screw up any other sample files I have so I've changed it:

Ah, nice!  Yes.  TIFF offsets are 32 bits.  (Format type 13 is an unsigned 32-bit IFD offset)

So with this change Darktable can now read the file correctly?

- Phil

[pedrocr]

(Is is just me, or is the dcraw code difficult to follow?)

Very much so, which is one of the reasons we've moved to 100% rawspeed but it does have the most accumulated knowledge of all the raw formats out there. This is actually standard though. Only >4 byte values are stored at the offset, all the rest are stored inline, which makes sense (as offsets use 4 bytes anyway).

Ah, nice!  Yes.  TIFF offsets are 32 bits.  (Format type 13 is an unsigned 32-bit IFD offset)

Do you have a reference for that type 13? I couldn't find it in the standard and as far as I can tell SubIFDs are actually stored as LONG (format 4).

So with this change Darktable can now read the file correctly?

Seems to.

[Phil Harvey]

Hi Pedro,

Do you have a reference for that type 13? I couldn't find it in the standard and as far as I can tell SubIFDs are actually stored as LONG (format 4).

Yes.  See the reference in my solution to TIFF problems.  Olympus has actually implemented this solution, which is fantastic, and makes ExifTool much more forward compatible with future ORF versions.

So with this change Darktable can now read the file correctly?

Seems to.

Excellent!!  Thanks for your prompt response and work on this!

- Phil