Writing PhaseOne MakerNotes

[alepheiffel]

Hi Phil,

Could the Phase One maker notes be made writable in a future release, or is there something preventing this with the IIQ format ?

As you know they put a lot of things in this bloc, and I'd really like to experiment with some Phase One and PhaseOne SensorCalibration tags (colormatrix, flatfield, linearization values, etc).
Thanks for reading.

Best,
Aleph

[Phil Harvey]

Hi Aleph,

I looked at an IIQ file to see why ExifTool won't write the maker notes.  The thing that scares me is the raw data is stored in the maker notes of these files, which may be the reason why I haven't added write support yet.  But also, I only have two IIQ sample from two models (P40+ and P45+), which is certainly not enough to thoroughly test any writing feature.

- Phil

[alepheiffel]

Hi Phil,

Thanks for your answer.

Indeed, Phase One puts all the interesting stuffs in the maker notes.
If you want to have a look, I can prepare a couple of files from my IQ back as well as some from Leaf Credos (Leaf dropped the .MOS format and switched to IIQ in 2012) and send you a download link.

Best,
Aleph

[Phil Harvey]

Hi Aleph,

Thanks.  Any IIQ samples you have from other models and/or software would be appreciated.

- Phil

[alepheiffel]

Hi Phil,

PM sent with a link to various IIQ files. Will try to get some more if needed.

Aleph

[Phil Harvey]

Hi Aleph,

I got the samples, thanks!

- Phil

[alepheiffel]

Hi Phil,

If you need more files or someone to test, please don't hesitate to contact me.

Best,
Alpeh

[Phil Harvey]

Hi Aleph,

Thanks for the offer.  What platform are you using?  (Hoping it is Mac or Linux because it is more work for me to produce a Windows version for testing.)

- Phil

[alepheiffel]

Hi Phil,

An OSX version is fine, I have access to 10.10 and 10.11 systems.

Alpeh

[alepheiffel]

Hello Phil,

There have been new models released as well as firmware updates. I can probably try and get you some files if still needed.

Best,
Aleph

[Phil Harvey]

Hi Aleph,

Files from new models would be appreciated.  This is top of my to-do list but I haven't had much time recently to work on this.

- Phil

[Phil Harvey]

I have just made a 10.08 pre-release available here.  This version has the ability to write the PhaseOne maker notes.  It would be great if you got a chance to test this out before the official release in a week or so.

- Phil

[alepheiffel]

Hi Phil,

That's great, I will test over the weekend and report back. I managed to get files from some newer backs too, not all of them though.

Aleph

[c++]

Hi Phil,

I'll hijack this thread as the question I have fits well in this sequence that does not seem to have been completed back then.

I am copying all meta data from RAW images to interpolated Tiff images. In case of Phase One *.IIQ files I just found that I copy the actual RAW image along with the maker notes.
When opening the Tiff in Photoshop the RAW data is encountered and Camera RAW opens the file on that basis - ignoring the Tiff as a consequence.
Ther eis a specialized PS > File > Open As > Tiff command that circumvents this but...

I would thus like to be able to either remove the RAW data from the maker notes or copy everything but that.

Didn't get that to work. The only thing I could do was:

Code Select Expand

exiftool -MakerNotePhaseOne= *.IIQ

That is kind of desperate...  There is stuff in there I'd like to keep - like:

Code Select Expand

 
| | | 40) FirmwareVersions = P40+-M, Firmware: Main=4.8.8, Boot=2.3, FPGA=1.1.9, CPLD[snip]
  | | |     - Tag 0x0301 (100 bytes, string[100])
  | | | 41) Software = Camera back, Main firmware: 4.8.8
  | | |     - Tag 0x0203 (36 bytes, string[36])
  | | | 42) System = Digital Camera Back
  | | |     - Tag 0x0204 (36 bytes, string[36])
  | | | 43) SerialNumber = EG021076
  | | |     - Tag 0x0102 (16 bytes, string[16])
  | | | 44) ShutterSpeedValue = 9.64385986328125
  | | |     - Tag 0x0400 (4 bytes, float[1])
  | | | 45) ApertureValue = 6
  | | |     - Tag 0x0401 (4 bytes, float[1])
  | | | 46) ExposureCompensation = 0
  | | |     - Tag 0x0402 (4 bytes, float[1])
  | | | 47) FocalLength = 150
  | | |     - Tag 0x0403 (4 bytes, float[1])


[Phil Harvey]

Thanks for the suggestion.

I should be able to make it so that the raw data isn't copied to other files.  I'll test this when I get a chance.

- Phil

[Phil Harvey]

It turns out there are a number of other large tags (mostly unknown) which bulk up the maker notes.  Unless you see a problem with this, I'll limit the tag size to < 8 kB when copying PhaseOne makernotes to another file with ExifTool 11.04.

- Phil

[c++]

I should be able to make it so that the raw data isn't copied to other files.  I'll test this when I get a chance.

Great, that will solve this for me. Thank you so much.
An update here once that is in an official build would be much appreciated.

[c++]

It turns out there are a number of other large tags (mostly unknown) which bulk up the maker notes.  Unless you see a problem with this, I'll limit the tag size to < 8 kB when copying PhaseOne makernotes to another file with ExifTool 11.04.

Turns out I missed Page 2 

I don't see a problem with that but I'm really the wrong person to ask what a sensible size threshold would be.

[c++]

Phil,

with this PhaseOne business I'm also having a problem with large Tiff files (around 600 MiB) being truncated by ExifTool upon writing Meta data to them.
I'll send you a test case via Wetransfer, perhaps that saves you time rather that I try to write out how to make it all happen.

[Phil Harvey]

Sure.  Send me the file and the exact command you are using.  This should not happen.

- Phil

[Phil Harvey]

The file you sent is not a TIFF image.  It is some sort of RAW image that I haven't seen before.  The Software tag in this file says "dcraw v9.28", but I didn't know that dcraw had the ability to write RAW images.

Deleting all EXIF from a RAW image will remove proprietary information that may destroy the image.  In this case, it seems that the RAW data is lost.  Actually, just writing this file with ExifTool will remove the RAW data since it isn't referenced from anywhere in the TIFF structure.

Edit:  I figured out what is going on.  This is a TIFF image, but it is not written correctly...

Code Select Expand

> exiftool -stripbytecounts -imagewidth -imageheight -samplesperpixel -bitspersample CF011962.tiff
Strip Byte Counts               : 69623872
Image Width                     : 11608
Image Height                    : 8708
Samples Per Pixel               : 3
Bits Per Sample                 : 16 16 16

The StripByteCounts is wrong.  It reports 69623872 (69 MB, hex 0x4266040), which is what ExifTool is copying.  However, the value should be:

ImageWidth * ImageHeight * SamplesPerPixel * BytesPerSample = 11608 * 8708 * 3 * 2 = 606494784 (606 MB, hex 0x24266040).

So more than 500 MB (hex 0x20000000) of the image is lost.  I will add a check to detect this problem and avoid writing bad files like this.  This looks like an arithmetic bug in whatever software wrote this file.

Thanks for this report.

- Phil

[c++]

The StripByteCounts is wrong.

Thank you. That is curious. Out of hundreds of cameras the Phase One IQ3 100MP is the only one I ever saw where dcraw does this.
It is a fairly recent back, something must have changed in the IIQ format that causes this.

Edit: Since this also happens for an intermediate DNG file made from the IIQ, it can't be about the IIQ format as such.

PHASE ONE IQ180 derived tiffs are fine.

Code Select Expand

Make (271) ASCII (2) 10<Phase One\0>
Model (272) ASCII (2) 6<IQ180\0>
ImageWidth (256) LONG (4) 1<10328>
ImageLength (257) LONG (4) 1<7760>
BitsPerSample (258) SHORT (3) 3<16 16 16>
SamplesPerPixel (277) SHORT (3) 1<3>
RowsPerStrip (278) LONG (4) 1<7760>
StripByteCounts (279) LONG (4) 1<480871680>


I will add a check to detect this problem and avoid writing bad files like this.

Thank you this will fix the problem pragmatically. I also wrote to David so that he is aware of it and can update dcraw as he sees fit.

[c++]

Hi Phil,

just encountering this issue again, did you follow up on:

I will add a check to detect this problem and avoid writing bad files like this.  This looks like an arithmetic bug in whatever software wrote this file.

I get an error in the consequence of yet another wrong StripByteCounts coming out of dcraw

Code Select Expand

Error: [minor] Undersized IFD0 StripByteCounts (69623872 bytes, but expected 606
494784) - E:/Test/from_IQ3_100MP_Trichr.tiff
    0 image files updated
    1 files weren't updated due to errors

Is it possible to just correct the StripByteCounts?

Code Select Expand

"exiftool.exe" -StripByteCounts=606494784 -overwrite_original "E:/Test/from_IQ3_100MP_Trichr.tiff"
Warning: Sorry, StripByteCounts is not writable
Nothing to do.

[Phil Harvey]

Is it possible to just correct the StripByteCounts?

No.  This would be very dangerous.

- Phil

[c++]

Is it possible to just correct the StripByteCounts?

No.  This would be very dangerous.

...and if it were to be limited to be able to set StripByteCounts = ImageWidth * RowsPerStrip * SamplesPerPixel * BitsPerSample/8
That would fix the immediate predicament, dcraw acts up, I wrote to DC on June 28th 2018.
So, I don't see a fix from that end and I don't really intend to dig into dcraw.c to do it myself.

[Phil Harvey]

Unfortunately there are many other cases where cameras write "interesting" values for StripByteCounts in raw images.  Allowing this to be written, even with the value the ExifTool wants, could cause problems with the specific manufacturer software used to read these images.  Also, it would be somewhat difficult for me to implement your suggestion, and I don't see that I could properly test it without a LOT of effort to see what the effects are wrt manufacturer software.

- Phil

[c++]

Thanks for thinking about it Phil, it's not your bug to fix really. Had there been a viable way around my problem I would have aprechiated a solution by exiftool. As it is I simply can't do the meta data aspects I need to do without fixing that Tiff file first. Alas it seems I'll either dig in deeper into dcraw's code base or find another commandline app to just read and re-write the file. LibTiff might have something, ImageMagick for sure should. I'd prefer a solution without any additional files in need of redistribution or user installation though... Puzzling why this happens for the huge IIQ files only, perhaps it is about initial meta data interpreted by dcraw - maybe something worth going through. If I find anything useful I'll update this thread.

[c++]

As expected LibTiff has an executable "tiffcp"

Code Select Expand

tiffcp in.tiff out.tif
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.

That fixes it.

In case anyone is building LibTiff on Win here are some notes:

0) run "C:\Program Files (x86)\Microsoft Visual Studio XX.0\VC\bin\vcvars32.bat"
1) open a command windo inside of the root folder of the package
2) build with nmake "nmake /f C:\tiff-4.0.3\Makefile.vc"
3) "cd tools"
4) embed manifest "mt.exe –manifest tiffcp.exe.manifest -outputresource:tiffcp.exe;1"

=> now you can redistribute the *.exe file along with LibTIffs license.

In case anyone is building LibTiff on MacOS here are some notes to buid stand alone tiffcp from libtiff:

0) edit libtiff.h to disable JPEG_SUPPORT, PIXARLOG_SUPPORT, and ZIP_SUPPORT
1) configure libtiff for i386:
./configure  CFLAGS="-O -arch i386" CXXFLAGS="-O -arch i386"  LDFLAGS="-arch i386"  --disable-dependency-tracking
2) make
3) go to where you built libtiff /tools/.libs folder by terminal and copy out tiffcp
4) terminal: "otool -L tiffcp" to list its dependencies
5) go to where you built libtiff /libtiff/.libs folder by terminal and copy out libtiff.5.dylib
6) put it in the same folder as the tiffsplit executable
7) terminal: "install_name_tool -change /usr/local/lib/libtiff.5.dylib ./libtiff.5.dylib tiffcp"

=> now you can redistribute the two files along with LibTIffs license.

Guess that's enough for anyone with the same issue later...

[Phil Harvey]

Great.  Glad you found a solution.

- Phil