Writing PhaseOne MakerNotes

Started by alepheiffel, September 28, 2015, 05:04:01 PM

Previous topic - Next topic

Phil Harvey

It turns out there are a number of other large tags (mostly unknown) which bulk up the maker notes.  Unless you see a problem with this, I'll limit the tag size to < 8 kB when copying PhaseOne makernotes to another file with ExifTool 11.04.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

c++

Quote from: Phil Harvey on June 23, 2018, 07:49:22 AM
I should be able to make it so that the raw data isn't copied to other files.  I'll test this when I get a chance.
Great, that will solve this for me. Thank you so much.
An update here once that is in an official build would be much appreciated.

c++

Quote from: Phil Harvey on June 26, 2018, 07:08:26 AM
It turns out there are a number of other large tags (mostly unknown) which bulk up the maker notes.  Unless you see a problem with this, I'll limit the tag size to < 8 kB when copying PhaseOne makernotes to another file with ExifTool 11.04.
Turns out I missed Page 2  :)

I don't see a problem with that but I'm really the wrong person to ask what a sensible size threshold would be.

c++

Phil,

with this PhaseOne business I'm also having a problem with large Tiff files (around 600 MiB) being truncated by ExifTool upon writing Meta data to them.
I'll send you a test case via Wetransfer, perhaps that saves you time rather that I try to write out how to make it all happen.

Phil Harvey

Sure.  Send me the file and the exact command you are using.  This should not happen.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#20
The file you sent is not a TIFF image.  It is some sort of RAW image that I haven't seen before.  The Software tag in this file says "dcraw v9.28", but I didn't know that dcraw had the ability to write RAW images.

Deleting all EXIF from a RAW image will remove proprietary information that may destroy the image.  In this case, it seems that the RAW data is lost.  Actually, just writing this file with ExifTool will remove the RAW data since it isn't referenced from anywhere in the TIFF structure.


Edit:  I figured out what is going on.  This is a TIFF image, but it is not written correctly...

> exiftool -stripbytecounts -imagewidth -imageheight -samplesperpixel -bitspersample CF011962.tiff
Strip Byte Counts               : 69623872
Image Width                     : 11608
Image Height                    : 8708
Samples Per Pixel               : 3
Bits Per Sample                 : 16 16 16


The StripByteCounts is wrong.  It reports 69623872 (69 MB, hex 0x4266040), which is what ExifTool is copying.  However, the value should be:

ImageWidth * ImageHeight * SamplesPerPixel * BytesPerSample = 11608 * 8708 * 3 * 2 = 606494784 (606 MB, hex 0x24266040).

So more than 500 MB (hex 0x20000000) of the image is lost.  I will add a check to detect this problem and avoid writing bad files like this.  This looks like an arithmetic bug in whatever software wrote this file.

Thanks for this report.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

c++

#21
Quote from: Phil Harvey on June 28, 2018, 07:31:34 AM
The StripByteCounts is wrong.
Thank you. That is curious. Out of hundreds of cameras the Phase One IQ3 100MP is the only one I ever saw where dcraw does this.
It is a fairly recent back, something must have changed in the IIQ format that causes this.

Edit: Since this also happens for an intermediate DNG file made from the IIQ, it can't be about the IIQ format as such.

PHASE ONE IQ180 derived tiffs are fine.
Make (271) ASCII (2) 10<Phase One\0>
Model (272) ASCII (2) 6<IQ180\0>
ImageWidth (256) LONG (4) 1<10328>
ImageLength (257) LONG (4) 1<7760>
BitsPerSample (258) SHORT (3) 3<16 16 16>
SamplesPerPixel (277) SHORT (3) 1<3>
RowsPerStrip (278) LONG (4) 1<7760>
StripByteCounts (279) LONG (4) 1<480871680>


Quote from: Phil Harvey on June 28, 2018, 07:31:34 AMI will add a check to detect this problem and avoid writing bad files like this.
Thank you this will fix the problem pragmatically. I also wrote to David so that he is aware of it and can update dcraw as he sees fit.

c++

Hi Phil,

just encountering this issue again, did you follow up on:
Quote from: Phil Harvey on June 28, 2018, 07:31:34 AM
I will add a check to detect this problem and avoid writing bad files like this.  This looks like an arithmetic bug in whatever software wrote this file.

I get an error in the consequence of yet another wrong StripByteCounts coming out of dcraw
Error: [minor] Undersized IFD0 StripByteCounts (69623872 bytes, but expected 606
494784) - E:/Test/from_IQ3_100MP_Trichr.tiff
    0 image files updated
    1 files weren't updated due to errors


Is it possible to just correct the StripByteCounts?

"exiftool.exe" -StripByteCounts=606494784 -overwrite_original "E:/Test/from_IQ3_100MP_Trichr.tiff"
Warning: Sorry, StripByteCounts is not writable
Nothing to do.

Phil Harvey

Quote from: c++ on February 06, 2019, 11:03:22 AM
Is it possible to just correct the StripByteCounts?

No.  This would be very dangerous.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

c++

Quote from: Phil Harvey on February 08, 2019, 01:24:16 PM
Quote from: c++ on February 06, 2019, 11:03:22 AM
Is it possible to just correct the StripByteCounts?
No.  This would be very dangerous.
...and if it were to be limited to be able to set StripByteCounts = ImageWidth * RowsPerStrip * SamplesPerPixel * BitsPerSample/8
That would fix the immediate predicament, dcraw acts up, I wrote to DC on June 28th 2018.
So, I don't see a fix from that end and I don't really intend to dig into dcraw.c to do it myself.

Phil Harvey

Unfortunately there are many other cases where cameras write "interesting" values for StripByteCounts in raw images.  Allowing this to be written, even with the value the ExifTool wants, could cause problems with the specific manufacturer software used to read these images.  Also, it would be somewhat difficult for me to implement your suggestion, and I don't see that I could properly test it without a LOT of effort to see what the effects are wrt manufacturer software.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

c++

Thanks for thinking about it Phil, it's not your bug to fix really. Had there been a viable way around my problem I would have aprechiated a solution by exiftool. As it is I simply can't do the meta data aspects I need to do without fixing that Tiff file first. Alas it seems I'll either dig in deeper into dcraw's code base or find another commandline app to just read and re-write the file. LibTiff might have something, ImageMagick for sure should. I'd prefer a solution without any additional files in need of redistribution or user installation though... Puzzling why this happens for the huge IIQ files only, perhaps it is about initial meta data interpreted by dcraw - maybe something worth going through. If I find anything useful I'll update this thread.

c++

As expected LibTiff has an executable "tiffcp"

tiffcp in.tiff out.tif
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.

That fixes it.

In case anyone is building LibTiff on Win here are some notes:

0) run "C:\Program Files (x86)\Microsoft Visual Studio XX.0\VC\bin\vcvars32.bat"
1) open a command windo inside of the root folder of the package
2) build with nmake "nmake /f C:\tiff-4.0.3\Makefile.vc"
3) "cd tools"
4) embed manifest "mt.exe –manifest tiffcp.exe.manifest -outputresource:tiffcp.exe;1"

=> now you can redistribute the *.exe file along with LibTIffs license.

In case anyone is building LibTiff on MacOS here are some notes to buid stand alone tiffcp from libtiff:

0) edit libtiff.h to disable JPEG_SUPPORT, PIXARLOG_SUPPORT, and ZIP_SUPPORT
1) configure libtiff for i386:
./configure  CFLAGS="-O -arch i386" CXXFLAGS="-O -arch i386"  LDFLAGS="-arch i386"  --disable-dependency-tracking
2) make
3) go to where you built libtiff /tools/.libs folder by terminal and copy out tiffcp
4) terminal: "otool -L tiffcp" to list its dependencies
5) go to where you built libtiff /libtiff/.libs folder by terminal and copy out libtiff.5.dylib
6) put it in the same folder as the tiffsplit executable
7) terminal: "install_name_tool -change /usr/local/lib/libtiff.5.dylib ./libtiff.5.dylib tiffcp"

=> now you can redistribute the two files along with LibTIffs license.

Guess that's enough for anyone with the same issue later...

Phil Harvey

Great.  Glad you found a solution.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).