Hello,
i do have a problem with corrupt ORF files. I do not use exif tool as standalone software. I use the picture database software imatch 2020 which itself implemented exiftools. Recently i installed a RAW converter Silkypix Developer pro 10 to develop my RAW´s. Where Silkypix was able to open Canon CR2, Panasonic RW2, it failed opening all (thousands) of my Olympus ORF files. I found out that ORF files not altered with Imatch / exiftool are readable in Silkypix. But after minor changes for example in {File.MD.XMP::xmpRights\WebStatement\WebStatement\0} and write back operation in imatch Silkypix failed open the ORF. This is because imatch in standard configuration does not only writing changes in a XMP sidecar but also with the help of ExifTools in the Exif container at the ORF. During this writing process something happens to the ORF files that makes them unreadable for Silkypix.
I openen a thread in imatch. It was not possible to find out what imatch / exiftools is changing that makes the ORF unreadable for Silkypix.
https://www.photools.com/community/index.php?topic=10648.0
(https://www.photools.com/community/index.php?topic=10648.0)
So i also opened a thread at dpreview forum and asked other people if they can help finding out the reason. A more experienced user told us that imatch / exif tools should not touch a RAW in any way and got in discussion with the develper of imatch. It is assumed that exiftools changed the makernotes what makes the files unreadable.
https://www.dpreview.com/forums/thread/4520466?page=2 (https://www.dpreview.com/forums/thread/4520466?page=2)
I also contacted Silkypix. They gave me the answer "The data you provided was also different from the RAW data structure we envisioned". After i asked once more they agreed giving it a person in charge.
My question to you is if you please can give exiftool an update that edits my ORF RAW files which corrects the changes exiftools / imatch made my ORF files unreadable for Silkypix.
Please find a link where you can download a picture in two versions, 1. original which is readable for Silkypix, 2. with minor Exif change {File.MD.XMP::photoshop\Headline\Headline\0} and not readable anymore just for Silkypix
https://drive.google.com/drive/folders/1kgUsylStMfB9bT_axK9vdTD7mOg0YTqD?usp=sharing
You are welcome to reply in the thread at imatch forum if you like.
Thank you very much.
Quote from: spiff on September 27, 2020, 11:45:32 AM
I also contacted Silkypix. They gave me the answer "The data you provided was also different from the RAW data structure we envisioned". After i asked once more they agreed giving it a person in charge.
Your edited files load correctly in Adobe Bridge, DarkTable, and DigiKam. My copy of Lightroom is very old, so I can't test to see how a modern version of Lightroom would read it, though it's likely that it would because Bridge can.
Odds are that this means they are not properly parsing the file structure. Instead, they are making assumptions about what data is where and directly reading that location for said data. In other words, lazy programming.
QuoteMy question to you is if you please can give exiftool an update that edits my ORF RAW files which corrects the changes exiftools / imatch made my ORF files unreadable for Silkypix.
That would probably be impossible because any change, no matter how minor, will probably break Silkypix's file parsing. And this is unlikely to be fixed on Silkypix's end because it would probably mean re-writing their whole file parsing routines.
Just a thought, you might use Adobe's DNG Converter to turn it into a DNG file. You can even embed the original in case Silkypix fixes their software.
As an example, here (https://www.dropbox.com/s/uf71iist3vxe8ct/PB290252-edited.dng?dl=0) is your edited file, converted to DNG with the original orf embedded. Twice as large as the original due to the embedding.
Also, RawTherapee (https://rawtherapee.com/) also has no problem reading the edited ORF file.
My guess is that the problem could be that the Make and Model tags were changed in this file -- the trailing spaces have been removed. ExifTool will only do this if you tell it to. It is not recommended to do this for these reasons (https://exiftool.org/faq.html#Q8).
Try this command and tell me if SilkyPix can read the resulting file:
exiftool -make="OLYMPUS CORPORATION " -model="PEN-F " FILE
There should be 4 spaces after the Make, and 11 spaces after the Model.
- Phil
Quote from: Phil Harvey on September 29, 2020, 06:47:50 AM
ExifTool will only do this if you tell it to.
IMatch does not explicitly tell ExifTool to strip blanks from make and model names.
It uses the XMP2EXIF.args file to map from XMP back to EXIF during write-back.
Maybe this is the reason?
Is there a specific argument that I need to set in addition to retain extra spaces when the mapping occurs?
The ORF in question only fails in SilkyPix.
Lightroom, ACR, Affinity Photo, Windiows Photo, Olympus Studio (!), LibRaw and several other RAW processors have no issues at all with this ORF. Only SilkyPix.
If they have indeed hard-coded the make/model tags with blanks, your explanation makes sense. Not their programming, though.
Quote from: Mac2 on September 29, 2020, 07:12:42 AM
IMatch does not explicitly tell ExifTool to strip blanks from make and model names.
While this is true, it would be best if it didn't do a group copy of tags to a raw file. ExifTool will strip trailing spaces from Make and Model when reading.
QuoteIt uses the XMP2EXIF.args file to map from XMP back to EXIF during write-back.
Ah. This would do it if the XMP contained Make and Model tags. At a minimum, I would suggest adding
--make --model when doing this if the target is a raw file.
QuoteIs there a specific argument that I need to set in addition to retain extra spaces when the mapping occurs?
There is no option to do this.
QuoteThe ORF in question only fails in SilkyPix.
Lightroom, ACR, Affinity Photo, Windiows Photo, Olympus Studio (!), LibRaw and several other RAW processors have no issues at all with this ORF. Only SilkyPix.
If they have indeed hard-coded the make/model tags with blanks, your explanation makes sense. Not their programming, though.
Has anyone tested this to see if the spaces are indeed the problem?
- Phil
i'm sorry, but i don't know enough to use exif tools. Free Trial version of Silkypix is available:
https://silkypix.isl.co.jp/en/download/ (https://silkypix.isl.co.jp/en/download/)
i got it right; it is without risk editing exif in a RAW file but not make and model tag? Or is altering a RAW file general no good thing?
OK. I've edited the Make and Model tags and uploaded the file here (https://exiftool.org/tmp/PB290252.orf). Check to see if this works in SilkyPix.
As I mentioned, changing Make or Model isn't recommended for other file types as well because it may affect the way makernote information is interpreted.
There is always some risk involved with writing any file. But with raw files specifically, there are things you can do to make them unusable (for example, deleting all maker notes).
- Phil
Thank you very much. The file you provided is readable with Silkypix. I hope this helps Mario. Because there is a discussion at dpreview, i ask you if you suggest to write XMP / exif datas in a RAW file (but not maker notes), or do you recommend write the XMP sidecar only and let a RAW file untached? Your answer wil help me configuring imatch proper. Imatch standard configuration is writing Exif / XMP in RAW, but it is possible to change the settings. Also Mario warned that RAW and sidecar going out of sync doing so. Thank you.
I have followed this but I'm still unsure as how to make the XMP2EXIF args file provided by IMatch can be overridden to not remove spaces from the make/model tags.
Typically, a write back runs these arguments:
-overwrite_original_in_place
-charset
FILENAME=UTF8
-m
-use
MWG
-charset
ExifTool=UTF8
-ex
-tagsfromfile
c:\temp\image.nef
-@
v:\exiftool\arg_files\exif2xmp.args
--Exif:rating
-@
v:\exiftool\arg_files\iptc2xmp.args
-@
v:\exiftool\arg_files\gps2xmp.args
-sep
-XMP-lr:HierarchicalSubject=
-XMP-dc:Subject=
-XMP-lr:HierarchicalSubject=ORIGIN|studio-ffm
-XMP-lr:HierarchicalSubject=imatch
-XMP-lr:HierarchicalSubject=NIKON D850
-XMP-dc:Subject=ORIGIN|studio-ffm
-XMP-dc:Subject=imatch
-XMP-dc:Subject=NIKON D850
-XMP-xmp:Label=Red
-XMP-xmp:Rating=2
-XMP:CreatorTool=photools.com IMatch 20.9.0.4 (Windows)
-xmp:InstanceID=xmp.iid:e766d4e4-843b-4e70-a15f-4750e2a94566
-XMP:MetadataDate=now
-XMP:ModifyDate=now
c:\temp\image.xmp
-execute
-overwrite_original_in_place
-charset
FILENAME=UTF8
-m
-EXIF:ImageDescription=
-EXIF:Software=
-EXIF:Copyright=
-EXIF:Artist=
-EXIF:UserComment=
-IFD0:ModifyDate=
-IFD0:Rating=
-IFD0:ImageDescription=
-IFD0:Software=
-IFD0:Copyright=
-IFD0:Artist=
-IFD0:XPTitle=
-IFD0:XPComment=
-IFD0:XPAuthor=
-IFD0:XPKeywords=
-IFD0:XPSubject=
-GPS:all=
-tagsfromfile
c:\temp\image.xmp
-@
[color=red]v:\exiftool\arg_files\xmp2exif.args[/color] <===
-@
v:\exiftool\arg_files\xmp2gps.args
c:\temp\image.nef
-execute9999
The call to xmp2exif.args (indicated by <===) maps XMP back into EXIF.
Can I override this args file somehow from the args file?
Hi Mario,
Quote from: Mac2 on September 30, 2020, 12:24:14 PM
The call to xmp2exif.args (indicated by <===) maps XMP back into EXIF.
Can I override this args file somehow from the args file?
It would be safest to write only XMP to RAW files and avoid xmp2exif.args completely. But you can override any of the tags copied by this argfile by adding your own arguments afterward. eg) I suggested adding
--make --model to the command
Quote from: spiff on September 30, 2020, 10:29:57 AM
Thank you very much. The file you provided is readable with Silkypix. I hope this helps Mario. Because there is a discussion at dpreview, i ask you if you suggest to write XMP / exif datas in a RAW file (but not maker notes), or do you recommend write the XMP sidecar only and let a RAW file untached? Your answer wil help me configuring imatch proper. Imatch standard configuration is writing Exif / XMP in RAW, but it is possible to change the settings. Also Mario warned that RAW and sidecar going out of sync doing so. Thank you.
So it seems that SilkyPix requires the trailing spaces on Make and Model. :(
I can't make a recommendation on which technique is best. It depends entirely on the software you are using. Do whatever is most compatible with your software. There are advantages and disadvantages to either technique.
- Phil
But I want XMP EXIF mapped back to EXIF when IMatch is used with the "MWG compliance mode". IMatch does this for many years.
This includes data like user comment / description but but also date created / subject created.
And GPS data of course, which is mapped from XMP back to GPS in a similar fashion.
This was the first time in all these years, and with millions of updated images, that one buggy software product caused issues. I hope SilkyPix gets this fixed soon.
Except for the DxO version which could handle only one little Endian. But that was fixed quickly.
I will add the --make and --model to my output arguments.
My software discourages from changing these tags anyway, so this should affect only a few users who do this (e.g. in scanned images or to trick their RAW processor).
I'd really with camera vendors would abandon the crappy EXIF and GPS data and just store all the info in a proper XMP record.
Modern cameras are basically smart phones with much better optics and there should be enough computing power to write XMP instead of legacy EXIF/GPS data...
I understand. This makes sense.
- Phil
I contacted Silkypix and wrote them your finding. Also i sent links to the discussion here and at dpreview. Thanks again for your help.
Regards
Björn
Quote from: Phil Harvey on September 30, 2020, 12:34:28 PM
So it seems that SilkyPix requires the trailing spaces on Make and Model. :(
- Phil
Which leaves me wondering that Silkypix struggles only with ORF and not with other RAW file types imatch altered the same way. That´s the way it is now.
Quote from: Mac2 on September 30, 2020, 01:33:18 PM
But I want XMP EXIF mapped back to EXIF when IMatch is used with the "MWG compliance mode".
But you should actively point it out in future that is also possible just writing in sidecars and let RAW untached (=avoid xmp2exif.args completely?) as an alternative. Giving the decission how to proceed in your users hands. Make it clear in a dialog after the very first start of imatch please. They should have the ability to decide between preset "MWG compliance mode" or preset "XMP sidecar only". Should be discussed in imatch forum better.
This was never an issue before this SilkyPix bug/incident.
The vast majority of users is very comfortable and safe with keeping the EXIF/GPS/legacy IPTC embedded in the RAW file in sync with the corresponding copies of this data in the XMP sidecar file (or the embedded XMP record).
IMatch is around since 1998.
SilkyPix is, to my knowledge, the only software which seems to have the bug that they consider spaces or other blanks in the EXIF camera vendor name as relevant.
It appears that all other software on the market (IMatch users, dpreview.com users and I have tested all relevant applications out there) does the right thing and just checks for "OLYMPUS" instead of of OLYMPUS followed by a varying number of blanks. I wonder what happens with SilkyPix when Olympus updates their firmware and adds more (or none) blanks to the vendor name? Probably they do so, in other camera models or firmware versions.
Will SilkyPix also break and just refuse to load the files...?
How IMatch deals and implements, together with ExifTool, the XMP standard, the IPTC standard, the rules and recommendations of the Metadata Working Group, established industry and workflow standards is documented in the corresponding sections in the Metadata Settings (https://www.photools.com/help/imatch/#rmh_config_metadata2.htm) and related topics.
The standard settings IMatch uses have been carefully chosen and developed with many users from over 70 countries over the past decade.
These settings just work for the vast majority of use cases, home and commercial workflows. They also ensure the best possible metadata quality, conformity and compatibility.
The fact that a faulty software like SilkyPix fails to properly handle files under some conditions does not change this.
If SilkyPix just fixes their broken maker check,. all your files will load immediately.
As they do in Windows WIC, Photoshop, ACR, Lighjtroom, Capture One, LibRaw, Windows Photo, and any other software.
I just came back to this and implemented the suggestion by Phil.
IMatch now explicitly excludes make/model from being mapped back from XMP to EXIF via this sequence in the args file:
...
v:\exiftool\arg_files\xmp2exif.args
--make
--model
...
The output produced by this now retains the original additional blanks and spaces in the make and model name.
I've then installed an evaluation version of SilkyPix for testing.
SilkyPix has no problems opening files with metadata updated by IMatch. But it also has no problems with the original file modified by IMatch (with the blanks removed).
I assume that the version of SilkyPix I've used (10.0.8.0) already contains the bug fix and they no longer rely on a specific number of blanks in the model name.