Hi,
I'm using exiftool as part of my photo-editing workflow, what a great tool, thanks.
I now had the case where after editing exif/IPTC data, all my Sony ARW RAW files show up as all-black or otherwise corrupt. While JPG files in the same folder are all fine.
I ran various variations of the following command:
exiftool -overwrite_original_in_place \
-XMP-dc:Subject+="peru2022" /path/to/folder
Image::ExifTool 12.70
I've uploaded both the original and corrupted file here: https://we.tl/t-rs7x7bHOl8
I've also noticed the file size of the original is about 2x of the corrupt one.
This has never been a problem so far and I'm also not able to reproduce it.
Thanks for your help.
I can see the corrupt one doesn't display in Adobe Bridge.
I think the problem appeared between ver 12.41 and 12.42. Running your command on 12.41 displays correctly in Bridge and only a few bytes difference in file size. Version 12.42 doesn't display in Bridge and is the smaller file size.
Edit: I don't see anything off hand in the version notes (https://exiftool.org/ancient_history.html#v12.42) that jumps out as a change to Sony ARW files.
Also, it may not matter, but it looks like the firmware version in your camera is 1.00 and it looks like the newest version is 2.01 (https://www.sony.com/electronics/support/software/00288929), though it says you have to update to ver 1.05 first.
This could be serious. Do not edit any more ARW images until I have figured this out. It looks like it may affect only "Lossless Compressed RAW 2" images.
I find the difference between versions 12.44 and 12.45. Here is the relevant change:
Sept. 16, 2022 - Version 12.45
- Patched to avoid duplicating raw data when writing Sony ARW images where the
raw data is double-referenced as both strips and tiles (affects ARW images
from some newer models like the ILCE-1 when SonyRawFileType is "Lossless
Compressed RAW 2")
- Phil
I've just released version 12.75 which should fix this issue.
Having the original sample was necessary to figure this out. Thanks for providing this, and for reporting the issue. The issue is that 2 sets of raw data is stored in the same IFD in this file. I have no idea why Sony does this. In other models I have only seen where 2 sets of pointers exist in the RAW IFD but point to the same data, which was causing the raw data to be duplicated and was fixed in the 12.45 patch. But in your file there really 2 sets of different raw data in this IFD, one of which would be lost when rewriting with 12.45, but this is fixed in 12.75.
- Phil
Quote from: Phil Harvey on January 30, 2024, 07:04:07 PMI find the difference between versions 12.54 and 12.55. Here is the relevant change:
I assume you mean 12.44 and 12.45?
Double-checking, both 12.41 and 12.42 worked fine. Bridge didn't update unless the
FileModifyDate changed and Windows was also slow to update the file size. But 12.50 does have the problem. I don't have any versions between 12.42 and 12.50.
Yes. The difference was between 12.44 and 12.45 -- I've updated my previous post.
- Phil
Wow, thank you both so much for solving this so quickly! Much appreciated :)
Re: firmware; I've updated to 2.0 a while ago, these pictures were taken on the old firmware in 2022.
Is there any chance we can fix the corrupted ARW files? It would save me a ton of work. As I understood it, the corrupted files should still contain one of the two sets of RAW data in IFD?
Thanks again!
Another update: I've just realized that I ran some commands on version 12.3 on these ARW files a while ago, to adjust the time. So maybe the bug fixed in 12.45 caused the two sets of RAW in IFD in the first place?
This may be related to the problem. I'll run some more tests.
I think it may be possible to fix the corrupted files. I'll test that too.
I'm busy most of the day today, so this may have to wait until tomorrow.
- Phil
I was able to find some time today to look into this in more detail.
Version 12.75 fixed the corruption issue at the expense of duplicating the raw data (ie. back to the 12.44 behaviour).
I've just released 12.76 which avoids duplicating the raw data of other images.
You are right that the problem occurs only with ARW images that were edited with 12.44 or earlier, then subsequently edited with 12.45 through 12.74.
I will prepare a version of ExifTool that will fix these corrupted images for you. What distribution package do you use? (Windows .exe?) This version will also remove the duplicated raw data from ARW files edited using 12.44 and earlier.
- Phil
Thank you so much!
I'm using exiftool from Homebrew installed on macOS. More than happy to clone the repository and run it locally if you point me to a branch/commit :)
Cheers,
Simon
Hi Simon,
I've gone ahead and produced both .tar.gz and .zip releases of the "fix ARW" version:
Mac/Linux: Image-ExifTool-12.76_fix_arw.tar.gz (https://exiftool.org/Image-ExifTool-12.76_fix_arw.tar.gz)
Windows: exiftool-12.76_fix_arw.zip (https://exiftool.org/exiftool-12.76_fix_arw.zip)
On Mac, download the .tar.gz version and unzip it, then run exiftool directly from the distribution folder with this command to fix any ARW image with duplicated or corrupted raw data:
Image-ExifTool-12.76_fix_arw/exiftool -forcewrite=arw -ext arw -r DIR
This command will leave the files unchanged unless duplicate and/or corrupted raw data is found. So it may also be used to remove duplicate raw data from files written by 12.44 and earlier.
- Phil
Awesome, it worked like a charm :D
Again, thank you so much for all your work!
Cheers,
Simon