Sony ARW files all black

[srueg]

Hi,

I'm using exiftool as part of my photo-editing workflow, what a great tool, thanks.


I now had the case where after editing exif/IPTC data, all my Sony ARW RAW files show up as all-black or otherwise corrupt. While JPG files in the same folder are all fine.

I ran various variations of the following command:

Code Select Expand

exiftool -overwrite_original_in_place \
  -XMP-dc:Subject+="peru2022" /path/to/folder


Code Select Expand

Image::ExifTool 12.70
I've uploaded both the original and corrupted file here: https://we.tl/t-rs7x7bHOl8

I've also noticed the file size of the original is about 2x of the corrupt one.

This has never been a problem so far and I'm also not able to reproduce it.

Thanks for your help.

[StarGeek]

I can see the corrupt one doesn't display in Adobe Bridge.

I think the problem appeared between ver 12.41 and 12.42.  Running your command on 12.41 displays correctly in Bridge and only a few bytes difference in file size.  Version 12.42 doesn't display in Bridge and is the smaller file size.

Edit:  I don't see anything off hand in the version notes that jumps out as a change to Sony ARW files.

Also, it may not matter, but it looks like the firmware version in your camera is 1.00 and it looks like the newest version is 2.01, though it says you have to update to ver 1.05 first.

[Phil Harvey]

This could be serious.  Do not edit any more ARW images until I have figured this out. It looks like it may affect only "Lossless Compressed RAW 2" images.

I find the difference between versions 12.44 and 12.45.  Here is the relevant change:

Sept. 16, 2022 - Version 12.45
  - Patched to avoid duplicating raw data when writing Sony ARW images where the
    raw data is double-referenced as both strips and tiles (affects ARW images
    from some newer models like the ILCE-1 when SonyRawFileType is "Lossless
    Compressed RAW 2")

- Phil

[Phil Harvey]

I've just released version 12.75 which should fix this issue.

Having the original sample was necessary to figure this out.  Thanks for providing this, and for reporting the issue.  The issue is that 2 sets of raw data is stored in the same IFD in this file.  I have no idea why Sony does this.  In other models I have only seen where 2 sets of pointers exist in the RAW IFD but point to the same data, which was causing the raw data to be duplicated and was fixed in the 12.45 patch.  But in your file there really 2 sets of different raw data in this IFD, one of which would be lost when rewriting with 12.45, but this is fixed in 12.75.

- Phil

[StarGeek]

I find the difference between versions 12.54 and 12.55.  Here is the relevant change:

I assume you mean 12.44 and 12.45?

Double-checking, both 12.41 and 12.42 worked fine.  Bridge didn't update unless the FileModifyDate changed and Windows was also slow to update the file size.  But 12.50 does have the problem.  I don't have any versions between 12.42 and 12.50.

[Phil Harvey]

Yes.  The difference was between 12.44 and 12.45 -- I've updated my previous post.

- Phil

[srueg]

Wow, thank you both so much for solving this so quickly! Much appreciated

Re: firmware; I've updated to 2.0 a while ago, these pictures were taken on the old firmware in 2022.

Is there any chance we can fix the corrupted ARW files? It would save me a ton of work. As I understood it, the corrupted files should still contain one of the two sets of RAW data in IFD?

Thanks again!

[srueg]

Another update: I've just realized that I ran some commands on version 12.3 on these ARW files a while ago, to adjust the time. So maybe the bug fixed in 12.45 caused the two sets of RAW in IFD in the first place?

[Phil Harvey]

This may be related to the problem.  I'll run some more tests.

I think it may be possible to fix the corrupted files.  I'll test that too.

I'm busy most of the day today, so this may have to wait until tomorrow.

- Phil

[Phil Harvey]

I was able to find some time today to look into this in more detail.

Version 12.75 fixed the corruption issue at the expense of duplicating the raw data (ie. back to the 12.44 behaviour).

I've just released 12.76 which avoids duplicating the raw data of other images.

You are right that the problem occurs only with ARW images that were edited with 12.44 or earlier, then subsequently edited with 12.45 through 12.74.

I will prepare a version of ExifTool that will fix these corrupted images for you.  What distribution package do you use?  (Windows .exe?)  This version will also remove the duplicated raw data from ARW files edited using 12.44 and earlier.

- Phil

[srueg]

Thank you so much!

I'm using exiftool from Homebrew installed on macOS. More than happy to clone the repository and run it locally if you point me to a branch/commit

Cheers,
Simon

[Phil Harvey]

Hi Simon,

I've gone ahead and produced both .tar.gz and .zip releases of the "fix ARW" version:

Mac/Linux: Image-ExifTool-12.76_fix_arw.tar.gz

Windows: exiftool-12.76_fix_arw.zip

On Mac, download the .tar.gz version and unzip it, then run exiftool directly from the distribution folder with this command to fix any ARW image with duplicated or corrupted raw data:

Image-ExifTool-12.76_fix_arw/exiftool -forcewrite=arw -ext arw -r DIR

This command will leave the files unchanged unless duplicate and/or corrupted raw data is found.  So it may also be used to remove duplicate raw data from files written by 12.44 and earlier.

- Phil

[srueg]

Awesome, it worked like a charm   

Again, thank you so much for all your work!

Cheers,
Simon