Hello,
I used my iphone to shoot some photos in RAW mode with an App called "procamera". The photos is in DNG format. As I understand it, there supposed to be a low resolution preview embedded in the DNGs. However, I can't use ExifTool to extract it.
Here is an sample file:
https://1drv.ms/u/s!ApWbHSC9NoYSgl4Iigk0YDPJgl8Y
I put the DNGs in D:/test , and then use
exiftool -b -PreviewImage -w jpg -ext dng d:/test
and got nothing.
However, I tried to use another tool
http://www.fsoft.it/ERawP/
I got a small 852*640 resoluton, 212KB size file.
So What Am I Missing Here? Thank you!
I just had a look at your file and while it does contain a 852x640 image (apart from the full image data), this is not a "preview" image, at least not in the sense as exiftool sees it. So you won't be able to extract it with e.g. exiftool -b -preview:all -w _prv.jpg -ext dng d:/test.
But since the image data is there, there is probably a way to extract (only) this image. @Phil, (how) can this be done?
Note: when you edit and save the image in e.g. Adobe Camera Raw, it will add previews that are recognisable by exiftool (plus, the preview will be full-sized too – assuming you have the ACR dng options set to generate full sized previews).
Thanks for the sample. This JPEG preview is stored in an unusual place for a DNG, and ExifTool doesn't extract it. I will look into this, but unless I post back here you may assume that the next version of ExifTool (10.38) will extract this preview.
- Phil
Hayo, Phil, Thanks for the answers!
Here is my situation:
I want to shoot raw with my iphone. And since the quality of the embedded image is too low, I have to shoot raw+jpg. However, keeping 2 files for one image is a nightmare to me. So my goal is to batch replacing(using ExifTool) the embedded preview images with the higher quality jpgs, and keeping just 1 file for each photo.
From my experiment, I also find that by using "Adobe DNG Converter", I can batch convert my iphone DNGs to new DNGs that contain recognizable previews by exiftool. (And then, I can do the exiftool "-previewImage<=d:/test/zz.jpg" d:/test/zz.DNG)
If next version of ExifTool can handle the iphone DNGs directly, maybe I can skip the above conversion step?
I am not sure if it's necessary for you, but I just put more samples here, in case it's helpful for your development of the next version.
They are iPhone app "procamera" shooting(raw+jpg) in landscape/portrait mode. And another app "procam" shooting(raw+jpg) in landscape/portrait mode.
https://1drv.ms/u/s!ApWbHSC9NoYSgl8AyWBjuCBo5wT0
Merry christmas.
Quote from: namer900 on December 24, 2016, 04:23:58 PM
If next version of ExifTool can handle the iphone DNGs directly, maybe I can skip the above conversion step?
If it also supports writing the preview, I think you might!
@Phil, I did some experimenting on my own files and found my DNGs actually contain 4 previews.
JpgFromRaw, 2 times
PreviewImage, and
ThumbnailTIFF. So far I have not found a way to specify which of the two preview images to extract (one is in the makernotes). Is there a way to do this? I don't mind if I have to write the code for this myself, but I haven't yet found out how you actually extract the images ;)
Hi Hayo,
I'll try to make the DNG preview writable.
If there are 2 previews in the same group (presumably the Composite group), the commands to extract them are:
1. exiftool -previewimage -b in.dng > out1.jpg
2. exiftool -copy1:previewimage -b in.dng > out2.jpg
- Phil
Hi Phil,
Excellent, I reckon this
copyx: trick also work on other duplicate tags in the same group? Am I correct to further assume that when using the perl library I should use the
(x) suffix (like any other duplicate)?
Also, I did some more digging/testing and found the following interesting things:
My (edited) DNGs actually contain multiple
Reduced-resolution image entries, all with different sizes:
[EXIF:IFD0] Subfile Type : Reduced-resolution image
[EXIF:IFD0] Image Width : 256
[EXIF:IFD0] Image Height : 171
[EXIF:IFD0] Compression : Uncompressed
[EXIF:IFD0] Strip Offsets : 256764
[EXIF:SubIFD] Subfile Type : Full-resolution Image
[EXIF:SubIFD] Image Width : 3900
[EXIF:SubIFD] Image Height : 2616
[EXIF:SubIFD] Compression : JPEG
[EXIF:SubIFD1] Subfile Type : Reduced-resolution image
[EXIF:SubIFD1] Image Width : 1024
[EXIF:SubIFD1] Image Height : 685
[EXIF:SubIFD1] Compression : JPEG
[EXIF:SubIFD1] Preview Image Start : 388092
[EXIF:SubIFD1] Preview Image Length : 26059
[EXIF:SubIFD2] Subfile Type : Reduced-resolution image
[EXIF:SubIFD2] Image Width : 3872
[EXIF:SubIFD2] Image Height : 2592
[EXIF:SubIFD2] Compression : JPEG
[EXIF:SubIFD3] Subfile Type : Reduced-resolution image
[EXIF:SubIFD3] Image Width : 2048
[EXIF:SubIFD3] Image Height : 1371
[EXIF:SubIFD3] Compression : Lossy JPEG
[EXIF:SubIFD4] Subfile Type : Reduced-resolution image
[EXIF:SubIFD4] Image Width : 256
[EXIF:SubIFD4] Image Height : 171
[EXIF:SubIFD4] Compression : Lossy JPEG
[MakerNotes:PreviewIFD] Compression : JPEG (old-style)
[MakerNotes:PreviewIFD] Preview Image Start : 21472
[MakerNotes:PreviewIFD] Preview Image Length : 118558
It would be nice if we could extract any of these. Some notes:
- EXIF:SubIFD2 actually is the JPGFromRaw
- It looks like EXIF:SubIFD1 is the PreviewImage though for this one the sizes do not match, the resulting JPG is 570x375 not 1024x685 as shown?
- From the size, either EXIF:IFD0 or EXIF:SubIFD4 could be the ThumbnailTIFF
- EXIF:SubIFD looks like it is the raw data, but why is the compression for it listed as JPG?
- The preview in the makernotes is the copy1:PreviewImage (the filesize matches)
Furthermore, I noticed that while the sizes of most of the resulting files when extracting the previews are exactly as mentioned, the "normal" ImagePreview isn't:
[Composite] Jpg From Raw : (Binary data 500886 bytes, use -b option to extract)
[Composite] Preview Image : (Binary data 118558 bytes, use -b option to extract)
[Composite] Preview Image : (Binary data 26059 bytes, use -b option to extract)
[Composite] Thumbnail TIFF : (Binary data 131544 bytes, use -b option to extract)JpgFromRaw,
copy1:PreviewImage, and
ThumbnailTIFF are respectively 500886, 118558, and 131544 bytes in size.
PreviewImage itself, however, is not 26059, but 144617 bytes...
Quote from: Hayo Baan on December 26, 2016, 05:54:42 AM
I reckon this copyx: trick also work on other duplicate tags in the same group?
Yes.
QuoteAm I correct to further assume that when using the perl library I should use the (x) suffix (like any other duplicate)?
There are other ways to do this via the library -- you can use GetInfo to get all duplicates and sort through them yourself.
QuoteAlso, I did some more digging/testing and found the following interesting things:
My (edited) DNGs actually contain multiple Reduced-resolution image entries, all with different sizes:
Could you email me this sample so I can take a look? (philharvey66 at gmail.com)
- Phil
Sample files sent. Hope they enable you to add easy image extracting functionality to exiftool :)
Quote from: Hayo Baan on December 27, 2016, 03:02:20 AM
Sample files sent. Hope they enable you to add easy image extracting functionality to exiftool :)
Thanks. I don't think it will be easy though... I'm thinking that the algorithm could handle this better and may need revising.
- Phil
Hi Hayo,
As far as I can tell, ExifTool is extracting all previews properly from the DNG samples you sent. Your confusion about the preview image size is that you extracted both previews into a single file.
So I think only problem is that the preview isn't extracted from the iPhone DNG. I will see about fixing this.
- Phil
Quote from: Phil Harvey on December 28, 2016, 09:57:18 PM
As far as I can tell, ExifTool is extracting all previews properly from the DNG samples you sent. Your confusion about the preview image size is that you extracted both previews into a single file.
Ah, you are absolutely right! I have
-a enabled by default so it was indeed adding both to the same file. doh ::)
Quote from: Phil Harvey on December 28, 2016, 09:57:18 PM
So I think only problem is that the preview isn't extracted from the iPhone DNG. I will see about fixing this.
Yes! Hopefully this also enables a way to extract
any embedded image file :)
(and perhaps, removing as well ;))
Well, I've worked about 10 hours on this so far. I have made some substantial changes to ExifTool that still need testing. One big change is that the the thumbnail/preview/jpgfromraw images have been moved from the Composite group to the same groups as the tags from which they are derived (ie. EXIF:IFD1 for the ThumbnailImage of a JPEG). This may lead to a bit of confusion since they are in the Composite table of the documentation, but will no longer be in that group. However, the benefit is that now these images may be addressed by specific location when reading and writing. It was the writing that was the trickiest, and I still need to test this thoroughly. If all goes well then ExifTool 10.38 should contain this update.
- Phil
Wow, that was a big change! Sounds awesome though 8) Do you need some help testing?
Have you also been able to have a look at completely removing a complete exif:ifd?
I'll let you know about testing. Maybe I'll post a pre-release when it is closer to release.
I don't want to add the ability to remove an entire IFD right now -- I have to think about this more because it would be very easy to delete something important this way.
- Phil
@Hayo: I have uploaded a 10.38 pre-release here (https://exiftool.org/Image-ExifTool-10.38p.tar.gz). So far the testing has been going well, but I would be happy if you want to try it out.
- Phil
(note that the ExifTool web server is scheduled for some system updates today, which may result in some down time)
Quote from: Phil Harvey on January 02, 2017, 08:13:07 AM
@Hayo: I have uploaded a 10.38 pre-release here (https://exiftool.org/Image-ExifTool-10.38p.tar.gz). So far the testing has been going well, but I would be happy if you want to try it out.
Nice, the previews are now reported inside their own group and can be extracted from there 8)
What doesn't seem to work yet though is the ability to see/extract the "other" embedded images. I still only get the "normal" previews (from the makernotes and subifd1, as well as the jpg from raw in subifd2), but not the additional images in SubIFD3 and SubIFD4 :(
P.S. I noticed you changed the composite tag Lens into Lens Spec?
Hi Hayo,
Thanks for testing this.
Those things in SubIFD3 and SubIFD4 are not JPEG images. I don't know what they are. They are structured like a JPEG, but don't contain a viewable image.
Yes, I changed the LensSpec description from "Lens" to "Lens Spec". Does that affect you at all?
- Phil
Quote from: Phil Harvey on January 02, 2017, 09:42:02 AM
Thanks for testing this.
You're welcome!
Quote from: Phil Harvey on January 02, 2017, 09:42:02 AM
Those things in SubIFD3 and SubIFD4 are not JPEG images. I don't know what they are. They are structured like a JPEG, but don't contain a viewable image.
Those are added when you enable "add fast load data" when creating DNGs. It's a real pity they don't contain any viewable data :(
(but is it extractable in case there is a way to turn that data into something useful?)
As I mentioned, I actually would like to get rid of them; they take up quite some space and I don't think they are actually really useful other than perhaps for Lightroom (?) (which I don't really use anyway: I like Camera Raw much better).
I hope removing these additional SubIFDs is going to be possible (happy to help testing code for you for this ;))
Quote from: Phil Harvey on January 02, 2017, 09:42:02 AM
Yes, I changed the LensSpec description from "Lens" to "Lens Spec". Does that affect you at all?
Ah, right, I see, so you didn't change the name of the tag, just the description. In that case it doesn't affect me at all (besides, the new description is actually clearer and closer to the name of the tag).
Cheers,
Hayo
Quote from: Hayo Baan on January 02, 2017, 10:37:17 AM
Those are added when you enable "add fast load data" when creating DNGs. It's a real pity they don't contain any viewable data :(
(but is it extractable in case there is a way to turn that data into something useful?)
I'll have to look into this.
QuoteAs I mentioned, I actually would like to get rid of them; they take up quite some space and I don't think they are actually really useful other than perhaps for Lightroom (?) (which I don't really use anyway: I like Camera Raw much better).
I hope removing these additional SubIFDs is going to be possible (happy to help testing code for you for this ;))
I'll put this on the to-do list, but again it will take some thought because it is potentially dangerous.
- Phil
I have found a potential problem in the pre-release with identification of some embedded images. The web server is down for maintenance at the moment, so I'll have to update the uploaded version later. I'll post again here when the updated version is available.
- Phil
I have updated the 10.38 pre-release with a fixed version. The change only affects the NRW file type.
- Phil
Updated, but as I don't have NRW files this didn't affect me :)
I updated to the new 10.38 version, and done some testing. The PreviewImage extracting works perfectly. However, I have a question regarding writing/replacing the PreviewImage.
In windows 10, its "File Explorer" can display the original iPhone DNG's thumbnail, and its "Photos app" can open/view the DNG. But after I replaced the PreviewImage of the DNG with ExifTool, "File Explorer" can't display the new DNG's thumbnail anymore, and the "Photos App" can't open/view the new DNG(error message-"It looks like we don't support this file format.")
My test files:
https://1drv.ms/u/s!ApWbHSC9NoYSgns0p2UsyEk2p1x7
Thanks
I can't test in Windows, but the DNG you uploaded (zz.DNG) displays fine in Apple Preview and Adobe Lightroom 5.
I'll look into trying this in Windows on Monday.
- Phil
I can't test this on Windows either (and on a Mac everything seems to work well), but one thing I can think of is the fact the JPG you used as replacement preview image contained metadata. Perhaps if you remove all metatdata before using it as replacement it will work?
You can remove all metadata using exiftool -all= FILE
Note: as on a Mac e.g. the Preview application renders the original raw data in the file, I replaced the preview with one that was completely different. That way it was clearer to me when the preview image got used and when the raw data. But the conclusion was the same: it works fine under Mac OS. I tried Finder (=Explorer on Windows), Adobe Camera Raw (and thus Lightroom), Bridge, Preview, and PhotoMechanic. The results where as expected: thumbnail in Finder was my replacement preview, PhotoMechanic always used my replacement preview, and everywhere else the original raw is used.
Thanks for the suggestion! I removed the metadata of the JPG, but it's still not working.
My files:
https://1drv.ms/u/s!ApWbHSC9NoYSgnzqdLHCgm9zlRYO
To be sure, the new DNG is working everywhere else I tried, just not working in windows 10 native environment (by that I mean no other third-party raw codecs installed, like "adobe dng codec", "FastPictureViewer codec" etc.). If I placed the original DNG on windows 10 desktop, its thumbnail will automatically displayed. If I double click the file, it will be opened in "Microsoft Photos" app. The new DNG can't be opened and its thumbnail won't shown.
Another thought: since you replaced the original preview with a much higher resolution version, the recorded width and height of the embedded image are no longer correct. So perhaps it is only a matter of correcting those.
In the sample case this would mean running exiftool -EXIF:IFD0:ImageWidth=4032 -EXIF:IFD0:ImageHeight=3024 zz.DNG.
Hope this works!
OK. I tried this on my Windows 10 and it won't event display a preview of the original DNG (even after I renamed it back to .DNG again). And I don't have Windows Photo and don't want to install it on my clean Windows 10 system, so it looks like I won't be able to reproduce this problem.
But I have an idea:
Try writing the original preview back to the modified DNG to see if this works. This will tell you if it is the preview itself that is a problem, or if it is due to a change in the structure of the DNG.
- Phil
Thanks hayo, phil for the ideas!
After I further process the modified DNG(which embedded with a no metadata jpg) with exiftool -EXIF:IFD0:ImageWidth=4032 -EXIF:IFD0:ImageHeight=3024 zz.DNG
the resulting DNG still not working. Interestingly, if I try to open it with "Photos App", the error message is changed from "It looks like we don't support this file format."(for the modified DNG) to "We can't open this file."(for the further processed DNG).
"Writing the original preview back to the modified DNG" will result in a working DNG again, just like the original DNG(thumbnail displaying in win10, and can be opened in "Photos App").
Weird, so its not the structural change (due to writing with exiftool) that is the cause; writing back the original JPG using exiftool fixes it (which probably introduces those same changes). :o
Yup. So it is something that Windows doesn't like about the embedded previewimage itself. Hayo had good ideas about the metadata and the image size being different. It does seem that the size has something to do with the problem since the message changes when you change the ImageWidth/Height tags. The last test I can think of is to see if you can get Windows to recognize the DNG if you write a different PreviewImage with the same dimensions as the original preview.
- Phil
Done the tests:
If I take some random JPG file, resize it to 852*640 resoluton, and then embed it to the original DNG, the resulting DNG will work!(thumbnail displaying in win10, and can be opened in "Photos App"). Embedding other resolution JPGs will not work. I've tried 851*640, 426*320, 640*852, 5472*3648 JPGs.
That is very interesting. I have no idea why the Windows DNG conversion would be so dependent on the size of the preview.
- Phil
Wait. There looks to also be a RowsPerStrip tag in the IFD0 that needs changing to the same as the new preview height. Setting this as well as the IFD0 ImageWidth/Height may fix things.
- Phil
(1) JPG metadata removed.
(2) Embed JPG into original DNG
(3) changing IFD0 ImageWidth/Height
exiftool -EXIF:IFD0:ImageWidth=4032 -EXIF:IFD0:ImageHeight=3024 d:/test/zz.DNG
(4) changing IFD0 RowsPerStrip
exiftool -EXIF:IFD0:RowsPerStrip=3024 d:/test/zz.DNG
After above steps, the final DNG
https://1drv.ms/u/s!ApWbHSC9NoYSgn2xTEWRzRq01Lkp
still not working. (error message "We can't open this file.") ???
OK. Well thanks for trying. That was my last idea. :(
- Phil