Hello,
I would like to extract raw exif data from image to a file (to save space) and later read it with an exiftool. Is this possible?
Thanks.
Hi Linas,
This is possible in many ways. Read here (https://exiftool.org/metafiles.html) for some ways to create a metadata sidecar file. Specifically, I'm thinking you may want to use an EXIF sidecar file (https://exiftool.org/metafiles.html#exif).
- Phil
Nice, thanks.
p.s. Just to clarify, do all of these commands output the same data?
exiftool -exif -b SRCFILE > out.exif
exiftool -tagsfromfile SRCFILE -exif out.exif
exiftool -o out.exif -exif SRCFILE
Hm, but if I do extract EXIF sidecar file. Do I miss all other additional formats? (e.g. XMP, IPTC)
I would like to extract any raw metadata except an actual image. The purpose of this is to have small footprints of images, so I could easily re-read these when new ExifTool comes out. I have over 20 million images.
Quote from: Linas Pašviestis on February 18, 2021, 12:07:58 PM
Hm, but if I do extract EXIF sidecar file. Do I miss all other additional formats? (e.g. XMP, IPTC)
Yes. XMP can be copied to a XMP sidecar but IPTC would be left out. There is an XMP equivalent for most IPTC data.
QuoteI would like to extract any raw metadata except an actual image. The purpose of this is to have small footprints of images, so I could easily re-read these when new ExifTool comes out.
Take a look at exiftool's own MIE file in the link Phil gave above. You could copy EXIF, IPTC, and XMP, as well as the ICC_Profile and MakerNotes.
QuoteI have over 20 million images.
Ok, I thought I had a significant collection of images I've collected from the net and photos I've taken and family photos I've scanned, somewhere between 1-2 million, but you've taken it to a whole other level. Consider me impressed.
Quote from: StarGeek on February 18, 2021, 01:05:14 PM
Take a look at exiftool's own MIE file in the link Phil gave above. You could copy EXIF, IPTC, and XMP, as well as the ICC_Profile and MakerNotes.
So if I understand it correctly the following comand will extract all available metadata from an image into a file?
exiftool -tagsfromfile a.jpg -all:all -icc_profile a.mieQuote from: StarGeek on February 18, 2021, 01:05:14 PM
Ok, I thought I had a significant collection of images I've collected from the net and photos I've taken and family photos I've scanned, somewhere between 1-2 million, but you've taken it to a whole other level. Consider me impressed.
I do run a website https://explorecams.com (https://explorecams.com) , feel free to donate some of the original photos if you like it. :-)
That won't copy the MakerNotes or any tag that exiftool doesn't know how to write. Copying the groups as a block should save everything, even stuff that exiftool is unable to write. And you'll probably want to do it in batch, so try this (I can't remember if copying EXIF will also copy MakerNotes or not)
exiftool -tagsfromfile -o %d%f.mie -EXIF -XMP -IPTC -MakerNotes -icc_profile /path/to/files/
This will create the sidecars in the same directory as the files. If you want to use a different directory, change the %d to the correct path.
Quote from: StarGeek on February 18, 2021, 07:28:30 PM
That won't copy the MakerNotes or any tag that exiftool doesn't know how to write. Copying the groups as a block should save everything, even stuff that exiftool is unable to write. And you'll probably want to do it in batch, so try this (I can't remember if copying EXIF will also copy MakerNotes or not)
exiftool -tagsfromfile -o %d%f.mie -EXIF -XMP -IPTC -MakerNotes -icc_profile /path/to/files/
This will create the sidecars in the same directory as the files. If you want to use a different directory, change the %d to the correct path.
I thought -all:all is the replacement for -exif -xmp -iptc and -makernotes.
Quote from: Linas Pašviestis on February 18, 2021, 09:07:36 AM
Just to clarify, do all of these commands output the same data?
exiftool -exif -b SRCFILE > out.exif
exiftool -tagsfromfile SRCFILE -exif out.exif
exiftool -o out.exif -exif SRCFILE
Yes, but they differ in their behaviour if the file already exists. The first one overwrites out.exif if it already exists, and the other two fail if out.exif exists.
- Phil
Quote from: Phil Harvey on February 19, 2021, 07:55:40 AM
Quote from: Linas Pašviestis on February 18, 2021, 09:07:36 AM
Just to clarify, do all of these commands output the same data?
exiftool -exif -b SRCFILE > out.exif
exiftool -tagsfromfile SRCFILE -exif out.exif
exiftool -o out.exif -exif SRCFILE
Yes, but they differ in their behaviour if the file already exists. The first one overwrites out.exif if it already exists, and the other two fail if out.exif exists.
- Phil
Phil, would the following command would be the ultimate one to extract every metadata bit from the image? Or I will miss some extra meta formats e.g. MakerNotes, XMP, IPTC?
exiftool -tagsfromfile a.jpg -all:all -icc_profile a.mie
That command will get most of the tags that ExifTool can write.
- Phil
The key point being "can write". MakerNotes for example, cannot be copied individually. It must be copied as a block (see last paragraph of FAQ #8 (https://exiftool.org/faq.html#Q8)).
Another example would be any tag that requires a config file to write, such as the Pix4d tags (https://github.com/exiftool/exiftool/blob/master/config_files/pix4d.config). Without the config file, those tags would not be copied.
But that command should copy the MakerNotes as a block for most common makes.
- Phil
Quote from: Phil Harvey on February 19, 2021, 09:19:55 AM
That command will get most of the tags that ExifTool can write.
Okay, so lets say my goal is to store a raw metadata file without the actual image, so I could easily re-run / update the output when I upgrade exiftool. The most important bit of information is identification of Lens Id to me and other inconsistances in EXIF (if these even happen between versions). Should this command 100% completely cover my goal?
Sorry, for asking this again and again, as it would take a bit of time to extract things from 20M+ images and I want to be 100% certain. :-)
I can't answer this 100%. It depends on the models of cameras used to generate the images, and the softwares used to process them afterwards. Some cameras may not be supported, and some software may make a huge mess of the metadata. In both of these cases you may lose some information. I suggest trying this out on a cross-section of representative samples to satisfy yourself that it is doing what you want for your files.
- Phil
Quote from: Phil Harvey on February 19, 2021, 10:39:56 AM
But that command should copy the MakerNotes as a block for most common makes.
Ah, my mistake then.
Quote from: Phil Harvey on February 19, 2021, 11:04:52 AM
I can't answer this 100%. It depends on the models of cameras used to generate the images, and the softwares used to process them afterwards. Some cameras may not be supported, and some software may make a huge mess of the metadata. In both of these cases you may lose some information. I suggest trying this out on a cross-section of representative samples to satisfy yourself that it is doing what you want for your files.
- Phil
Hey Phil, so I did some testing and found the problem where the actual Lens information is lost when trying to parse it from MEI.
That's the test image:
https://cdn.explorecams.com/storage/test/explorecams-1347.jpg
So when I do the following:
exiftool -z -tagsfromfile explorecams-1347.jpg -all:all -icc_profile tamron.mie
exiftool tamron.mie
Lens ID output is "15.0-30.0mm f/2.8"
but when I am doing:
exiftool explorecams-1347.jpg
Lens ID output is "Tamron SP 15-30mm f/2.8 Di VC USD (A012)"
What I am missing here? Please advice.
Hm, but when I do this I can read the correct lens from the MIE file.
exiftool -tagsfromfile explorecams-1347.jpg -exif -xmp -iptc -makernotes -icc_profile tamron6.mie
I thought -all:all is the same as -exif -xmp -iptc -makernotes -icc_profile ... is there any other directives that could be included to get any other possible metadata bits extracted? It is pretty hard to find anything about it in the documentation. :-)
XMP-aux:LensID isn't being copied with All:All.
Quote from: StarGeek on February 27, 2021, 10:39:01 AM
XMP-aux:LensID isn't being copied with All:All.
Do you have any idea why? Is there any additional command that would complement all:all command and extract raw metadata bits from the image into a file?
Ah. ExifTool rejects the XMP-aux:LensID because it is negative. I'll patch the next version to allow negative numbers.
- Phil
Quote from: Phil Harvey on February 27, 2021, 11:26:28 AM
Ah. ExifTool rejects the XMP-aux:LensID because it is negative. I'll patch the next version to allow negative numbers.
- Phil
Oh, are there any other rules / limitations that get rejected when all:all is used? This does not happen when I do define -xmp in the command line.
-all:all copies tags individually, so each tag has a chance to do some validity checking if it wants. Also, only writable tags will be copied with -all:all, so for XMP it is more complete to copy it as a block (provided its size is below 64kB because you run into complications with JPEG segment size limitations if it is larger than this).
- Phil
Quote from: Phil Harvey on February 27, 2021, 11:56:50 AM
-all:all copies tags individually, so each tag has a chance to do some validity checking if it wants. Also, only writable tags will be copied with -all:all, so for XMP it is more complete to copy it as a block (provided its size is below 64kB because you run into complications with JPEG segment size limitations if it is larger than this).
- Phil
Oh I see. I missed the whole concept of the -all:all . I thought it was an alias for all the available metadata blocks (-XMP, -IPTC, etc...). Is there any such exiftool command line that would check for existing blocks and copy them as a whole?
Hi Linas,
Quote from: Linas Pašviestis on February 27, 2021, 12:10:10 PM
Is there any such exiftool command line that would check for existing blocks and copy them as a whole?
No, but there aren't many blocks available for copying. There are all listed in the Extra tags (https://exiftool.org/TagNames/Extra.html).
- Phil
Quote from: Phil Harvey on February 27, 2021, 05:28:18 PM
No, but there aren't many blocks available for copying. There are all listed in the Extra tags (https://exiftool.org/TagNames/Extra.html).
Thanks Phil.
So I did come up with the following command line. Would it get the most of the metadata from the images, or I am missing some bits?
exiftool -tagsfromfile explorecams-1347.jpg -exif -xmp -a -iptc -makernotes -icc_profile -adobe -canondr4 -canonvrd -trailer -comment -z final.mie
Also, by reading the documentation and if I understood it correctly '-a' option would extract extended XMP block. Is this right?
I would have to try this to see what happens with the extended XMP block, but I don't think it will handle this properly. However, extended XMP blocks are very rare.
Also, I think -makernotes is unnecessary because it is contained in the EXIF block.
- Phil
Quote from: Phil Harvey on February 28, 2021, 08:21:57 AM
I would have to try this to see what happens with the extended XMP block, but I don't think it will handle this properly. However, extended XMP blocks are very rare.
Also, I think -makernotes is unnecessary because it is contained in the EXIF block.
Thank you for your help!
I have one more question. Is it somehow possible to write a MIE sidecar and also give JSON output of the generated MIE file in a single command?
curl -s "https://images.unsplash.com/photo-1594819167477-290dd33b4daa" | exiftool -fast -tagsfromfile - -exif -xmp -iptc -icc_profile -adobe -canondr4 -canonvrd -comment -z test.mie
No.
Is it possible to have a conditional metadata extraction?
I have tried the following command without much success.
curl -s 'https://images.unsplash.com/photo-1594819167477-290dd33b4daa' | exiftool -fast -tagsfromfile - -if '$Model' -exif -xmp -iptc -icc_profile -adobe -canondr4 -canonvrd -comment -z test1.mie
If possible I would like to avoid creating a sidecar MIE file if EXIF data does not have a Model.
The -if condition operates on tags from the source file, not the -tagsFromFile file, so I don't think you can easily do what you want.
- Phil