Extracting raw EXIF data from image

[ExploreCams]

Hello,

I would like to extract raw exif data from image to a file (to save space) and later read it with an exiftool. Is this possible?

Thanks.

[Phil Harvey]

Hi Linas,

This is possible in many ways.  Read here for some ways to create a metadata sidecar file.  Specifically, I'm thinking you may want to use an EXIF sidecar file.

- Phil

[ExploreCams]

Nice, thanks.

p.s. Just to clarify, do all of these commands output the same data?

exiftool -exif -b SRCFILE > out.exif
exiftool -tagsfromfile SRCFILE -exif out.exif
exiftool -o out.exif -exif SRCFILE

[ExploreCams]

Hm, but if I do extract EXIF sidecar file. Do I miss all other additional formats? (e.g. XMP, IPTC)

I would like to extract any raw metadata except an actual image. The purpose of this is to have small footprints of images, so I could easily re-read these when new ExifTool comes out. I have over 20 million images.

[StarGeek]

Hm, but if I do extract EXIF sidecar file. Do I miss all other additional formats? (e.g. XMP, IPTC)

Yes.  XMP can be copied to a XMP sidecar but IPTC would be left out.  There is an XMP equivalent for most IPTC data.

I would like to extract any raw metadata except an actual image. The purpose of this is to have small footprints of images, so I could easily re-read these when new ExifTool comes out.

Take a look at exiftool's own MIE file in the link Phil gave above.  You could copy EXIF, IPTC, and XMP, as well as the ICC_Profile and MakerNotes.

I have over 20 million images.

Ok, I thought I had a significant collection of images I've collected from the net and photos I've taken and family photos I've scanned, somewhere between 1-2 million, but you've taken it to a whole other level.  Consider me impressed.

[ExploreCams]

Take a look at exiftool's own MIE file in the link Phil gave above.  You could copy EXIF, IPTC, and XMP, as well as the ICC_Profile and MakerNotes.

So if I understand it correctly the following comand will extract all available metadata from an image into a file?

Code Select Expand

exiftool -tagsfromfile a.jpg -all:all -icc_profile a.mie

Ok, I thought I had a significant collection of images I've collected from the net and photos I've taken and family photos I've scanned, somewhere between 1-2 million, but you've taken it to a whole other level.  Consider me impressed.

I do run a website https://explorecams.com , feel free to donate some of the original photos if you like it. :-)

[StarGeek]

That won't copy the MakerNotes or any tag that exiftool doesn't know how to write.  Copying the groups as a block should save everything, even stuff that exiftool is unable to write.  And you'll probably want to do it in batch, so try this (I can't remember if copying EXIF will also copy MakerNotes or not)
exiftool -tagsfromfile -o %d%f.mie -EXIF -XMP -IPTC -MakerNotes -icc_profile /path/to/files/

This will create the sidecars in the same directory as the files.  If you want to use a different directory, change the %d to the correct path.

[ExploreCams]

That won't copy the MakerNotes or any tag that exiftool doesn't know how to write.  Copying the groups as a block should save everything, even stuff that exiftool is unable to write.  And you'll probably want to do it in batch, so try this (I can't remember if copying EXIF will also copy MakerNotes or not)
exiftool -tagsfromfile -o %d%f.mie -EXIF -XMP -IPTC -MakerNotes -icc_profile /path/to/files/

This will create the sidecars in the same directory as the files.  If you want to use a different directory, change the %d to the correct path.

I thought -all:all is the replacement for -exif -xmp -iptc and -makernotes.

[Phil Harvey]

Just to clarify, do all of these commands output the same data?

exiftool -exif -b SRCFILE > out.exif
exiftool -tagsfromfile SRCFILE -exif out.exif
exiftool -o out.exif -exif SRCFILE

Yes, but they differ in their behaviour if the file already exists.  The first one overwrites out.exif if it already exists, and the other two fail if out.exif exists.

- Phil

[ExploreCams]

Just to clarify, do all of these commands output the same data?

exiftool -exif -b SRCFILE > out.exif
exiftool -tagsfromfile SRCFILE -exif out.exif
exiftool -o out.exif -exif SRCFILE

Yes, but they differ in their behaviour if the file already exists.  The first one overwrites out.exif if it already exists, and the other two fail if out.exif exists.

- Phil

Phil, would the following command would be the ultimate one to extract every metadata bit from the image? Or I will miss some extra meta formats e.g. MakerNotes, XMP, IPTC?

Code Select Expand

exiftool -tagsfromfile a.jpg -all:all -icc_profile a.mie

[Phil Harvey]

That command will get most of the tags that ExifTool can write.

- Phil

[StarGeek]

The key point being "can write".  MakerNotes for example, cannot be copied individually.  It must be copied as a block (see last paragraph of FAQ #8).

Another example would be any tag that requires a config file to write, such as the Pix4d tags.  Without the config file, those tags would not be copied.

[Phil Harvey]

But that command should copy the MakerNotes as a block for most common makes.

- Phil

[ExploreCams]

That command will get most of the tags that ExifTool can write.

Okay, so lets say my goal is to store a raw metadata file without the actual image, so I could easily re-run / update the output when I upgrade exiftool. The most important bit of information is identification of Lens Id to me and other inconsistances in EXIF (if these even happen between versions). Should this command 100% completely cover my goal?


Sorry, for asking this again and again, as it would take a bit of time to extract things from 20M+ images and I want to be 100% certain. :-)

[Phil Harvey]

I can't answer this 100%.  It depends on the models of cameras used to generate the images, and the softwares used to process them afterwards.  Some cameras may not be supported, and some software may make a huge mess of the metadata.  In both of these cases you may lose some information.  I suggest trying this out on a cross-section of representative samples to satisfy yourself that it is doing what you want for your files.

- Phil

[StarGeek]

But that command should copy the MakerNotes as a block for most common makes.

Ah, my mistake then.

[ExploreCams]

I can't answer this 100%.  It depends on the models of cameras used to generate the images, and the softwares used to process them afterwards.  Some cameras may not be supported, and some software may make a huge mess of the metadata.  In both of these cases you may lose some information.  I suggest trying this out on a cross-section of representative samples to satisfy yourself that it is doing what you want for your files.

- Phil

Hey Phil, so I did some testing and found the problem where the actual Lens information is lost when trying to parse it from MEI.

That's the test image:
https://cdn.explorecams.com/storage/test/explorecams-1347.jpg

So when I do the following:

Code Select Expand


exiftool -z -tagsfromfile explorecams-1347.jpg -all:all -icc_profile tamron.mie
exiftool tamron.mie

Lens ID output is "15.0-30.0mm f/2.8"

but when I am doing:

Code Select Expand


exiftool explorecams-1347.jpg

Lens ID output is "Tamron SP 15-30mm f/2.8 Di VC USD (A012)"


What I am missing here? Please advice.

[ExploreCams]

Hm, but when I do this I can read the correct lens from the MIE file.

Code Select Expand


exiftool -tagsfromfile explorecams-1347.jpg -exif -xmp -iptc -makernotes -icc_profile tamron6.mie



I thought -all:all is the same as -exif -xmp -iptc -makernotes -icc_profile ... is there any other directives that could be included to get any other possible metadata bits extracted? It is pretty hard to find anything about it in the documentation. :-)

[StarGeek]

XMP-aux:LensID isn't being copied with All:All.

[ExploreCams]

XMP-aux:LensID isn't being copied with All:All.

Do you have any idea why? Is there any additional command that would complement all:all command and extract raw metadata bits from the image into a file?

[Phil Harvey]

Ah.  ExifTool rejects the XMP-aux:LensID because it is negative.  I'll patch the next version to allow negative numbers.

- Phil

[ExploreCams]

Ah.  ExifTool rejects the XMP-aux:LensID because it is negative.  I'll patch the next version to allow negative numbers.

- Phil

Oh, are there any other rules / limitations that get rejected when all:all is used? This does not happen when I do define -xmp in the command line.

[Phil Harvey]

-all:all copies tags individually, so each tag has a chance to do some validity checking if it wants.  Also, only writable tags will be copied with -all:all, so for XMP it is more complete to copy it as a block (provided its size is below 64kB because you run into complications with JPEG segment size limitations if it is larger than this).

- Phil

[ExploreCams]

-all:all copies tags individually, so each tag has a chance to do some validity checking if it wants.  Also, only writable tags will be copied with -all:all, so for XMP it is more complete to copy it as a block (provided its size is below 64kB because you run into complications with JPEG segment size limitations if it is larger than this).

- Phil

Oh I see. I missed the whole concept of the -all:all . I thought it was an alias for all the available metadata blocks (-XMP, -IPTC, etc...). Is there any such exiftool command line that would check for existing blocks and copy them as a whole?

[Phil Harvey]

Hi Linas,

Is there any such exiftool command line that would check for existing blocks and copy them as a whole?

No, but there aren't many blocks available for copying.  There are all listed in the Extra tags.

- Phil

[ExploreCams]

No, but there aren't many blocks available for copying.  There are all listed in the Extra tags.

Thanks Phil.

So I did come up with the following command line. Would it get the most of the metadata from the images, or I am missing some bits?

Code Select Expand


exiftool -tagsfromfile explorecams-1347.jpg -exif -xmp -a -iptc -makernotes -icc_profile -adobe -canondr4 -canonvrd -trailer -comment -z final.mie


Also, by reading the documentation and if I understood it correctly '-a' option would extract extended XMP block. Is this right?

[Phil Harvey]

I would have to try this to see what happens with the extended XMP block, but I don't think it will handle this properly.  However, extended XMP blocks are very rare.

Also, I think -makernotes is unnecessary because it is contained in the EXIF block.

- Phil

[ExploreCams]

I would have to try this to see what happens with the extended XMP block, but I don't think it will handle this properly.  However, extended XMP blocks are very rare.

Also, I think -makernotes is unnecessary because it is contained in the EXIF block.

Thank you for your help!

I have one more question. Is it somehow possible to write a MIE sidecar and also give JSON output of the generated MIE file in a single command?

Code Select Expand


curl -s "https://images.unsplash.com/photo-1594819167477-290dd33b4daa" | exiftool -fast -tagsfromfile - -exif -xmp -iptc -icc_profile -adobe -canondr4 -canonvrd -comment -z test.mie


[Phil Harvey]

No.

[ExploreCams]

Is it possible to have a conditional metadata extraction?

I have tried the following command without much success.

Code Select Expand


curl -s 'https://images.unsplash.com/photo-1594819167477-290dd33b4daa' | exiftool -fast -tagsfromfile - -if '$Model' -exif -xmp -iptc -icc_profile -adobe -canondr4 -canonvrd -comment -z test1.mie


If possible I would like to avoid creating a sidecar MIE file if EXIF data does not have a Model.

[Phil Harvey]

The -if condition operates on tags from the source file, not the -tagsFromFile file, so I don't think you can easily do what you want.

- Phil