Importing custom binary data to MakerNotes field

[joshcomley]

Hi there, I've searched a fair chunk of forum history, read as much as I can and experimented as much as possible and I'm stuck

I'm trying to set the MakerNotes to my own custom binary data. Is this possible with ExifTool? If so, how can I do it? I understand I have to copy it as a block, but how can I do this? Can I block import from a file, for example? I cannot find a command line action for this.

Thank you!!

[Phil Harvey]

Hi Josh,

The only way to do this is to override the ExifTool MakerNotes with a user-defined tag and a config file like this:

Code Select Expand

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::Exif::Main' => {
        0x927c => {
            Name => 'MyMakerNotes',
            Writable => 'undef',
            Binary => 1,
        },
    },
);
1; # end

See the sample config file for instructions on using a config file.

- Phil

[joshcomley]

Hi Phil,

Thank you so much for your response!

I have created the following .ExifTool_config file:

Code Select Expand

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::Exif::Main' => {
        0x927c => {
            Name => 'MyMakerNotes',
            Writable => 'undef',
            Binary => 1,
        },
    },
);
1;  #end

And I am trying to run it like this:

Code Select Expand

exiftool -MyMakerNotes= image.jpg

But the output says:

Code Select Expand

0 image files updated
1 image files unchanged

Where do I place the binary I'd like to use? Is it the

Code Select Expand

Binary => 1, part of the config file? Currently I have a small file I'd like to embed in the MakerNotes section.

Many thanks again,
Josh

[Phil Harvey]

Hi Josh,

Code Select Expand

exiftool -MyMakerNotes= image.jpg

But the output says:

Code Select Expand

0 image files updated
1 image files unchanged

This command deletes the maker notes.  It will give this message if they didn't exist.

Where do I place the binary I'd like to use?

For example, to get the maker notes from file "DATA.BIN", do this:

exiftool "-mymakernotes<=DATA.BIN" file.jpg

- Phil

[joshcomley]

Thanks again Phil, that works perfectly!

I just tried on a bunch of files. It worked for some but some exceed 64kb in size; is there anything I can do (save making it smaller, which is theoretically impossible at the moment)?

In my reading I learned about the ability to exceed 64kb if the data spans multiple segments but I couldn't find any real explanations of how to achieve this or any hint of pseudocode to adapt.

I noticed here:

http://www.exiftool.org/writing.html#JPEG

You mention

This multi-segment information is handled properly by ExifTool.

So is there a way I can split my file up or do some magic to achieve this?

Many thanks again, I really appreciate ExifTool and your excellent help.

Josh

[Phil Harvey]

Hi Josh,

The EXIF specification limits EXIF data to a single segment in JPEG images, so it can not exceed 64 kB.  (And the MakerNotes you are writing are contained in the EXIF.)  If you want to store more, you need to go to something other than EXIF.

- Phil

[joshcomley]

Thanks Phil, I was worried that would be the case.

Do you have a suggestion of what to Google?

Are you talking about APP2, for example?

[Phil Harvey]

Hi Josh,

If you are talking about the ICC_Profile APP2 information, then no.

The XMP APP1 would be the place, except the XMP doesn't support binary data directly, so it would need to be encoded (probably Base64).  For example, you could hide it in XMP:ThumbnailImage, which does the Base64 encoding.

There really aren't any extensible alternatives for storing >64kB binary data directly in a JPEG image.

You could put it in a MIE trailer, but only ExifTool understands those, so it would be of limited use.

- Phil

[joshcomley]

Ah, yes, I can encode it to Base64 no problem.

XMP:ThumbnailImage seems like a good solution.

Interestingly, I've tried injecting >64kb in the MakerNote field (actually base 64 encoded) using a few other mechanisms, one which I brute force the content in there regardless of size.

Sometimes the data is >100kb and it resulted in a functioning JPEG but with a "Bad MakerNote" section reported if I let ExifTool analyse it.

But crucially the some of the JPEGs with this large MakerNote data would still render fine in all viewers. I just wish I could make it consistent across all JPEG/MakerNote combinations; I'm not sure why some work and some don't.

[Phil Harvey]

You must have split the EXIF into multiple JPEG segments, which is not allowed by the standard, and ExifTool can not be used to read this.

But XMP sounds like the way to go.  ExifTool will do the Base64 encoding for you.  If you use XMP:ThumbnailImage, it is done already.  If you create a user-defined tag, then add these conversions:

Code Select Expand

        ValueConv => 'Image::ExifTool::XMP::DecodeBase64($val)',
        ValueConvInv => 'Image::ExifTool::XMP::EncodeBase64($val)',

- Phil

[joshcomley]

Hi Phil,

I've been experimenting with the config file and so on and I've got something working with one problem.

Here is my config file:

Code Select Expand

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::XMP::xmp' => {
        JoshTag => {
            SubDirectory => {
                TagTable => 'Image::ExifTool::UserDefined::JoshTag',
                # (see the definition of this table below)
            },
            Name => 'JoshTag',
            Writable => 'string',
        },
    },
);

%Image::ExifTool::UserDefined::JoshTag = (
    GROUPS => { 0 => 'XMP', 1 => 'XMP-JoshTag', 2 => 'Image' },
    WRITABLE => 'string',
    # example structured XMP tag
    NewXMPJoshTagStruct => {
        # the "Struct" entry defines the structure fields
        Struct => {
        },
    },
);
1;  #end

I actually do the base 64 encoding myself outside and then use:

Code Select Expand

exiftool "-JoshTag<=base64.txt" image.jpg

It works, but roughly every 64kb is split up by something that looks like this:

Code Select Expand

<?xpacket end='w'?>[b]ÿáÿÿhttp://ns.adobe.com/xmp/extension/ 3AFF02DE3FEB5AF9C895191EF2F39234 ¦$    [/b]<x:xmpmeta xmlns:x='adobe:ns:meta/' x:xmptk='Image::ExifTool 9.47'>

Is there anyway to prevent this or does XMP get split up into 64kb chunks like this? I will be parsing the XMP out of the jpeg completely manually so I am trying to work out how to read the string value without this being in the way. I could parse out the value of the namespace above, but I've tried that and what is inbetween the xpacket end tag and <x:xmpmeta does not exactly match what appears in the string.

Another option I have is to have separate custom tags (JoshTag1, JoshTag2 etc.) and split it up into 60kb chunks, but this seems even more hacky than what I'm already doing

Can I tell ExifTool to not place this marker in? Or perhaps specify what the marker should be so I can predict it?

Many thanks again and again!!
Josh

[Phil Harvey]

Anything stored inside the JPEG must be spit into maximum 64kB segments.  If you don't want it to be split, the only option is to put it in a trailer.   In the most simple form, you can do this:

cat data.bin >> image.jpg

(assuming you are running Mac or Linux this will work.  In Windows, you will need some other technique to concatenate the two files.)

By the way, your user-defined tag definition is confused.  You are writing it as a simple tag in the xmp namespace, so the user-defined namespace and structure are superfluous, and this will do the same thing:

Code Select Expand

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::XMP::xmp' => {
        JoshTag => { },
    },
);
1; # end

- Phil

[joshcomley]

Phil just wanted to say thank you so much for your help, it's vitally appreciated. If I make millions I'll send something your way!!

Your tool is fantastic and your support somehow even better.