Computer Vision Labels are Metadata - they belong in the EXIF

[voxleone]

Greetings everyone

I'm about to start using ExifTool in a labeling pipeline for computer vision where the annotations would be stored in the tags of the image file itself. The idea is to experiment with simplifying the dataset file system, by eliminating the sidecar text files, and check if there are any important gains that justify changes in the current training process paradigm at least for small datasets and fine-tuning tasks. Sidecar files by definition store data (often metadata) which is not supported by the format of a source file. That obviously isn't true with modern day digital image files, as the whole corpus in this site can attest.

ExifTool proves to be instrumental in this process, with its ability to manipulate and, chiefly, create UserDefined Tags [days ago @Phil and @StarGeek cleared things up about the use of that feature in the 'Create New Tag' thread and I've understood].

In the scheme I envisioned, the image label annotation would be serialized to a UserDefined tag – OR, as suggested, to a renamed standard one –  named 'Label' [1] in the XMP table – assuming the tag have been created or renamed in the 

exif.config

[right?]

I'm a bit ashamed to say this in this house, but I come from Python background. I'm writing these brief snippets just to get my point across, sorry.

What I want is

Code Select Expand

       
def writeToEXIFtag(data)
    #some pseudocode for now
    Image.Exif.Label = data

instead of the usual paired text file [json, xml, csv, etc.]

Code Select Expand

def writeToJSONFile(path, fileName, data):
    fileName = fileName.split(".")[0]
    filePathName = path + '/' + fileName + '.json'
    with open(filePathName, 'w') as fp:
        json.dump(data, fp)

as in what has become the canonical process.

I said I come from Python background. In this project, for greater practicality [integration with other modules, etc.], the best way for me to go seems to be using virtual environments such as virtualenv and conda. As you probably know, it is hardly possible to bring together the exact same packages on both platforms.

At the moment I'm using environments I configured with modules put together through the not very clean practice of mixing conda+pip. I still have things to figure out – not much proficient in Pearl [used it in web oriented tasks back in the day] and I'm having a hard time having everything [namely exiftool + wrapper] working together – I'm starting to think of ditching the wrapper altogether and work with native Pearl. I have to study a little bit more.

Sorry for the long post. This forum, despite being more of a support site for ET has such a cool web 1.0 vibe and I just couldn't resist.

(*) I got a post about this on my Blog: https://voxleone.com/2023/10/14/cv-labels-are-metadata-lets-treat-them-as-such/

Peace all

[StarGeek]

A minor nitpick with your title, but I would strongly be against including such things in EXIF. EXIF data should, for the most part, only be data directly from the camera.  It's not very expandable and EXIF tags don't include the name of the tag.  Any proprietary tags would not contain any hints as to the actual usage.  One of my commonly repeated phrases is
All EXIF data is Metadata, but not all Metadata is EXIF data.

Creating new tags should always be created in XMP.  It is significantly more flexible and expandable.

(ok, maybe not minor nitpick, as this is obviously something I feel very strongly about)

Sidecar files by definition store data (often metadata) which is not supported by the format of a source file. That obviously isn't true with modern day digital image files, as the whole corpus in this site can attest.

I would disagree with that.  All modern RAW file types can include metadata, but there are a lot of reasons that nearly all (if not all) DAM programs use sidecar files.

First, writing to RAW file types correctly can be difficult to keep up with, as, for some reason, camera companies keep changing minor bits of their format with each new camera.  This is why there is a delay with RAW development programs such as Lightroom in adding support for new cameras.

Because of this, many photographers are vehemently against editing their RAW files in any way.

Another issue is backups.  Changes to sidecar files are much easier and quicker to backup.  This post by @obetz shows their reason why they use sidecars.  And they make a very good point about changes possibly breaking some (IMO, poorly written) software.  Programs included by camera companies are notorious *cough*Nikon*cough* for doing this.

In the scheme I envisioned, the image label annotation would be serialized to a UserDefined tag – OR, as suggested, to a renamed standard one –  named 'Label' [1] in the XMP table – assuming the tag have been created or renamed in the 

exif.config

[right?]

If you're willing to create the infrastructure need to read a user defined tag, then that would be the way to go.  But such data would not be displayed by nearly every DAM.  Re: repurposing, I would say Label wouldn't be the best tag, as most of the time it is considered to have a limited selection of options and is usually used by DAMs to add a color to the thumbnail of an image.

Sidenote, dealing with user defined tags is something IMatch excels at.  See the GUANO.config thread where a user needed to access a specific tag regarding bat sounds and it ended up adding about 30 new tags to IMatch.  This post displays the result.

Also, there are image formats still used today that have limited support for metadata.  The obvious one is BMAP, which Windows still uses extensively and cannot support any type of metadata.  GIF has the ability to contain XMP data, but I have never found a program that is able to read it.  PNG is a modern file type, but still has hardly any software support for metadata.  I believe Adobe finally added support for it within the past 4? 5? years?  And with WebP I don't think I've seen any program beyond Google's original webpmux command.

I'm a bit ashamed to say this in this house, but I come from Python background.

Nothing wrong with Python, as a lot of kids these days are using it   Personally, I think the whole indent block thing is a big step backwards.  Reminds me of when I was trying to learn RPG (dating myself here).  Stupid easy language and I couldn't get the hang of how each line needed to be in an exact column.

I'm having a hard time having everything [namely exiftool + wrapper] working together – I'm starting to think of ditching the wrapper altogether and work with native Pearl.

If I recall correctly, the PyExiftool wrapper options end up being closer to command line tools.  One problem is making sure that you have the correct PyExiftool, as there is a fork that is much more up to date. Ah, found it.  I believe the sylikc/pyexiftool is the up to date version.