tags, metadata and those programs that manage images for the newcomer

[rappel]

Background.
 - I have a collection of around 80,000 images, never tagged, simply categorised by date and time in the filename and grouped by camera. I've looked around before at tagging and never quite found the right way to manage this. I appreciate that I will probably just have to suck it and see with a few weeks worth of new images before starting in to the big pile of history, as I need to find a way to categorise that suits me and the other half as a way of working. That said once I do so I don't want to box myself in a corner with software.

 - We don't use Adobe products because we can't at this time afford them but we may head that way in the future if we're lucky enough and/or have other needs. We are currently trying to work out what desktop image programs to use and as usual we have different preferences and as I shoot Raw I'm tending to Darktable with Lighttable and Gimp for final edits as I'm a 10 year user of the editor. The other half doesn't at the moment bother with Raw and quite likes Digikam and seems to be heading toward Krita for the image editing as the interface sits better for her. The point is though the image management and editors may change over time as our photography changes or our needs change or...

So, this isn't really about the products themselves but rather the interoperability of the products to use and exchange metadata without loss and seeking to gain more understanding about the limitations of the current methods.

 - I've spent a day trawling all over looking for advice on tagging with user created tags. but while there's a ton of info on why you should use tags, what you can put in them, how to roundtrip in some specific circumstances, there's little info on where various tags are stored, how they are stored and bugger all on whether different programs interpret them the same. Some metadata e.g. colours I am reading are not consistent between programs presumably as they all regad them as interal flags rather than true metadata. - Yes I found the large list of XMP tags on this site but that didn't really help.

Question

In order to plan for the future as best I can

Is there an idiots guide anywhere that:

• Provides a definitive as of (say) last year list of tags that all programs should be able to access and use?
• Explains clearly whether or not user defined tags can be used and transferred between current programs and what if any current limitations are.
• Explains clearly what tags can be stored in which file formats. In an ideal world I'd like to use just one file, but reading today sugegsts that there are limitiations to jpeg and Raw formats ability to have lots of user defined stuff in there.
• ...so if one has to use extension files such as XMP do all programs have the same naming conventions for these files, again reading today suggests that not to be the case.

With this info I'm looking to understand if its possible to use 2 image management programs to access and manipulate the same image data set while also processing raw images to feed into this as well.
And in the long term if we need or want to change image mgt or editors what is the basic set of tags and user defined tags today that I need to stick with to minimise the workload in moving.

[StarGeek]

That said once I do so I don't want to box myself in a corner with software.
...
 I'm tending to Darktable with Lighttable and Gimp for final edits as I'm a 10 year user of the editor. The other half doesn't at the moment bother with Raw and quite likes Digikam and seems to be heading toward Krita for the image editing as the interface sits better for her.

Both Darktable and DigiKam are fine. They will save metadata either in the file, such as in JPEGs, or in an XMP sidecar file, as with RAW file types.

There may be a slight difficulty with, I think, DigiKam and the XMP sidecar files (see below), as I think it includes the file extension in the sidecar file filename, i.e. it would use file.CR2.xmp where most programs use file.xmp, but the only difficulty there is renaming the file to remove the middle extension.

So, this isn't really about the products themselves but rather the interoperability of the products to use and exchange metadata without loss and seeking to gain more understanding about the limitations of the current methods.

Both of these are following the standards for saving metadata and will be transferable to other programs, such as Lightroom.

Editing programs such as GIMP are less likely to keep all the original metadata, but it's not something I've looked at in detail. It should be fine as long as you keep the originals.

I've spent a day trawling all over looking for advice on tagging with user created tags. but while there's a ton of info on why you should use tags

There may be a misunderstanding of the term here, but user created tags should never be a default. The standards exist for a reason. To quote Phil (exiftool author) about programs that create their own tags

(A note to software developers: Re-inventing your own private tags instead of using the equivalent tags in standard XMP namespaces defeats one of the most valuable features of metadata: interoperability. Your applications mumble to themselves instead of speaking out for the rest of the world to hear.)

User defined tags should never be created unless you have a specific purpose and plan on creating scripts/programs to read such data because no other program is going to be able to read them.

To clarify a bit, Tags are the locations where metadata is stored. A description would be stored in the Description. A short description would be in the Headline tag, and so forth.

The word "tag" is often used to describe the keywords associated with an image. An image with a cat, a dog, and little Timmy next to a well would have the keywords "Dog", "Cat", "Timmy", and "Well". Some programs will list these as "tags". This is what Windows Properties does.

what you can put in them, how to roundtrip in some specific circumstances, there's little info on where various tags are stored, how they are stored and bugger all on whether different programs interpret them the same.  Some metadata e.g. colours I am reading are not consistent between programs presumably as they all regad them as interal flags rather than true metadata. - Yes I found the large list of XMP tags on this site but that didn't really help.

Overall, you shouldn't worry about the inner details, as that path lies madness. Let Darkroom/DigiKam take care of the details behind the scene.

Provides a definitive as of (say) last year list of tags that all programs should be able to access and use?

Tags don't change that often. The standards are pretty well set and are stable. The most useful list would be the IPTC Photo Metadata Standard. But that is a technical document and a dry read.

Explains clearly whether or not user defined tags can be used and transferred between current programs and what if any current limitations are.

As mentioned, they cannot. If you create your own tag, nothing will be able to read it.

Explains clearly what tags can be stored in which file formats.

See the Supported File Types table on the front page.

In an ideal world I'd like to use just one file, but reading today sugegsts that there are limitiations to jpeg and Raw formats ability to have lots of user defined stuff in there.
...so if one has to use extension files such as XMP do all programs have the same naming conventions for these files, again reading today suggests that not to be the case.

There is a lot of misinformation out there about what metadata can be put into what file type. This is especially the case around RAW file types, with many people saying it can't be done (it can be done). But again, you shouldn't worry about it. Good programs such as DigiKam/Darktable/LightRoom will take care of these things behind the scenes. They'll embed the data in the file when appropriate, such as with JPEGs, and as XMP sidecars when that is appropriate, such as with RAW files.

With this info I'm looking to understand if its possible to use 2 image management programs to access and manipulate the same image data set while also processing raw images to feed into this as well.

The one thing I would suggest is to set DigiKam to set the Sidecar file names are compatible with commercial programs option (I had to look this up). That will make DigiKam use standard XMP naming which will be useable by other programs.

I can't comment on how much metadata GIMP will save from exported images, as I haven't looked into that. It's best to keep the originals just in case.[/quote]

[rappel]

Many thanks for the quick and full reply.

I've skimmed the IPTC doc you referenced and the exiftool support page you linked and from the inital read it looks like most of my basic questions are answered (but I'll read properly later on). I conclude that as you suggest I will have to trust to the image mgt programs' abilities.

You were correct, I didn't mean tag, I meant keyword when talking about user defined. Unfortuantely tag is used everywhere with different meaning now and I didn't realise that the "tag" I was interested in was keyword.

I do have a couple of questions left however which are more practical.

• The round tripping between programs and multiple instances of these working on the same source directories on a network file system. I assume it's pretty much suck it and see whether the programs read and write info to the same metadata containers or just trust that they can work it out if they don't? i.e. one defaults to write to a field in the EXIF data of a jpeg whereas another defaults to write to the XMP file.
• With keywords, Digikam can make use of hierarchical keywords, which presumably is based on it parsing the text strings and using the results in a smart way. Is this a standards approach for all image mgt programs or is it replicated across multiple programs anyway as a de facto approach? OR not at all...
• With the other markers used in programs to manage user workflow - colours, ratings etc. are these also catered for in any of the metadata stored so that they can be shared between progams and users or is this specific to the program in question?

[StarGeek]

You were correct, I didn't mean tag, I meant keyword when talking about user defined. Unfortuantely tag is used everywhere with different meaning now and I didn't realise that the "tag" I was interested in was keyword.

To be fair, context matters. Tag/Tagging is fine for casual use, but more precise words are needed when digging into the details of metadata. It's similar to how many people call all metadata EXIF data, when EXIF data is just one type of metadata. (I just noticed that this is what you seem to be doing below )

I think what you were really interested in would be a Controlled Vocabulary.

The round tripping between programs and multiple instances of these working on the same source directories on a network file system. I assume it's pretty much suck it and see whether the programs read and write info to the same metadata containers or just trust that they can work it out if they don't? i.e. one defaults to write to a field in the EXIF data of a jpeg whereas another defaults to write to the XMP file.

For the most part, this shouldn't be a problem with a good program, of which DigiKam/Darktable/Lightroom are (as long as you change the DigiKam setting). By default, they should be correctly saving directly to the file or to a sidecar file as the file type demands. And they will be able to correctly read a sidecar file if it exists for a file where the data would normally be embedded, like a JPEG.

But you wouldn't want to be running two or more programs on the same set of files at the same time. If one program makes a change to the metadata, the other program may not detect that there has been a change and would still be operating on the old data.

Also, there are some programs that do things differently, such as ACDSee. Overall, it's a decent program and saves the data in the file, but it is the cause of the above comment about private tags. It writes data to a bunch of tags that are only read by ACDSee. I can't remember if it also writes to standard tags as well, as it's been some years since I tested it out.

With keywords, Digikam can make use of hierarchical keywords, which presumably is based on it parsing the text strings and using the results in a smart way. Is this a standards approach for all image mgt programs or is it replicated across multiple programs anyway as a de facto approach? OR not at all...

Hierarchical keywords is a bit of a messy situation, as the separator for the hierarchy isn't standardized. But some programs are flexible and let you change the separator in the settings. This isn't an issue I've looked into in details. It's something that should be tested with the software used by writing a hierarchy with one program, and then loading the file into the other to see if it works.

If needed, exiftool can be used to change the separator in bulk.

With the other markers used in programs to manage user workflow - colours, ratings etc. are these also catered for in any of the metadata stored so that they can be shared between progams and users or is this specific to the program in question?

This is a similar situation. There isn't a standard as to how these are to be set, though it seems likely to me that most programs use the same or similar values as Lightroom might use. Part of it is that these things are not part of the standards. But the values might be changeable in the settings. Again, this isn't something I've looked at in detail.

A quick check shows that DigiKam has its own tag called ColorLabel label, so if it's saving the colors there, then that might not transfer. My (very) old version of Lightroom saved colors to the XMP:Label tag.