The Darwin Core is an XML schema used in science world-wide.
http://rs.tdwg.org/dwc/
and
http://rs.tdwg.org/dwc/terms/guides/xml/index.htm for an XML schema.
Apparently so far IDImager was the only software which supported this namespace in XMP as well. Since IDImager has been recently discontinued, there is now a real need to support XMP-dwc in other applications.
ExifTool can extract the DarwinCore data from XMP like all other unknown XMP data, but does not include it in -listx schema output. My application uses the output from listx to setup reference tables in the database, and cannot work with or store tags "unknown" to ExifTool, even if Exiftool can extract the data from XMP as text. I need somehow to integrate this.
How could I make this namespace known to ExifTool so that ExifTool supports it on the same level as, for example, XMP-dc or XMP-photoshop?
What information would I need to request from Darwin Core people to add support for it to ExifTool?
I know I can declare "custom namespaces" (somehow, did not do that yet) but this is not an application-specific namespace so maybe we should integrate it tighter?
Adding this schema is certainly a possibility. What I need is an XMP sample containing all of the available properties.
- Phil
Hi, Phil
this is also new to me. I have one sample image with a subset of the data in an XMP namespace.
I will try to get the full spec and more info and then come back to you.
Dear Phil,
I have actually used the software IDImager to create that custom DarwinCore XMP namespace.
Photo Supreme (now marketed as replacement of IDImager) can still apparently read the XMP but no longer permits customization or modifications.
With IDImager Pro now being discontinued I am quite desperate to find alternatives that are future-proof, supporting the Darwin Core XMP (quite unfortunately all information about the DarwinCore XMP was also even removed from the IDImager user forum and their WikiSite has been deleted !!! Not sure if that was intentional, but it is all the more reason for me to try finding alternatives...).
Thus I am highly interested in other software that at least reads, better writes to that schema. I have been in touch with the developer of iMatch and he mentioned that he is using ExifTool for reading XMP. He even suggested he could get in touch with you and I sent him quite a bit of information about the DarwinCore. Not sure if you ever got this...
Attached that info...
One problem with the DarwinCore: it is very extensive, many, many fields. At the research station where I work we have essentially only used a subset of it, so I can send you images with the fields that we are regularly using, but it would take quite a bit of effort to create a photo that makes use of every single field. Still, if that is what is required, I am more than happy to work on this and send you an example...
Cheers,
Frank
Hi Frank,
I'll read your documents tomorrow.
Even without a full set of pre-defined Darwin core tags, ExifTool is future proof in this respect. It can read any XMP as-is, and you only need to create user-defined tags for any undefined XMP tags that you want to write.
- Phil
Hi Frank,
Thanks for the documents, Mario didn't pass them along to me.
I am more than happy to create an ExifTool config file to allow writing of these properties. I will need at minimum a small example with a cross-section of the properties to verify that I have the correct general formatting. But a sample with all properties would be preferable, and would allow validation of all DarwinCore tags.
- Phil
Hi Phil,
that's great. OK - I will work on generating a photo that has content for all XMP fields then and send you that.
(It will take a bit because I will need to make sure I add all fields - quite a few...)
Frank
Hi Phil,
sorry for the long delay. Here comes an example with all DarwinCore XMP fields filled in. Essentially all nonsense data, i.e., the examples that are given on the Darwin Core website about what kind of data the fields should contain.
Let me know if you need anything more.
Thanks,
Frank
Hi Frank,
Here is the config file that gives you the ability to write these Darwin Core tags (I have also added it as an attachment):
#------------------------------------------------------------------------------
# File: ExifTool_dwc.config --> ~/.ExifTool_config
#
# Description: ExifTool configuration file to add Darwin Core XMP tags
#
# Usage: exiftool -config ExifTool_dwc.config ...
#
# Revisions: 2013/01/28 - P. Harvey Created
#
# References: http://rs.tdwg.org/dwc/index.htm
#------------------------------------------------------------------------------
use Image::ExifTool::XMP;
%Image::ExifTool::UserDefined = (
'Image::ExifTool::XMP::Main' => {
dwc => {
SubDirectory => { TagTable => 'Image::ExifTool::UserDefined::dwc' },
},
},
);
%Image::ExifTool::UserDefined::dwc = (
GROUPS => { 0 => 'XMP', 1 => 'XMP-dwc', 2 => 'Other' },
NAMESPACE => { 'dwc' => 'http://rs.tdwg.org/dwc/index.htm' },
WRITABLE => 'string',
Event => {
Struct => {
day => { Writable => 'integer' },
earliestDate => { %Image::ExifTool::XMP::dateTimeInfo },
endDayOfYear => { Writable => 'integer' },
eventID => { },
eventRemarks => { Writable => 'lang-alt' },
eventTime => { %Image::ExifTool::XMP::dateTimeInfo },
fieldNotes => { },
fieldNumber => { },
habitat => { },
latestDate => { %Image::ExifTool::XMP::dateTimeInfo },
month => { Writable => 'integer' },
samplingEffort => { },
samplingProtocol => { },
startDayOfYear => { Writable => 'integer' },
verbatimEventDate => { },
year => { Writable => 'integer' },
},
},
GeologicalContext => {
Struct => {
bed => { },
earliestAgeOrLowestStage => { },
earliestEonOrLowestEonothem => { },
earliestEpochOrLowestSeries => { },
earliestEraOrLowestErathem => { },
earliestPeriodOrLowestSystem=> { },
formation => { },
geologicalContextID => { },
group => { },
highestBiostratigraphicZone => { },
latestAgeOrHighestStage => { },
latestEonOrHighestEonothem => { },
latestEpochOrHighestSeries => { },
latestPeriodOrHighestSystem => { },
lithostratigraphicTerms => { },
lowestBiostratigraphicZone => { },
member => { },
},
},
Identification => {
Struct => {
dateIdentified => { %Image::ExifTool::XMP::dateTimeInfo },
identificationID => { },
identificationQualifier => { },
identificationReferences => { },
identificationRemarks => { },
identifiedBy => { },
typeStatus => { },
},
},
MeasurementOrFact => {
Struct => {
measurementAccuracy => { Format => 'real' },
measurementDeterminedBy => { },
measurementDeterminedDate => { %Image::ExifTool::XMP::dateTimeInfo },
measurementID => { },
measurementMethod => { },
measurementRemarks => { },
measurementType => { },
measurementUnit => { },
measurementValue => { },
},
},
Occurrence => {
Struct => {
associatedMedia => { },
associatedOccurrences => { },
associatedReferences => { },
associatedSequences => { },
associatedTaxa => { },
behavior => { },
catalogNumber => { },
disposition => { },
establishmentMeans => { },
individualCount => { },
individualID => { },
lifeStage => { },
occurenceRemarks => { },
occurrenceDetails => { },
occurrenceID => { },
occurrenceStatus => { },
otherCatalogNumbers => { },
preparations => { },
previousIdentifications => { },
recordNumber => { },
recordedBy => { },
reproductiveCondition => { },
sex => { },
},
},
Record => {
Struct => {
basisOfRecord => { },
collectionCode => { },
collectionID => { },
dataGeneralizations => { },
datasetID => { },
datasetName => { },
dynamicProperties => { },
informationWithheld => { },
institutionCode => { },
institutionID => { },
ownerInstitutionCode => { },
},
},
ResourceRelationship => {
Struct => {
relatedResourceID => { },
relationshipAccordingTo => { },
relationshipEstablishedDate => { %Image::ExifTool::XMP::dateTimeInfo },
relationshipOfResource => { },
relationshipRemarks => { },
resourceID => { },
resourceRelationshipID => { },
},
},
Taxon => {
Struct => {
acceptedNameUsage => { },
acceptedNameUsageID => { },
class => { },
family => { },
genus => { },
higherClassification => { },
infraspecificEpithet => { },
kingdom => { },
nameAccordingTo => { },
nameAccordingToID => { },
namePublishedIn => { },
namePublishedInID => { },
nomenclaturalCode => { },
nomenclaturalStatus => { },
order => { },
originalNameUsage => { },
parentNameUsage => { },
parentNameUsageID => { },
phylum => { },
scientificName => { },
scientificNameAuthorship => { },
scientificNameID => { },
specificEpithet => { },
subgenus => { },
taxonConceptID => { },
taxonID => { },
taxonRank => { },
taxonRemarks => { },
taxonomicStatus => { },
verbatimTaxonRank => { },
vernacularName => { Writable => 'lang-alt' },
},
},
dctermsLocation => {
Struct => {
continent => { Groups => { 2 => 'Location' } },
coordinatePrecision => { Groups => { 2 => 'Location' } },
coordinateUncertaintyInMeters => { Groups => { 2 => 'Location' } },
country => { Groups => { 2 => 'Location' } },
countryCode => { Groups => { 2 => 'Location' } },
county => { Groups => { 2 => 'Location' } },
decimalLatitude => { Groups => { 2 => 'Location' } },
decimalLongitude => { Groups => { 2 => 'Location' } },
footprintSRS => { Groups => { 2 => 'Location' } },
footprintSpatialFit => { Groups => { 2 => 'Location' } },
footprintWKT => { Groups => { 2 => 'Location' } },
geodeticDatum => { Groups => { 2 => 'Location' } },
georeferenceProtocol => { Groups => { 2 => 'Location' } },
georeferenceRemarks => { Groups => { 2 => 'Location' } },
georeferenceSources => { Groups => { 2 => 'Location' } },
georeferenceVerificationStatus => { Groups => { 2 => 'Location' } },
georeferencedBy => { Groups => { 2 => 'Location' } },
higherGeography => { Groups => { 2 => 'Location' } },
higherGeographyID => { Groups => { 2 => 'Location' } },
island => { Groups => { 2 => 'Location' } },
islandGroup => { Groups => { 2 => 'Location' } },
locality => { Groups => { 2 => 'Location' } },
locationAccordingTo => { Groups => { 2 => 'Location' } },
locationID => { Groups => { 2 => 'Location' } },
locationRemarks => { Groups => { 2 => 'Location' } },
maximumDepthInMeters => { Groups => { 2 => 'Location' } },
maximumDistanceAboveSurfaceInMeters => { Groups => { 2 => 'Location' } },
maximumElevationInMeters => { Groups => { 2 => 'Location' } },
minimumDepthInMeters => { Groups => { 2 => 'Location' } },
minimumDistanceAboveSurfaceInMeters => { Groups => { 2 => 'Location' } },
minimumElevationInMeters => { Groups => { 2 => 'Location' } },
municipality => { Groups => { 2 => 'Location' } },
pointRadiusSpatialFit => { Groups => { 2 => 'Location' } },
stateProvince => { Groups => { 2 => 'Location' } },
verbatimCoordinateSystem => { Groups => { 2 => 'Location' } },
verbatimCoordinates => { Groups => { 2 => 'Location' } },
verbatimDepth => { Groups => { 2 => 'Location' } },
verbatimElevation => { Groups => { 2 => 'Location' } },
verbatimLatitude => { Groups => { 2 => 'Location' } },
verbatimLocality => { Groups => { 2 => 'Location' } },
verbatimLongitude => { Groups => { 2 => 'Location' } },
verbatimSRS => { Groups => { 2 => 'Location' } },
waterBody => { Groups => { 2 => 'Location' } },
},
},
);
1; #end
As an example, you can use this command to generate an output XMP containing all DWC tags:
exiftool -config ExifTool_dwc.config -xmp-dwc:all=2013 out.xmp
Note that I have left most of the tags as open text format, and have only restricted the formatting of a few tags (ie. the date-format tags).
Just one question: Is the namespace URI really "http://rs.tdwg.org/dwc/index.htm"? The normal convention would be to leave off the "index.htm".
- Phil
Hi Phil,
Thanks a lot. Excuse my ignorance: Does that mean that people can only use the DWC if they load that config file into ExifTool or will it be part of future ExifTool releases?
>>> Just one question: Is the namespace URI really "http://rs.tdwg.org/dwc/index.htm"? The normal convention would be to leave off the "index.htm".
I must admit that I am not too familiar with the conventions. When I used IDImager to define the XMP for the DarwinCore I assumed that the full URL would be the URI. If the convention would be to use ""http://rs.tdwg.org/dwc/", I guess that would better be changed? However, I do have tons of images already with DarwinCore XMP embedded and it would be near impossible to re-write that XMP to all those images...
I am really no expert in all this, but I would definitely prefer not to have to re-write those metadata.
Frank
Hi Frank,
Currently this means that people can only use DWC if they download and apply this config file. I am open to including this in a future ExifTool release if there is enough demand, either by including the config file with the distribution, or by building definitions into the XMP module. Maybe a significant benefit of the latter is that your tags would then be documented in the ExifTool XMP tag name documentation (https://exiftool.org/TagNames/XMP.html), which would give the Darwin Core namespace broader exposure.
But for now, I think it is appropriate to take this one step at a time.
If the config file is activated by renaming it to ".ExifTool_config" and placing it in your home directory, or the directory of the exiftool application, or the current directory, then you don't need to use the -config option on the command line.
About the URI: Definitely do not change it if this is what you have used already. I just wanted to be sure that I had it right.
- Phil
Hi Phil,
>>>Currently this means that people can only use DWC if they download and apply this config file. I am open to including this in a future ExifTool release if there is enough demand, ...
The demand for the DwC will never be huge. By its very nature it is geared mostly towards specialists, i.e., scientists who work with collection data and photos of specimens in natural history collections. I strongly believe in XMP and think that it is a huge advantage, if these kind of metadata can be embedded as XMP inside image files. That is why I am pushing this a bit. I am actually quite surprised that hardly anyone has yet discovered the advantage of embedding Darwin Core Metadata into image files. Huge projects that manage tons of images of species (e.g., Encyclopedia of Life) use things like Flickr Tags to store this kind of information...
This is a bit of a "vicious circle". If you say it should only be included "if there is enough demand", big projects like the Encyclopedia of Life, GBIF, Symbiota, etc. likely will continue to ignore DwC XMP. I think that would be really quite a shame, given its huge potential.
>>>...either by including the config file with the distribution, or by building definitions into the XMP module. Maybe a significant benefit of the latter is that your tags would then be documented in the ExifTool XMP tag name documentation, which would give your namespace broader exposure.
I think building the namespace definitions into the XMP module would be fantastic. Like I said: I believe DwC XMP has a huge potential to become a de facto standard for image management of specimens, yet it won't likely be adopted, if there are no tools out there that support it - and ExifTool is THE most widely distributed XMP tool around...
Cheers,
Frank
Hi Frank,
Understood.
The barrier to inclusion in ExifTool is really a performance issue. ExifTool is pure Perl, so the modules are compiled at run time, and each new namespace that I add incurs a slight performance penalty, which is more significant for namespaces with a large number of tags such as DwC.
But perhaps the solution is to break up the XMP module into smaller sections (currently it is in 3 parts), so only the required part(s) of the module get compiled. If I can do this without a lot of work, I would be in a better position to include large and less common namespaces. Let me think about this.
- Phil
Hi Phil,
OK, I understand the challenge. The DwC is indeed huge and like I said before, we essentially use only a small part of it at the Charles Darwin Foundation (CDF) in the Galapagos.
>>> But perhaps the solution is to break up the XMP module into smaller sections
You will have noticed by now that the DwC is divided into several XMP structures (Record, Event, Occurrence, Location, Geological Context, Identification , Taxon, ResourceRelationship, MeasurementOrFact). Of those, we (CDF) regularly use only some fields in Record, Event, Occurrence, Location, and Identification , Taxon.
So, perhaps ExifTool could read those XMP structures as separate sections?
Then again: even of this smaller set of DwC XMP structures most people will hardly use all fields. The documents that I sent you before summarize pretty well which fields inside those XMP structures we regularly fill in at CDF.
In the DwC XMP there is also some repetitiveness with other XMP: fields like "country" or "province in "dwc::dcterms::Location::country" store essentially the same information as "photoshop::country". So, in a way these fields are duplication and somewhat redundant.
Perhaps focusing on a "slimmed down" version of the most commonly used fields would be another option?
Cheers,
Frank
Hi Frank,
You've convinced me that this is worth the work... I'll see about building the DwC tags into ExifTool in a separate module to avoid the performance penalty. Give me a few days to work on this.
- Phil
Perfect !!!
Hi Frank,
I've got the DarwinCore module added to my working version now (OK, so I was a bit quicker than a couple of days), and have been spending some time trying to tame the wild tag names to make them a bit more user-friendly. By default, ExifTool generates the flattened tag names by combining the structure name with the field name, but in some cases this gives unwieldy names like ResourceRelationshipRelationshipEstablishedDate, so I have tweaked some of the names by hand to shorten them, and some to avoid conflicts with other ExifTool tags.
I did discover one error in the sample you sent: "occurenceRemarks" was misspelt, and should have been "occurrenceRemarks". I found this error by chance, and haven't attempted to validate all of the property names.
Anyway, here is the ExifTool Darwin Core tag name documenation (https://exiftool.org/TagNames/DarwinCore.html) as it currently stands. Let me know if you are unhappy at all with the liberties I have taken with the tag names. To be clear: The ExifTool tag names are NOT what is written to the XMP file -- they are just what the ExifTool interacts with when reading/writing this information. (You may want to read the preamble to the XMP Tags documentation (https://exiftool.org/TagNames/XMP.html) to understand more about how this documentation is organized.)
- Phil
Hi Phil,
wow - that was quick!
Now, I will actually probably need a few days to check thoroughly. I had a quick glance but would like to make sure there are no major glitches ;-)
From what I did see, I do have one question:
You mentioned before that you configured most fields to be strings and few to be dates or time fields. When I put together the XMP using IDImager, I believe I also used string for almost all fields. There are some fields, however, where it makes sense to permit entry of several language options. An example is "dwc:taxon:vernacularName". A vernacular name of an animal could be "Eagle", "Aguila", "Adler" - each language has its own vernacular name of an animal. So, the field type is still string, but you have different alternatives that you can enter. In Idimager the property type would be "alternatives", and the data type "string".
There are a few other fields like that. I believe I sent you a summary of how the fields were defined?
Since I am no expert but simply used IDImager to set this up, I am not sure how important these details are... Do you need me to go through those field types too?
>>> I did discover one error in the sample you sent: "occurenceRemarks" was misspelt, and should have been "occurrenceRemarks"
Good catch. I do hope there are no other spelling errors like that.
Will try to get back to you quickly...
Thanks so much,
Frank
PS: One more thing - IDI distinguishes basically these these three:
Property type
Data type
Edit type
Most fields I have defined as
Property type: variable
Data type: string
Edit type: default
Multi-language fields:
Property type: variable
Data type: string
Edit type: alternative
And very few fields are number fields, e.g. dwc:Event:year
Property type: variable
Data type: integer
Edit type: default
Fields that sometimes need fairly long text entries, like your "occurrenceRemarks" would however be defined as:
Property type: variable
Data type: string
Edit type: memo
The dates would be
Property type: variable
Data type: date
Edit type: default
Hi Frank,
Quote from: Frank Bungartz on January 29, 2013, 11:17:57 AM
There are some fields, however, where it makes sense to permit entry of several language options. An example is "dwc:taxon:vernacularName".
Yes. In ExifTool these are denoted by "lang-alt" in the
Writable column.
All of these types are explained in the preamble to the XMP tags documentation (https://exiftool.org/TagNames/XMP.html) as I mentioned. I have already assigned date, integer and lang-alt types for all of the tags I thought needed them. There is no distinction between a long and short string as you had in IDI. In ExifTool the structures themselves ("struct" type) are also accessible, but in general it is usually easiest to interface with the flattened tags.
- Phil
Hi Phil,
I started too look at this now.
So, let me see if I understand this corrrectly. Your automatically generated tag name would be "ResourceRelationshipRelationshipRemarks" because both the structure tag name and the axtual XMP tag get combined, that means, for the XMP Tag "RelationshipRemarks" that belongs to "ResourceRelationship", you get this unwieldy name and therefore suggest to shorten it to "RelationshipRemarks" as the name displayed in ExifTool, though not changing how this is written as XMP, right?
I think that should be no problem, even for most things where there is no repetition, so "HighestBiostratigraphicZone" for "GeologicalContextHighestBiostratigraphicZone" should be alright too. I think anyone who would use the Darwin Core should understand "biostratigraphic" is a geological term.
What I do not understand though:
Sometimes you have not listed that repetition of the structure tag name in the Notes column.
For example:
EventTime date_ (EventEventTime)
but
EventFieldNotes string_
and NOT
EventFieldNotes string_ (EventEventFieldNotes)
OR
OccurrenceLifeStage string_
OccurrenceDetails string_ (OccurrenceOccurrenceDetails)
Any reason why?
Thanks,
Frank
Hi Frank,
Quote from: Frank Bungartz on January 30, 2013, 06:46:49 PM
therefore suggest to shorten it to "RelationshipRemarks" as the name displayed in ExifTool, though not changing how this is written as XMP, right?
Correct.
QuoteSometimes you have not listed that repetition of the structure tag name in the Notes column.
Right. The combined name is only listed in the Notes if it differs from the Tag Name.
QuoteEventFieldNotes string_
and NOT
EventFieldNotes string_ (EventEventFieldNotes)
This is the "Event" structure "fieldNotes" field. So there is no duplicate "Event" as there would be for the "eventTime" field for example. Take a look at the field names in your structures: Sometimes the structure name appears in the field name, and sometimes it doesn't.
- Phil
Hi Phil,
OK, I got it.
I think the list of XMP tags looks complete and I do think that for your naming conventions it is not a problem that the structure tags are not repeated by ExifTool. I have read through the list twice now and not caught anything that looks out of place.
One problem of course is: I am not a native speaker and glitches like "occurenceRemarks" instead of "occurrenceRemarks". I really do hope I have not overlooked anything else, there are so many fields. It also does not help that I have now read this stuff so many times, that even if some spelling errors were there I probably would not catch them, things that you read and are familiar with, you tend to overlook these glitches.
Still, since most of that stuff was copied from the Darwin Core Website, when I originally put it together for IDimager, I am quite confident this is ready to g now...
Sorry for the delay...
And THANKS again!!! I really do hope the DwC XMP will reach more followers now that ExifTool documents it.
Cheers,
Frank
PS: when is the new version with that documentation coming out? (Just asking because iMatch apparently supports XMP through ExifTool and thus, if ExifTool officially supports DwC XMP, iMatch should be able to read it - and I would really like to see if perhaps I could use iMatch as an alternative to IDimager, now that IDimager is a dead end...)
Hi Frank,
Great, thanks for going over this.
Quote from: Frank Bungartz on February 02, 2013, 10:50:09 PM
PS: when is the new version with that documentation coming out?
It is available now.
- Phil