Good-day,
I was in touch with Mario Westphal at IMatch regarding the possibility of indexing bat call .wav files via the widely used (amongst bat folks) GUANO metadata format (see here: https://github.com/riggsd/guano-spec/blob/master/guano_specification.md). He suggested I contact you to see if this type of metadata might be included in the exiftool. I do work with bats and have multiple bat call recordings begging for indexing, but am only a GUANO user and unassociated with its development. It is used by several bat call analysis programs and often includes, not only bat ID info. and field notes, but also time/date, GPS location, detector ID and similar data.
Thank you,
Conrad.
GUANO - Grand Unified Acoustic Notation Ontology
I love it
Can you link/share some sample files?
Phil will have to comment on this but I suspect that the weather is especially good in his area right now and he's usually out birding in good weather.
Quote from: StarGeek on May 15, 2023, 10:59:36 AMPhil will have to comment on this but I suspect that the weather is especially good in his area right now and he's usually out birding in good weather.
Correct. :) Nice weather AND spring bird migration... Spending all of my time outdoors recently.
Sample files would be necessary so we can see what GUANO is all about.
- Phil
Sorry, for the slowness - good field weather on my end and was out chasing insects and doing other fieldwork. I have attached an audio file from a bat detector. This one might not actually have a bat on it, but it does have the GUANO metadata describing date, detector etc.
Let me know if you need something more, and I'll try to respond more rapidly.
Thanks for thinking about this!
Conrad.us_2023-04-13_20-45-02.wav
It appears to be simple text in the Unknown_guan tag and can be extracted with the -u (-unknown) option (https://exiftool.org/exiftool_pod.html#u--unknown) and the -b (-binary) option (https://exiftool.org/exiftool_pod.html#b---b--binary---binary).
C:\>exiftool -G1 -a -s -e --file:all Y:\!temp\aaaa\us_2023-04-13_20-45-02.wav -u -b -Unknown_guan
GUANO|Version: 1.0
Make: Titley Scientific
Model: Anabat Express FS
Firmware Version: Sonic v2.4.5.23030
Serial: 660081
Timestamp: 2023-04-13T20:45:02
Loc Position: 42.360092 -73.592216
Loc Elevation: 203.3
Temperature Int: 26.0
Filter HP: 10.00
Anabat|Battery voltage: 5.06
Anabat|Microphone: Ultrasonic
Anabat|Activation: Triggered
Anabat|Zc Sensitivity: 18
Anabat|Trigger min freq: 15000
Anabat|Trigger max freq: 250000
Anabat|Min event: 2
Anabat|Trigger Window: 2000
Anabat|Maximum File Duration: 10000
SB|Version: 4.4.5
SB|Species Auto ID: noID
Species Auto ID: noID
Species Manual ID:
Samplerate: 500000
Length: 0.0
TE:
SB|Filter HP: 0
Note:
SB|Region: Northeastern NA
SB|Classifier: northeastern US
There's also 8 bytes in Unknown_LIST_gbin, though that's not as obvious as to what it is.
Thanks. Yep, that looks like the right data. I am a metadata user not a programmer, if I just pass along your comments to Mario of IMatch, is that all he needs to know in order to make IMatch be able to read those data? I think he said that IMatch couldn't read it (I sent him samples), but that he would be happy to incorporate it into IMatch if the exiftool included it.
Conrad.
Mario is on here as well, but with a different name.
I wasn't sure if IMatch could support custom tags in exiftool, but I found it on this page (https://www.photools.com/help/imatch/tech_exiftool.htm). Editing the imatch_et.config file looks like it would include custom tags, though I'm not sure how you would do so.
I don't think off hand he would want to write a definition for exiftool himself. I haven't tried to create a custom tag for the RIFF group so I'm not sure what else is required, as the example config file (https://exiftool.org/config.html) doesn't have an example for RIFF.
I reached out to Mario and told him that, as he suggested, I had started this thread and asked if maybe he could pop in to review it and give feedback. Let's see if he has any thoughts. Thanks again.
I'll add a new "Guano" tag in ExifTool 12.63:
> exiftool ~/Desktop/us_2023-04-13_20-45-02.wav -guano -b
GUANO|Version: 1.0
Make: Titley Scientific
Model: Anabat Express FS
Firmware Version: Sonic v2.4.5.23030
Serial: 660081
Timestamp: 2023-04-13T20:45:02
Loc Position: 42.360092 -73.592216
Loc Elevation: 203.3
Temperature Int: 26.0
Filter HP: 10.00
Anabat|Battery voltage: 5.06
Anabat|Microphone: Ultrasonic
Anabat|Activation: Triggered
Anabat|Zc Sensitivity: 18
Anabat|Trigger min freq: 15000
Anabat|Trigger max freq: 250000
Anabat|Min event: 2
Anabat|Trigger Window: 2000
Anabat|Maximum File Duration: 10000
SB|Version: 4.4.5
SB|Species Auto ID: noID
Species Auto ID: noID
Species Manual ID:
Samplerate: 500000
Length: 0.0
TE:
SB|Filter HP: 0
Note:
SB|Region: Northeastern NA
SB|Classifier: northeastern US
- Phil
P.S. @StarGeek: all other options except -u in your command are ignored when using -b
Quote from: Phil Harvey on May 20, 2023, 01:51:38 PMP.S. @StarGeek: all other options except -u in your command are ignored when using -b
Yeah, but the start of that is my text replacement string for viewing data. I just type "et(enter)" on the command line and it's expanded to
exiftool -G1 -a -s -e --file:all All I need to do is add the filepath, usually from control+v, and any other options I need.
Thanks so much. I know bat biologists are a niche audience, but the ability to organize and find calls easily by their GUANO tags is really helpful! I look forward to 12.63.
Just so you know, this will return all the data in a block. If you need to extract the individual lines, that can be done with user-defined.
Quote from: Phil Harvey on May 20, 2023, 01:51:38 PMI'll add a new "Guano" tag in ExifTool 12.63:
That's very cool, thank you :)
When the new tag comes in and the user rescans the files, the Guano data will be imported by IMatch.
This means it can be displayed, searched, accessed via variables etc., like any other tag value.
But as a
block of text, as shown in your post above.
I have never written a user-defined tag group for ExifTool.
And looking at the Guano metadata specification I'd guess this would take some time.
I think, for now, having the data as a block of text is sufficient.
If I get many requests from biologists for better GUANO support, I will look into this again.
Thanks everybody. This has definitely gotten us farther along. Once I can play with this sorting in IMatch, I'll try to evaluate the utility of a 'custom tag group', although if you all describe that as a complex undertaking, I'm sure it will be beyond my personal abilities.
I am, however, going to reach out to some fellow bat biologists to better understand how broad interest might be in such a tool. As background, current bat monitoring can produce 1000s of call recordings, each automatically tagged with time, temperature, date, location and detector serial number. These calls can then be run through auto-ID software which adds purported species tags to the metadata. Calls should then be hand vetted, and most such vetting software has a way of adding manual IDs together with commentary on habitat or social interactions... you can thus see how being able to manage that metadata efficiently could be quite helpful.
Quote from: Mac2 on May 22, 2023, 05:05:57 AMI have never written a user-defined tag group for ExifTool.
And looking at the Guano metadata specification I'd guess this would take some time.
Don't worry, I already have one written. It's basically just a regex match against the block of text. Then copy/paste that for every line.
Cool. Once the new EXIFtool is out and Mario can include it in the next IMatch iteration, we'll see where we are at. Thanks again.
Quote from: StarGeek on May 22, 2023, 10:56:45 AMDon't worry, I already have one written. It's basically just a regex match against the block of text. Then copy/paste that for every line.
Awesome. I'll give this a try with the sample images I have once the .63 release is out.
May take some time, 150% busy with the final stage of IMatch 2023 and all that's involved in shipping the new release.
Quote from: Conrad Vispo on May 22, 2023, 08:37:18 PMCool. Once the new EXIFtool is out and Mario can include it in the next IMatch iteration, we'll see where we are at.
I think all you would have to do is update the version of exiftool that IMatch is using in
C:\Program Files\photools.com\imatch6, and then Preferences->Metadata 2->Tag Manager, though I could be wrong
Here's the config file. From the example file, I took the text from before the colon as the name with a "Guano" prefix. Everything after the colon and spaces is the value.
#------------------------------------------------------------------------------
# File: Guano.config
#
# Description: User-defined Composite tag definitions to extract individual entries
# from the RIFF:Guano text block.
#
# Requires exiftool version 12.63+
#
# See GUANO - Grand Unified Acoustic Notation Ontology
# https://github.com/riggsd/guano-spec/blob/master/guano_specification.md
#
# The code itself is simply a regex match. If other entries are needed,
# then all that needs to be done is to copy/paste one of the blocks, change
# the name and edit the regex
#
# Examples:
# This will list the Make and Model from the Guano text block, if they exist
# exiftool -config Guano.config -GuanoModel -GuanoMake /path/to/files/
#
# Lists all the Guano entries in the Guano text block
# exiftool -config Guano.config -Guano* /path/to/files/
#
# Revisions: 2023/06/09 - Bryan K. Williams (aka StarGeek) Created
#------------------------------------------------------------------------------
%Image::ExifTool::UserDefined = (
'Image::ExifTool::Composite' => {
GuanoVersion => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/GUANO\|Version:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoMake => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Make:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoModel => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Model:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoFirmwareVersion => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Firmware Version:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoSerial => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Serial:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoTimestamp => {
Require => {
0 => 'Guano',
},
Groups => { 2 => 'Time' },
PrintConv => '$self->ConvertDateTime($val)',
ValueConv => q{
return (($val[0]=~m/Timestamp:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoLocPosition => {
Require => {
0 => 'Guano',
},
Groups => { 2 => 'Location' },
ValueConv => q{
return (($val[0]=~m/Loc Position:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoLocElevation => {
Require => {
0 => 'Guano',
},
Groups => { 2 => 'Location' },
ValueConv => q{
return (($val[0]=~m/Loc Elevation:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoTemperatureInt => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Temperature Int:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoFilterHP => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/^Filter HP:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoAnabatBatteryVoltage => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Anabat\|Battery voltage:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoAnabatMicrophone => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Anabat\|Microphone:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoAnabatActivation => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Anabat\|Activation:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoAnabatZcSensitivity => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Anabat\|Zc Sensitivity:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoAnabatTriggerMinFreq => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Anabat\|Trigger min freq:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoAnabatTriggerMaxFreq => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Anabat\|Trigger max freq:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoAnabatMinEvent => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Anabat\|Min event:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoAnabatTriggerWindow => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Anabat\|Trigger Window:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoAnabatMaximumFileDuration => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Anabat\|Maximum File Duration:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoSBVersion => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/SB\|Version:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoSBSpeciesAutoID => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/SB\|Species Auto ID:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoSpeciesAutoID => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Species Auto ID:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoSpeciesManualID => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Species Manual ID:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoSampleRate => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Samplerate:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoLength => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Length:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoTE => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/TE:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoSBFilterHP => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/SB\|Filter HP:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoNote => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/Note:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoSBRegion => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/SB\|Region:\s+(.*)/m)) ? $1 : undef;
},
},
GuanoSBClassifier => {
Require => {
0 => 'Guano',
},
ValueConv => q{
return (($val[0]=~m/SB\|Classifier:\s+(.*)/m)) ? $1 : undef;
},
},
},
);
#------------------------------------------------------------------------------
1; #end
Example output
C:\>exiftool -config Guano.config -G1 -a -s -Guano* us_2023-04-13_20-45-02.wav
[RIFF] Guano : GUANO|Version: 1.0.Make: Titley Scientific.Model: Anabat Express FS.Firmware Version: Sonic v2.4.5.23030.Serial: 660081.Timestamp: 2023-04-13T20:45:02.Loc Position: 42.360092 -73.592216.Loc Elevation: 203.3.Temperature Int: 26.0.Filter HP: 10.00.Anabat|Battery voltage: 5.06.Anabat|Microphone: Ultrasonic.Anabat|Activation: Triggered.Anabat|Zc Sensitivity: 18.Anabat|Trigger min freq: 15000.Anabat|Trigger max freq: 250000.Anabat|Min event: 2.Anabat|Trigger Window: 2000.Anabat|Maximum File Duration: 10000.SB|Version: 4.4.5.SB|Species Auto ID: noID.Species Auto ID: noID.Species Manual ID: .Samplerate: 500000.Length: 0.0.TE: .SB|Filter HP: 0.Note: .SB|Region: Northeastern NA.SB|Classifier: northeastern US..
[Composite] GuanoAnabatActivation : Triggered
[Composite] GuanoAnabatBatteryVoltage : 5.06
[Composite] GuanoAnabatMaximumFileDuration : 10000
[Composite] GuanoAnabatMicrophone : Ultrasonic
[Composite] GuanoAnabatMinEvent : 2
[Composite] GuanoAnabatTriggerMaxFreq : 250000
[Composite] GuanoAnabatTriggerMinFreq : 15000
[Composite] GuanoAnabatTriggerWindow : 2000
[Composite] GuanoAnabatZcSensitivity : 18
[Composite] GuanoFilterHP : 10.00
[Composite] GuanoFirmwareVersion : Sonic v2.4.5.23030
[Composite] GuanoLength : 0.0
[Composite] GuanoLocElevation : 203.3
[Composite] GuanoLocPosition : 42.360092 -73.592216
[Composite] GuanoMake : Titley Scientific
[Composite] GuanoModel : Anabat Express FS
[Composite] GuanoNote : SB|Region: Northeastern NA
[Composite] GuanoSBClassifier : northeastern US
[Composite] GuanoSBFilterHP : 0
[Composite] GuanoSBRegion : Northeastern NA
[Composite] GuanoSBSpeciesAutoID : noID
[Composite] GuanoSBVersion : 4.4.5
[Composite] GuanoSampleRate : 500000
[Composite] GuanoSerial : 660081
[Composite] GuanoSpeciesAutoID : noID
[Composite] GuanoSpeciesManualID : Samplerate: 500000
[Composite] GuanoTE : SB|Filter HP: 0
[Composite] GuanoTemperatureInt : 26.0
[Composite] GuanoTimestamp : 2023-04-13T20:45:02
[Composite] GuanoVersion : 1.0
Hi, StarGeek
thanks for the support here and the configuration file.
I've just installed the 12.63 edition of ExifTool and ran it with the two sample videos provided to me by Conrad.
ExifTool imports the data into the RIFF::Main\guan\Guano tag and IMatch imports this data without problems.
As far as I know, the -config option can be used only once, and IMatch already uses this to specify its own config file, with some custom IPTC tags and options.
I have copied your configuration setup into the default config file IMatch uses and it works perfect.
Do I have your permission to include your configuration with IMatch?
I have copied a link to this thread and your Revisions info with your name and community tag into the IMatch ExifTool config file. Let me know if this is OK for you.
This is how IMatch (https://www.photools.com/imatch/) displays the GUANO metadata in the Metadata Panel:
Image1.jpg
I'll add guano.config to the official ExifTool distribution.
Thanks StarGeek.
- Phil
Quote from: Mac2 on June 11, 2023, 09:06:08 AMDo I have your permission to include your configuration with IMatch?
Anything I post here I consider to be public domain, to be freely spread, edited, folded, spindled, mutilated as desired.
Thanks!
Quote from: Phil Harvey on June 11, 2023, 09:33:55 AMI'll add guano.config to the official ExifTool distribution.
One quick edit I feel should be done first, but I can't really figure it out on my own.
The Loc Position is a GPS coordinate, Lat & Long separated by a space.
GuanoLocPosition => {
Require => {
0 => 'Guano',
},
Groups => { 2 => 'Location' },
ValueConv => q{
return (($val[0]=~m/Loc Position:\s+(.*)/m)) ? $1 : undef;
},
},How would this be changed so that the numbers are affected by the
-c (
-coordFormat) option (https://exiftool.org/exiftool_pod.html#c-FMT--coordFormat)? Do they need to be separate tags first?
I think this PrintConv should do it.
PrintConv => q{
require Image::ExifTool::GPS;
my @v = split ' ', $val;
return Image::ExifTool::GPS::ToDMS($self, $v[0], 1, "N") . ', ' .
Image::ExifTool::GPS::ToDMS($self, $v[1], 1, "E");
},
Attached is a slightly cleaned up config file with this addition.
- Phil
Mario did a great job of integrating this into IMatch. I had one follow-up question - ExifTool reports the GUANO metadata as read-only, and so they can't be edited within IMatch, is this necessary or could it be opened in an editable way? The data can of course be edited in the bat-call specific software, but being able to also edit it from within IMatch would add flexibility.
Thanks,
Conrad.
The original file was a wav file and exiftool cannot edit wav files
Got it. Thanks again.