Guano Metadata

[Conrad Vispo]

Good-day,

I was in touch with Mario Westphal at IMatch regarding the possibility of indexing bat call .wav files via the widely used (amongst bat folks) GUANO metadata format (see here: https://github.com/riggsd/guano-spec/blob/master/guano_specification.md). He suggested I contact you to see if this type of metadata might be included in the exiftool. I do work with bats and have multiple bat call recordings begging for indexing, but am only a GUANO user and unassociated with its development. It is used by several bat call analysis programs and often includes, not only bat ID info. and field notes, but also time/date, GPS location, detector ID and similar data.

Thank you,

Conrad.

[StarGeek]

GUANO - Grand Unified Acoustic Notation Ontology
I love it

Can you link/share some sample files?

Phil will have to comment on this but I suspect that the weather is especially good in his area right now and he's usually out birding in good weather.

[Phil Harvey]

Phil will have to comment on this but I suspect that the weather is especially good in his area right now and he's usually out birding in good weather.

Correct.   Nice weather AND spring bird migration... Spending all of my time outdoors recently.

Sample files would be necessary so we can see what GUANO is all about.

- Phil

[Conrad Vispo]

Sorry, for the slowness - good field weather on my end and was out chasing insects and doing other fieldwork. I have attached an audio file from a bat detector. This one might not actually have a bat on it, but it does have the GUANO metadata describing date, detector etc.

Let me know if you need something more, and I'll try to respond more rapidly.

Thanks for thinking about this!

Conrad.

[StarGeek]

It appears to be simple text in the Unknown_guan tag and can be extracted with the -u (-unknown) option and the -b (-binary) option.

Code Select Expand

C:\>exiftool -G1 -a -s -e --file:all Y:\!temp\aaaa\us_2023-04-13_20-45-02.wav -u -b -Unknown_guan
GUANO|Version:  1.0
Make:  Titley Scientific
Model:  Anabat Express FS
Firmware Version:  Sonic v2.4.5.23030
Serial:  660081
Timestamp:  2023-04-13T20:45:02
Loc Position:  42.360092 -73.592216
Loc Elevation:  203.3
Temperature Int:  26.0
Filter HP:  10.00
Anabat|Battery voltage:  5.06
Anabat|Microphone:  Ultrasonic
Anabat|Activation:  Triggered
Anabat|Zc Sensitivity:  18
Anabat|Trigger min freq:  15000
Anabat|Trigger max freq:  250000
Anabat|Min event:  2
Anabat|Trigger Window:  2000
Anabat|Maximum File Duration:  10000
SB|Version:  4.4.5
SB|Species Auto ID:  noID
Species Auto ID:  noID
Species Manual ID: 
Samplerate:  500000
Length:  0.0
TE: 
SB|Filter HP:  0
Note: 
SB|Region:  Northeastern NA
SB|Classifier:  northeastern US
There's also 8 bytes in Unknown_LIST_gbin, though that's not as obvious as to what it is.

[Conrad Vispo]

Thanks. Yep, that looks like the right data. I am a metadata user not a programmer, if I just pass along your comments to Mario of IMatch, is that all he needs to know in order to make IMatch be able to read those data? I think he said that IMatch couldn't read it (I sent him samples), but that he would be happy to incorporate it into IMatch if the exiftool included it.

Conrad.

[StarGeek]

Mario is on here as well, but with a different name.

I wasn't sure if IMatch could support custom tags in exiftool, but I found it on this page.  Editing the imatch_et.config file looks like it would include custom tags, though I'm not sure how you would do so.

I don't think off hand he would want to write a definition for exiftool himself.  I haven't tried to create a custom tag for the RIFF group so I'm not sure what else is required, as the example config file doesn't have an example for RIFF.

[Conrad Vispo]

I reached out to Mario and told him that, as he suggested, I had started this thread and asked if maybe he could pop in to review it and give feedback. Let's see if he has any thoughts. Thanks again.

[Phil Harvey]

I'll add a new "Guano" tag in ExifTool 12.63:

Code Select Expand

> exiftool ~/Desktop/us_2023-04-13_20-45-02.wav -guano -b
GUANO|Version:  1.0
Make:  Titley Scientific
Model:  Anabat Express FS
Firmware Version:  Sonic v2.4.5.23030
Serial:  660081
Timestamp:  2023-04-13T20:45:02
Loc Position:  42.360092 -73.592216
Loc Elevation:  203.3
Temperature Int:  26.0
Filter HP:  10.00
Anabat|Battery voltage:  5.06
Anabat|Microphone:  Ultrasonic
Anabat|Activation:  Triggered
Anabat|Zc Sensitivity:  18
Anabat|Trigger min freq:  15000
Anabat|Trigger max freq:  250000
Anabat|Min event:  2
Anabat|Trigger Window:  2000
Anabat|Maximum File Duration:  10000
SB|Version:  4.4.5
SB|Species Auto ID:  noID
Species Auto ID:  noID
Species Manual ID: 
Samplerate:  500000
Length:  0.0
TE: 
SB|Filter HP:  0
Note: 
SB|Region:  Northeastern NA
SB|Classifier:  northeastern US
- Phil

P.S. @StarGeek: all other options except -u in your command are ignored when using -b

[StarGeek]

P.S. @StarGeek: all other options except -u in your command are ignored when using -b

Yeah, but the start of that is my text replacement string for viewing data. I just type "et(enter)" on the command line and it's expanded to exiftool -G1 -a -s -e --file:all   All I need to do is add the filepath, usually from control+v, and any other options I need.

[Conrad Vispo]

Thanks so much. I know bat biologists are a niche audience, but the ability to organize and find calls easily by their GUANO tags is really helpful! I look forward to 12.63.

[StarGeek]

Just so you know, this will return all the data in a block.  If you need to extract the individual lines, that can be done with user-defined.

[Mac2]

I'll add a new "Guano" tag in ExifTool 12.63:

That's very cool, thank you

When the new tag comes in and the user rescans the files, the Guano data will be imported by IMatch.
This means it can be displayed, searched, accessed via variables etc., like any other tag value.
But as a block of text, as shown in your post above.

I have never written a user-defined tag group for ExifTool.
And looking at the Guano metadata specification I'd guess this would take some time.
I think, for now, having the data as a block of text is sufficient.

If I get many requests from biologists for better GUANO support, I will look into this again.

[Conrad Vispo]

Thanks everybody. This has definitely gotten us farther along. Once I can play with this sorting in IMatch, I'll try to evaluate the utility of a 'custom tag group', although if you all describe that as a complex undertaking, I'm sure it will be beyond my personal abilities.

I am, however, going to reach out to some fellow bat biologists to better understand how broad interest might be in such a tool. As background, current bat monitoring can produce 1000s of call recordings, each automatically tagged with time, temperature, date, location and detector serial number. These calls can then be run through auto-ID software which adds purported species tags to the metadata. Calls should then be hand vetted, and most such vetting software has a way of adding manual IDs together with commentary on habitat or social interactions... you can thus see how being able to manage that metadata efficiently could be quite helpful.

[StarGeek]

I have never written a user-defined tag group for ExifTool.
And looking at the Guano metadata specification I'd guess this would take some time.

Don't worry, I already have one written.  It's basically just a regex match against the block of text.  Then copy/paste that for every line.