BirdVoxDetect v0.5: hierarchical consistency, TaxoNet confidence, and β-testing

61 views
Skip to first unread message

Vincent Lostanlen

unread,
Jan 11, 2021, 7:18:40 AM1/11/21
to bir...@googlegroups.com

Dear members of the BirdVox discussion group,

First of all, let me wish you all a Happy New Year 2021!

It is my pleasure to announce the release of the fifth version of the BirdVoxDetect software. I thank the β-testers of BirdVoxDetect v0.4 and everyone who provided feedback since my last email from November 2020, announcing the v0.4 series. Special thanks to β-tester Phil Taylor who helped me chase bugs and corner cases for BirdVoxDetect v0.5 :)

Below my signature is a detailed description of what has changed since BirdVoxDetect v0.4.


You may already install the β version of BirdVoxDetect v0.5 via the Python Package Index

pip install birdvoxdetect==0.5.0b1

Or from the source code: https://github.com/BirdVox/birdvoxdetect

We are actively looking for beta-testers, so please contact me on this discussion group or via the GitHub issue tracker to report feedback.

Once the most urgent feedback is addressed, we'll release BirdVoxDetect v0.5 in its "stable" form.


Sincerely,

Vincent.


I. Hierarchical consistency

In BirdVoxDetect v0.4, we had introduced a new species classifier, named TaxoNet. The gist of TaxoNet is to perform predictions at multiple levels of the avian taxonomy: order, family, and species.

Our numerical experiments on 300 hours of annotated audio have demonstrated that TaxoNet outperforms the classifier of BirdVoxDetect v0.3 (see previous release notes for details). However, we have noted that the output of TaxoNet is occasionally inconsistent across levels. For example, the "family" level may return "Turdidae" (Thrushes) while the "species" level would return "WTSP", i.e. White-Throated Sparrow.

Although infrequent, this lack of consistency may be confusing to users. To address it, we have implemented a new algorithm for hierarchical classification which ensures consistency by design: see https://github.com/BirdVox/birdvoxdetect/pull/73

The key idea is to query TaxoNet in a coarse-to-fine fashion, while progressively narrowing down the list of candidate species corresponding to the detected flight call. Thanks to Justin Salamon for proposing this elegant solution.

We are currently in the process of evaluating the impact of hierarchical consistency on species classification at the full-season scale. We expect this impact to be neutral or slightly favorable. The main reason behind implementing hierarchical consistency wasn't so much classification accuracy than the principle of least surprise from a user's perspective.



II. TaxoNet confidence

We have taken the opportunity of rewriting the classify_species function (see #73) to enrich the information that is stored in the BirdVoxDetect checklist. In addition to textual labels for order, family, and species, we have added three new columns, corresponding to "order confidence", "family confidence" , and "species confidence".

As an example, here's the typical output of BirdVoxDetect v0.5.0b1 with a file from the test suite:

Time (hh:mm:ss)

Detection confidence (%)

Order

Order confidence (%)

Family

Family confidence (%)

Species (4-letter code)

Species confidence (%)

00:00:08.78

70.14839172363280

Passeriforme

92.61552095413210

Turdidae

98.76762628555300

SWTH

99.93199110031130


Compare with v0.4.1:

Time (hh:mm:ss) Species (4-letter code) Family    Order         Confidence (%)
00:00:08.78     SWTH                    Turdidae  Passeriforme  70.14839

These values of confidence all range between 50% and 100% and provide insight on the prediction of TaxoNet. Users should be warned, however, that the percentages are not calibrated, and thus do not come with any probabilistic interpretation. Rather, they are simply the output values of the sigmoid units in TaxoNet. This is unlike "Detection confidence (%)", which is carefully calibrated to match the precision of the detector (on a precision--recall curve).

Do not hesitate to contact me for more details about these new columns in the checklist. I'd be happy to help.

Wim van Dam

unread,
Jan 11, 2021, 1:26:58 PM1/11/21
to Vincent Lostanlen, bir...@googlegroups.com
Thank you for humoring this (likely) elementary question. 

When I try to use the latest version I get the error message: 
-bash: birdvoxdetect: command not found

On macOS Big Sur I have installed things with the following sequence:
conda create -n birdvoxdetect python=3.8
conda activate birdvoxdetect
pip install birdvoxdetect==0.5.0b1
This all went well, so I must be forgetting something simple. 

My PATH is: 
$ echo $PATH
/Users/wimvandam/opt/anaconda3/envs/birdvoxdetect/bin:
/Users/wimvandam/opt/anaconda3/condabin:/anaconda3/bin:
/Library/Frameworks/Python.framework/Versions/3.6/bin:
/Library/Frameworks/Python.framework/Versions/3.6/bin:
/Library/Frameworks/Python.framework/Versions/3.6/bin:
/usr/local/bin:
/usr/bin:
/bin:
/usr/sbin:
/sbin:
/Library/TeX/texbin:
/Library/Frameworks/Mono.framework/Versions/Current/Commands:
/Users/wimvandam/opt/anaconda3/envs/birdvoxdetect/bin/python

Thanks in advance. 

Wim van Dam
Solvang, CA, USA

--
You received this message because you are subscribed to the Google Groups "birdvox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to birdvox+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/birdvox/ca991f15-5d37-876a-2f1b-4ecfbb73c8bc%40ls2n.fr.
For more options, visit https://groups.google.com/d/optout.

Wim van Dam

unread,
Jan 23, 2021, 2:02:38 PM1/23/21
to birdvox
To answer my own question: my problem disappeared when I switched from python 3.8 to 3.6. 

Wim van Dam
Solvang, CA

Kevin Tolan

unread,
Jan 24, 2021, 5:49:14 PM1/24/21
to birdvox
Similar to Wim, I also had trouble using 3.8 (I kept getting the error code "Command errored out with exit status 1"). When I switched to 3.6 it worked fine. 
Reply all
Reply to author
Forward
0 new messages