Using WI in an automated bird species detector

[Thorsten von Eicken]

I'm working on a project that attempts to identify unusual bird species coming to a bird bath camera trap. I have something working and I'm wondering whether participating in WI makes sense. So I have a bunch of questions :-).

In brief, what I have so far is a networked camera pointed at a bird bath. The video feed is processed by an open source software package called Frigate that detects motion and performs AI object detection using a Coral EdgeTPU accelerator. Video images that have birds detected are further processed using a custom classification model that I put together based on the Caltech birds dataset. Here's a full-frame shot from the camera:

Here are some sample classification outputs:

(These images are the exact ones used as input to the classification model. And yes, it's not 100% accurate :-) )

I have a couple of other cameras I'm getting ready to deploy to capture some of the mammalian wildlife roaming around here.

Now to the questions I have:

- I believe it would be relatively simple for me to upload the images I use for classification to WI and have WI classify them, is that acceptable use? (I realize your bird species coverage is currently very limited.)

- I will be tagging images that are classified incorrectly in order to improve my model. Is there a requirement to manually tag/correct images classified by WI?

- Overall, I don't quite understand how WI manages the training input to the AI model. Do you expect that the users of WI provide this input by tagging images? If so, how do you ensure the accuracy of the tagging? It seems to me that other projects, such as eBird or iNaturalist actually spend a tremendous effort on this aspect alone.

- I'm an amateur birder and researcher interested in all this from several angles, the project I've described is my own, not part of some non-profit or other organization, can I use WI in the first place?

Thanks much!

[Wildlife Insights]

Hi Thorsten,

Thanks for getting in contact with us and sharing details on your setup! In short - yes! You can definitely use Wildlife Insights. The platform is open to anyone collecting camera trap images, regardless if you’re part of a larger organization. You should be aware of a few caveats though:

• We currently only accept image files though we’re actively discussing support for video.
• It sounds like you’ve converted some of the video to images already - these images should be very easy to upload to Wildlife Insights. We have a lot of resources on our website to guide you through the upload process.
• After the images are uploaded to WI, you will need to manually correct or approve the classifications. We rely on these reviewed images & IDs to feed into our training model. Additionally, we do have processes in place to verify that the datasets we’re using in our training model meet our gold standard, so the data going into any one of our training datasets is actually a subset of data in Wildlife Insights. 

We’d love to have you contribute to Wildlife Insights! Let me know if you have any other questions. Otherwise, feel free to sign up for an account at https://app.wildlifeinsights.org/join. 

Nicole

Wildlife Insights team

[Thorsten von Eicken]

Thanks for the info Nicole! :-) You are correct that I'm already extracting images from the video. I have more questions:

- I end up detecting a short video clip with an animal and then extracting a bunch of frames from that clip. The classification is really for the clip, not for each individual image. Is that how it currently works with the bursts?

- Sometimes the clips end up with different birds, so I end up with a burst that may have 5 images from a gold finch and then 6 images from a jay. How does WI handle that? (Other than expecting me to fix/disqualify the burst.)

- What is the minimum resolution you expect for images usable for training? I'm capturing XGA or HD video but then extract images that are are a little larger than the bounding box of the detected animal and then further scale to roughly 300x300 to feed into my classification model (exact size depends on the model). So the resolution tends to be relatively low, but the animal does fill the frame.

[Wildlife Insights]

You can create 2 types of projects in Wildlife Insights: 

1. Image projects
   • With image projects,  each image is tagged with exactly what is seen in that individual image. One image = 1 record
   • Bursts are simply a tool to group images to help process them more quickly, since images taken in within some short period of time have a high likelihood of capturing the same animal or object. 
   • Let's say there are 10 images grouped in a burst. 4 images show only a gold finch, 5 images show only a jay and 1 image shows both a gold finch and a jay. You would tag the 4 images with 1 goldfinch, 5 images with 1 jay and the last image would be tagged with both 1 goldfinch and 1 jay
   • This tagging process may be more time consuming than tagging in sequences
2. Sequence projects
   • In a sequence project, the tag is applied to the entire sequence. One sequence (which could contain many images) = 1 record.
   • Sequences are set at a 60 second interval in Wildlife Insights by default.
   • Let's say you have the same scenario as mentioned above. With a sequence project you would tag the entire sequence with 1 goldfinch and 1 jay.
   • A potential downside of creating a sequence project is that the images from sequence projects currently can't be used to retrain the AI model.

I've seen as small as ~640x480 uploaded to Wildlife Insights but I'll need to get back to you on minimum requirements. Hope this helps!

Nicole

Wildlife Insights team