Training with multiple flies and per_frame features created outside of JAABA

[ARPITA MONDAL]

Hi,

We are using sleap to track multiple flies keeping their identities for the whole experiment.

Then we create multiple features including 1-fly features (e.g. Fly1_forwardVelocity) and features that involve 2 flies (e.g. Dist_fly1_fly5).

We like to ctrain JAABA classifiers to detect behaviors such as chasing. 

Is there a way to do it? I guess that there are three issues:
1. How can I choose during training in the JAABA GUI which fly chases which fly?
2. Is there a way to name the per-frame features such that JAABA will recognize which frames are relevant to use for training given that fly x is chasing fly y

3. Once trained, can JAABA generalize? Meaning find chasing between any pair of flies?

Thanks a lot - hope my description was clear.

Arpita  

[Mayank Kabra]

Hi Arpita,

Chasing is one of the common behaviors detected using JAABA, and in fact, it is the behavior we demonstrate in the JAABA documentation video. Have you checked out the documentation and the video (https://jaaba.sourceforge.net/GettingStarted.html#DemoVideo)?

As for adding custom animal type with your own per-frame features, it was answered on the mailing list (https://groups.google.com/g/jaaba/c/puc5p_G8qRU/m/1GS-WY9OAgAJ).

Mayank

--
You received this message because you are subscribed to the Google Groups "JAABA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jaaba+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/jaaba/1d005fb5-9d46-48bf-a90c-390ce996d807n%40googlegroups.com.

[Dudi Deutsch]

Hi Mayank,

Thanks for the reply! Hope you are doing well these days.

We are aware that chasing is commonly classified with JAABA. I used JAABA myself to classify chasing before. We also know that we can use per-frame features, which is also something I already used.

What is not clear to us is how (or if) we can use our own per-frame features with a movie that has multiple flies, and when we want, for the same movie, to give examples of different pairs, such that (this is the critical part) JAABA will only use the per-frame features that are relevant for those two flies. So, for example, if we have the per-frame feature fly1fly6Dist (being the distance between fly1 and fly6), we don't want this feature to be used when we train on, for example, fly4 and fly2. Then, of course, for the inference - it needs to apply the classification for all the possible pairs, so instead of creating a binary output size (1,nFrames) as in the case of 2 flies, the output will be size (M,M,nFrames), with M being the number of flies (I guess that the diagonal will be something like NaN).

We can go around this by only looking at one pair at a time and then running the inference over all possible pairs in a loop - this is our current plan, but we were wondering if the option to do it for all the flies at once exists.

Hope it was clear :)

and thanks again for the support!

Dudi and Arpita  

To view this discussion visit https://groups.google.com/d/msgid/jaaba/CAGgnnKzWe6ZyWrtwLnYNKLxdxMJMpQZ9aQFRh7PKTvn8QPjZVA%40mail.gmail.com.

[Mayank Kabra]

Ah, ok!

In the per-frame features that are designed in JAABA, we circumvented this by using dnose2ell, dnose2ell_angle_30tomin30, and other social per-frame features that look at the distances from the current fly to the closest fly -- dnose2ell, or the closest fly within +/- 30 degrees cone of the current fly head -- dnose2ell_angle_30tomin30 and so on. We computed this distance irrespective of the closest fly's (the fly being chased) identity, which was sufficient for detecting chase. Once you have detected a chase using these features, you can post-hoc try to match the fly doing the chasing with the fly being chased by finding the closest fly or the closest fly in front of the current fly. If this post-hoc identity assignment gets confused and selects a fly that is not being chased, you can try to train a "being chased" detector, and select pairs that are "chasing" and "being chased".

Of course, this is not fool-proof compared to looking at all the pairs during the inference, but the accuracy is likely sufficient for most practical purposes. It'll depend on how often chasing involves more than two flies. You can easily test if this works using the existing JAABA animal type and per-frame features. 

HTH,

Mayank