simple way to access crossing blob data

[Samuel Snow]

Hi Friends!

I am working with 3D neural-net-based tracking data on medium-sized groups of individuals (10). The detections can be messy at times so I am using idtracker on the top-down videos to validate the IDs (inspired by this recent paper). In general, this means I have 3D traces that run through most of the top-down crossing blobs, so I am primarily using idtracker as the groundtruth identity when the blobs are all well-defined individuals. In fact, discrepancies between the 3D centers of mass and idtracker's interpolated cnetroids inside crossing blobs are causing issues and even introducing new id switches! 

I am wondering if there is a way to use the output data to flag when a centroid is inside a crossing blob so I can drop them from my trajectories. 

The "with gaps" trajectory data is no good (if I understand correctly) because it still includes instances of single centroids inside crossing blobs that are poorly centered on an individual. Does the list_of_blobs.pickle file do this? Alternatively, do you have any advice for cleaning up the way the Validator outputs interpolated data?

Thanks so much!!

-Sam 

[Samuel Snow]

Update: I was initially incorrect about the cause of the issue. The problem stems from the fact that sometimes the crossing detector calls a crossing blob an individual blob in cases when one animal largely occludes the other. When this happens in the middle of a crossing event, there can sporadically be a few or a single "good" points for one of the animals, but the centroid will often be pretty far off the mark and cause problems. I had previously been filtering by nans for id probability, but this ony works when the blobs are correctly defined as crossing blobs. 

So I suppose my new question is: is there a way to tweak the parameters of the crossings detector to make it slightly more conservative? Or, perhaps, a way to manually force crossing blobs in the validator?

Thanks again, and happy holidays!

[idtrackerai]

Hi Sam,

Very interesting setup!

Crossings are detected by a self-supervised trained model. This model is feeded with images of very obvious crossings and very obvious individuals (based on the blob size and the overlap with other blobs in consecutive frames). Then, this trained model is used to classify all blobs in the video. When one animal (fish?) completely occludes another, they looks like one single individual and the crossing detector assigns them with the "individual" label.

There's no easy fix for this. There are some advanced parameters to tweak the obvious crossigns/individual blobs clasification used to train the model but not to directly affect its final predictions.

In the Validator you cannot change the individual/crossing classification. What you can do is to interpolare the trajectory gaps and then generate the desired trajectories in the following way:

When you add a centroid using the interpolator in the Validator, this centroid is assigned to the closest blob (individual or crossing, it does not matter) in its specific frame (all centroids are always assigned to blobs). Then, this blob (if it already contained another centroid), now contains 2 centroids. With this, you can write your own script to generate trajectory of, for example, only blobs that have exactly one centroid (mainly, individual blobs). The following script (from the source code https://gitlab.com/polavieja_lab/idtrackerai/-/blob/master/src/idtrackerai/base/postprocess/trajectories_creation.py?ref_type=heads#L102) does this.

import numpy as np

from idtrackerai import ListOfBlobs, Session

blobs = ListOfBlobs.load("path/to/list_of_blobs.pickle")

session = Session.load("path/to/session_folder")

trajectories = np.full((session.number_of_frames, session.n_animals, 2), np.nan)

for blob in blobs.all_blobs:

# # original code

# for identity, centroid in blob.final_ids_and_centroids:

# if identity not in (None, 0):

# trajectories[blob.frame_number, identity - 1] = centroid

# filtering out blobs with multiple identities (crossings)

valid_ids_centroids = [

(id, centroid)

for (id, centroid) in blob.final_ids_and_centroids

if id not in (None, 0)

]

if len(valid_ids_centroids) == 1: # only one valid centroid

identity, centroid = valid_ids_centroids[0]

trajectories[blob.frame_number, identity - 1] = centroid

This way you can generate trajectories with NaN gaps whenever there are more than one centroid in the blobs.

I hope this helps, keep asking if this didn't really reply to your question,

Jordi

[Samuel Snow]

Hi Jordi,

Wow, thanks so much for the insight and the code snippet! If I understand correctly, the main thing that manually needs to happen is I must accept interpolations for all "missing ID" situations so that occlusion blobs gain a second centroid, regardless of their classification as "individual" blobs.  I think this might do the trick- I will give it a try.

Thanks again!

[idtracker.ai]

Exactly, yes!

--
You received this message because you are subscribed to the Google Groups "idtracker.ai users group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to idtrackerai_us...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/idtrackerai_users/c8f80489-ffa8-42f2-a189-e3f6d3230e58n%40googlegroups.com.