Converting multi-animal, multi-part DeepLabCut CSV to JAABA-compatible track files

[Kevin Chen]

Hi,

We're using DeepLabCut to track mouse body parts. This outputs a CSV with XY coordinates of each body part for each animal at each frame. We are now trying to convert the CSV into a trx file to use in JAABA. We are taking over this project from another student who graduated. They already added the custom input xml and want us to use .ufmf video files. We are using trackConverter2.py to convert the "36 Optimus Progesterone DoseDLC_resnet50_RatLabTest3Aug2shuffle1_44500_el_filtered (1).csv" into p.mat. This conversion uses the registered_trx.mat as a template replacing only the trx struct. We've included the files in a Google Drive folder. When we try to create a project in JAABA using the .mat file and .ufmf we get the following error.

Thank you for any help you may be able to provide. Along with this, do you have any recommendations for how we may better convert our CSV. Thanks again.

Kevin Chen

[Mayank Kabra]

Hi Kevin,

Thanks for describing the problem in great detail and providing the files!

I don't see any issue with the movie and the trx file. I was able to create a JAABA project and also load the movie. So I don't think the issue is with the files. When you created the JAABA project, what did you give as the movie filename and trx filename? In your case it should be OptimusProgesteroneDose36.ufmf and registered_trx.mat. Since the name seems to include the experiment conditions, I would suggest renaming the movie file to movie.ufmf and move it into a folder named OptimusProgesteroneDose along with the trx file. More information on the directory structure required by JAABA is at http://jaaba.sourceforge.net/DataFormatting.html#ExperimentDirectoryStructure

Other than that, the movie quality looks quite bad. I imagine this is because you converted the movie into ufmf format. I don't think ufmf is an ideal format for your movie. You should use mp4 or avi file format with h.264 or any other commonly used video codec. JAABA works with any file format that Matlab can read. Also, the tracking results that I saw in the registered_trx.mat are completely wrong. I didn't check the csv output. You might want to double-check your conversion from csv to registered_trx.mat. With the current tracking results in registered_trx.mat you won't be able to detect any behavior.  

Mayank

--
You received this message because you are subscribed to the Google Groups "JAABA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jaaba+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jaaba/b8073d6f-861f-4920-9aec-c710a9e6c962n%40googlegroups.com.

[Mayank Kabra]

Hi Kevin,

If tracking is missing for some of the frames, you will have to add NaNs to x,y, and other track fields to indicate that the tracking is missing for those frames. For JAABA, trx(n).x(t) is the x position of animal "n" in frame (t+trx(n).firstframe-1). After padding track fields with NaNs, (endframe-firstframe+1) will match "nframes" for each trx. "dt" field merely indicates the time difference between the current frame and the next frame. It is not used to interpret whether frames are being skipped. It is mostly used to just get a more accurate measurement for velocity etc. In most cases, it's value will be just 1/fps.

Please include the google groups in your replies because the conversation is likely to help others who encounter this problem.

HTH,

Mayank

On Fri, Sep 24, 2021 at 6:29 AM Kevin Chen <che...@carleton.edu> wrote:

Hi Mayank,

Could you elaborate more on why we need to fix those fields? The output of DeepLabCut returns the locations of the body parts of both rats, but we don't have the position of every rat for every frame. As such, we included the frames within the first and last frames that we have a position.  As a result, the values for firstframe, endframe, and nframes differ between the two rats. Is that the issue you were referring to? Because of these gaps in the tracking, the dt is also not always consistent. 

Thank you for your help and expertise,

Kevin Chen & Etienne Richart 

On Wed, Sep 22, 2021 at 2:39 AM Mayank Kabra <mayank...@gmail.com> wrote:

Hi Kevin,

Ah, sorry didn't realize that. Yes, the p.mat file's format is wrong. The details of trx files format can be found here: http://jaaba.sourceforge.net/DataFormatting.html#TracksFileStructure . You will have to update the conversion code to comply with the trx file format. Specifically, fields like firstframe, lastframe, and off. 

Mayank

On Tue, Sep 21, 2021 at 9:55 PM Kevin Chen <che...@carleton.edu> wrote:

Hi,

First of all, thank you so much for taking the time to help us out, we really appreciate it!

It seems that there may have been a misunderstanding; the registered_trx.mat file is the template that we used to create the new tracks file: p.mat. We changed the "trx" struct within registered_trx.mat to a struct that contains our data( based off of JAABA's documentation for preparing data), but didn't change the "timestamps" field. 

We are unsure on what "timestamps" does, could you let us know what we should put for that?

To be clear, the tracks file that we used is called "p.mat", not "registered_trx.mat".

Thanks again for your help,

Kevin Chen

Carleton College '23

Computer Science & Statistics

[Kevin Chen]

Hello Mayank,

Thanks for your earlier advice. Once we implemented the changes you suggested we got past the original issue but are now running into a new one. We have updated the drive we previously shared with the new conversion code (“trackConverter2.py”), a new trx.mat file ( “p.mat”) and a .mp4 version of our movie ( “movie.mp4”) as well as a folder ( testBehavior1) which is our experiment directory that we load into JAABA. The link to the files if you need it: https://drive.google.com/drive/folders/1Z1rHoflMsMsN9Lk8C_TN2sSugRYFP9M_?usp=sharing

When we now try to create a new project we can load the movie, but before we get past the black screen, the software throws an error, which we have included below. 

Do you have any suggestions for how we can resolve this error? We believe it could be a problem with how the trx.mat data is formatted. We’ve checked the dimensions of the fields in the output file (“p.mat”) thoroughly and don’t see an issue. We also speculate that perhaps our extensive use of NaN values might be the cause of the error; we’ve had some difficulty in converting our DeepLabCut tracks into some of the features in the trx.mat file because the documentation is vague. For instance, we were unsure on what values to put in for “theta”, “x_mm”, “theta_mm”.

Also, we believe the issue might also originate from the feature config file(featureConfig_logan_rat.xml). We were given that xml file from the previous person who worked on this project, but we don’t know what any of it means. Is there some documentation on how the xml file should be formatted? 

Thanks again for your help.

Sincerely,

Kevin Chen & Etienne Richart

[Mayank Kabra]

Hi Kevin,

So first, there were a few bugs that were introduced into JAABA because Matlab changed how they handle empty arrays recently. I've fixed the bugs that I encountered but you might encounter a few more. Do let us know about them! You should pull the latest version from github.

While testing out the remaining stuff on your data, I found that the firstframe, endframe, and off fields in the trx are defined as int64 which lead to a lot of errors. These (and all the other fields) should be double.

The other problem in the trx file is that the dt field in p.mat is not formatted properly. Its size should be 1 x(nframes-1) because it is the time difference between the frames. As for x_mm (and other *_mm) fields, you can keep them the same as the original x field. The reason JAABA has two versions is that pixel size (in mm) can be different between different videos based on the zoom. The x_mm and other mm fields thus tell JAABA about measurements in absolute terms which are essential for training classifiers that generalize. The x, y (and other non mm fields) are required in order to plot the trajectories in the interface for labeling. Theta and theta_mm should be set to 0 (or animal's actual orientation) for frames where you have tracking. Right now it was set to 0 for all frames.

As for the feature config file, I found that it is using only x & y perframe features. With this, the classifier will get information only about the location of the animal within the arena and as such would be able to detect extremely simple behaviors such as whether the animal is the top-right quadrant or similar. I'm not sure what types of behavior you are interested in so it might be enough. However, for any slightly more complex behavior when creating your JAABA project you might want to start with default "fly" as the animal type. The animal type just signifies the perframe features that are computed. You might want to look at https://www.nature.com/articles/nmeth.2281 to find more details of the perframe features. You can also look at https://groups.google.com/g/jaaba/c/KLu6-QYsLYE/m/8ba3srxwk4oJ to get more details about the feature config files.

Overall though, I found that the tracking has a lot of gaps and as such JAABA will have a lot of trouble in detecting behaviors. With the deeplearning based tracking methods, it should be possible to get better accuracy by labeling more data. That might help you in getting better behavior classifiers with JAABA.

HTH,

Mayank

To view this discussion on the web visit https://groups.google.com/d/msgid/jaaba/77bb2c9b-2f90-4360-a038-db2a58712323n%40googlegroups.com.

[Mayank Kabra]

And I forgot to mention that you need to delete the perframe directory whenever you change the trx file.

Mayank