uFMF (version?) not able to be read by other software.

[Aaron Allen]

Good afternoon,

 

I had a question about recording directly to uFMF. With the release of 0.11.1, I am now able to record videos with strand-cam (https://groups.google.com/g/multicams/c/9Cj8ZqNOdzg). But I’m having issues with tracking/playing-back these videos with other tracking software. Are their multiple different versions of uFMF that may be causing compatibility issues?

 

A little background: with our previous setup we had been recording uncompressed .avi videos with LabView and then converting these uncompressed videos to uFMF using the “any2ufmf” software (http://ctrax.sourceforge.net/any2ufmf.html). These uFMF videos were then tracked with Caltech FlyTracker (http://www.vision.caltech.edu/Tools/FlyTracker/). These videos from any2ufmf are read in and tracked just fine.

 

But, when we try to open the uFMF videos recorded with strand-cam in Caltech Flytracker it is unable to open the video. Does anyone happen to know what might be the issue? The error message from Caltech FlyTracker’s calibrator.m function is below.

 

Thanks.

Aaron.

 

 

Error message:

Index exceeds the number of array elements (28).

Error in ufmf_read_header (line 79)

    header.timestamps < header.mean_timestamps(i+1);

Error in video_open (line 55)

      vinfo.ufmf = ufmf_read_header(filename);

Error in calibrator (line 24)

    vinfo = video_open(f_vid,1);

Error in tracker>tracker_interface/runCalibrator (line 646)

       calib_success = calibrator(f_vid,f_calib);

 Error while evaluating UIControl Callback.

 

[Andrew Straw]

Dear Aaron,

(I know CTrax but have used it little.) uFMF is a flexible container format (like, e.g. a zip file or an HDF5 file) but this then requires writer and reader to agree on what should be in the container. It seems that CTrax expects to read something different than what Strand Camera writes. I presume it would be fairly straightforward either to adapt Strand Camera to write a format understood by CTrax (the any2ufmf source may be helpful here). My main concern is that the present ufmf of Strand Camera is the same format we and colleagues have been using for years and I wouldn't want to break compatibility with past videos we have recorded or with our own tools. We could potentially have a second uFMF output for "CTrax compatible uFMF" but that seems error-prone for users.

It may also be possible to adapt CTrax to read the files as written by Strand Camera. That I can say less about but also would seem quite feasible, and with fewer compatibility concerns, as long as the required information is already saved in the uFMF files saved by Strand Camera. If that seemed straightforward, it would be the option I would favor.

It seems the CTrax google group would probably be the best place to continue this discussion with the CTrax developers. Can you start a thread there on the topic? I am a member of that group, too (although don't seem to be getting the emails, so I will manually check discussion online in the next days and also check for any email filters).

Best,

Andrew

--
You received this message because you are subscribed to the Google Groups "multi-camera software from the Straw Lab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to multicams+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/multicams/5123edfb-a4fd-4242-8b91-6333dda5abbdn%40googlegroups.com.

[Aaron Allen]

 Dear Andrew, 

 

Thanks for the rapid response.  

I agree that adding support for CTrax style uFMF to Strand Camera may not be the best (for the reasons you cite), and that it would be better to add support for Strand Camera uFMF videos to FlyTracker.  

Not sure how much use posting on the CTrax google group would be, as they don’t maintain the code for FlyTracker (which has its own google group - https://groups.google.com/g/flytracker). That being said, neither group seems to have any activity from either the original developers or anyone who may be maintaining the software... 

It does seem that Kristin Branson might have officially/unofficially taken over some maintenance of FlyTracker (https://github.com/kristinbranson/FlyTracker). Or at the very least is making some changes/updates available for what they are using internally. Perhaps I will ask there, to see if they are willing to add compatibility for Strand Camera uFMF videos. 

Do you have any documentation on the Strand Camera uFMF definition/description? I’ve attached a copy of the latest version (I think … it doesn’t appear as if it has changed for over a decade ...) of the CTrax-uFMF description, and the source code for any2ufmf software is available here https://github.com/kristinbranson/any2ufmf (again, I think … not clear if this is the most recent version). Using the `ufmf_read_header.m` MATLAB function from FlyTracker (https://github.com/kristinbranson/FlyTracker/blob/main/tracking/utilities/ufmfread/ufmf_read_header.m) to read the header of a CTrax-uFMF, and the `read.ufmf.header.R` function from Greg Jefferis’ fmfio package in R (https://github.com/jefferis/fmfio/blob/master/R/read.ufmf.R) to read the header of a Strand Camera uFMF, I noticed that they seem to differ in the `mean_timestamps`. For the CTrax uFMF videos `nmeans` appears to be equal to the length of `mean_timestamps`, however in the Strand Camera uFMF videos `nmeans` appears to be 2x the length of `mean_timestamps`. This seems to be what’s causing the issue (or at least first issue) when trying to read Strand Camera uFMF with the CTrax style readers, based on the error messages previously shown. The also differ in how the dimensions of the video are specified with CTrax style having 2 separate fields for `max_height` and `max_width` and Strand Camera style having a paired value for the field `max_imdims`. 

  

I'm curious (in super broad strokes) how Strand Camera goes about making the uFMF video. Is it “keeping” the lossless information from all regions that are different enough from the mean/median background? Or is it only focusing on coordinates of the tracked individual? Since you need the real time tracking turned on and it appears to be setup to only track one individual, I’m wondering if the Strand Camera generated uFMF videos appropriate for recording multiple individuals at a time? I’ve included a sample image of our arenas that we currently record below, which typically have 20-60 flies. 

 

On another related note, what would you say is the best way to playback and view the uFMF videos generated by Strand Camera? 

 

 

Thanks again for all your assistance. 

Aaron.