BTK server (Matlab) issue with long collections?

[Jordan Sturdy]

I have recently begun using the BTK functionality with Matlab and have been very pleased so far. 

One issue I seem to have is that when I open a large file (36000 marker points and 360000 analog points) all my data structures are zero matrices. For markers I get a 1x3 set ([0,0,0]) for xyz and for analog I get a 10x1 zero matrix. 

This doesn't happen to my other files which are under 1500 marker points and 15000 analog points.

I have version 0.2.0

The functions I use are: srv = btkEmulateC3Dserver(), srv.Open(file,3), btkGetMarkers(srv.GetHandle()), and btkGetAnalogs(srv.GetHandle()).

I use btkAppendAnalog, btkAppendForcePlatformType2, and srv.SaveFile as well, but the issue is present before those are used.

Does anyone know if there is a max length of file that imports properly, or am I missing some specification or special handling?

Thanks!

Jordan

[Arnaud BARRE]

Hi Jordan,

Welcome to the BTK community! Could you send me one of the file where you have this problem? Do you use Visual3D from C-Motion to create these C3D file? They decided to use a different method than the one proposed in the C3D file format to handle large file (for historical reason). It should not take time to add/fix this.

Thanks a lot,

Arnaud

Jordan

--
You received this message because you are subscribed to the Google Groups "BTK Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to btk-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Eric Walker]

I have had success using btkReadAcquisition in the matlab wrapper to open up long collections. most of my collections are sount 3 minutes, but I have also run some 7 minute trials, with analog data collected at 1000Hz and marker at 100Hz. The c3d files are created in Vicon Nexus, and actually BTK seems to handle the fies better.

Eric

[Arnaud BARRE]

Thanks for the feedback Eric. If the large files used by Jordan don't come from Visual3D, then there is maybe an error in the Matlab function btkEmulateC3Dserver.

Jordan, can you try to use directly the Matlab function btkReadAcquisition instead of btkEmulateC3Dserver?

Arnaud

On Wed, May 15, 2013 at 4:34 AM, Eric Walker <erwal...@gmail.com> wrote:

I have had success using btkReadAcquisition in the matlab wrapper to open up long collections.  most of my collections are sount 3 minutes, but I have also run some 7 minute trials, with analog data collected at 1000Hz and marker at 100Hz.  The c3d files are created in Vicon Nexus, and actually BTK seems to handle the fies better.

Eric

[Jordan Sturdy]

Thanks Arnaud and Eric,

I'm uploading one of the files to my Adrive and will post the link. Its too big to attach to this or an email. http://www.adrive.com/public/CUtYtx/10min_1.c3d

The c3d is created in Cortex (motion analysis), I don't use Visual 3d for collections.

I will try a little more with btkReadAcquisition, but I just ran it and I think the warning message may indicate something that is the problem.

the following is copied from my matlab command window:

[H, BYTEORDER, STORAGEFORMAT] = btkReadAcquisition('10min_1.c3d')

Warning: btkC3DFileIO.cpp(673): '10min_1.c3d' - The last frame index wrote in the

header is different than in the parameter TRIAL:ACTUAL_END_FIELD. The number of frames is modified

by keeping the value in the parameter. 

Warning: btkAcquisition.cpp(942): Impossible to set the frame number to 0 or lower. The number of

frames is now equals to 1. 

H = 4.8436e-316

BYTEORDER =IEEE_LittleEndian

STORAGEFORMAT =Float

No warning message was given when I ran the same request on my shorter files (10 seconds vs 10 minutes).

I will continue digging into the using the ReadAcquisition as that was my first exposure to it.

[Jordan Sturdy]

Quick follow up. I just loaded my original c3d into visual 3d to check my start and end frames. 

For ACTUAL_END_FIELD it gives me:

Parameter Data (1) = -29536(36000)

Parameter Data (2) = -3584(61952)

for ACTUAL_START_FIELD I have

Parameter Data (1) = 1

Parameter Data (2) = 0

My shorter files are more like this:

Parameter Data (1) = 1150

Parameter Data (2) = 0

and

Parameter Data (1) = 1

Parameter Data (2) = 0

Something is going on with that, and I'm not sure it has anything to do with the btk files now, but is there a way I can bypass the Parameter Data check and just try to access the marker and analog data?

[Arnaud BARRE]

Thanks Jordan for this digging. It is clearly related to the way BTK is managing the parameters ACTUAL_START_FIELD and ACTUAL_END_FIELD.I don't have my computer right now, but using the file you uploaded, I will correct the C3D reader in consequence.

I was compiling the binaries for Matlab and BTK 0.2.1 when you reported this issue. I will fix it and I will create new packages at the same time. You should be able to read your large files with BTK in the next days!

Thanks again

Arnaud

[Jordan Sturdy]

Thanks very much Arnaud!

Please let me know if you have any more questions once you look at the file. If you think it may be the way Cortex exports the large files, I will try to get information from them. They have in the past adapted specific exporting functions for us, so they may be able to update this too.

If it helps, I have also noticed that the label 'TRIAL' only appears in the MetaData for the larger files. My static and shorter gait files don't have this field when I open them with btk functions.

I look forward to what you find out in the next few days.

Jordan

[Arnaud BARRE]

Hi Jordan,

As you reported the problem is in the construction of the parameter TRIAL:ACTUAL_END_FIELD. This parameter should be used to create an unsigned integer stored on 32-bit (from two unsigned integer stored on 16-bit; uint16) as it is mentionned in the official documentation of the C3D file format: http://c3d.org/HTML/actualendfield.htm

To create the final number you have to do: total_number = uint16[1] * 2^16 + uint[0].

However, Motion Analysis used it differently. They used the first uint16 to store the last supported frame (36000 for them) and (i guess) the total number of frames in the second uint16 (i.e. 61952). The correct use of this parameter should be {61952, 0} (if 61952 is you total number of frames). If your total number of frames is not 61952 but 97952 (36000+61952), then the content of the parameters should be {32416,1}.

I propose you this solution. First, I will create a "hook" in the code to manage correctly this. A warning will be displayed by default to mention that. But the Motion Analysis company should fix this problem as no external C3D reader will be able to read correctly their files when they use the parameters TRIAL:ACTUAL_(START|END)_FIELD. If you try with to read your uploaded file with Visual3D, it is limited to 36000 instead of your true total number of frames. Moreover, with their solution, they will be limited to small number of frames compared to what is proposed in the C3D file format (i.e. more than 4 billion of frames...)

I propose you to send them a bug report to correct this.

Can you also tell me what is the total number of frames in your uploaded files? 61952 or 97952? I need this information to know how to manipulate data in the hook.

Arnaud

[Arnaud BARRE]

I implemented the hook based on what I guessed is the total number of frames (i.e. 61952 or 97952). I checked both solution with a modified version of Mokka and they doesn't work as expected. Actually, there is only 36000 frames in your C3D file. So I don't understand how Motion Analysis uses the parameter TRIAL:ACTUAL_END_FIELD. Can you ask them? Or maybe you have more information on how they do? Is it only a part of you acquisition (starting at frame 61952 and containing 36000 frames)? Have you the equivalent TRB file?

Arnaud

[Jordan Sturdy]

Arnaud, 

I believe the number of frames should be 36000 exactly. The 10min file is collected at 60hz with a x10 multiplier for the analog rate. 

Our shorter files are all collected at 120hz with the same x10 for the analog.

I've uploaded the trb and anb files for the 10 minute file as well as a set of files for a short "Static" trial if you need to see what's different between the two.

http://www.adrive.com/public/W3KfRj/files for btk.rar

I will ping Motion Analysis and see if they can answer what's going on with TRIAL:ACTUAL_END_FIELD because it only seems to show up in the longer collections. 
If I have an older version of Cortex, I will try to see if the file comes out the same way.

I truly appreciate your help.

Jordan

[Jordan Sturdy]

Ok this does seem to be a version issue. I have a 10 minute file that was exported with a previous release of Cortex, and the TRIAL:ACTUAL_END_FIELD parameter does not exist. Both the analog and marker data load fine with the toolkit functions. I'm not sure why anything would have changed, but hopefully Motion Analysis can figure it out quickly.

Jordan

[Arnaud BARRE]

Thanks for these feedbacks Jordan. Let me know what is the response of Motion Analysis.

For your information, the use of the parameter TRIAL:ACTUAL_(START|END)_FIELD are optional if your acquisition contains less than 65535 frame. But I guess Motion Analysis uses it now by default in their latest version.

Arnaud

[Jordan Sturdy]

Motion Analysis is working on this issue, and hope to resolve it soon.

Via MA support: "The TRIAL parameters were added to 4.0 to handle long C3D files. The parameter is two 2-byte numbers that need to be combined in software to be a single 4 byte number. See the c3d docs for this (http:// www.c3d.org/). We export this file when the capture exceeds 32767 frames."

I have sent them some files, and they have seen the issue. Clearly the implementation into Cortex 4 has some miscalculation that is generating negative values in the parameter. Once I get the fix, I will make sure it imports with btk and let you know all is well!

Jordan

[Jordan Sturdy]

Arnaud,

Motion Analysis has fixed the issue, and they are finalizing a build of the software to send out.

Once I get the new version, I will check to make sure everything handles properly in BTK.

Thanks.

Jordan