Performance Evaluation - A sample program

[Simon Ruffieux]

Dear challengers,

I have uploaded a sample program to evaluate the output of your algorithms. 

I have also updated the required output format for your algorithms (as indicated in the page "Rules" of the website).  

The program is available for download on the server containing the Dataset in the folder "Program".

Please read the readme.txt file for more information.

In case of any problem, do not hesitate to contact us.

Hope this will help you for your research !

Simon Ruffieux

[Ying Yin]

Hi Simon,

Thank you for the evaluation program. I just did a quick test using the example output and I changed MainSensor to Xsens. There is an exception on line 230 of MainWindow.xaml.cs. I think that's because hitList is empty and so are listofStartErrors and listofStopErrors.

Thanks,
Ying

[Simon Ruffieux]

Hello Yin,

I am out of office for a conference. Therefore I only have limited time and access to internet these days. 

I will try to do my best to check this issue. But it may take a few days.

Note that if you use the Xsens as Mainsensor, you must use the related sample output file. (for xsens).

Sorry for the inconvenience and thank you for pointing out the issue.

Best regards,

Simon

[Ying Yin]

Also should the first letter of the MainSensor name be lowercase according to this line?

if (sensor.Equals("kinectRGB") || sensor.Equals("xSens"))

Both the website and the sample input have uppercase. 

[Ying Yin]

Maybe it's better to change these comparisons to 

if (sensor.Equals("KinectRGB") || sensor.Equals("Xsens"))

[Simon Ruffieux]

The version 1.1 of the program has been uploaded to the server. The bugs should have been corrected. Sorry for the inconvenience.

Best Regards,

Simon Ruffieux

[Ying Yin]

Hello,

Seems like if one completely missed all the gestures in a batch, the ground truth gestures in that batch will not be counted. Is this the case?

Thanks,
Ying

[ChAirGest Challenge]

Hello Ying,

If someone completely missed all the gesture in a batch, of course that will not count be counted.

But, from my knowledge, this should not happen in the dataset.

Could you send me a mail to indicate which bacth is incorrect ?

(I am currently in vacation with limited connexion access and will have a look as soon as I come back)

Best regards,

Simon Ruffieux

[Ying Yin]

Hi Simon,

The data set is fine. I'm asking about if someone's algorithm somehow misses all the gestures in a batch (no output at all) and this will make the recall rate high because the total number of ground truth gestures is not complete. Right now, it seems that the ground truth calculation only checks the files in the user's output file.

Thanks,
Ying