Is this project still active?

[Jay]

I wrote a blog post on the need for a good, broad ml benchmarking site/infrastructure (http://blog.empathybox.com/post/18810157226/does-off-the-shelf-machine-learning-need-a-benchmark), and a few people pointed me to your site, which I had not previous heard of. This seems to be very close to what I was looking for. Are you guys still interested in actively pursuing this? Are you taking code contributions? The site is very nice but seems like it might need a few more features to be really useful. Do you guys have an idea of features or bugfixes you are looking for? Is there any instructions on how to get set up to do development on the code base?

Also, perhaps more importantly there seems to be a bit of a bootstrapping problem here to get people to be aware of this and use it. Did you guys do a paper to help researchers be aware that this exists? Any other thoughts on how to market it more effectively?

-Jay

[Percy Liang]

Hi Jay,

Nice post - many of the issues you brought up are precisely what
motivated us to create MLcomp. Currently, the site is somewhat
active, but we would love to have more people contribute! All the
code is open-sourced and available here:
https://github.com/percyliang/mlcomp
The README should have instructions on how to get set up.
Here are a bunch of bug/feature requests that's on our plate:
https://github.com/percyliang/mlcomp/issues

The site currently is quite functional and general, but getting
widespread adoption remains a challenge. Often ML workshops will have
competitions with standard benchmarks, and a good first step would be
to get them use MLcomp as the platform. We have been talking to
various groups, but there's a lot of inertia: people want to be able
to run their algorithms using their own custom setup (even though
MLcomp allows arbitrary binaries to execute - it just has to be
self-contained). Another thing is that we should probably do is just
seed MLcomp with more standard datasets/algorithms that researchers
are actively working on - once there is critical mass, more people are
likely to use it because it'll be convenient to do comparisons. A lot
of people download datasets/algorithms from the site without running
them there - maybe because they have private data. Please let us know
if you have any other ideas.

Cheers,

-Percy