Challenges in Machine Learning Workshop at NIPS 2014
Isabelle
Challenges in Machine Learning
NIPS 2014 workshop
December 12, 2014
http://ciml.chalearn.org/
Call for abstracts
We welcome 2-page extended abstracts on one of the topics of the workshop. The abstracts should be submitted before October 9th, 2014 by email to nips...@chalearn.org.
Topics of interest:
Methods:
- Novel or atypical challenge protocols, particularly to tackle complex tasks with very large datasets, multi-modal data, and data streams.
- Methods and metrics of entry evaluation, quantitative and qualitative challenges.
- Methods of data collection, "ground-truthing", and preparation including bifurcation/anonymization, data generating models.
- Teaching challenge organization.
- Hackatons and on-site challenges.
- Challenge indexing and retrieval, challenge recommenders.
Theory:
- Experimental design, size data set, data split, error bounds, statistical significance, violation of typical assumptions (e.g. i.i.d. data).
- Game theory applied to the analysis of challenge participation, competition and collaboration among participants.
- Diagnosis of data sanity, artifacts in data, data leakage.
Implementation:
- Re-usable challenge platforms, innovative software environments.
- Linking data and software repositories to challenges.
- Security/privacy, intellectual property, licenses.
- Cheating prevention and remedies.
- Issues raised by requiring code submission.
- Challenges requiring user interaction with the platform (active learning, reinforcement learning).
- Dissemination, fact sheets, proceedings, crowsourced papers, indexing post-challenge publications.
- Long term impact, on-going benchmarks, metrics of impact.
- Participant rewards, stimulation of participation, advertising, sponsors.
- Profiling participants, improving participant professional and social benefits.
Applications:
- Where to venture next: opportunities for challenge organizers to organize challenges in new domains with high societal impact.
- Successful challenge leading to significant breakthrough or improvement over the state-of-the-art or unexpected interesting results.
- Rigorous study of the impact of challenges, analyzing topics and tasks lending themselves to high impact machine learning challenges.
- Challenges as an educational tool.
- Challenges organized or supported by Government agencies, funding opportunities.
Motivations
Challenges in Machine Learning have proven to be efficient and cost-effective ways to quickly bring to industry solutions that may have been confined to research. In addition, the playful nature of challenges naturally attracts students, making challenge a great teaching resource. Challenge participants range from undergraduate students to retirees, joining forces in a rewarding environment allowing them to learn, perform research, and demonstrate excellence. Therefore challenges can be used as a means of directing research, advancing the state-of-the-art or venturing in completely new domains.
Yet, despite initial successes and efforts made to facilitate challenge organization with the availability of competition platforms, little effort has been put into the theoretical foundations of challenge design and the optimization of challenge protocols. This workshop will bring together workshop organizers, platform providers, and participants to discuss best practices in challenge organization and new methods and application opportunities to design high impact challenges. The themes to be discussed will include new paradigms of challenge organization to tackle complex problems (e.g. tasks involving multiple data modalities and/or multiple levels of processing).
Important dates
Abstract submission deadline: October 9th, 2014.
Acceptance decisions: October 23rd, 2014.
Finalized program: October 30th, 2014.
Early bird registration: November 7, 2014
Invited speakers
Percy Liang, Stanford, USA: Coopetitions
Joyce Noah-Vanhoucke, Kaggle, USA: Making data science a sport
Olga Russakovsky, Stanford USA:The ImageNet project
Rinat Sergeev, Harvard, USA: The NASA tournament lab
Gabor Melis, Franz Inc., Hungary: A serial winner
Tim Salimans, Algoritmica, The Nethelands: A brilliant data scientist
Michele Sebag, LRI, France: Pascal challenges
Gustavo Stolovitzky, IBM, USA: DREAM challenges
Important dates
Abstract submission deadline: October 9th, 2014.
Acceptance decisions: October 23rd, 2014.
Finalized program: October 30th, 2014.
Early bird registration: November 7, 2014
Committee
Kristin Bennett, RPI, New-York, USA
Gideon Dror, Yahoo!, Haifa, Israel
Hugo Jair Escalante, INAOE, Puebla, Mexico
Sergio Escalera, University of Barcelona, Catalonia, Spain
Bram van Ginneken, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands
Jayashree Kalpathy-Cramer, Harvard University, Boston, Massuchussetts, USA
Hugo Larochelle, Université de Sherbrooke, Canada
Vincent Lemaire, Orange research, Lannion, Britany, France
Percy Liang, Stanford University, Palo Alto, California, USA
Chih Jen Lin, National Taiwan University, Taiwan
Simon Mercer, Microsoft, Redmond, Washington, USA
Jin Paik, Harvard University, Boston, Massuchussetts, USA
Florin Popescu, Fraunhofer First, Berkin, Germany
Mehreen Saeed, University of Lahore, Pakistan
Danny Silver, Acadia University, Wolfville, Nova Scotia, Canada
Alexander Statnikov, American Express, New-York, USA
Rinat Sergeev, Harvard University, Boston, Massuchussetts, USA
Ioannis Tsamardinos, University of Crete, Greece
Organizers
Isabelle Guyon ChaLearn, Berkeley, California, USA
Evelyne Viegas Microsoft, Redmond, Washington, USA
Sponsors
ChaLearn, Microsoft, DREAM, IBM