IJCNN 2006 Model Selection + Performance Prediction Competition

[Simon Lucas]

IJCNN 2006 is running a competition of interest to this group:

http://clopinet.com/isabelle/Projects/modelselect/

Some more details below...

Best regards,

Simon Lucas

Performance Prediction Challenge

Tuesday, July 18, 2006
Vancouver, British Columbia, Canada

Background
Model selection is a problem in statistics, machine learning, and data
mining. Given training data consisting of input-output pairs, a model
is built to predict the output from the input, usually by fitting
adjustable parameters. Many predictive models have been proposed to
perform such tasks, including linear models, neural networks,
classification and regression trees, and kernel methods. The selection
of an optimal model, which should perform best on test data, is the
object of this workshop. A related problem is to find an optimal
ensemble of models forming a committee and voting for the final
decision according to given scores. Contributions to ensemble methods
are also within the scope of the workshop.

Part of the workshop will be devoted to the results of a challenge on
performance prediction:
How good are you at predicting how good you are?
In most real world situations, it is not sufficient to provide a good
predictor, it is important to assess accurately how well this predictor
will perform on new unseen data. Before deploying a model in the field,
one must know whether it will meet the specifications or whether one
should invest more time and resources to collect more data or develop
more sophisticated models. The performance prediction challenge is
connected to model selection because accurate performance predictions
are good model ranking criteria. We formatted five data sets for the
purpose of benchmarking performance prediction in a controlled manner.
The data sets span a wide variety of domains and have sufficiently many
test examples to obtain statistically significant results.

Challenge
The WCCI 2006 performance prediction challenge is to obtain a good
predictor AND predict how well it will perform on a large test set.
Entrants must provide results on ALL five data sets provided. To
facilitate entering results for all five data sets, all tasks are
two-class classification problems. During the development period,
participants may submit validation set results on a subset of the data
sets.

How to participate:
The challenge will open soon. Check our tentative schedule. To be
informed of the status of the challenge organization, send email to
model...@clopinet.com.

Participation
Participation in the workshop is not conditioned to entering the
challenge. Likewise, challenge entrants are not required to attend the
workshop nor to publish the methods they employed. Challenge entrants
may remain anonymous during the development period, but only identified
entrants will be included in the final competition ranking.
The best contributions will be invited to submit a paper to a special
issue of the Journal of Machine Learning Research.

Schedule
- September 2005: Start of the challenge.
- January 31st 2006: Deadline for WCCI paper submission. Challengers
doing well on the validation set will be solicited to submit papers.
- February 1st 2006: Release of the validation set labels.
- March 1st 2006: Final challenge entry submission deadline.
- March 15th 2006: Announcement of the results of the challenge. Paper
decision notification.
- April 15th 2006: WCCI camera ready submission.
- Tuesday July 18, 2006: WCCI workshop.
- Wednesday July 19, 2006: Award banquet at WCCI, price for the winner.