Possible project idea - looking for mentors

194 views
Skip to first unread message

Mason Gallo

unread,
Feb 25, 2016, 8:23:19 PM2/25/16
to gsoc-r
Hi! I'm a graduate student in computer science. I'm looking for potential mentors and/or advice on my idea below:

Designing an R package for the creation of model evaluation curves. This would include both learning curves and validation curves. 

When writing research, I often need to create learning curves to demonstrate the bias/variance tradeoff of a model, and I haven't found any package that addresses this. I also think validation curves would be interesting to include.

Bernd Bischl

unread,
Feb 29, 2016, 5:13:16 AM2/29/16
to gso...@googlegroups.com, mason...@gmail.com, Julia Schiffner, Lars Kotthoff, Zachary M. Jones
Hi,

we from mlr might be interested. But a few comments:


1) We had a very succesful gosc 2015 project on model visualization. I added Zach in CC who worked on this and the mentors.

2) mlr _already_ supports learning curves. See here:
http://mlr-org.github.io/mlr-tutorial/devel/html/learning_curve/index.html

3) What do you mean by "validation" curves (a somewhat fuzzy term).

In summary:
We have a lot of interest in model visualization, and quite a lot of code already in place. If you could write down a somewhat more detailed outline what you wanna do, we could talk.

Best

Bernd
--
You received this message because you are subscribed to the Google Groups "gsoc-r" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gsoc-r+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Mason Gallo

unread,
Feb 29, 2016, 5:11:33 PM2/29/16
to gsoc-r, mason...@gmail.com, Julia.S...@uni-duesseldorf.de, larsko...@gmail.com, z...@zmjones.com
Thanks for the note! Let me provide more specific details and clarity on my idea:

Situation: many of the common model visualization / model evaluation plots either require lots of boilerplate code to visualize or are sparsely implemented across packages
e.g. learning curves are implemented in sklearn but you need to write the boilerplate code yourself to visualize; caret provides a means to plot the performance of tuning parameters (http://topepo.github.io/caret/training.html) but doesn't provide many of the other options provided by mlr; mlr provides the performance of tuning parameters but doesn't automatically generate curves.

Overall goal: provide easy to use methods for many of the common model visualization / evaluation plots

Quickly looking at what's available in mlr, here are some of my ideas to add (I am not an expert on the needs/features of the mlr codebase, so I'm happy to incorporate any ideas/feedback from you and the team):

  • Plotting a tuning parameter against the output of a scoring function (this is what I meant above by validation curve) - we use this on my team frequently especially for evaluating clustering (ie k in k-means) and dimensionality reduction. As discussed above, this could leverage the already implemented $opt.path.
  • Training/prediction time curves (depending on eager/lazy learner). This could potentially leverage the already implemented $opt.path but may need to be spit based on the type of learner.
  • Model complexity curves (ie plotting # of non-zero coefficients against the output of a scoring function)
  • Model visualization on pca-reduced data to visualize separation boundaries or highly dimensional data in 2D
  • Support for std bands for learning curves and any other model eval visualizations where a mean is calculated over several CV iterations for the scoring function
  • Support for epochs instead of train size (ie for neural nets) for applicable model visualizations
  • All model eval visualizations would ideally include "benchmarking", such as sklearn's Dummy model (looks like this is referenced issue #712 on github)

Best,
Mason

Bernd Bischl

unread,
Mar 1, 2016, 6:43:35 AM3/1/16
to gso...@googlegroups.com, mason...@gmail.com, Julia.S...@uni-duesseldorf.de, larsko...@gmail.com, z...@zmjones.com
Hi,

On 29.02.2016 23:11, Mason Gallo wrote:
> Thanks for the note! Let me provide more specific details and clarity
on my idea:
>
> Situation: many of the common model visualization / model evaluation
plots either require lots of boilerplate code to visualize or are
sparsely implemented across packages
> e.g. learning curves are implemented in sklearn but you need to write
the boilerplate code yourself to visualize; caret provides a means to
plot the performance of tuning parameters
(http://topepo.github.io/caret/training.html) but doesn't provide many
of the other options provided by mlr; mlr provides the performance of
tuning parameters but doesn't automatically generate curves.

Totally agreed regarding the observations.
We try to improve the situation a bit in mlr by always define a cleanly
specified "container" object for the plot data.
And to separate the code of generating that data (which at the end
produces the container) and the plotter.

This has couple of advantages:

a) I you dislike our plots, you can implement your own, but have the
well-structured container (and can still use mlr to gather that data)
b) If your data comes from another source, but you would still like to
use the plots, you can simply transform them in the container format.


>
> Overall goal: provide easy to use methods for many of the common
model visualization / evaluation plots
>
> Quickly looking at what's available in mlr, here are some of my ideas
to add (I am not an expert on the needs/features of the mlr codebase, so
I'm happy to incorporate any ideas/feedback from you and the team):
>
> Plotting a tuning parameter against the output of a scoring
function (this is what I meant above by validation curve) - we use this
on my team frequently especially for evaluating clustering (ie k in
k-means) and dimensionality reduction. As discussed above, this could
leverage the already implemented $opt.path.

Yes, the optpath is already a nice container. And we do have _some_ code
in ParamHelpers to plot "traces" from this. Which could be _heavily_
improved I guess. But it's a starting point.
Another option is to look at specific parameters, where the model
fitting itself already produces a "curve" or "trace".
Prominent examples are iterations vs OBB performance in bagging /
forests / boosting type methods.

We would really like to "know" though the software whether a learner
supports this (eg using mlr's tagging mechansim) and have a common API
call for this.
A downside to this is that is usually implemented always a bit
differently, and only for very "normal" performance measures, so
currently we are working on having an API call for the OOB _predictions_
for the whole trace.
Then you can compute anything on this, post-hoc.



> Training/prediction time curves (depending on eager/lazy
learner). This could potentially leverage the already implemented
$opt.path but may need to be spit based on the type of learner.
Yup
> Model complexity curves (ie plotting # of non-zero coefficients
against the output of a scoring function)
Yup
> Model visualization on pca-reduced data to visualize separation
boundaries or highly dimensional data in 2D

Also nice. Another option might be multim. scaling. We dont really have
plotting options for both currently.

> Support for std bands for learning curves and any other model
eval visualizations where a mean is calculated over several CV
iterations for the scoring function

This would be easy to add what we currently have.

> Support for epochs instead of train size (ie for neural nets) for
applicable model visualizations
> All model eval visualizations would ideally include
"benchmarking", such as sklearn's Dummy model (looks like this is
referenced issue #712 on github)

Dummy models I guess you mean? So the most naive model as a base line?
Yes, this was asked a couple of times.

Mason Gallo

unread,
Mar 2, 2016, 2:25:05 AM3/2/16
to gsoc-r, mason...@gmail.com, Julia.S...@uni-duesseldorf.de, larsko...@gmail.com, z...@zmjones.com
Thanks for the feedback and POV!

 And to separate the code of generating that data (which at the end produces the container) and the plotter...
I definitely agree here and would want to make sure to maintain this methodology for all of my ideas mentioned above. 

We would really like to "know" though the software whether a learner supports this (eg using mlr's tagging mechansim) and have a common API call for this. 
To make sure I'm following, are you referring to the property tags for a given learner? Such as mentioned in https://github.com/mlr-org/mlr/blob/master/R/RLearner.R. 

Dummy models I guess you mean? So the most naive model as a base line?
Exactly. A dummy classifier could do things like mimic the training set's class distribution, predict the most frequent class, among other options.

Do you and the rest of the team have any other ideas on the roadmap that might complement the ideas mentioned so far? Anything I might have left out?

Bernd Bischl

unread,
Mar 2, 2016, 3:28:57 AM3/2/16
to gso...@googlegroups.com, mason...@gmail.com, Julia.S...@uni-duesseldorf.de, larsko...@gmail.com, z...@zmjones.com
Hi.


On 02.03.2016 08:25, Mason Gallo wrote:
Thanks for the feedback and POV!

 And to separate the code of generating that data (which at the end produces the container) and the plotter...
I definitely agree here and would want to make sure to maintain this methodology for all of my ideas mentioned above. 

We would really like to "know" though the software whether a learner supports this (eg using mlr's tagging mechansim) and have a common API call for this. 
To make sure I'm following, are you referring to the property tags for a given learner? Such as mentioned in https://github.com/mlr-org/mlr/blob/master/R/RLearner.R. 


Yes. And the OOB stuff is in an unfinished PR here.
https://github.com/mlr-org/mlr/pull/693


Dummy models I guess you mean? So the most naive model as a base line?
Exactly. A dummy classifier could do things like mimic the training set's class distribution, predict the most frequent class, among other options.

Yup.

Do you and the rest of the team have any other ideas on the roadmap that might complement the ideas mentioned so far? Anything I might have left out?


I could iterate over this. (currently abraod on a workshop)
But it would be great if Julia, Lars or Zach would also respond wrt to their interest.
I would only mentor this, if one of them does as well.

Best

Bernd

Zachary M. Jones

unread,
Mar 2, 2016, 9:36:17 AM3/2/16
to Bernd Bischl, gso...@googlegroups.com, mason...@gmail.com, Julia.S...@uni-duesseldorf.de, larsko...@gmail.com
I probably won't have time to do much else this summer. I am an RA on a grant that will require a lot of programming and am going to try to get close to finishing my dissertation so I can apply for jobs in the fall. I would be happy to participate when I can though. This of course would be awesome to have. 

Mason Gallo

unread,
Mar 7, 2016, 12:08:52 AM3/7/16
to gsoc-r, bernd_...@gmx.net, mason...@gmail.com, Julia.S...@uni-duesseldorf.de, larsko...@gmail.com, z...@zmjones.com
Thanks Bernd and Zach! Just following up to check if mentoring this would be feasible for mlr. I would love to work with you on what we discussed, and I think mlr would be a great home for my ideas.

Please let me know as soon as you can if this is feasible. The student application period is fast approaching, and I'd like to start working with you on next steps.

Best,
Mason

Lars Kotthoff

unread,
Mar 7, 2016, 8:02:05 PM3/7/16
to Mason Gallo, gsoc-r, bernd_...@gmx.net, Julia.S...@uni-duesseldorf.de, z...@zmjones.com
Hi Mason,

 sorry for not getting back to you earlier. I think what you're proposing sounds great and I would definitely be willing to mentor you. We should probably narrow the scope a bit though as it's currently very broad. I like the first idea in your second email best -- plotting the progress of tuning wrt to a parameter. We don't have this at the moment and it would be very helpful to see this. There would also be scope for doing more sophisticated things, i.e. visualising more than one parameter/scoring function or doing ablation analysis. This would be relevant to the other things you propose to some extent as well, but in my opinion it's the most interesting from your list.

How does that sound? It would involve working with the ParamHelpers package as well (actually probably a 50-50 split between mlr and ParamHelpers).

Cheers,

Lars


Mason Gallo

unread,
Mar 8, 2016, 11:39:34 AM3/8/16
to Lars Kotthoff, gsoc-r, bernd_...@gmx.net, Julia.S...@uni-duesseldorf.de, z...@zmjones.com
Hi Lars,

No worries - thanks for your feedback! Your thoughts on scope are very helpful. I will give more thought to scope this week.

Based on what we have discussed, I will work on a rough draft of the overall project design goals and send to the team this weekend for feedback. Here's my outline for next steps up until the application deadline:

- Align on final overall project design goals/theme
- Align on rough draft of project schedule for summer period
- Submit rough draft of project proposal for team feedback
- Incorporate feedback to finalize project proposal prior to Google deadline

In the meantime, please let me know if you or Bernd (or anyone on the team) has any additional thoughts I can incorporate. 

Best,
Mason


Reply all
Reply to author
Forward
0 new messages