Hi,
On 29.02.2016 23:11, Mason Gallo wrote:
> Thanks for the note! Let me provide more specific details and clarity
on my idea:
>
> Situation: many of the common model visualization / model evaluation
plots either require lots of boilerplate code to visualize or are
sparsely implemented across packages
> e.g. learning curves are implemented in sklearn but you need to write
the boilerplate code yourself to visualize; caret provides a means to
plot the performance of tuning parameters
(
http://topepo.github.io/caret/training.html) but doesn't provide many
of the other options provided by mlr; mlr provides the performance of
tuning parameters but doesn't automatically generate curves.
Totally agreed regarding the observations.
We try to improve the situation a bit in mlr by always define a cleanly
specified "container" object for the plot data.
And to separate the code of generating that data (which at the end
produces the container) and the plotter.
This has couple of advantages:
a) I you dislike our plots, you can implement your own, but have the
well-structured container (and can still use mlr to gather that data)
b) If your data comes from another source, but you would still like to
use the plots, you can simply transform them in the container format.
>
> Overall goal: provide easy to use methods for many of the common
model visualization / evaluation plots
>
> Quickly looking at what's available in mlr, here are some of my ideas
to add (I am not an expert on the needs/features of the mlr codebase, so
I'm happy to incorporate any ideas/feedback from you and the team):
>
> Plotting a tuning parameter against the output of a scoring
function (this is what I meant above by validation curve) - we use this
on my team frequently especially for evaluating clustering (ie k in
k-means) and dimensionality reduction. As discussed above, this could
leverage the already implemented $opt.path.
Yes, the optpath is already a nice container. And we do have _some_ code
in ParamHelpers to plot "traces" from this. Which could be _heavily_
improved I guess. But it's a starting point.
Another option is to look at specific parameters, where the model
fitting itself already produces a "curve" or "trace".
Prominent examples are iterations vs OBB performance in bagging /
forests / boosting type methods.
We would really like to "know" though the software whether a learner
supports this (eg using mlr's tagging mechansim) and have a common API
call for this.
A downside to this is that is usually implemented always a bit
differently, and only for very "normal" performance measures, so
currently we are working on having an API call for the OOB _predictions_
for the whole trace.
Then you can compute anything on this, post-hoc.
> Training/prediction time curves (depending on eager/lazy
learner). This could potentially leverage the already implemented
$opt.path but may need to be spit based on the type of learner.
Yup
> Model complexity curves (ie plotting # of non-zero coefficients
against the output of a scoring function)
Yup
> Model visualization on pca-reduced data to visualize separation
boundaries or highly dimensional data in 2D
Also nice. Another option might be multim. scaling. We dont really have
plotting options for both currently.
> Support for std bands for learning curves and any other model
eval visualizations where a mean is calculated over several CV
iterations for the scoring function
This would be easy to add what we currently have.
> Support for epochs instead of train size (ie for neural nets) for
applicable model visualizations
> All model eval visualizations would ideally include
"benchmarking", such as sklearn's Dummy model (looks like this is
referenced issue #712 on github)
Dummy models I guess you mean? So the most naive model as a base line?
Yes, this was asked a couple of times.