matplotlib, code convention: what do plots return, fig or None, or fig_or_ax?

[josef...@gmail.com]

One question for the mailing list, triggered by Ralf's pull request

We are starting to converge to a more consistent handling of plot
arguments and returns.

plot functions have an optional ax argument. If it is None, then a
figure is created inside the function.

Now, the question is what the function should return.

I converged to returning fig_or_ax which is either a figure or an axes
instance. If ax was given as function argument, I'm returning the same
instance.
Ralf is returning fig or None: if the figure was created inside the
function, then fig is returned. If ax was specified as function
argument, then None is returned.

I essentially have no preference between the two, but I would like
consistency, and, if possible, follow a standard matplotlib pattern.

Any opinions?

Josef

[Josh Hemann]

Josef, I am glad you are asking this question. I cannot say anything
about the existing matplotlib standards, but in my own work I have
found it very helpful to make my plotting functions return the fig
object. I try to make the function definitions take the usual keywords
users would expect to see (e.g. setting axes titles, line colors etc),
but return fig in case they want to modify something else after the
plot has been created.

Josh

[Skipper Seabold]

Maybe we should ask on the matplotlib mailing list as well? Would
probably get more feedback there.

Question, if we use given axes or use plt.gca(), why do we need to
return anything? Can't we think of this as working in-place? If we
create a figure, it makes sense to return it though. So I think I
prefer Ralf's way.

It also depends on what the function itself does. The rugplot example
from yesterday, for instance, wouldn't return anything because it's
intended to act on an existing axes. Similar with (TBD) line plots for
qqplot. They're intended to add to an existing graph rather than
create one.

Skipper

[josef...@gmail.com]

On Tue, Dec 20, 2011 at 10:00 AM, Josh Hemann <josh_...@yahoo.com> wrote:

> Josef, I am glad you are asking this question. I cannot say anything
> about the existing matplotlib standards, but in my own work I have
> found it very helpful to make my plotting functions return the fig
> object. I try to make the function definitions take the usual keywords
> users would expect to see (e.g. setting axes titles, line colors etc),
> but return fig in case they want to modify something else after the
> plot has been created.

Thanks for replying, Yes, I think we want users to get hold of the
figure or axis, (gcf, gca smells to much like globals to me)

The question is also, whether we should return ax, if the user already
has hold of it

def aplot(x, ax=None):
if ax is None:
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
else
fig = None
ax.plot(.....)
return fig #you might get None

or

def aplot(x, ax=None):
if ax is None:
fig_or_ax = plt.figure()
ax = fig.add_subplot(1,1,1)
fig_or_ax = fig
else
fig_or_ax = ax

ax.plot(.....)
return fig_or_ax #you might get your ax back

------
fig = plt.figure()
ax = fig.add_subplot(2,1,1)
whatsthis1 = aplot(x1, ax=ax)
ax = fig.add_subplot(2,1,2)
whatsthis2 = aplot(x2, ax=ax)

ax.set_title('')

Since ax cannot be used in chains, ax.set_title(..).set_..., I think I
prefer now also Ralfs, standard python for inplace modification.

Going through some matplotlib code, matplotlib returns the individual
plot elements, e.g. ax.xcorr,

if usevlines:
a = self.vlines(lags, [0], c, **kwargs)
b = self.axhline(**kwargs)
else:

kwargs.setdefault('marker', 'o')
kwargs.setdefault('linestyle', 'None')
a, = self.plot(lags, c, **kwargs)
b = None
return lags, c, a, b

or plt.hist returns the list of patches at the end

but I never used those plot elements.

Josef

[josef...@gmail.com]

Ok, I will rewrite my functions in Ralf's way.
(I guess I just didn't like to sometimes return nothing (None) and
sometimes something.)

>
> It also depends on what the function itself does. The rugplot example
> from yesterday, for instance, wouldn't return anything because it's
> intended to act on an existing axes. Similar with (TBD) line plots for
> qqplot. They're intended to add to an existing graph rather than
> create one.

returning the individual plot element could then be a useful
alternative after all. It would avoid having to walk through the
figure to find the right element if a user wants to change anything.
(which I did several times).

This raises then the question, how serious we want to get with the plots.
Initially I considered them as a quick interactive aid and recipes,
but with additions like Ralf's, the plots are not trivial, so given
the user more options to fine-tune instead of copying the function
will become more useful

Josef

>
> Skipper

[Skipper Seabold]

True.

I'm still wary of shipping too much plotting from our end because of
inevitable bit rot (like scikits.timeseries), though if they're tested
then it should be fine. I usually stay pretty up-to-date with MPL.

Skipper

[josef...@gmail.com]

Testing would be another point in favor of returning the main
individual plot elements.

>>> this = ax.plot(x)
>>> np.testing.assert_equal(x, this[0].get_ydata())

finding which is which in the figure tree was the main reason I gave
up on testing the content of the plot.

It won't make sense for gridplots, which have too many plot elements.

Josef

>
> Skipper

[Skipper Seabold]

I wouldn't expect us to need tests like this though. I would just
think smoke tests, running the plots from time to time to check for
deprecation warnings or errors rather than testing elements (ie., the
warnings we're seeing now with master MPL...). The timeseries scikit
ended up choking on some of its own plotting functions related to
visual customization because of MPL deprecation and internal changes,
rather than plotting being incorrect. I think we can always expect
that what we're doing is what we think we're doing, it's just the how
that *could* change.

Skipper

[John Hunter]

On Tue, Dec 20, 2011 at 9:31 AM, Skipper Seabold <jsse...@gmail.com> wrote:

> Maybe we should ask on the matplotlib mailing list as well? Would
> probably get more feedback there.

My advice would be to be consistent about what you return: sometimes
returning and ax, sometimes a fig, or sometimes None, would be
confusing. I suggest always returning the figure. Since the axes has
a reference to the figure (ax.figure), you can always return the
figure even if the user passes in the axes. Also, the figure has a
reference to the axes (fig.axes) so if the user gets back the figure
he can always access the axes that were created.

Finally, I would avoid anything in pyplot (gca, gcf) because you can't
assume your user has imported pyplot (eg they may be embedding your
stats plots in a GUI and importing pyplot could cause conflicting GUI
mainloops). I would only import pyplot if the axes is None

if ax is None:
import matplotlib.pyplot as plt
fig = plt.figure()
ax = fig.add_subplot(111)
else:
fig = ax.figure

JDH

[josef...@gmail.com]

Thanks, I like this way of always returning the same thing.
And thanks for clarifying the gca, gcf story, I didn't know why I
don't like them (except that they sound too much like matlab globals).

Ralf wrote two helper functions that we can use then across the board,
after adding ax.figure
https://github.com/statsmodels/statsmodels/pull/122/files#L7R13

Josef

>
> JDH

[John Hunter]

On Tue, Dec 20, 2011 at 11:53 AM, <josef...@gmail.com> wrote:

> Ralf wrote two helper functions that we can use then across the board,
> after adding ax.figure
> https://github.com/statsmodels/statsmodels/pull/122/files#L7R13

Cool, I added a couple of comments.

[Ralf Gommers]

On Tue, Dec 20, 2011 at 6:53 PM, <josef...@gmail.com> wrote:

On Tue, Dec 20, 2011 at 12:30 PM, John Hunter <jdh...@gmail.com> wrote:
> On Tue, Dec 20, 2011 at 9:31 AM, Skipper Seabold <jsse...@gmail.com> wrote:
>
>> Maybe we should ask on the matplotlib mailing list as well? Would
>> probably get more feedback there.
>
> My advice would be to be consistent about what you return: sometimes
> returning and ax, sometimes a fig, or sometimes None, would be
> confusing.  I suggest always returning the figure.  Since the axes has
> a reference to the figure (ax.figure), you can always return the
> figure even if the user passes in the axes.  Also, the figure has a
> reference to the axes (fig.axes) so if the user gets back the figure
> he can always access the axes that were created.
>
> Finally, I would avoid anything in pyplot (gca, gcf) because you can't
> assume your user has imported pyplot (eg they may be embedding your
> stats plots in a GUI and importing pyplot could cause conflicting GUI
> mainloops).  I would only import pyplot if the axes is None

Thanks, this is an important point. I was also using pyplot to grab colormaps, guess I should do this simply through matplotlib.cm. Am I correct in saying that the only thing that I'd need pyplot for is the figure() and show() functions? Anything else I can import from elsewhere?
 

>
> if ax is None:
>    import matplotlib.pyplot as plt
>    fig = plt.figure()
>    ax = fig.add_subplot(111)
> else:
>    fig = ax.figure

Thanks, I like this way of always returning the same thing.

I think returning None is a normal pattern in Python too, but either way is okay with me. I'd prefer to return ax though instead of fig since the figure can be easily obtained from ax. ax is the object you usually work with.

Ralf

[John Hunter]

On Tue, Dec 20, 2011 at 2:55 PM, Ralf Gommers
<ralf.g...@googlemail.com> wrote:

> Thanks, this is an important point. I was also using pyplot to grab
> colormaps, guess I should do this simply through matplotlib.cm. Am I correct
> in saying that the only thing that I'd need pyplot for is the figure() and
> show() functions? Anything else I can import from elsewhere?

Yes, you can get everything else from the API. In my library code I
tend not to call show, and just let the user do it if they are calling
plot functions. I would suggest just calling plt.figure and nothing
else from pyplot. I can advise if you have specific use cases you are
trying to duplicate w/ the API.

JDH

[Fernando Perez]

On Tue, Dec 20, 2011 at 9:30 AM, John Hunter <jdh...@gmail.com> wrote:
> On Tue, Dec 20, 2011 at 9:31 AM, Skipper Seabold <jsse...@gmail.com> wrote:
>
>> Maybe we should ask on the matplotlib mailing list as well? Would
>> probably get more feedback there.
>
> My advice would be to be consistent about what you return: sometimes
> returning and ax, sometimes a fig, or sometimes None, would be
> confusing.  I suggest always returning the figure.  Since the axes has
> a reference to the figure (ax.figure), you can always return the
> figure even if the user passes in the axes.  Also, the figure has a
> reference to the axes (fig.axes) so if the user gets back the figure
> he can always access the axes that were created.

One point to consider in light of the release of the ipython notebook:
in the notebook, figure objects know how to display themselves, so you
get the following kind of behavior:

In: foo() # foo returns a figure
Out:
#
# A figure is drawn here
#

which works fairly well, *as long as foo()* doesn't call show().
Because if it calls show(), then you get a duplicate figure, the one
from show() and the return value.

With the builtin methods from pyplot, while most of them do call
show() fortunately I don't think any of them returns the figure, they
all return axes or orther objects.

I'm not saying that you need to constrain your design to the notebook,
but I think we may wan to explore, now that this environment is
available and people seem to like it, how to make plotting use
patterns as easy and pleasant as possible.

Cheers,

f

[josef...@gmail.com]

can you explain this a bit more?
we never call show() in the plot functions, and I thought pyplot
doesn't call show itself either.
what's the connection between show and returning a figure?

Functions that build a plotgrid, with many subplots don't have any
options for creating a figure or axis instance. Figure is always
created, It's just

fig =plt.figure()
for i in range(......):
ax = fig.add_subplot(nrows, ncols, i+1)
...

>
> I'm not saying that you need to constrain your design to the notebook,
> but I think we may wan to explore, now that this environment is
> available and people seem to like it, how to make plotting use
> patterns as easy and pleasant as possible.

That's the reason, besides consistency, that I started this and
similar discussions on the usage of matplotlib. I would like to avoid
a pattern that restricts what users can do with the plots, whether
interactive, printing, in a gui or in a web interface (if possible
without jumping through too many hoops to accomplish this). Except for
following occasionally related discussions on matplotlib user, I only
have experience with interactive pyplot and printing.

So, any feedback what works and what doesn't is very welcome.

Josef

>
> Cheers,
>
> f

[John Hunter]

On Dec 20, 2011, at 5:56 PM, josef...@gmail.com wrote:

> can you explain this a bit more?
> we never call show() in the plot functions, and I thought pyplot
> doesn't call show itself either.
> what's the connection between show and returning a figure?

I think Fernando misspoke. As far as I know, none of the pyplot methods call show, but the do call draw if interactive.

I've assumed the notebook inspects the internal pyplot datastructures to see if any figures were created, and if so, inline them. This is how the sphinx plot directive works. But I haven't looked at the ipython internals.

JDH

[Fernando Perez]

On Tue, Dec 20, 2011 at 4:23 PM, John Hunter <jdh...@gmail.com> wrote:
>
> I think Fernando misspoke. As far as I know, none of the pyplot methods call show, but the do call draw if interactive.

Yes, sorry: draw_if_interactive is what triggers the display.

> I've assumed the notebook inspects the internal pyplot datastructures to see if any figures were created, and if so, inline them. This is how the sphinx plot directive works. But I haven't looked at the ipython internals.

Yup, our inline backend records any calls to draw_if_interactive and
then flushes out any figures at the end of the execution of user code:

https://github.com/ipython/ipython/blob/master/IPython/zmq/pylab/backend_inline.py

(ignore the docstring that says it's SVG for Qt only, now it does SVG
and png for both the notebook and the qt console, we just need to
update the docstring).

Cheers,

f

[Steve Genoud]

Hello,

What would you think of introducing a helper decorator to do the grunt work? The decorator automatically fetches the axes and feed them to the function. Also, it adds the figure as first return value. This would also easily create a common interface for all the functions.

Is it something that would be useful?

Best,

Steve

The code will be something like that, it rests on the decorator module:

from decorator import decorator

import inspect

def _insert_in_return_value(to_insert, return_value):

    if isinstance(return_value, tuple):

        return tuple([to_insert] + list(return_value))

    elif return_value is None:

        return to_insert

    else:

        return to_insert, return_value

@decorator

def get_axes(f, *args, **kwargs):

    ax_index = inspect.getargspec(f)[0].index('ax')

    ax = args[ax_index]

    fig, ax = _create_mpl_ax(ax)

    args_out = list(args)

    args_out[ax_index] = ax

    out = f(*args_out, **kwargs)

    return _insert_in_return_value(fig, out)

[John Hunter]

On Tue, Dec 20, 2011 at 7:24 PM, Fernando Perez <fpere...@gmail.com> wrote:
> Yup, our inline backend records any calls to draw_if_interactive and
> then flushes out any figures at the end of the execution of user code:
>
> https://github.com/ipython/ipython/blob/master/IPython/zmq/pylab/backend_inline.py
>
> (ignore the docstring that says it's SVG for Qt only, now it does SVG
> and png for both the notebook and the qt console, we just need to
> update the docstring).

Just in case all this is a bit arcane for those not wrangling with
ipython and mpl internals, the takehome message is

* it is probably best to return a figure instance so that ipython
notebook knows how to render it, and so the users have a consistent
return value for all your plotting functions whether they create one
or more axes.

* don't call show or draw_if_interactive in your library code

* don't use pyplot for anything but plt.figure and only then when the
user passes ax=None

JDH

[josef...@gmail.com]

On Wed, Dec 21, 2011 at 8:11 AM, Steve Genoud <steve....@gmail.com> wrote:
> Hello,
>
> What would you think of introducing a helper decorator to do the grunt work?
> The decorator automatically fetches the axes and feed them to the function.
> Also, it adds the figure as first return value. This would also easily
> create a common interface for all the functions.
>
> Is it something that would be useful?

I think it would be useful, but I'm not sure it's worth it (at this
stage) do save one or two lines and on having to follow the new
convention.

roughly what I'm thinking

plots would be good candidates for decorators because they are more
isolated and standalone functions (or methods),
time overhead is not a problem

automatic inserting of return arguments makes code inspection more
difficult. (My main way of quickly checking some functions in
different packages, for example scipy and matplotlib is to call up the
source in Spyder.) This will be more difficult or misleading if a
decorator changes what is returned.

using the decorator module adds another dependency, or we would have
to maintain a copy inside statsmodels

If we add the decorator module, it would be useful to see if we can
use the same idea of decorators for consistent interfaces to other
functions, e.g. statistical tests, a memoize decorator and similar.

On the other hand, if I remember correctly, Michele Simionato, when he
introduced the decorator module, mentioned that there is quite a bit
of overhead involved, and these milliseconds here and there for
convenience methods might start to add up, or not.

pros and cons, and I'm open either way.

For my personal taste in coding, I don't mind having to write a bit of
boiler plate if it makes reading the code easier, but the cache
decorator turned out to be very useful.

Josef

[josef...@gmail.com]

Thanks for the nice summary, we can add this to our developer documentation.

I didn't know about the last point and need to check my code for this.

Another question then for the case when we create several subplots in
a figure, and don't have an ax argument:

Do these rules also apply if we do have a `fig` instead of an `ax`?
Does it make sense for the user to pass in a fig instance as argument
and we attach the subplots, instead of always creating fig=plt.figure
inside the function?

Josef

>
> JDH

[John Hunter]

On Wed, Dec 21, 2011 at 9:21 AM, <josef...@gmail.com> wrote:

> Another question then for the case when we create several subplots in
> a figure, and don't have an ax argument:
>
> Do these rules also apply if we do have a `fig` instead of an `ax`?
> Does it make sense for the user to pass in a fig instance as argument
> and we attach the subplots, instead of always creating fig=plt.figure
> inside the function?

I should have said

* don't use pyplot for anything but plt.figure and only then when the

user passes ax=None or fig=None. Ie, if the user passes a figure, you
can the figure API to create axes. Once you have an axes or figure,
you can do everything you need with it and don't need pyplot for
anything. But if the user gives you neither, use pyplot.figure to
create one and then use the API from that point forward.

Elaborating a bit, if you have a plot that uses multiple axes, eg a
plotgrid, then I usually do a "fig=None" in the function signature
rather than an "ax=None". This will allow the user who wants to use
their own Figure instance (eg a GUI application writer) to do so. The
usage is the same:

if fig is None:

import matplotlib.pyplot as plt
fig = plt.figure()

JDH

[John Hunter]

On Wed, Dec 21, 2011 at 9:11 AM, <josef...@gmail.com> wrote:

> automatic inserting of return arguments makes code inspection more
> difficult. (My main way of quickly checking some functions in
> different packages, for example scipy and matplotlib is to call up the
> source in Spyder.) This will be more difficult or misleading if a
> decorator changes what is returned.

+1 - IMO, the decorator makes the code less readable and only saves
you a line or two per function.

JDH

[josef...@gmail.com]

very good, all clear now, I think. I will adjust my code.

Thanks,

Josef

>
> JDH

[Ralf Gommers]

This is the only one I can't find:
        plt.setp(ax.get_xticklabels(), fontsize='small', rotation=45,
                 horizontalalignment='right')

Of course I can change this to
    labels = ax.get_xticklabels()
    for label in labels:
        label.set_fontsize='small'
        ...<etc>
 but is there anything more compact?

Thanks,
Ralf

[John Hunter]

On Wed, Dec 21, 2011 at 4:04 PM, Ralf Gommers
<ralf.g...@googlemail.com> wrote:
Yes, you can get everything else from the API.  In my library code I
>> tend not to call show, and just let the user do it if they are calling
>> plot functions.  I would suggest just calling plt.figure and nothing
>> else from pyplot.  I can advise if you have specific use cases you are
>> trying to duplicate w/ the API.
>
>
> This is the only one I can't find:
>         plt.setp(ax.get_xticklabels(), fontsize='small', rotation=45,
>                  horizontalalignment='right')
>
> Of course I can change this to
>     labels = ax.get_xticklabels()
>     for label in labels:
>         label.set_fontsize='small'
>         ...<etc>
>  but is there anything more compact?

One teach-a-man-to-fish solution is to open up matplotlib.pyplot and
search for "def setp". You will find

@docstring.copy(_setp)
def setp(*args, **kwargs):
ret = _setp(*args, **kwargs)
draw_if_interactive()
return ret

So setp just calls _setp. If you search for _setp, you will find

from matplotlib.artist import setp as _setp

So that is the API version. pyplot.setp is a thin wrapper that calls
matplotlib.artist.setp and then calls draw_if_interactive. That is
fairly standard pyplot boilerplate.

JDH

[Ralf Gommers]

On Wed, Dec 21, 2011 at 11:08 PM, John Hunter <jdh...@gmail.com> wrote:

On Wed, Dec 21, 2011 at 4:04 PM, Ralf Gommers
<ralf.g...@googlemail.com> wrote:
 Yes, you can get everything else from the API.  In my library code I
>> tend not to call show, and just let the user do it if they are calling
>> plot functions.  I would suggest just calling plt.figure and nothing
>> else from pyplot.  I can advise if you have specific use cases you are
>> trying to duplicate w/ the API.
>
>
> This is the only one I can't find:
>         plt.setp(ax.get_xticklabels(), fontsize='small', rotation=45,
>                  horizontalalignment='right')
>
> Of course I can change this to
>     labels = ax.get_xticklabels()
>     for label in labels:
>         label.set_fontsize='small'
>         ...<etc>
>  but is there anything more compact?

One teach-a-man-to-fish solution is to open up matplotlib.pyplot and
search for "def setp".  

I did just that, saw "def setp" and stopped reading. Sorry for the noise.

Ralf