gonum plot redesign: the next generation

[Dan Kortschak]

During discussion in a PR[1] we decided that conversation should move to
gonum-dev. This is that movement (including an advertisement to
plotinum-discuss to participate - please join on gonum-dev).

The discussion at the PR conversation was prompted by planned changes to
plot.New that lazily error, hopefully result in a usable zero plot.Plot
value. However, I believe that the decisions made here impact on
features that we have planned in the previous design discussion
(including panels and more configurable axes).

A significant amount of change has happened to the API since the fork
from plotinum and I think we are making the suite a lot stronger, but I
also think we need to have some discussions about the direction to make
sure that continues (also bearing in mind that we have a practical
cut-off date for API change of January 2016 when plotinum will cease to
exist except in local repositories).

I would like to hear what users would like to see in the plot API wrt
those features, but also have dev input into how to sanely achieve the
kinds of things that are being discussed at the PR.

Please read the PR discussion first, and probably also have a re-read of
the design wiki page[2] before commenting here - if you have wiki edit
permissions and you see a planned feature that has been implemented,
please edit to add a strike through.

thanks

[1]https://github.com/gonum/plot/pull/190
[2]https://github.com/gonum/plot/wiki/Plotinum-redesign

[Brendan Tracey]

Thanks for starting the conversation, and to all who have put in much work so far.

A couple of short thoughts now:

- When using plotinum, there were three main things that were frustrating (everything else was minor). 1) No colored scatter. 2) Hard to scale the plot resolution 3) Using []struct{X,Y float64} instead of struct{X,Y []float64}. 1) is a feature request, so irrelevant here, and I believe 2 has already been fixed. I have mentioned 3 in the past, but I don't remember seeing any decisions on fixing on one or the other or both. It should be decided before the API is fixed. Personally, I much more often have data as slices (for example, a single set of x data with multiple y sets)

- Coming up with a good design for Panels seems like a good idea. Another use is heat maps/colored scatter to show the legend of colors. We should at least think about it enough that we know we can implement it without backward incompatibilities. For example, maybe a Plot is a series of panels, one of which is the main plot area. This may require changing some of the canvas stuff

- I don't like the design of Palettes. Fundamentally, it should be along the lines of
type Palette interface {
    Color(float64) color.Color
}
i.e. a Palette is a function for associating a color to each value. This could be a set of banded colors (like the available heat maps), could be a set of continuous colors, or could be something like "Use Rainbow except for NaN/inf which make black" or "make 1.267 green because it's special". These are easy to encode in a function, and hard to encode with just a list of colors. The plotter then just needs to query the pallette to find the appropriate color.

- On the error issue, we might consider some form of non-error indication. For example, the fmt package is really nice in that something happens even if you have a mismatch in the number of elements and the number of format strings. Often with print statements, it's really nice that a basic error there doesn't crash my program, since it wasn't that important to begin with. I feel similarly to plotting. Most of the plots I make are not for publication. The point is to see how something looks, for example to verify that things are working correctly. In this case, I'd rather have the plot write something ugly that's along the lines of my request than to panic and crash the program. So, in WriterTo for example, maybe write the plot even if there are errors (probably with some sort of on-plot moniker that errors were present), and also return the error. On a second pass, I can fix the problem with the plot routine, but I also got to see the results I wanted.

[Dan Kortschak]

On Wed, 2015-04-15 at 22:42 -0700, Brendan Tracey wrote:
> - When using plotinum, there were three main things that were
> frustrating (everything else was minor).

> 1) No colored scatter.

Have you filed an issue for this? If not, please do - I vaguely recall
discussion about this and I think there is was debate about its merits,
but an issue presumably would help it along.

> 2) Hard to scale the plot resolution

> 3) Using []struct{X,Y float64} instead of struct{X,Y []float64}.

Because e.g. XYer is an interface, you can use the second type if you
implement Len and XY on that type:

type structXY struct {
X, Y []float64
}

func (xy structXY) Len() int {
if len(xy.X) != len(xy.Y) {
panic("data slice length mismatch")
}
return len(xy.X)
}

func (xy structXY) XY(i int) (x, y float64) {
return xy.X[i], xy.Y[i]

}


> 1) is a feature request, so irrelevant here, and I believe 2 has
> already been fixed. I have mentioned 3 in the past, but I don't
> remember seeing any decisions on fixing on one or the other or both.
> It should be decided before the API is fixed. Personally, I much more
> often have data as slices (for example, a single set of x data with
> multiple y sets)
>
> - Coming up with a good design for Panels seems like a good idea.

Yes, this was the primary motivator for the OP.

> Another use is heat maps/colored scatter to show the legend of
> colors. We should at least think about it enough that we know we can
> implement it without backward incompatibilities. For example, maybe a
> Plot is a series of panels, one of which is the main plot area. This
> may require changing some of the canvas stuff

Chris Tessum expressed an interest in implementing a legend for HeatMap
(adding a Thumbnailer to HeatMap) - I think this is probably an easier
approach to this problem.

> - I don't like the design of Palettes. Fundamentally, it should be
> along the lines of
> type Palette interface {
> Color(float64) color.Color
> }
> i.e. a Palette is a function for associating a color to each value.

This is a fundamental issue with how palettes work. There has been some
discussion about formalising the notion of sequential and qualitative
palettes as interfaces in the palette package. I think this better
handles the variety of behaviours of the three different classes of
palettes; I would be very worried if a client could get a qualitative
palette and call Color(float64) on that.

I propose for palette:

// Sequential represents a sequentially ordered color palette.
type Sequential interface {
Palette

// Color returns a fractional color from the
// palette over the interval [0,1]. If f does
// not correspond to an element of the palette,
// it is calculated by interpolation.
Color(f float64) color.Color
}

// Qualitative represents a qualitative unordered color palette.
type Qualitative interface {
Palette

// Label returns the label of the ith color.
Label(i int) string
}

and the following addition to Diverging (renamed from DivergingPalette):

// Diverging is a collection of colors ordered into a palette
// with a critical class or break in the middle of the color range.
type Diverging interface {
Palette

// Color returns a fractional color from the
// palette over the interval [0,1]. If f does
// not correspond to an element of the palette,
// it is calculated by interpolation.
Color(f float64) color.Color

// CriticalIndex returns the indices of the lightest
// (median) color or colors in the Diverging, this
// corresponds to the colors at or adjacent to the
// color returned by Color(0.5).
// The low and high index values will be equal when
// there is a single median color.
CriticalIndex() (low, high int)
}

The reason these are not composable (Color() is not wrapped by its own
interface) is that I don't believe that it is reasonable to compose
these types of palettes in the real world.

> This could be a set of banded colors (like the available heat maps),
> could be a set of continuous colors, or could be something like "Use
> Rainbow except for NaN/inf which make black" or "make 1.267 green
> because it's special". These are easy to encode in a function, and
> hard to encode with just a list of colors. The plotter then just needs
> to query the pallette to find the appropriate color.
>
> - On the error issue, we might consider some form of non-error
> indication. For example, the fmt package is really nice in that
> something happens even if you have a mismatch in the number of
> elements and the number of format strings. Often with print
> statements, it's really nice that a basic error there doesn't crash my
> program, since it wasn't that important to begin with. I feel
> similarly to plotting. Most of the plots I make are not for
> publication. The point is to see how something looks, for example to
> verify that things are working correctly. In this case, I'd rather
> have the plot write something ugly that's along the lines of my
> request than to panic and crash the program. So, in WriterTo for
> example, maybe write the plot even if there are errors (probably with
> some sort of on-plot moniker that errors were present), and also
> return the error.

I quite like this idea - sort of like the diagonal banded DRAFT you see
on draft documents.

[Volker Dobler]

Hello

I do like the design of gonum/plot and thought about adding some

useful features.

Autoscaling Tics and GlyphBoxer:

Currently there is just full autoscaling, after automatic determination of min

and max you may manually change it afterwards. This works well for

all kind of stuff but has two drawbacks:

1. It is hard (or impossible) to expand the range of an axis to the next

    major tick marks

2. The range is not covering the added space if the glyphs require extra

    space.

Both produce visually suboptimal plot if you try to draw a border.

Date/Time axes:

Currently unsupported.

Using floats for time requires a suitable reference time and some

helper functions, that would be easy. The more complicated stuff

is in generating ticks and labels.

Faceted Plots:

Unsupported, "undoable".

I think all three points are doable but require a fancier range setup

mechanism: Axis ranges are a function of

 - fix limits, constrained or full autoscaling based on data

 - constraints from other plots (e.g. in a faceted plot)

 - constraints from glyphs (e.g. dodged bars or large bubbles)

 - constraints from the ticks (e.g. min and max must fall on

   minor ticks).

I think the axis range setup code just needs one more round after autoscaling

with the DataRanger.

Are these improvements interesting? Should I come up with a design and

a PoC implementation?

Regards

Volker

Background: I contributed gnuplot's constrained autoscaling and the

user defined palette. I'm the author of github.com/vdobler/chart

(more a toy project while learning Go) and github.com/vdobler/plot

(even more a toy aimed at understanding ggplot2 better).

[Dan Kortschak]

I would be interested in these things.

[Ethan Burns]

I like these too.

If you decide to work on them, I recommend you check out the issue tracker. I know that at least one of these (date/time tick marks) has an open issue that you should assign to yourself.

Ethan

--
You received this message because you are subscribed to the Google Groups "plotinum-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to plotinum-discu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Volker Dobler]

Thanks for the support.

I started to play around starting with faceted plots (which are especially ugly)

and reached the following state:

which looks promising.

I now think the main obstacle in flexible axis and plots is the padding of the plot canvas

to prevent glyph clipping. I have to admit I do not understand how this padding

really works. I carefully studied the bubbles tutorial, but I simply do not understand

how the internals work (padX, padY) and how this interferes with axis drawing.

This padding also has one drawback: It doesn't expand the axis so drawing mirrored

axis is almost impossible (same for gray backgrounds, etc.).

I thus think, that this expansion of the plot needs rework. I'd like to propose to change this

padding of the draw area to an expansion of the axis until all glyphs can be drawn.

This would require (my limited understanding) to allow inversion of what Normalizer

does: Compute all glyphs and see if their bounding boxes fall into the axis ranges:

If not expand axis.

The other solution would be:

  A) Have the Plotters expand the axis ranges. If a plotter generates a glyph which

     doesn't lie inside the data range the data range is expanded. This would work

     very well for Plotters who generate glyphs based on data range (e.g. some kind

     of tiling) but requires inverting Normalisers output (e.g. for bubble plots in log scale).

 B) Ignore ultra large glyphs which are not data range based (e.g. bubbles) and just

    expand the axis range so that min and max lie at 5% and 95% of the plot area

    (that's what ggplot2 does amongst other things).

The most flexible would be a combination of A and B but this would require and

additional training step of the axis where all plotters are queried if the need to

expand the current data range.

Would such major rework of the API be possible?

Regards,

Volker

[Ethan Burns]

I just skimmed your email (I'll read it in more detail this weekend), and I see that you had difficulty understanding the padding code. I have difficulty with it too. In fact, I can never remember how it works. Luckily I got sick of re-deriving it each time and added some notes here: https://github.com/gonum/plot/issues/100. Maybe you'll find them useful. 

Ethan

[Ethan Burns]

Hi Volker,

The demo plot you sent looks very promising. I'm happy that someone is working on this, because it's something that I always wanted in Plotinum but never got to. (By the way, I'm curious what the API looks like to create such a plot.)

I think that your initial proposal of inverting Normalizer in order to determine the ranges sounds good. It won't require extra burden on Plotter creators like option A will (as you noticed, the GlyphBoxer code is already way too complex, hence why it needs a special tutorial). In my opinion option B is a non-starter, because I don't want the plot to clip a glyphs. At least, I don't want it to clip when automatically setting the axis ranges. I think it's OK to clip if the user overrides the automatic ranges; if we fit glyphs by extending the axes, and we can't extend the user-set axes, then how can we possibly prevent clipping?

Anyway, feel free to make backwards-incompatible changes. When I sent the announcement email about gonum/plot, I stated that we plans on making breaking changes, at least up until January 2016, when code.google.com comes down and Plotinum with it.

Ethan

[Volker Dobler]

I did some more experiments and implemented the simple ideas I had.

You can find the current state of my PoC in the redesign of my fork of plot here

https://github.com/vdobler/plot-1/tree/redesign

See the design.md https://github.com/vdobler/plot-1/blob/redesign/design.md

document for the current state and the proposed API additions (several) and

changes (few).

I'll be on holidays for the next three weeks and will be offline for extended

periods of time. But do not hesitate to contact me if you have questions or

suggestions, just do not expect timely answers.

The following examples have been generated by some sample scripts

in testdata.

​

--

Dr. Volker Dobler

[Dan Kortschak]

This looks good. Do you have a roadmap for adding these (are you back)? I'll look at the code more closely in review, so the only comment I have right now is that I think the justification of time ticks on the X axis would be better the hours were aligned (currently the day tick has the hour one line above the rest).

[Volker Dobler]

Thanks for the kind words! Yes I'm back :-)
I do not have a timeline for adding this stuff, currently it is more
a prove of concept than something reviewable.
I thought about the following process:
1. I'll finish the PoC adding at least common color axis
2. I'll come up with an API proposal; probably not everything
can be done in an backwards compatible way.
3. If API seems reasonable I'll prepare several PRs

One thing needs some major rework: Handling fonts.
Currently if you want to draw text you need a font, this may
fail which is reflected in functions like makeAxis or plot.New:
They return an error.
Assume a faceted plot. Due to a special plotter a color (or size)
scale is needed which needs axis labels which need a font.
Suddenly all methods/functions may fail due to missing fonts.

I thought about some kind of theming: Let a Plot have a Theme
and the Theme contains all the fonts. Creating a Theme can fail
due to missing fonts but plot creation won't return errors, at least
not due to missing fonts. Settings in a Theme could be organised
hierarchically (like in CSS or how ggplot2 does it) so that there
is no need to set the line color of the minor y-axis tics to black
once the main line color is set; but you can still do
yaxis.tics.minor:color = green if you need too.

Volker

On Sun, Jul 19, 2015 at 1:05 AM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> This looks good. Do you have a roadmap for adding these (are you back)? I'll look at the code more closely in review, so the only comment I have right now is that I think the justification of time ticks on the X axis would be better the hours were aligned (currently the day tick has the hour one line above the rest).

--
Dr. Volker Dobler

[Dan Kortschak]

On Mon, 2015-07-27 at 12:50 +0200, Volker Dobler wrote:
> One thing needs some major rework: Handling fonts.
> Currently if you want to draw text you need a font, this may
> fail which is reflected in functions like makeAxis or plot.New:
> They return an error.

Do we have any further ideas on this aspect of the redesign?

[Volker Dobler]

Hi,

I do have an idea.

Problem:
The problem is, that a plot has a huge amount of places where
you will want to tweak its appearance e.g. fonts are used in
- plot title
- axis title
- axis label
- legend entries
- legend title (not jet)
- facet labels (not jet)
- text glyphs (not jet)
and each requires a font. Switching from Arial for plots used on
the web to Concrete Roman for publishing is painful: If you forget
to change to font on the plot title your plot mixes different fonts.
(Gnuplot suffers from this issue.)
Additional ugliness for font comes from fonts being loaded and
the necessity to deal with the associated failures.

Solution:
What works well are themes and a hierarchical organisation.
CSS is not that bad, it just has a complex box model.

How about packing all styling information into a new type Theme
which provides hierarchical settings with fallback to settings
"higher up"? Creating (or instantiating) a theme may fail (e.g.
because fonts are unavailable) but using a theme will is always
possible.
This would allow e.g. an axis to retrieve the tic label font
(typeface, size, color) if tics labels have to be drawn. If no special
tic label size is specified in the theme the value is looked up
in one of the parent nodes.
Such a hierarchy could look like the one ggplot2 uses, see
http://docs.ggplot2.org/current/theme.html

Examples:
E.g. theme["/axis.line.color"] = blue could set the color of
any line related to an axis to be blue while
theme["/axis/x/tics/major.font.size"] = 8 would fix the font size
of the major tics label to 8. Unset values propagate up and may
end accessing theme["/.font.size"] which is the global font size.

Maybe an autogenerated color axis (needed because some
ColorBubbles Plotter is used) could be set up pragmatically by
a plot. It would use theme["/axis/color/tics/label.font..."] to
draw the tic labels; if unset this might use some global setting.

Such theming would allow to provide high quality themes, e.g.
Traditional, Ggplot2, Gnuplot to mimic common plot appearances.

Benefit:
Having themed plot would:
- replace lots of draw.{Text,Line}Style fields with either a theme
or a theme parameter to the drawing methods.
- allow to add new visual stuff easily, e.g. a box around legends
or a background to legends or a title to the legends: You need
one switch to turn it on/of and entries in the default themes.
- allow to have non-failing functions/methods, e.g. a constructor
plot.NewFrom(theme Theme) *plot.Plot would be possible.

TODO: Find a typesafe representation of various style settings and
a sensible hierarchy.

Drawbacks:
Lots of code changes, and incompatible API changes.


What do you think?

V.

--
Dr. Volker Dobler

[Sebastien Binet]

that sounds really good.
wrt lots of code changes: we have unit tests :)
wrt incompatible API changes: the window of opportunity for breaking
API is still open (but shrinking: end of 2015, IIRC)
of course, breaking people code should be justified but IMHO having a
nice theming facility would easily satisfy.

-s