Re: [julia-dev] Gadfly: a prototype grammar of graphics implementation

[Jeff Bezanson]

This is just awesome. Beautiful work.

With this, I think we've hit a critical mass of graphics-related
projects. We should try to factor out the most common low-level
definitions, such as abstract color and shape types, Point, Line,
Polygon, etc. into a Graphics or GraphicsBase package. This would be
really minimal; the goal would just be to avoid having 6 different
Point types floating around.

It would be great to add your color.jl to the Color package that
currently exists. Since I'm looking at that file, a couple random
comments:
- values are automatically converted to field types when assigning to
or initializing an object, so all the convert(Float64, x) are not
necessary.
- I believe I usually see the HLS colorspace referred to as HSL; is
there a standard order for the components?
- is there a use for ordering colors (isless)?

Feel free to nominate useful utilities for inclusion in base. I like lerp().

On Thu, Dec 20, 2012 at 3:33 AM, Daniel Jones <daniel...@gmail.com> wrote:
>
> I've been having a go at implementing a plotting interface along the lines
> of Hadley Wickham's ggplot2 and Leland Wilkinson's book. Though far from
> complete (or bug free), it's gradually starting to come together and worth
> showing off. I have a short demo here.
>
> What I have working is point, line, and (partially) bar geometries, basic
> continuous and discrete scales, discrete color scales, and continuous scale
> transformations.
>
> The primary backend is SVG, which it generates itself. The entire system is
> Julia all the way down, with the exception of a dependency on pango which is
> used to precompute text extents. There is a separate (not quite up to date)
> cairo backend, but generating my own SVG had some advantages. Smaller,
> simpler (i.e., easier to load up in illustrator or inkscape and edit) SVG is
> one, but more interestingly it opens the possibility of adding id and class
> attributes to plot elements and embedding javascript to add interactivity.
>
> The primary approach I'm aiming for is plots that are viewable without a
> browser, but improved with one. Obviously such an approach is limited (it's
> not meant to compete with d3), but an improvement over totally static
> plotting tools.
>
> I'm still feeling this interface out, deciding on a direction to push this.
> Feel free to have a say.
>
> --
>
>
>

[Tom Short]

Agreed. Looks awesome, Daniel!

> --
>
>
>

[Harlan Harris]

This is outstanding!

Concur with Jeff's suggestion to coordinate with the other graphics projects! Could you tell us a bit more about your Compose package?

How transparent is the mapping from the syntax to the internal Plot object? In ggplot2 it's a bit of a mess, with half of the computation done in the initial call, but most of it delayed until rendering. As a result, you can't easily debug a plot that's not doing what you expect. Have you thought about text-based ways to describe the Plot object? 

One note is that we're considering annotations on DataFrame columns for levels of measurement (ordinal, nominal, etc) https://github.com/HarlanH/DataFrames.jl/issues/6 . This would likely supersede R's notion of a Factor type, and abstract away from implementation details. Lord knows if/when this will happen...

 -Harlan

--

[Tim Holy]

Agreed, this is great. Compose looks particularly interesting, and I'd love to
know more about how it works. In particular, I'm guessing it will allow
something I've always wanted to do with Matlab, which is to let panel A be a
drawing I made in Inkscape, and panel B a plot rendered in Matlab. Currently I
have to stitch those together post-hoc, but it looks like Compose will, either
now or in the future, allow that to be done nicely?

I'm curious to know how flexible Compose will be, but lack time right now to
dive into the code. Will it be usable in, say, something being rendered with
OpenGL??

Nice work!
--Tim

[Jason Knight]

Awesome! 

I love your other work too (notably Quip), it's great to see so many bioinformatics/computational biologists getting involved with Julia.

Jason

[Viral Shah]

I agree. This is indeed beautiful. We should factor out a common API. I would love to include a simple API for interacting with plots zoom and pan to start with, but there could even be mouse events and such.

-viral

[Tim Holy]

On Thursday, December 20, 2012 07:00:56 AM Viral Shah wrote:
> I
> would love to include a simple API for interacting with plots zoom and pan
> to start with, but there could even be mouse events and such.

Agreed, we should make interactivity a primary focus from the beginning.
Otherwise we could build a nice edifice, but it won't be flexible enough to meet
common needs.

wheelmouse support (including shift-scroll, ctrl-scroll, etc) in particular
seems important to test early, as is sometimes seems to be an afterthought but
is often the nicest interface for rapid zooming.

--Tim

[John Myles White]

This is a wonderful start, Daniel. Great work!

-- John

> --
>
>
>

[Brian Mattern]

This looks fantastic. I particularly like the following (from Compose's readme:

"Besides coordinate transformations, Compose also handles mixtures of relative and abosule coordinates. For example, 1w - 10mm is a well formed expression, giving the width of the parent canvas minus ten millimeters."

This mixing of relative and absolute coords is essential for good layout and often not very convenient to do. Julia's syntax makes this particularly nice looking.

Brian

[Stefan Karpinski]

This is absolutely beautiful. I particularly enjoyed the Sierpinski gasket example in the Compose readme and this quote from Color:

Whenever a range of colors is used to convey information, problems arise. Human perception of color was shaped largely by the need for our hairier, tree-dwelling ancestors to tell when fruit is ripe.

I am very, very excited about this. Thank you!

[Stefan Karpinski]

On Thu, Dec 20, 2012 at 6:48 AM, Jeff Bezanson <jeff.b...@gmail.com> wrote:

We should try to factor out the most common low-level definitions, such as abstract color and shape types, Point, Line, Polygon, etc. into a Graphics or GraphicsBase package. This would be really minimal; the goal would just be to avoid having 6 different Point types floating around.

This is one really nice aspect of multiple dispatch: many different pieces of functionality can be applied to shared data types like shapes or statistical distributions. In C++/Java-style OO, each library ends up having their own types with methods stuck into them.

[Jeff Bezanson]

Exactly; we can share all kinds of common representations and
definitions without dictating anything about how things work or what
underlying graphics devices/libraries are present.

> --
>
>
>

[Daniel Jones]

Thanks for the feedback everyone!

I'll think about what parts can be generalized and moved out of gadfly/compose. I think merging the color code in compose into the Color package is an easy first start. I'm tempted to try to integrate some of code handling units with Tim's units.jl in extras, since I'm borrowing ideas from there anyway.

I very much want to to see zooming and panning happen. It's difficult, but possible. I've thought of three ways.

1. Client/server: events get sent to Julia which redraws the plot and sends it back.

2. A lot more javascript: move most of the rendering to javascript where re-rendering on the fly is easier.

3. A little more javascript: pre-generate all the svg in julia, use javascript to pan/zoom/clip this prepared svg.

These are ranked by decreasing power but increasing convenience: (1) requires julia to view a plot, (2) requires a browser, (3) doesn't strictly require either. I'm inclined to try (3), partially since a like the idea of having one plot that you can submit to a journal and post on your blog. 

[Daniel Jones]

On Thursday, December 20, 2012 6:55:36 AM UTC-8, Jason Knight wrote:

Awesome! 

I love your other work too (notably Quip), it's great to see so many bioinformatics/computational biologists getting involved with Julia.

I'm really excited to contribute to a Bio module, when such a thing exists. Julia could be such a great platform for genomics, transcriptomics, etc.

[Brian Mattern]

FYI, if anyone else wants to try this, you need to update DataFrames to master. The version in METADATA has a bogus convert method that breaks things.

[Stefan Karpinski]

(3) sounds like the most promising option to me. Seems like Units, Color and Time need some integration.

--
 
 
 

[Tim Holy]

Hi Daniel,

As you know, I've been exploring your very nice Compose package. When this
appeared, I had just started thinking about the problem of graphics layout
management for Julia. I'm interested in understanding a bit more about what
can and can't be done with Compose, where you see it going, and how well it
will serve some of the use cases I've been contemplating. I've started to try
to figure it out on my own, but I came to the conclusion that there may be
important elements of your "vision" that are not yet implemented, so I decided
just to pester you instead :-).

Probably the best way to start is to give you an example of a graphic that has
some of the constraints I was starting to try to wrap my head around:

http://postimage.org/image/s6pskvknf/

This is an unpolished "mockup" created in Matlab. Here are the important
elements:

1. A row of buttons along the bottom. The height of the buttons should be
determined from the fontsize of the text inside.
2. At the left, a scatterplot and the marginal distributions along each axis.
It's obviously important that the "frames" for the marginal distributions be
aligned with the frame for the scatterplot. The positioning of this region has
to be sensitive to the needs to have room for all the labels.
3. At right, an image. It's crucial that the image preserve its actual
dimensions, and not look squashed along one axis or another.
4. Let's also say I want to allocate the right 40% for the image, and I'd like
the image's frame to be aligned vertically with the frame of the scatterplot
image. (One could make different choices for this step, too, but let's go with
this one.)

The key point: we demand that these constraints are satisfied even as the
window is resized by the user. (Matlab doesn't support this well, so I'm not
saying this is a trivial problem.) Smells a lot like a linear programming
problem to me.

Is this possible with the Compose framework? I started trying to put it
together, and came to the conclusion that it might not yet be possible. If not
available now, is this kind of thing on your radar? Or might that kind of
layout management conflict with some of the very impressive strengths of
Compose, in which case one might want to take a different approach for GUIs?

--Tim

[Daniel Jones]

Not all of this is possible currently. I've thought a little about it, but not enough to be sure how to proceed.

Another simple problem I want to solve that gets at some of this is automatically rotating x-axis labels, or word-wrapping, when they wouldn't otherwise fit. There's no way to do that now. There should be a way to do it, but I think you're right that it would involve solving some sort of optimization problem (as opposed to just blindly transforming coordinates). This sort of automatic layout problem could get very complex (e.g., reimplementing latex), but maybe a very limited notion of automatic layout would fit our needs. The example you show is helpful to elucidate this.

One roundabout way of tackling layout problems is trying to generate HTML and let browsers do the layout, since they already solve complex layout problems. But that would harm the ability to render directly to postscript or pdf, for example.

[Tim Holy]

On Saturday, December 22, 2012 12:08:24 PM Daniel Jones wrote:
> Not all of this is possible currently. I've thought a little about it, but
> not enough to be sure how to proceed.

I agree, it's not obvious how to solve this problem. After writing that last
email I thought, "wow, using LP to solve the layout problem is such an obvious
idea, I bet it's been thought of before." Indeed it has, numerous times.
Here's a link to a relatively recent paper:
https://www.cs.auckland.ac.nz/courses/compsci705s2c/lectures/geraldpapers/reading1_LutterothStrandhWeber.pdf

I haven't finished reading the paper yet, but I've thought a little about how
one might integrate this into Compose. It seems that one could define a new
measure, TabStop, where the parent of any canvas can have an array of x and y
tabstops. Position of a frame within the parent could be specified as

canvas(tx[3], ty[2], tx[5]-tx[3], ty[7]-ty[2])

The tabstops would be specified by the LP problem.

This architecture implies that when two canvases need to preserve some kind of
relative alignment, they have to share the same parent so they're using the
same tabstops. This kind of argues for a "shallow" representation of the
graphical scene, as opposed to the nesting which is the coolest feature of
Compose. This is where I worry that there may be competing goals for this
problem. I suppose one could basically say that when you need that kind of
alignment, you have to use a common parent, but within an aligned region you
could nest as deeply as you wish. But it seems hard to use Gadfly's current
organization of wrapping the axis frame and the label frames into a single
"panel" frame, because alignment would usually have to be with respect to the
axis frame (which therefore needs to be a child of the window itself).

> Another simple problem I want to solve that gets at some of this is
> automatically rotating x-axis labels, or word-wrapping, when they wouldn't
> otherwise fit. There's no way to do that now.

Hmm, yes, that is a very important problem, and one to solve early. It seems
like there also needs to be an "askchild" facility, so that canvases can get
some of their geometric information from their contents. I haven't thought
about this yet at all, so I don't have anything that could be mistaken for
wisdom here.

> One roundabout way of tackling layout problems is trying to generate HTML
> and let browsers do the layout, since they already solve complex layout
> problems. But that would harm the ability to render directly to postscript
> or pdf, for example.

Yes, I've also thought about punting an putting this off to other toolkits.
Like HTML, Tk (used for visual display in Winston) has a built-in layout
manager. So one option is to do direct support of the native toolkit. But as
you say, it's not obvious that we will be happy targeting only a single
backend, especially because not all backends have a layout manager. It seems
better to do the layout in Julia, and then hope that either (1) one can map
the Julia layout problem to the toolkit manager framework, or (2) hope that
the toolkit supports a callback interface so that Julia can just feed position
information in real time.

Peter Wang, any thoughts on this matter? Any possibilities for collaboration
among projects here, either at the level of ideas or code??

--Tim

[david tweed]

On Saturday, December 22, 2012 8:08:24 PM UTC, Daniel Jones wrote:

Not all of this is possible currently. I've thought a little about it, but not enough to be sure how to proceed.

Another simple problem I want to solve that gets at some of this is automatically rotating x-axis labels, or word-wrapping, when they wouldn't otherwise fit. There's no way to do that now. There should be a way to do it, but I think you're right that it would involve solving some sort of optimization problem (as opposed to just blindly transforming coordinates). This sort of automatic layout problem could get very complex (e.g., reimplementing latex), but maybe a very limited notion of automatic layout would fit our needs. The example you show is helpful to elucidate this.

One roundabout way of tackling layout problems is trying to generate HTML and let browsers do the layout, since they already solve complex layout problems. But that would harm the ability to render directly to postscript or pdf, for example.

 

I haven't really been following this discussion, but at this point I'd like to see if I understand something:

 

Is this backend proposed for simple "live" rendering? As an example, suppose you've set up some code to produce a kernel density estimate from some data, and you've connected it to a "kernel width" slider bar on screen so that as you move the slider bar the density is recomputed and then replotted. (I'm not thinking of any kind of full-blown GUI application, but the kind of thing you might want to do very simple interactions with.)

 

Regards,

Dave

 

[Tim Holy]

On Sunday, December 23, 2012 01:18:34 AM david tweed wrote:
> Is this backend proposed for simple "live" rendering?

You mean, the (hypothetical) HTML backend? Definitely, simple (and hopefully
even not-so-simple) interaction is part of my own personal graphics roadmap,
and it seems to be for most everyone else, too. I think we're still in the
process of deciding the right backends/implementations/etc. Feel free to chime
in.

--Tim

[Brian Mattern]

Many years ago, I used to use a library called 'edje' for scalable
interface layout. (See [1] for a wiki page that does a mediocre job
describing the library or [2] for an example screenshot of an
application I wrote using it.) Edje was originally designed for
window-manager theming, but has some nice features that I haven't seen
in other layout systems.

The basic idea is that the layout is described as a tree of nested
canvases (I'll adopt Compose's nomenclature here). Each canvas contains
a list of parts whose geometry is specified by the top left and bottom
right corner of their respective bounding boxes. The corner locations can
be given relative to the current canvas, or to a different part in the
same canvas, and is specified by a percentage of the "relative to"
part's size and an absolute offset. The two corners can be relative to
different parts and additional constraints such as min/max size or
aspect ratios can be defined. (Text parts can also be constrained to fit
to the rendered font size). Alignment of a constrained-part size within
its specified bounding box can also be given.

See [3] for a very basic example.

I may try mocking up Tim's example to see if the system if the system is
flexible enough for this, but I believe it should be. Of course, this
layout would require the image and scatter plot (all three panels) to
live within a single canvas (you can't specify geometry relative to a
sub-part of a canvas).

Anyway, I thought I'd throw this out there as at least something to draw
a few ideas from regrading constraint specification.

[1]: http://trac.enlightenment.org/e/wiki/Edje
[2]: http://rephorm.com/files/old_files/elicit2.png
[3]: http://trac.enlightenment.org/e/wiki/EdjePartPositioning

> --
>
>
>

[Stefan Karpinski]

It makes sense that constrained layout logic could be factored out to be common between both plot layout and UI layout. I suspect that any approach that didn't use linear programming might be doomed to either not work that well or not be sufficiently general.

--

[Tim Holy]

I think your description of edje is pretty close to what I was proposing as an
extension of Compose.

I've spent a little time today looking at Tk's "grid" layout manager. grid is
clearly doing something similar, and is a subset of the functionality in that
"Auckland Layout Model" paper in that link I sent. (The main difference seems
to be the lack of support for "partial ordering," Figs 6-8, for anyone who
cares.) I'm beginning to think that almost all modern layout managers have
quite a lot in common. That would be good news. I'm unclear on whether there
are performance advantages in using a toolkit's layout manager directly, or
whether one could do just as well by having Julia do the layout via a window
resize callback.

Clearly the most "interesting" problem is how one specifies the constraints in
a way that feels natural. Examples of how edje handles this would be great to
see. I have a sneaking suspicion that Julia's support for expressions might be
leveraged to make this pretty sweet, but nothing beyond a gut feeling at this
point.

--Tim

[Brian Mattern]

I put together a quick mockup of Tim's example in edje. I've attached
the text input file to give an idea of how the constraints and relations
between parts are defined. Screenshots of various (somewhat arbitrary)
sizes are available at
the following links:

http://i.imgur.com/C9pOb.png
http://i.imgur.com/Hh2uB.png
http://i.imgur.com/CIWUE.png
http://i.imgur.com/pK63h.png
http://i.imgur.com/VFF1S.png
http://i.imgur.com/PcerG.png

I made some ad-hoc choices about how to define the constraints as well,
which aren't necessarily optimal. Also, I didn't take the time to put in
the labels. But, it *mostly* works.

I should mention that edje has reasonable defaults for most parameters,
so they can just be left off. ("rel1" is the top-left corner and "rel2"
is the bottom right).

Also, there are a few tricks here. For the "peppers" image, the geometry
set by rel1 and rel2 has 0 width. The width of the image is instead
fixed by the height and the aspect ratio constraint. The iamge is
horizontally aligned to the right, which puts its right edge at the
correct location.

Also of note is that parts can have cross constraints. The peppers'
vertical geometry is relative to the y-histogram, while the
y-histogram's horizizontal geometry is relative to the peppers...

There's probably a better way to define the constraints, but this is
what I came up with in a few minutes (and within the confines of edje's
allowed constraints).

(Oh, and there's a bug in the library that causes rotated text to occupy
the space of the unrotated text. Hence the extra padding on the y-axis
label)

Brian

> --
>
>
>

[Daniel Jones]

Looking into edje, the impression I have is that it isn't doing more than coordinate transformation. What it supports, that Compose doesn't is a way of specifying coordinates relative to some other part (e.g., A is to the right of B which is to the right of C). Compose only supports relative coordinates in terms of the containing canvas (e.g., A is at the left of the canvas, B in the middle, C to the right). Or put more simply, Compose uses a tree of coordinate transformations, edje uses a directed graph.

Maybe I don't have that completely right, I don't entirely understand the system yet. For example, what happens if there is a cycle in the graph (e.g., A is to the left of B, B is to the left of A)? Can I only specify coordinates relative to something that was declared prior?

The Aukland Layout Model is also very interesting. I really like the idea of a tabstop measure in Compose. That seems like a totally manageable thing to implement that would add a lot of power. One problem is a figuring out a syntax for constraints that isn't completely dreadful. The AUL examples use sytax like

ls.AddConstraint(new double[] { 2, -1 }, new Variable[] { x1, x2 }, 
	OperatorType.EQ, 0);

Besides not fitting with the functional style of Compose, it's not too pretty.

I should also point out, that not even linear programming is sufficiently powerful to solve all the relevant layout problems. One I have in mind is the problem of labeling points in a plot such that labels do not overlap, which is NP-hard, though apparently simulated annealing tends to do a good job. There are potentially other sorts of combinatorial optimization problems. I mentioned deciding whether to rotate the labels on the x-axis, which I don't think would fit into a linear program. (Ok, that's one boolean, hardly an optimization problem, but still.) Though these are probably isolated special cases that should be treated as such.

[Stefan Karpinski]

If I'm allowed to use ILP for the package manager, I think we can justify the same for UI layout calculations ;-)

--
 
 
 

[Tim Holy]

On Monday, December 24, 2012 02:03:16 AM Daniel Jones wrote:
> The Aukland Layout Model is also very interesting. I really like the idea
> of a tabstop measure in Compose. That seems like a totally manageable thing
> to implement that would add a lot of power. One problem is a figuring out a
> syntax for constraints that isn't completely dreadful. The AUL examples use
> sytax like
>
> ls.AddConstraint(new double[] { 2, -1 }, new Variable[] { x1, x2 },
> OperatorType.EQ, 0);
>
> Besides not fitting with the functional style of Compose, it's not too
> pretty.

Agreed wholeheartedly. In the spirit of encouraging collective development,
I've pushed a "skeleton" of a system here:
https://github.com/timholy/Layout.jl
There is quite literally no code here that I've even tried executing. But this
may be the most efficient way to describe the kind of API I'm contemplating. In
particular, the test/scatter_image.jl file might be particularly helpful in
illustrating usage. My first target for getting a functional system is
"simple.jl"

Comments on the API are most desired. I have no idea whether it's even
feasible. The core idea is to build the constraints as expressions, and then
have a parser that turns this into actual LP variables.

I also have no particular need to develop all of this myself :-), so anyone
who wants to run with it is invited to do so! I will keep plugging away at
this, too, but I want to make it clear that I view this as an experiment in
progress. Other experiments (either starting from here or trying something
totally different) would be most welcome.

> I should also point out, that not even linear programming is sufficiently
> powerful to solve all the relevant layout problems.

Yep, this is probably right. I'm quite skeptical that a simulated annealer is
what we want running for our general window resizing, but in special cases
you're right that it would be good to allow even more powerful algorithms.

--Tim

[Tim Holy]

It occurs to me that much of that functionality in my "API demo" should
probably be in a symbolic fronted to LP, rather than buried in the graphics
layout manager.

--Tim

[Brian Mattern]

A few comments below:

On Dec 24, 2012, at 2:03 AM, Daniel Jones <daniel...@gmail.com> wrote:

> Looking into edje, the impression I have is that it isn't doing more than coordinate transformation. What it supports, that Compose doesn't is a way of specifying coordinates relative to some other part (e.g., A is to the right of B which is to the right of C).

Yes, I meant to be more clear that I was primarily proposing that we consider relative constraints like this (and didn't mean to imply that edje's layout algorithm was particularly interesting in other regards).

> Compose only supports relative coordinates in terms of the containing canvas (e.g., A is at the left of the canvas, B in the middle, C to the right). Or put more simply, Compose uses a tree of coordinate transformations, edje uses a directed graph.
>
> Maybe I don't have that completely right, I don't entirely understand the system yet. For example, what happens if there is a cycle in the graph (e.g., A is to the left of B, B is to the left of A)? Can I only specify coordinates relative to something that was declared prior?

Declaration orde is not important. But, unresolvable constraints such as cycles are invalid. The description format is compiled into a binary representation, which reports an error if cycles are present.

Brian

[david tweed]

On Sunday, December 23, 2012 11:06:46 AM UTC, Tim wrote:
You mean, the (hypothetical) HTML backend? Definitely, simple (and hopefully  

even not-so-simple) interaction is part of my own personal graphics roadmap,
and it seems to be for most everyone else, too. I think we're still in the
process of deciding the right backends/implementations/etc. Feel free to chime
in.

Thanks: I wasn't sure if this was purely a framework for doing "high-quality offline layouts" or was going to be used for all graphics. The main thing that concerns me is that the structure of the layout specification makes it very easy to see that nothing has been changed (including the ranges of the data, which may imply axis rescaling) so that the involved ILP processing for figuring out where to put what can be skipped (and ideally only the actually changed elements need "repainting", at whatever level this is done).

A more minor point is that, although I have no experience with HTML 5, I'm constantly amazed at how slow HTML rendering is. This isn't about the network latency, even when the data is local the layout for CSS is annoyingly non-snappy. (I do tend to work on netbooks with relatively slow processors, so I possibly have a different user experience to desktop users).

But the main thing is seeing that the semantics make it easy to determine "no change needed". I'll have to find the time to look at the layout language in more detail.

Regards,
Dave

[Tim Holy]

On Tuesday, December 25, 2012 03:45:26 AM david tweed wrote:
> Thanks: I wasn't sure if this was purely a framework for doing
> "high-quality offline layouts" or was going to be used for all graphics.
> The main thing that concerns me is that the structure of the layout
> specification makes it very easy to see that nothing has been changed
> (including the ranges of the data, which may imply axis rescaling) so that
> the involved ILP processing for figuring out where to put what can be
> skipped (and ideally only the actually changed elements need "repainting",
> at whatever level this is done).

Refresh is an important problem in general, and you're right that it does
interact with layout. (I'm using layout to mean purely "where things are
placed," not "drawing".) I think it should be pretty straightforward to say,
"if a canvas hasn't had its data change, and its position in the layout hasn't
changed, don't redraw it." But I haven't gotten that far yet.

> A more minor point is that, although I have no experience with HTML 5, I'm
> constantly amazed at how slow HTML rendering is. This isn't about the
> network latency, even when the data is local the layout for CSS is
> annoyingly non-snappy. (I do tend to work on netbooks with relatively slow
> processors, so I possibly have a different user experience to desktop
> users).

Interesting. What renderer are you using? I'd guess it depends on whether it's
Gecko/WebKit/IE/etc. I don't have any experience here. If all HTML renderers
are substantially slower than native toolkits, then I for one am going to be
pretty interested in the toolkits.

> But the main thing is seeing that the semantics make it easy to determine
> "no change needed". I'll have to find the time to look at the layout
> language in more detail.

The layout language is still a work-in-progress. But excitingly, this much is
working:
https://github.com/timholy/Layout.jl/blob/master/test/simple.jl
Of course that file is likely to change over the next few days, so this
statement only applies at a particular commit (as of 68bb32a).

Up next: working on how to specify "references" to tabstops in other layouts
(presumably all within the same window), and to "compile" everything into one
big merged LP. That "compiling" (which is done by that "parse" command in the
demo) typically only needs to be done once when you create the layout---
afterwards, you just solve the numerical LP each time the window resizes. In
setting up the LP, it's very nice to have symbols.

--Tim

[Peter Wang]

On Sat, Dec 22, 2012 at 2:43 PM, Tim Holy <tim....@gmail.com> wrote:

On Saturday, December 22, 2012 12:08:24 PM Daniel Jones wrote:
> Not all of this is possible currently. I've thought a little about it, but
> not enough to be sure how to proceed.

I agree, it's not obvious how to solve this problem. After writing that last
email I thought, "wow, using LP to solve the layout problem is such an obvious
idea, I bet it's been thought of before." Indeed it has, numerous times.

Here's a link to a relatively recent paper:
https://www.cs.auckland.ac.nz/courses/compsci705s2c/lectures/geraldpapers/reading1_LutterothStrandhWeber.pdf

This paper is interesting, but what I've seen used successfully is the Cassowary constraint solver: http://www.cs.washington.edu/research/constraints/cassowary/

This is the constraint solver that Apple chose to use for OS X.  It's also what the Enthought folks are using in ENAML, their declarative UI framework.  

ENAML: https://github.com/enthought/enaml/

Cython wrapper for Cassowary: https://github.com/enthought/casuarius

All that being said, I think it's important to note that constraint-based layout specification for UI widgets is kind of a different problem than providing accessible layout facilities for data analysts who want to make some plots with minimal fuss.  The problem of UI widget layout is usually centered around optimizing free space allocation, lining up visual elements across visibility groups, and doing this with interactive resize events.

For most plots, it's pretty obvious where the free space should be allocated, and it's usually sufficient to align the panels or frames.  Plot layout can oftentimes be schematized with box layouts.  (Your example above, for instance, can be achieved with nested box/grid sizers.)

The undistorted image is a separate plotting feature that needs to be handled by specifying data-space "aspect ratio".  Here is a video demonstrating this feature with Chaco: http://www.youtube.com/watch?v=jL6oLzQON8A  (Bear with it; I initially show locking a fixed screen-space aspect ratio, and about 1 minute in, I show pinning a data aspect ratio.)

> Another simple problem I want to solve that gets at some of this is
> automatically rotating x-axis labels, or word-wrapping, when they wouldn't
> otherwise fit. There's no way to do that now.

 

The general axis labeling problem is a constraint-solving problem in its own right.  For a given amount of screen space and text metrics, there are any number of ways to tick and format those ticks.  This is more straightforward when you use standard ticking heuristics, but when you want to allow users to specify arbitrary tick label formats (e.g. for dates), it gets complicated very quickly, because you might want to move up or down different levels of resolution, based on how much text space the formatted tick labels take.  If you want to make this interactive for panning, zooming, and window resize, there is a certain amount of hysteresis you need to build into the optimization so that you don't get unbearable flickering as the window size or data window approaches certain ticking boundaries.  Etc.

I spent a solid week tackling this problem many years ago for Chaco, and devised the approach in the chaco.scales sub-package: https://github.com/enthought/chaco/tree/master/chaco/scales  This subpackage was purposely designed to not rely on any other dependencies, so it can be used by other projects pretty easily.  It was also designed to support a "weird" labeling case, when people want to format the edge ticks or edges of a plot more verbosely than the format for intermediate ticks.  (This occurs if you're a high-frequency trader that is looking at microsecond level resolution data, but want to know exactly what year/month/day/HMS you're looking at.)

Hmm, yes, that is a very important problem, and one to solve early. It seems
like there also needs to be an "askchild" facility, so that canvases can get
some of their geometric information from their contents. I haven't thought
about this yet at all, so I don't have anything that could be mistaken for
wisdom here.

A common pattern in GUI layout systems is for inner components to indicate their width & height to parents, along with one of several possible conditions for each dimension, e.g. Fixed, Minimum, Maximum, etc., and then to also optionally indicate some priority number on how much of available free space it should consume.  (See Qt's layout system, which is very well thought-out: http://doc.qt.digia.com/qt/layout.html and http://doc.qt.digia.com/qt/qsizepolicy.html)  Then the top-level parent allocates space using a simple algorithm, and then recurses to each of its contained components, etc.  It's kind of a poor-man's heuristic optimizer.

Even if you use a constraint-based layout system, you generally still need to build an easier-to-use end-user facing API for it, especially if we are talking about plot layouts.  I think that a lot of people are going to fall into the "I'm used to box layouts, give me those", and then there are others who want the full Protovis/d3 treatment where they can link data to screen anchors of panels and subplots and such.

 

> One roundabout way of tackling layout problems is trying to generate HTML
> and let browsers do the layout, since they already solve complex layout
> problems. But that would harm the ability to render directly to postscript
> or pdf, for example.

Yes, I've also thought about punting an putting this off to other toolkits.
Like HTML, Tk (used for visual display in Winston) has a built-in layout
manager. So one option is to do direct support of the native toolkit. But as
you say, it's not obvious that we will be happy targeting only a single
backend, especially because not all backends have a layout manager. It seems
better to do the layout in Julia, and then hope that either (1) one can map
the Julia layout problem to the toolkit manager framework, or (2) hope that
the toolkit supports a callback interface so that Julia can just feed position
information in real time. 

Peter Wang, any thoughts on this matter? Any possibilities for collaboration
among projects here, either at the level of ideas or code??

The issue of layout, like almost every other aspect of the "graphics in Julia" discussion, depends very much on just how much of the problem you'd like to tackle, and how quickly you want a solution in place.  The general pattern is that if you just want to Get Something Working, you're going to have a much easier time if you do a very specific thing for a subset of the larger, general problem.  But this reduces the amount of potential interop with other projects, because they have already Gotten Something Working, and are looking to improve on that.  For instance, there are many possible avenues of development that punt the layout to the underlying toolkit (although I wouldn't recommend HTML).  But that would put you near feature parity with things like Matplotlib and Chaco (both of which handle their own layout), but wouldn't get you to the d3 level of flexible layout.

For Bokeh, we have mostly decided that for layout, we're going to stick with a schema-like approach to start with.  After looking closely at many visualizations composed with Protovis and d3, I've come to the conclusion that for most of the *useful* infographics, all that is truly required is flexible composition and embedding of data subspaces, not necessarily giving users unfettered access to mix data- and screen-space transformations.  The latter can make for interesting infographics, but it seems difficult to do that and maintain any semblance of a graphical pipeline which can be used to reduce data volumes between a server rendering "big data" and a client across the network.

Hope all of the above made sense...

-Peter

[Viral Shah]

Hi Daniel,

On OS X (using homebrew), there is no libpangoft-1.0. I changed this to use libpango-1.0. Anyone knows how libpangoft is different from libpango?

I now get this:

julia> require("Gadfly")

file_path not defined

 in include_from_node1 at util.jl:244

 in reload_path at util.jl:254

 in require at util.jl:223

 in include_from_node1 at util.jl:244

 in reload_path at util.jl:254

 in require at util.jl:223

at /Users/viral/.julia/DataFrames/src/DataFrames.jl:391

-viral

[John Myles White]

Can you update DataFrames? It needed to be updated today to the new file API.

I'm also interested in libpangoft, since that is what's keeping me from using Gadfly.

 -- John

--
 
 
 

[Daniel Jones]

Try updating now.

Pango needs a backend to work. I was using freetype (libpangoft2), but I guess that's not necessarily built with homebrew. I've switched that to cairo (libpangocairo) which should be.

[Viral Shah]

Ok, that got me a lot further.

I am trying to run the examples in README.md, but cannot get an image to be produced.

-viral

> --
>
>
>

[Daniel Jones]

Do you see an error message, or just no file being written?

[Viral Shah]

Sorry for the delayed response. I did find the files being written.

Thanks,

-viral