Intended usage?

[Larry White]

Hi,

I'm building an open source java data-frame / column-store (https://github.com/lwhite1/tablesaw) and would like to add some basic exploratory graphics capability through integration with another library. Glimpse looks very impressive, and appears to scale well, but I'm not sure how well it fits my use case. I would like to:

1. Render charts interactively without needing to embed them in a larger GUI application. For example, it would be great if the user could type something like:

columnX.histogram();

and a new swing window would open and display the histogram.

(of course, I would do all the integration :)

2. Be able to render all the usual exploratory graph types: scatter, line, bar, histogram. And some less usual, but really helpful ones (box plot)

3. Have good support for time series. (be able to render labels ordered by date, on multiple time scales, etc.)

Any advice on whether or not Glimpse is a fit would be greatly appreciated.

thanks.

[Mike Hogye]

Cool project! The thought of a database with visualizations built-in warms my heart. :-)

1. Very doable. We have something sort of similar in-house for using Glimpse from Matlab. I don't know if that will ever get open-sourced -- but I can say that it works great, and was easy to implement. The only tricky thing was threading -- calls come in on a Matlab thread, so Glimpse operations needed to be either synchronized, or invoked on the Swing thread.
   
2. Glimpse has really good scatter, line, bar, and histogram plots. I don't know if we have box plots yet, but we would be interested in adding them.
3. Glimpse is fantastic for visualizing timeseries data.

Another question to consider is whether OpenGL is okay for your visualizations. This works out fine on decent graphics cards (even the ones built into recent Intel CPUs), but it's not as universally supported as something like Java2D, which has no hardware acceleration.

[Mike Hogye]

If you clone the Glimpse code, there are a bunch of examples you can run, both to get a feel for what the various plots can do, and to see the code required to create them. Take a look at:

• ScatterplotExample (be sure to try dragging the handles on the color axis!)
  
• LinePlotExample (try dragging/zooming the y-axis on the left edge, and also the y-axis on the right edge)
  
• CollapsibleTimelinePlotExample
• HistogramPlotExample
• TaggedHeatMapExample (you didn't mention heatmaps, but you might find it useful anyway)
  

There are other cool examples too, but those are the ones that seem most relevant.

[Larry White]

This sounds really good. I'm not too concerned about requiring openGL, as tablesaw is designed for programmers and data scientists, and I suspect OpenGL is one of relatively few ways to get plots that support large amounts of data. I will try some scenarios and see how it goes :) 

Thanks very much

On Tuesday, June 21, 2016 at 10:07:34 AM UTC-4, Mike Hogye wrote:

[Mike Hogye]

One pattern we find ourselves using over and over is an x,y plot combined with a timeline. As the user drags the timeline selection window back and forth, the x,y plot updates to show the x,y data that falls in the selected time window. This is great for visualizing geographic data that changes over time.

The idea can be generalized to work for other kinds of multi-dimensional data. Instead of x,y,time, it might be x,y,depth,temperature -- show an x,y plot and a depth,temperature plot, where the depth,temp plot shows datapoints inside the x,y selection box. By watching the depth,temp plot while moving the mouse around the x,y plot, you can get a sense of x,y variations in the depth,temp curve.

Good luck with Tablesaw -- let us know if you find ways we can improve Glimpse!

[Larry White]

Mike,

Thank you for the help and encouragement. I integrated several charts (scatter, line, timeline, histogram) with Tablesaw this weekend and have been really pleased with the library so far. I expect to add the TreeMap soon as well.

I have a pile of questions (sorry), but there's no big hurry.  I feel really good about building on Glimpse and hope it will let more people get the benefit of all your hard work.

Here are the questions: 

Are their any examples of BarCharts? I assume they're similar to Histograms but with af categorical x axis, but couldn't find any code. An example of a horizontal bar chart would be a plus. 

I looked at the HeatMap, but I think the more common use-case for my users would be for non-continuous (integral and/or category) x and y variables. Is that possible? 

I built my ScatterPlots using a point accumulator. Does that help performance if you add all the points before rendering? 

Is there a way to change the point size in ScatterPlot when using an Accumulator that doesn't require creating an array of sizes. I often want them all to be the same size?

Is there a way to change the point shape in scatterplot?

Is there an example of brushing? Seems like that kind of what you suggested can be done with the timeline/scatterplot combo. Not sure if there's a general way to select a group of points using a click-drag rectangle.

Is there a way to get standard cross-hairs in the time-series plot? I can't find a reference to the cursor (which displays as a vertical line only in my code).  I'm asking because I found it confusing that the display by the cursor showed the vertical position of the cursor, rather than the intersection of the timeline at that horizontal position. I thought having a standard cross hair would make it clearer what the numbers meant.

Finally, it seemed like my CPU was working hard even when displaying a static visualization - So much that my laptop gets really hot whenever a couple of plots are left on the screen. Is there anything I can do about that?

Thanks again for your help and for an awesome library.

larry 

 

[Geoffrey Ulman]

Hello!

Answers to your questions inline below.

Geoff

 

Are their any examples of BarCharts? I assume they're similar to Histograms but with af categorical x axis, but couldn't find any code. An example of a horizontal bar chart would be a plus. 

There isn't a GlimpsePainter class which builds a nice bar chart automatically unfortunately. That said, you can use PolygonPainter to efficiently draw large numbers of arbitrary polygonal shapes. That's the easiest way to draw a bar chart at the moment. You just have to do a few more calculations to compute the four corners of each bar. You could combine that with the example I link below which demonstrates custom categorical axis labels: https://gist.github.com/ulmangt/21dd09076618e7acb7eda4b75c086f2f

 

It's certainly not as easy as we'd like for such a common case as a bar chart. Definitely in the future this should be wrapped into a nicer painter class.

I looked at the HeatMap, but I think the more common use-case for my users would be for non-continuous (integral and/or category) x and y variables. Is that possible? 

You can modify HeatMapExample to use a discrete color scale and have custom axis labels corresponding to the data categories. Here's an example: https://gist.github.com/ulmangt/21dd09076618e7acb7eda4b75c086f2f

 

In the example, look for the //XXX comments to indicate the locations where something has changed from the base HeatMapExample.

Note: internally you'd still have to assign integer values to each of your categories.

I built my ScatterPlots using a point accumulator. Does that help performance if you add all the points before rendering? 

What Glimpse painter class are you using to draw your data? If you're using DynamicPointSetPainter and adding points with BulkPointAccumulator, then yes, the bulk methods are there to improve performance when adding / removing large number of points. I'd also point out that depending on your needs, ShadedPointPainter (see the example: com.metsci.glimpse.examples.basic.ScatterplotExample) may be more appropriate. It has the advantage of allowing dynamic adjustment of the color/size of the points based on an associated color/size scale axis.

 

Is there a way to change the point size in ScatterPlot when using an Accumulator that doesn't require creating an array of sizes. I often want them all to be the same size?

It depends on how you're drawing the data. If you're following com.metsci.glimpse.examples.basic.ScatterplotExample and using ShadedPointPainter, you can call ShadedPointPainter.setConstantPointSize( ) to set a fixed size for all the data points.

  

Is there a way to change the point shape in scatterplot?

The way to do this efficiently in OpenGL would be via a Fragment Shader (https://people.freedesktop.org/~idr/OpenGL_tutorials/03-fragment-intro.html#id2827467). But the Glimpse painters don't currently provide an easy way to do this.

 

Is there an example of brushing? Seems like that kind of what you suggested can be done with the timeline/scatterplot combo. Not sure if there's a general way to select a group of points using a click-drag rectangle.

It's supported on a per-painter basis. So it depends on which GlimpsePainter class you're using to display your data. For example, DynamicPointSetPainterExample uses a selection based on the mouse cursor selection box and prints the IDs of all the data points contained inside it. Which GlimpsePainter are you looking to select data from?

 

Is there a way to get standard cross-hairs in the time-series plot? I can't find a reference to the cursor (which displays as a vertical line only in my code).  I'm asking because I found it confusing that the display by the cursor showed the vertical position of the cursor, rather than the intersection of the timeline at that horizontal position. I thought having a standard cross hair would make it clearer what the numbers meant.

 

Finally, it seemed like my CPU was working hard even when displaying a static visualization - So much that my laptop gets really hot whenever a couple of plots are left on the screen. Is there anything I can do about that?

By default, Glimpse currently uses a fixed frames-per-second rendering loop. It's definitely been on our list to provide support for event-based redrawing (only updating the screen when data, axes positions, etc... change). It's not officially supported yet. However, you can pretty easily get most of the way there.

Check out this example: ManualRepaintExample.java. Basically, you set up the GlimpseCanvas like you would from the examples, but don't do the step where you create an FPSAnimator. Instead, you add an AxisListener1D which repaints the canvas when the axes change. Note: you'll now have to manually call that repaint method whenever you make programmatic changes to your Canvas (like modifying the data).

[Geoffrey Ulman]

I wasn't completely clear on the cross-hair cursor question. Could you describe a bit more what you expect to see? Maybe a screenshot would be useful?

Geoff

[Larry White]

I can't screenshot it because the issue only appears when the mouse is over the glimpse window, but it doesn't work if another app is controlling the mouse.

Basically  a box appears next to the cursor, and it gives the horizontal and vertical position of the cursor, but often there is no actual data under the cursor. The data point represented in the time series at that time value may be much higher or lower than the current cursor position. I originally thought the numbers displayed in the box were the x and y values for the series at that x coordinate on the chart, but it's just the cursor location.

thank you for the quick answers. much appreciated.

[Larry White]

I think I missed the critical parts: I see a single blue vertical line, as i have set the time selection width to 0, I no longer see the pale blue rectangle over the plot. I would like to see a crosshair pointer as that would make it clearer that the displayed values reflect the pointer position and not the value of the series where it crosses the vertical blue line that follows the cursor position.