using webgl(2D) to boost d3js performance

[Phyo Arkar Lwin]

Hello D3js!

This is my first post here but  we were using D3js for a few months already , and it works wonders.

From what i see d3js mainly support declarative elements to be manipulated (DOM,Canvas and SVG as speciality)

Have anyone tried combining D3js with javascript object-based libraries ?

What i want to do is to use d3js in a combination with pixijs or cocos2d-html (they are 2D frameworks that uses webgl - for performance and they are multiple times faster than Canvas element which d3 supports well)

What do you think people?

What i want to do is 2d-scatterplot using webgl(2d) frameworks and i want functionalities like zoom/pan and rect selection box.

But d3.svg.brush is specifically SVG so i am not sure its going to work on those libraries.

Thanks a lot for d3js! Looking for your reply!

Phyo.

PS: 

here is my stackoverflow question  with bounty :

http://stackoverflow.com/questions/16236220/suitable-library-for-combining-with-d3js-to-allowing-drawing-to-webgl-2d

here are libraries i mentioned:

https://github.com/GoodBoyDigital/pixi.js

http://www.cocos2d-x.org/

[David Kim]

Hello, 

This is a very interesting question to me too, and would like to know how as well.

(I was looking at three.js (basically, it is for 3D,though) to see if there's a way to visualize 

millions of images for the Viz module of a content-based image retrieval system, and 

planning to play with it & D3 during this summer.)

---

By the way, (your link to the scatter plot matrix example reminds me of this), 

I could draw this with D3js: each dot represents speech - 163 speeches, and 

each cell chart shows the co-relational points between two topics for the speeches - Tp stands for Topic.

And, as you asked, having Zoom-in/-out functionality will be fantastic especially when the data to plot is big.

http://davidkim.info/viz/multivariate/1/speech_topic.png

David

--
You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Phyo Arkar]

hey david

nice to have someone in same boat. thanks!
yes we had draw one using  d3js for kmeans clusters using that example including zoom and select box but when reaching 20k plots it slows down in frame rate.
so we planning to change rendering from svg to webgl.

[Phyo Arkar]

wow that huge amount of data is what you plot in d3js?or non js lib?how much it slows for you?

On May 3, 2013 2:42 AM, "David Kim" <davi...@gmail.com> wrote:

[marc fawzi]

Is Vega open source? I can't recall, but I do remember that it comes with an abstraction layer than makes rendering using X technology possible.

[Phyo Arkar]

hi mark

what is vega?js lib?

[David Kim]

Yep, nice to see you, Phyo.

It takes 3 seconds on Chrome and Safari, but FF can't finish the process. 

Oh, I had one for k-means for a simple document clustering, and with D3, 

I chose a simple force directed graph instead of the standard scatter plot with centroids ~ : D. 

[Phyo Arkar]

3secs means both generation and rendering?how many points in total? i must be doing something wrong. with my graph it is only kmeans document clustering of and scatterplot of 20000 points making browser sluggish.

[David Kim]

Mine should have 163*20*20 = 65200 dots. 

3 secs doesn't include the k-means process, though. K-means is done by mahout, and the Viz just imports 

the result data after massaging it~

[David Kim]

Oh, the scatter plot matrix is not about k-means. The data is about topic modeling, which is 

done by LDA (Latent Dirichlet Allocation).

[marc fawzi]

Vega is a product of the same team that brought us D3 ... google for it, it looks great for standard customizable visualizations

[Phyo Arkar]

we are slowing at 20k so something is off . it is just 20k * 1

[Phyo Arkar]

i found it but only have canvas vs svg right?from looking from description.webgl is faster tho.

[Phyo Arkar]

i am doing with sklearn python. but the prob is serverside

[Phyo Arkar]

urhg.i meant browser side not server

[Ian Johnson]

A lot of the power of d3 is binding to the DOM, which has performance limits for elements in the hundreds or few thousands. You probably want to avoid this part of d3 for these applications, but you can still use many other powerful pieces of d3 to assist you with webgl.

You can use scales to map your data into spatial and color dimensions. This is pure javascript and shouldn't be slow at those numbers.

You could also use the brush component or zoom behavior, simply layer it over the top of your webgl visualization using the same scales.

D3 is like a toolbelt for making data visualizations with web standards, it is at its most elegant when using SVG, just keep in mind you can mix and match the standards and the tools!

[Kai Chang]

The brush doesn't depend on the elements being rendered in SVG. Only the brush itself is SVG.

WebGL-2d test with SVG brushes for parallel coordinates: http://bl.ocks.org/syntagmatic/5023284

WebGL-2d and Scatterplot transitions: http://bl.ocks.org/ZJONSSON/raw/3229856/

[Phyo Arkar]

Ah thanks alot  that explains me a lot! I am going to check those out and try abit.

[Phyo Arkar]

is there any examples with mapping data to spatial and color dimensions? 

[Ian Johnson]

almost every svg based d3 example is about mapping data to spatial or color dimensions.

perhaps some of my videos might help here:

color scales:

http://www.youtube.com/watch?v=PFxWmjiUWII&list=PLI_sHchSmdCDLfLl5uTnsaRB54tDlRubK&index=1

brush:

http://www.youtube.com/watch?v=KhONK158L_A&list=PLI_sHchSmdCC2Yg2IXzhGUc2V6dfZK_dV&index=4

(code examples are linked in the descriptions)

Ian Johnson - 周彦

http://enja.org

[Phyo Arkar]

ah i understand now. the dimensions are called spatial dimensions

[Kai Chang]

The difference with canvas is to "map to a color dimension" actually means to change the canvas style state, then do some rendering.

ctx.fillStyle = "red";

// render your red circle here

ctx.fillStyle = "green";

// render your green circle here

So there's quite a bit more imperative code where execution order matters. You also can't use d3's selections and transitions to change the colors of circles-- you'll need to create custom tweens and transitions and render each frame.

[Dave Edelhart]

I would suggest that pairing d3 with a higher level canvas manipulator like Easel might be the way to go. 

Canvases don't have a native persistent library of objects, whereas Easel /kinetic does. 

[Phyo Arkar]

Hi dave , 

does Easel support webgl?

[Phyo Arkar]

I checked Kinectic and Easel , looks quite cool. What i read so far in kinetic , it have full OO support which i am looking for too.

But i am not sure they have Webgl(2D) backend  , which have far superior performance than Canvas.

[Stardrive Engineering]

Fabric has support for SVG path plus a robust scenegraph layer on top of the standard canvas and looking at the different canvas libraries it looks like the best option:

http://fabricjs.com

The Fabric roadmap todo list also has WebGL as a next target.

The canvas v5 spec has also been out for a while and some of the awesome features should be showing up in browsers soon. The author of Fabric has made following feature test available:

http://kangax.github.io/jstests/canvas-v5

Eventually, hopefully things will speed up. Until then D3 is to slow to run well on mobile platforms!

Thom

[Dave Edelhart]

https://github.com/CreateJS/EaselJSRenderers

is the plugin/abstration for webGL -- it looks like they have it but its not actively pursued. 

I used Kinetic for a while. There is ittle practical difference between Kinetic and Easel - you get an abstraction layer in both cases -- but I'd put my money behind Easel as Adobe has a vested interest in maintaining it. 

I think if people expressed a strong interest in that area they might plug it more actively. 

Keep in mind their target audience is Flash refugees who don't have a huge demand on performance -- farms grow slowly :D

[Phyo Arkar]

Looks interesting too! Tooo many js Canvas libraries .. would be cool if they join efforts ..

[Phyo Arkar]

After a few researches on google i found this , quite interesting performance!

http://bl.ocks.org/pjanik/5238326

Performance test of Pixijs(WebGL) + D3js! 

Animating 5000 circles goes at 60fps on my laptop!

[marc fawzi]

i didn't spot D3 in that example

but Mike (author of D3) has been playing with WebGL per his latest bl.ocks

http://bl.ocks.org/mbostock

not to say that the two things are related, but just noting the increasing interest in WebGL overall as the ultimate rendering layer for the web

correct me if I'm wrong but my impression is that mobile implementations remain too slow or non existent

[Kai Chang]

GLSL.js is a nice library for getting started with fragment shaders. The intro video assumes no knowledge.

https://github.com/gre/glsl.js/

And then Mike's recent examples show you how to combine vertex/fragment shaders without a library.

[Phyo Arkar]

yeah sorry it is just comparisons between d3svg vs webgl pixi vs kinetic canvas

[Phyo Arkar]

can you give me link of mike's experiment?

[Phyo Arkar]

we are testing  with pixijs and threejs
webgl2d with textures implementation of pixi is alot faster than svg but threejs partical system is very fast. we plot 500000 particles without dropping a frame.

[karth...@gmail.com]

We are planning to build a high performance visualization using D3 and WebGL - not sure, which type of chart we will use as of yet - parallel coordinates or scatter plot or cluster.
Can you share your code used? We can probably use it as a reference.

[Kai Chang]

Here's my experiment with parallel coordinates and WebGL-2d http://bl.ocks.org/syntagmatic/5023284

For more options, visit https://groups.google.com/d/optout.

[karth...@gmail.com]

Thank you so much!

[cheemengs ho]

'm trying to use images in d3.js but webgl rendering, following is the code in jsfiddle that shows d3 force layout with simple shapes rendered with webgl. i'm trying to use different images/textures for each of the circles, and i'm lost trying out the shaders/vertexes.

Here is the jsfiddle http://jsfiddle.net/bb4Le5w9/

[Chris Viau]

You should start a new thread, preferably on StackOverflow. But here is a list of WebGL and X3DOM examples that may help:

http://bl.ocks.org/camio/5087116

http://bl.ocks.org/hlvoorhees/5986172

http://bl.ocks.org/hlvoorhees/5376764

http://jsdo.it/makc/djN0/fullscreen

http://bl.ocks.org/ZJONSSON/1291667

http://css.dzone.com/articles/render-geographic-information

https://vida.io/documents/N4jSip7n68yQ48DXp

http://codepen.io/QuanFengnan/pen/oDaig

http://blog.thematicmapping.org/2013/11/showing-gps-tracks-in-3d-with-threejs.html

http://www.delimited.io/blog/2014/3/14/d3js-threejs-and-css-3d-transforms

https://github.com/asutherland/d3-threeD

ChrisV

[cheemengs ho]

Thanks Chris!!!!

[Phuoc Do]

Chris included a link to my d3 force layout example (https://vida.io/documents/N4jSip7n68yQ48DXp). I've used d3 with three.js and x3dom. three.js is very fast. Much faster than x3dom and svg. I generated visualizations with 7k elements. You can make rendering faster if you decrease frame rate.

You can adjust lighting and turn off shading to make objects look like 2D. Clicks are a bit harder to handle than svg, but manageable. Bonus points are you get zoom, pan for free. three.js documentation is also good.

[cheemengs ho]

Hi Phuoc Do,

Nice work on combining three.js with d3,
just wondering if its possible to do http://bl.ocks.org/mbostock/3231298
with three.js ?

[cheemengs ho]

I've managed to apply the collision detection plus your three.js implementation.
But its very jittery and doesn't follow the mouse too well,
here's the jsfiddle but its not showing online, any way to make it less jitter?

[cheemengs ho]

here's the jsfiddle link
http://jsfiddle.net/Lgtvnhet/3/

[Phuoc Do]

Hi cheemengs,

The issue is with coordinate transform from 2D mouse.x, mouse.y to 3D coordinate. You'll need something like this:

        var vector = new THREE.Vector3();

        vector.x = ( event.clientX / WIDTH ) * 2 - 1;

        vector.y = - ( event.clientY / HEIGHT ) * 2 + 1;

        vector.z = 0.5;

        var projector = new THREE.Projector();

        projector.unprojectVector( vector, camera );

        

        var dir = vector.sub( camera.position ).normalize();

        var distance = - camera.position.z / dir.z;

        var pos = camera.position.clone().add( dir.multiplyScalar( distance ) );

        // now pos.x, pos.y are 3D coordinates to use in force layout

Also when translate to force layout position, use x and y scales. For example, x(pos.x) and y(pos.y). See this updated fiddle:

http://jsfiddle.net/Lgtvnhet/4/

Phuoc

[cheemengs]

Hi Phuoc,

Wow, awesome work, thank you so much :)

This is actually for in-moderation.com, a social awareness we came up with (www.wunderfauks.com)
just our own in-house project.

--
You received this message because you are subscribed to a topic in the Google Groups "d3-js" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/d3-js/bup6gwrymZE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to d3-js+un...@googlegroups.com.

[da...@ayasdi.com]

At Ayasdi, we faced a similar constraint with D3 for rendering large networks (thousands of nodes and edges). 

We switched to WebGL using Pixi.js and our own API (github.com/ayasdi/grapher).

You can read more about our results and methods here:
http://dcochran.com/home/107616419392

[Eric Forgy]

Hi Danny,

Thanks for the heads up. As I was learning d3, the presentation by Josh Lewis on how you guys are using d3 and backbone was pretty inspiration for some things I'm working on, so it is a small bummer to hear you moved away from d3 for the viz (although you still use it for transforms). I'll have a look at Grapher, but, so far, I'm not using node.js server side (I went Spring MVC), but I might set up node for use with Julia. I'm having trouble getting Julia, Java, and d3 to play nice together so may set up a Julia server via node and access it from Java using RESTful API.

Cheers,

Eric