things got much slower between 3.1.10 and 3.2.0

74 views
Skip to first unread message

Pierce Krouse

unread,
Apr 22, 2016, 4:43:10 PM4/22/16
to d3-js

Apologies for bringing up something that happened in 2013, but while running down a performance problem, this is where things changed.

We are building a scatter plot function with d3 and decided to go with a tooltip implementation that is maybe a bit different from most. Instead of mousing over a point to get the popup, our users will click in the plot area and we bring up the tooltip at the nearest point.  We also maintain/redraw the tooltip when the user zooms or pans.  So, no mouse events are attached to the datapoints, just a click event capture where we find the nearest point and put the tooltip in.

The problem is, on my machine with a scatter plot of around 3000 points, the popup does not appear until about 3 seconds have passed.  I timed the nearest-neighbor calculations and they only take about 22 msec.  All the time is taken up _before_ my mouse click routine gets called.  This problem scales with the data load, I checked that too.  So while putting all of this in jsfiddle, the problem went away entirely.  Somehow I had loaded a much earlier version than 3.5.16, and stumbled on this.  So I went back to my standalone html file that showed the problem, and loaded an earlier d3 version (2.8.1 instead of 3.5.16) and the problem was indeed gone.  I switched back and forth a few times just to be sure. So I started spot checking d3 versions between the two to see where the problem came in.  Here is what I found:


https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.16/d3.min.js // slow ...
https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.10/d3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/3.4.13/d3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/3.4.5/d3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/3.4.0/d3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/3.3.13/d3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/3.3.7/d3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/3.2.1/d3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/3.2.0/d3.min.js    // ... slow
https://cdnjs.cloudflare.com/ajax/libs/d3/3.1.10/d3.min.js   // ok...
https://cdnjs.cloudflare.com/ajax/libs/d3/3.1.8/d3.min.js   
https://cdnjs.cloudflare.com/ajax/libs/d3/3.0.4/d3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/3.0.1/d3.v3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/3.0.0/d3.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/2.10.0/d3.v2.min.js
https://cdnjs.cloudflare.com/ajax/libs/d3/2.8.1/d3.v2.min.js // ...ok

To recap, versions 3.1.10 and older handle 3000 points in a fraction of a second.  Later than that, 3 or 4 seconds.

Now another frustrating thing is that I put together 2 versions on codepen, and the difference is not that dramatic there, but the difference really is apparent in standalone files.

And I cannot stay back on 3.1.10 because I depend on zoom.center, which came into being later.
If there is a way to implement zoom.center in 3.1.10, that might be a stopgap solution, but there is some seriously cool stuff in d3 that I want to take advantage of, and 3.1 will cut me off from 2.5 years of improvements.


So here are codepen demos.
fast one:
http://codepen.io/pkrouse/details/JXBpKo/

slow one
http://codepen.io/pkrouse/details/ONwQmz/

Apologies to all, this code is kind of a mess.  And I am happy to attach standalone files for people if you want, but the codepen examples will hopefully show the problem.

--PK

Pierce Krouse

unread,
May 4, 2016, 10:49:49 AM5/4/16
to d3-js
I am adding the relevant code to this post to see if it generates any interest.

First up is the svg initialization:
var svg = d3.select("body").append("svg")
               
.attr("width",viewportWidth)
               
.attr("height",viewportHeight)
               
.call(zoom)
               
.append("g")
       
.attr("class", "main")
               
.attr("transform","translate("+(left_shift)+","+(top_shift)+")");

Next is the mouse click registration. We use the click to find the nearest data point and put the tooltip there. I have some timing code in there
 
d3.select('svg').on("click",  function() {
     
var startTime = new Date();
     
var coords = [0, 0];
      coords
= d3.mouse(this);
     
var isClicked = isPointClicked(point_clicked);
     
if(!isClicked) clicked(coords[0], coords[1]);
      point_clicked
= false;

     
var finalEndTime = new Date();
     
var finalTimeDiff = finalEndTime - startTime;
      console
.log('all click actions completed in '+finalTimeDiff+' milliseconds.');
   
});

We use the mouse right-click to turn off tooltip actions. Skipping that code, showing the plotting code. 
I think this part is relevant only in that it shows that I am not attaching any mouse events to the data itself.
var side = 2;     // takes place of radius
   
// shiftrect needs to be half the square size to center square on data point
   
var shiftrect = 1;
   
for(var k=0;k<plot_data.series.length;++k) {
       
if(plot_data.series[k].visible === true) {
           
var y_list = plot_data.series[k].ydata;
           
var x_list = plot_data.series[k].xdata;
            svg
.selectAll("dot")
               
.data(x_list)
               
.enter().append("rect")
           
.attr("id", function(d, i) { return "rect"+k+i;})
               
.attr("x", function(d) {return x(parseFloat(d))-shiftrect;})
               
.attr("y", function(d,i) {return y(parseFloat(y_list[i]))-shiftrect;})
               
.attr("height", side)
               
.attr("width", side)
               
.attr("fill",function() { return color(k);})
               
.attr("clip-path", "url(#plot-clip)");
       
}
   
}


So that's basically it.  When I click the mouse in a scatter plot with only a few data points, the tooltip code is called almost immediately.
If I have a few thousand entries, there is a significant delay before my code is even reached.  This delay exists in D3 3.2.0 and later. 
D3 3.1.10 does not exhibit this behavior, and as you can see my data has no direct relation to the mouse click itself.

That's why I think this is a bug introduced in the 3.2.0 code.


Can anyone think of a different approach to get around this significant slowdown?  Right now, the slowdown is stopping me from delivering this functionality.

Is there something I can do that gets the mouse click in pixel space relative to the svg element without actually tying it to the svg?  Can I place an empty svg in the same position on the page that serves as a ghost or placeholder?  All I need is a way to quickly capture this mouse click without having the data get in the way, since the data is not directly responsible for this mouse click.






Mike Bostock

unread,
May 4, 2016, 12:11:36 PM5/4/16
to d3...@googlegroups.com
Can you post a version that reproduces the problem? Perhaps JSFiddle’s wrapper frame is interfering somehow; I recommend using GitHub Gist and bl.ocks.org.

The only relevant change I can think of in 3.2 is that the zoom behavior was changed to apply user-select: none to the document element on mousedown, rather than immediately calling event.preventDefault.


It seems like that might be triggering a redraw of your scatterplot, which is then slow because it has thousands of elements. It might be faster if D3 registered a selectstart listener on the window and cancelled selection instead (which is what D3 does if user-select: none isn’t available).

If you want to draw thousands of elements interactively, you might have better luck moving those elements to Canvas, while still using SVG for things like axes (and perhaps HTML for tooltips). That’s a pretty common optimization technique. See this, for example:


Although it rarely matters, you can also use D3’s quadtree to accelerate finding the closest point to the mouse (which should be a lot faster than a Voronoi overlay if you have thousands of points):


Mike

Mike Bostock

unread,
May 4, 2016, 12:19:17 PM5/4/16
to d3...@googlegroups.com

Pierce Krouse

unread,
May 5, 2016, 10:27:57 AM5/5/16
to d3-js, mi...@ocks.org
Hi Mike, thanks for weighing in on this issue.

There are two codepen links in my original post.  The only difference between the codepens is the version of D3 that is referenced.  JSFiddle was causing problems, so I went with codepen instead. The difference in speed, for reasons I do not understand, not as dramatic when I ran the codepens compared to seeing this in production on my server.

I doubt the scatter plot is being redrawn during the mouse click detection, but I will check just to be sure.

Pierce Krouse

unread,
May 5, 2016, 5:52:44 PM5/5/16
to d3-js, mi...@ocks.org
OK Mike, to your triggering a redraw comment, I logged the beginning and end of my plot loop, and the loop itself did not fire when I clicked the mouse.  Is it possible
that D3 itself is replotting the data somehow internally?  I have no idea how D3 works under the covers.

The codepens were giving me a warning since I created them before I created my codepen account.  Hopefully you did not run into that.

OK, now for the big update:  This is only a problem on firefox (but the problem did not exist in 3.1.0).  FF is my default dev browser, and I forgot to check IE (chrome is another story, for another day,  I'll buy you a beer and we can talk). 

IE does not show this problem. 

Mike Bostock

unread,
May 5, 2016, 9:18:34 PM5/5/16
to Pierce Krouse, d3-js
By “repaint” I meant the browser is redrawing the page. You can only see this is you enable repaint flashing in the developer tools on Chrome. (Firefox might have a similar feature.)

Mike

Pierce Krouse

unread,
May 13, 2016, 11:43:12 AM5/13/16
to Mike Bostock, d3-js
OK, now I see what you mean.
I don't have a good handle on getting this to work in chrome, but chrome is working well anyway.  I checked repaint flashing in IE and it looks like it is repainting everything.

Is there a way to prevent this in D3?

The other option I see is to place an empty svg exactly on top of the svg used/created by D3, capture the mousedown, mouseup, and click events there, and call the functions for my tooltips from there.  This should sidestep whatever the browser is doing with the svg elements in the svg that D3 has created, and I am sure that geometry is a big part of the problem here.

How will that affect the scroll function in the d3 element though?  On mouse down and mouse up, I do some calculations that are used on mouse click.  Can I pass the mousedown and mouseup directly to d3 when I am finished with them?  I really only want to sequester the click action since that is the one I am most concerned about.

Pierce Krouse

unread,
May 13, 2016, 5:30:51 PM5/13/16
to d3-js, mi...@ocks.org
OK, so I made a lot of progress today, and have one problem left.

I went ahead and overlaid another SVG element on top of the d3 SVg element in my page. I did it with nested divs and relative/absolute positioning:
 div.plot_container {
     position
: relative;
     margin
:0px; padding:0px; /* insurance */
     border
: none;
 
}
 div
.ghost_container {
     position
: absolute;
     margin
:0px; padding:0px; /* insurance */
     top
: 0px;left: 0px;
     width
:0px; height:0px; /* width and height are irrelevant */
     overflow
: visible; /* insurance */
     border
: none;
 
}
<div class = "plot_container" id = "thing_plot_container">
   
<div class = "ghost_container" id = "thing_ghost_container"></div>
</div>


This was straightforward, and works in ie, ff and chrome.
This next snippet is just my standard d3 js svg object creation:
var svg = d3.select("#thing_plot_container").append("svg")
               
.attr("id", "mySVG")

               
.attr("width",viewportWidth)
               
.attr("height",viewportHeight)
               
.call(zoom)

               
.on("dblclick.zoom", null)

               
.attr("transform","translate("+(left_shift)+","+(top_shift)+")");

And finally my custom SVG that overlays the main one:
    var use_overlay = true;
   
var catcherWidth = viewportWidth;
   
var catcherHeight = viewportHeight;
   
var svgns = "http://www.w3.org/2000/svg";
   
if (use_overlay == true){
     
var mouse_div = document.getElementById('thing_ghost_container');
     
var mouse_svg_container = document.createElementNS(svgns, "svg");
      console
.log('replace these magic numbers with viewportWidth and height');
      mouse_svg_container
.setAttribute("width",catcherWidth+"px"); // clips on chrome/ff if not done this way
      mouse_svg_container
.setAttribute("height",catcherHeight+"px");
      mouse_div
.appendChild(mouse_svg_container);
     
var rect_mouse = document.createElementNS(svgns, "rect");
      rect_mouse
.setAttribute("x",0);
      rect_mouse
.setAttribute("y",0);
      rect_mouse
.setAttribute("fill","blue");
      rect_mouse
.setAttribute("fill-opacity",0.05); // set this to 0.0 when working
      rect_mouse
.setAttribute("width", catcherWidth+"px");
      rect_mouse
.setAttribute("height", catcherHeight+"px");
      rect_mouse
.setAttribute("onclick", "ghost_click(evt)"); // this HAS to be evt to get all browsers cooperating
     
// leaving these mouse down/mouseup events to the original svg does not work. The overlay svg smothers it
      rect_mouse
.setAttribute("onmousedown", "ghost_mousedown(evt)");
      rect_mouse
.setAttribute("onmouseup", "ghost_mouseup(evt)");
      mouse_svg_container
.appendChild(rect_mouse)
   
}

This works well in that it completely gets around the wait time in IE to receive the mouse click event.  With a  3000-point scatter plot, my tooltip render times went from 2.5 seconds (ouch) to 1.3 seconds.  I know I can squeeze more out of it, but I had to get around the please-oh-please-give-me-the-mouse-event problem first.

The problem with this approach is what I feared.  I cannot pan my chart with this overlay in place.  How would I take the mouse events (I guess just mouse motion?) and pass it to d3 so that it knows to pan?


Pierce Krouse

unread,
May 16, 2016, 6:20:22 PM5/16/16
to d3-js
Here is a fiddle of the latest.  I purposely shortened the overlay SVG to allow you to grab with the mouse and pan the plot.  This functionality does not work if you do that to the part of the plot covered with the overlay SVG, and this is the problem.  All I need to do now is get the mouse motion hooked into the panning code and I will have a good speedup technique.

https://jsfiddle.net/pkrouse/5sbqchnn/

Can anyone point me in the right direction on this?

Pierce Krouse

unread,
May 25, 2016, 5:37:05 PM5/25/16
to d3-js
Has anyone taken a look at this?  As it stands, registering the mouse click in the svg on a scatterplot with 3000 points is not too great in IE.  Mike, you asked if IE was redrawing the page.  It is.  Is there anything I can do about that? 

If not, how about my partial solution with the separate svg?  All I need to do is let some of the mouse events go on to the original svg, or directly into the d3 framework somehow, so that normal mouse panning  and zooming will work.

Mike Tahani

unread,
Jun 1, 2016, 11:38:22 AM6/1/16
to d3...@googlegroups.com
I skimmed your code and have a couple suggestions:

Use a quadtree or another more efficient proximity algorithm instead of doing what you're doing in `clicked()`. You will see immense speedups from this.

In the `changeArrow*()` functions, it looks like you're appending CSS `style` tags as strings to the DOM? This is an extremely inefficient way to apply styles (why not just change the class on the elements?) and will trigger reflows/repaints.

Try using canvas instead of SVG.

The inefficiencies in your code are causing the library to stutter. Fix your code first-- if you're still seeing problems, THEN start disassembling the library.

Mike T



--
You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages