Shiny + Leaflet Bottlenecks

[Malcolm Charles]

This is a pretty high-level question, so not sure if there is an appropriate answer...

I've been building some Shiny+Leaflet mapping apps for my job. Thus far, I've been limiting the # of points I'm plotting in order for the app to run smoothly and be responsive. In a few instances, I've tried to scale up to hundreds of thousands of points and see the dramatic slow down.

I'm beginning to get interest from company higher-ups in developing these apps further. I was wondering if anyone here could tell me what they know about what bottlenecks a Shiny leaflet app? I was thinking of doing a Proof of Concept using a dedicated Shiny server with a beefy amount of RAM and processing power (relative to my dinky laptop), but wanted to know what to expect in terms of scaling and bottlenecks. I'm not an experience software developer so excuse me if I'm using incorrect terminology.

[Chris Lee]

Hey Malcolm, first of all congrats - that's an exciting stage of a project.

You are going to have serious bottlenecks when trying to show tens of thousands of points let alone "hundreds of thousands of points" with Leaflet or OpenLayers, etc (not due to R RStudio/leaflet or your server). It will be worse once you put it on a server because then it will have to transmit the data to the client (which is actually a worse bottleneck ...). However, you have many options and I'll highlight only a few (I'm going to assume the styling needs to change based on R calculations or user inputs):

Stick with what you have:

• only show what is within map_bounds (could be slow data transmission)
• Yelp-style click to analyze new region and then show points (might not be good UI design ...)
• marker clustering 
• depending on the zoom level, aggregate to an appropriate jurisdictional boundary (zip+4, county, etc) 

Add Geoserver:

• set up GeoServer on the same server and use a pre-existing shapefile of points (or host using PostGIS). You will need to send custom styling from the app to Geoserver which still won't be perfect. I'd be happy to help with this off-line.

Ditch it all:

• If none of these solutions do the trick (and your users have WebGL enabled browsers) you may have to jump ship: https://github.com/mapbox/mapbox-gl-js
  

Chris

[Malcolm Charles]

Thanks for your response, Chris.

I've been working with your first few suggestions - sticking with what I have. I hadn't considered implementing that 'Yelp-style' click. That sounds like it could be a good option for me. I'll search around to see if there is some related code up on a github (unless you know of an example?). I've used marker clusters before, because I don't really need to see every distinct point at a wide zoom, but there doesn't seem to be much of an impact on speed in any case.

I have a little bit of experience with setting up GeoServer & PostGIS, but my 'coding' skills are limited primarily to R. So I've been trying to squeeze as much juice out of the Shiny/Leaflet fruit as I can, so to speak.

The more people I share this little app with, the more the excitement around it grows. Consequently, the more concerned I get that I'm going to be asked to do something beyond the limitations of this framework - hence my initial question. In truth, the app isn't really for analysis or even for concurrent users (I do the geospatial analysis in R itself, without needing to visualize it). It's more of way to market capabilities that look a little less sexy as lines of code :)  

[Joe Cheng]

There's a plugin coming for heatmapping, you can also do simple rasters now using addRasterImage. Hexbinning would be cool too (I haven't thought about it much though).

--
You received this message because you are subscribed to the Google Groups "Shiny - Web Framework for R" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shiny-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/cba66e7f-ef69-41d5-89a5-b7b89c40eb1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Lee]

Yelp-click will just be an actionButton tied to a reactive that does a gBuffer + gIntersection (sp object) from the subsequent map_click. I'm sure the framework will work for you but you may have to add on other tools - it's certainly worked for us. E.g., if the app is really for visualizing static data you could make your own basemap with mapbox or cartodb and just add it into leaflet as such.

Joe, hexbinning is cool, and FWIW Cartodb has open-sourced their PostGIS extensions including that capability: https://github.com/CartoDB/cartodb-postgresql/blob/224b2ce39504f974cc47602fd49c3fecb60a5b27/scripts-available/CDB_Hexagon.sql. While we are the topic of showing/sending a lot of data, I'm wondering if this would help: https://github.com/rstudio/leaflet/issues/237.

Chris

[Matt Schumwinger]

Hi Malcolm,

I'm new to developing leaflet/shiny apps too and really appreciate your question and this discussion thread. 

Will your point data change very often? If not, I wonder if one approach to your problem may make raster tiles of the point data with the attributes of each point baked into the UTF grid. I know this is old school, but I would do that in TileMill and host the tiles on a server which your shiny app can then access. If your data changes, but not too frequently, perhaps you can write a script to make the tiles overnight. Perhaps this is related to what Joe Cheng is getting with his addRasterImage solution. 

- Matt

On Monday, May 16, 2016 at 2:07:22 PM UTC-5, Malcolm Charles wrote: