Shiny and parallel processing

[Laz C. Peterson]

Hello everyone,

Has anyone had luck getting decent performance out of a foreach (with %dopar%) loop of data used to create a page of Shiny tags?

It almost seems like it’s not going parallel at all … I can post code tomorrow (on the way out of the office right now).

Or — possibly, what would be the .combine parameter that should be used?  I am using paste0(), which is the best I can think of.  tagList() does not work.

Thanks!

~ Laz Peterson
Paravis, LLC

[Joe Cheng]

It's usually not the tag creation that takes a long time, it's converting the tree of tags to HTML (which happens when you print it or as.character or htmltools::renderTags or...)

What exactly are you doing that involves constructing such a large tree of tags?

--
You received this message because you are subscribed to the Google Groups "Shiny - Web Framework for R" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shiny-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/35D3C28F-E86E-4B78-8AE1-57B2D7A135BA%40paravis.net.
For more options, visit https://groups.google.com/d/optout.

[Laz C. Peterson]

Hey there Joe ~

We had a spreadsheet for end-of-year gifts for referring businesses.  This data was then put into a database, and we are now rendering “tiles” through Shiny for each referring business that show the amount of referrals and payments they have generated throughout 2015 and 2016.  These tiles also have a list of the gifts given at the end of 2015, and then buttons to select the gifts they wish to send at the end of 2016.

There are roughly 260 total referring business.  I’m getting closer by making lists of lists, storing them as reactive values (and then unlist()-ing them), so we’re at about 17 seconds now to create the full page.

But realistically, this processing should actually be done in less than 2 seconds on the very very very slow end.  withTags() and the related operations are wonderful, though they would be even better if there was something we can do for performance. :-)

My only issue is that I’m dealing with very impatient people, and when something pops up and it takes more than a couple seconds, they get really angry.  No worries — if there’s nothing we can do to speed up the tags, we’ll deal with it as it is.  But after this project, I’m actually thinking we will be using this method more and more, as it is AWESOME for putting together basic apps!

Thanks Joe!

~ Laz Peterson
Paravis, LLC

[Joe Cheng]

That's an astoundingly long time--that doesn't seem right. Have you tried profvis? I'd love to know what your profvis trace looks like. Or else, save your tag tree using saveRDS and email it to me--I'd love to take a look.

[Laz C. Peterson]

Ok sounds good.  It’s dinner time now, but I can send first thing in the morning.

The weird thing is that if I run it parallel versus single thread, the performance is exactly the same.  So I feel like it’s something with how I’m combining the foreach data.  Since tagList() does not work, we’re using paste0, which may or may not be the right choice.

Either way, I’ll do a profvis and let you know how it goes first (yes, I promise I will start remembering to do these when troubleshooting performance issues), and if I can’t nail it quickly, I’ll then send you the data.

Thanks Joe!

~ Laz Peterson
Paravis, LLC

To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/CAF_jjKqKnC_aiC-hvLsAvo_1LzjkDHb47uKbHMQMndczdF2Qcw%40mail.gmail.com.

[Laz C. Peterson]

Joe, so we ran the profvis and it definitely is hitting a major bottleneck when taking the tagList and preparing it for display.

I’ll send you the rds file, you should be able to simply load that up into your RStudio and take it from there.

We are taking this data (which is a list of all the tags), and doing a tagList() of the object.  Is there a better way to render this?

~ Laz Peterson
Paravis, LLC

Ph: 951.319.3240 x201

To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/5ABE25FC-26A7-4E49-BDC4-8643D2E6701A%40paravis.net.

[Laz C. Peterson]

Also, the fact that it does take so long to render as HTML definitely answers the question about why %dopar% was not effective.  Since the actual rendering is done after all that is finished.

~ Laz Peterson
Paravis, LLC

To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/D6BAF9F1-7FF7-4341-A929-09C09B673FC8%40paravis.net.

[Laz C. Peterson]

Any thoughts on this one, Joe?  (Hopefully the RDS file containing the data we use with HTML() made it to your inbox ok?)

We have this gift app that builds each respective gift giver's (5 individuals) tiles based on only the live data that applies to them (roughly 150 average for each person).  So, there really is no way to do it statically.

And then there is an administrative gift user, who gets tiles of all of the gift givers (an aggregated total of about 300).

So the more tiles we create, the processing time increases exponentially.  I'm sure there is a better way to accomplish this task, but I'm at a loss.

If the HTML output performance from the server side worked as fast as the native Shiny functions, I would probably run around in circles for joy.  It would increase our possibilities infinitely, and I would actually start rebuilding each of our Javascript/PHP apps to be built on Shiny's framework.

Much thanks for your insight.
~Laz Peterson
Paravis, LLC