How to run long R Scripts with Shiny

[Shelley Macneil]

Hello,

I created a shiny APP that runs an R script. It can can take anywhere from 1 minute to a few hours to run, depending on the dataset size of the user.  Once the script is done, it sends the results to the user in an email. 

My questions is:

1) Do the commands halt once the user closes the window or the server disconnects?

2)  Is there was a "nohup" like command I can use for the long runs so the user can close the window.

Thanks,

Shelley

[Laz C. Peterson]

Hello Shelley —

Where does the user’s datasets come from?  Do they upload them through the Shiny interface, or is it something they have in a database or on a network share somewhere?

There are a lot of limitations on the user’s connection and session.  You probably don’t want to leave it up to chance if the user closes their browser window.

We have a few things that run processes that may take a bit of time (2 hours is probably the longest).  So on our Shiny server (note: we should probably have this actually running on a different server), we have a cron job that runs once every 5 minutes and checks for CSV files in a folder, which is a mounted network share.  If the cron script sees that a new file exists and is no longer open or accessed, it it moves the file elsewhere and then starts processing.  We also do this with data in MySQL databases.  If there’s something waiting to be dealt with, it goes into a queue and gets processed.

In your case, the Shiny side of it could take the user’s data and put it in a “holding tank” on a database server somewhere to await processing, in a format quite native to R, and also index that user’s request with the data (email address, requested job task, data, etc) to send the results to when it’s done, or do whatever other task would be next.  The cron script would run an R script that would check if there are any data waiting to be processed, figure out which data would be processed the quickest, then prioritize and start the processing (of course sending the results to each respective user when it’s done).

Just an idea.

~ Laz Peterson

Paravis, LLC

--
You received this message because you are subscribed to the Google Groups "Shiny - Web Framework for R" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shiny-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/621a91ad-a44a-4359-9b81-e321def5311c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shelley MacNeil]

Hi Laz,

Thanks for your response!  The dataset is being upload by the user.

I will try cron. Do you think it would be wise to be using cron for every single user run just in case they close the window? I was thinking to have it running every 5-10 seconds. 

Do you know how I would go about storing the files onto the server? Can you explain the mounted network share? Can I just store the data on the Shiny server, or should I use a different/bigger server?  I was thinking this may be how to do it.  https://www.r-bloggers.com/sending-data-from-client-to-server-and-back-using-shiny/

Also, do you think I should move my app to different server with more CPUs in case multiple users want to use the software at once? I don't expect much use at first though. Right now I am using the shiny server. 

Thanks again,

Shelley

Paravis, LLC

To unsubscribe from this group and stop receiving emails from it, send an email to shiny-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/621a91ad-a44a-4359-9b81-e321def5311c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Shelley MacNeil

PhD Candidate

University of Utah 

Huntsman Cancer Institute 

Ph: 505-280-4987 

smacn...@gmail.com

[Laz C. Peterson]

Hey there Shelley --

Your users could have (for example) access to some network share or somewhere on the server that they could save their files. And your script could start (before the Rscript part) by doing a "find" command for any files sitting in that folder. That's just an idea ... but you could definitely have it run on a single cron job.

Either make sure their data gets stored as a file in a specific location on the server, or I would recommend setting up a MySQL server and then using DbWriteTable (or is it DbCopyTable?) functions to just throw it into the database server. Much easier accessed that way.

If your front end app could simply accept their data and place it as a file on the server or into a database, the cron script could run and just check those locations for anything that needs to be processed. You'll just need a way of defining which user is waiting for the results from which files - either putting this information as metadata inside the file name, or having a database table linked with that information. That's a discussion we could have another time. (If you're interested in the database method, I can explain how we do ours in private emails.)

The big issue you're facing is making sure that if they do not wait for processing to finish before closing their browser, the data could go into the oblivion. So to have a reliable system, it will definitely have to have a few moving parts. A Shiny front end is great, with a cron-style R back end.

We used to have our data stored as files, but the more you work with, the harder it is to deal with. Databases are great because we can index each dataset and flag it as new, processed, processed but not reported yet, etc.

Learning dplyr, DBI and pool package functions will get you sorted proper with the database input and output.

It's been a long week, so I'm probably not making any sense at this point ha ha. But feel free to ask any questions you like.

Have a great weekend.

~ Laz Peterson
Paravis, LLC

[Shelley MacNeil]

Hi Laz,

I totally took your input and now just use shiny to save the user files/parameters to the server. Once shiny copies the files over, shiny is done with. 

I ended up not running cron though.  On the sever, I just crated an R script that is constantly running using nohup, to check to see if the "Jobs To Complete" folder has a new job to run. 

If a job exists, the script moves the file [with the oldest date] to a a folder called, "In process", calls the main R script that processes the files, and then sends an email to the user once it's completed. If the job failed, it sends the user the stdrr file. Finally, the script moves these files to a "completed" folder, and the loop starts over to process the next job. 

Thank you so much for your help and input! 

[Laz C. Peterson]

Rock on, sounds like you have it sorted out!

One thing to pay special attention to, just ensure that your background script has some way of verifying that the file is 100% finished being copied over from the client.  I am not sure how to do that properly in R, (or even bash script for that matter).  But on our Windows servers, the easy test we use to check if a file has finished being written is a “copy file.txt+NUL file.txt >NUL” … And if it errors out saying “The process cannot access the file because it is being used by another process.”  I think linux shell scripts are a little more involved to work with checking file access.

We’ve had to deal with that issue, where users' data files were being moved prematurely, or causing issues with the scripts.  So just make sure that your setup is good to go in that regard.

Cheers.

~ Laz Peterson
Paravis, LLC