ShinyApp unexpectedly exits (seemingly when a large dataset uploaded)

[Namaloom Martian]

Hi,

I am working on an application which should allow users to upload data and then do some textual analysis thereof.
Here is what I want to achieve:

1. Allow users to upload text dataset using any from the list provided, or from their local space.
2. Perform pre-processing of the dataset.
3. To generate features from the dataset.
...

But I face errors in 2) and 3). The errors are such that:

i. Sometimes they occur, sometimes do not occur (though it appears to me the error occurs more often when a large dataset is used. For example, I have 3 datasets - UAE Expat forum, UAE TripAdvisor, and Middle East Politics Forum - and the error occurs more often with the Middle East Politics forum dataset).
ii. Sometimes no error is sent to the Javascript console, sometimes the following

Listening on port 59669
Read 2230 items
Warning in parallel::mclapply(x, FUN, ...) :
  all scheduled cores encountered errors in user code
Warning in parallel::mclapply(x, FUN, ...) :
  all scheduled cores encountered errors in user code

The above occurs when the pre-processing phase fails and the application exits unexpectedly.

Warning in parallel::mclapply(x, FUN, ...) :
  all scheduled cores encountered errors in user code
Warning in parallel::mclapply(x, FUN, ...) :
  all scheduled cores encountered errors in user code
Warning in parallel::mclapply(x, termFreq, control) :
  all scheduled cores encountered errors in user code
Warning in is.na(x) :
  is.na() applied to non-(list or vector) of type 'NULL'
Warning in TermDocumentMatrix.VCorpus(preprocessedCorpus(), control = list(weighting = function(x) weightSMART(x,  :
  invalid document identifiers
Warning in simple_triplet_matrix(i = i, j = j, v = as.numeric(v), nrow = length(allTerms),  :
  NAs introduced by coercion
Error in simple_triplet_matrix(i = i, j = j, v = as.numeric(v), nrow = length(allTerms),  :
  'i, j, v' different lengths
Error in simple_triplet_matrix(i = i, j = j, v = as.numeric(v), nrow = length(allTerms),  :
  'i, j, v' different lengths

The above occurs when the pre-processing phase is accomplished, but the feature generation phase results in an error. The above error must not occur as I have tested the same code locally, and it works fine each time I try.

I have also noticed shiny-server-pro.js in the console, but I am afraid I don't know if that is relevant.

Here is the link to my app: https://aliakazmi.shinyapps.io/sampleTextAnalysisApp/
Here is the link to the ui.R file: https://gist.github.com/noobuseR/711a39df5ce16e0093e6#file-ui-r
Here is the link to the server.R file: https://gist.github.com/noobuseR/711a39df5ce16e0093e6#file-server-r

Please do let me know if there is more I can do to assist in the process of solving this problem.

Kind Regards,
AK

[Andy Kipp]

Hi,

The problem you describe sounds like it could be because your application instance is running out of memory. I would suggest reading the Application Instances section of the guide, and trying increasing your instance since to a larger type.

-Andy

[Marcus Schwemmle]

My app also quit unexpectedly over and over again even though it worked fine locally... quite frustrating. Then I found this thread and tried:

shinyapps::configureApp(APPNAME, size="medium")

Now it works a charm. Better error messages would be useful e.g. "Insufficient memory. 256mb available 426mb required."

Best

Marcus

[Andy Kipp]

Marcus,

Thanks for the feedback. I agree we need better error reporting here. We also intend to expose utilization metrics for people's apps so they can see how much memory they are using. I've also come to conclude that 256MB of memory is probably too small for a lot of people's apps. We may make the default 512MB from now on.

-Andy