Re: [shinyapps-users] Disconnected from Server after text input

98 views
Skip to first unread message
Message has been deleted

Tareef Kawaf

unread,
May 29, 2016, 9:39:03 AM5/29/16
to Joel Polanco, ShinyApps Users
We have a long vacation weekend so it will be a few days before our engineers can confirm but I suspect that your application is running out of memory.  The only recourse that you could try now would be to increase the application instance size and see if that makes a difference.  


On Saturday, May 28, 2016, Joel Polanco <jpo...@gmail.com> wrote:
Hello,

Help Needed! I've been troubleshooting all day. My app loads on the server and I am able to successfully load the application via my load button. Upon inputting a partial sentence into the text box the app crashes ... I get a "Disconnected from Server" error. The logs are no help.

File Size of dataframes being searched total ~351MB

Here is a link to my app: https://joelpolanco.shinyapps.io/test2/



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Logs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2016-05-28T23:18:19.890185+00:00 shinyapps[104965]: jsonlite version: 0.9.19
2016-05-28T23:18:19.890186+00:00 shinyapps[104965]: RJSONIO version: NA
2016-05-28T23:18:19.890195+00:00 shinyapps[104965]: htmltools version: 0.3.5
2016-05-28T23:18:20.075257+00:00 shinyapps[104965]: Using jsonlite for JSON processing
2016-05-28T23:18:20.081011+00:00 shinyapps[104965]: 
2016-05-28T23:18:20.081013+00:00 shinyapps[104965]: Starting R with process ID: '12'
2016-05-28T23:18:20.103541+00:00 shinyapps[104965]: Loading required package: NLP
2016-05-28T23:18:49.178541+00:00 shinyapps[104965]: 
2016-05-28T23:18:49.178544+00:00 shinyapps[104965]: Listening on http://0.0.0.0:41390
2016-05-28T23:19:25.455272+00:00 shinyapps[104965]: Server version: 0.4.3.1780
2016-05-28T23:19:25.455365+00:00 shinyapps[104965]: R version: 3.3.0
2016-05-28T23:19:25.455366+00:00 shinyapps[104965]: shiny version: 0.13.2
2016-05-28T23:19:25.455367+00:00 shinyapps[104965]: rmarkdown version: NA
2016-05-28T23:19:25.455388+00:00 shinyapps[104965]: knitr version: NA
2016-05-28T23:19:25.455389+00:00 shinyapps[104965]: jsonlite version: 0.9.19
2016-05-28T23:19:25.455391+00:00 shinyapps[104965]: htmltools version: 0.3.5
2016-05-28T23:19:25.623323+00:00 shinyapps[104965]: Starting R with process ID: '26'
2016-05-28T23:19:25.645527+00:00 shinyapps[104965]: Loading required package: NLP
2016-05-28T23:19:25.455390+00:00 shinyapps[104965]: RJSONIO version: NA
2016-05-28T23:19:25.617358+00:00 shinyapps[104965]: Using jsonlite for JSON processing
2016-05-28T23:19:25.623321+00:00 shinyapps[104965]: 
2016-05-28T23:19:54.687756+00:00 shinyapps[104965]: 
2016-05-28T23:19:54.687759+00:00 shinyapps[104965]: Listening on http://0.0.0.0:59129
2016-05-28T23:23:08.085859+00:00 shinyapps[104965]: Server version: 0.4.3.1780
2016-05-28T23:23:08.259866+00:00 shinyapps[104965]: Using jsonlite for JSON processing
2016-05-28T23:23:08.085933+00:00 shinyapps[104965]: R version: 3.3.0
2016-05-28T23:23:08.085944+00:00 shinyapps[104965]: shiny version: 0.13.2
2016-05-28T23:23:08.085945+00:00 shinyapps[104965]: rmarkdown version: NA
2016-05-28T23:23:08.266108+00:00 shinyapps[104965]: Starting R with process ID: '37'
2016-05-28T23:23:08.085946+00:00 shinyapps[104965]: knitr version: NA
2016-05-28T23:23:08.085948+00:00 shinyapps[104965]: jsonlite version: 0.9.19
2016-05-28T23:23:08.085978+00:00 shinyapps[104965]: RJSONIO version: NA
2016-05-28T23:23:08.086004+00:00 shinyapps[104965]: htmltools version: 0.3.5
2016-05-28T23:23:08.266106+00:00 shinyapps[104965]: 
2016-05-28T23:23:08.288323+00:00 shinyapps[104965]: Loading required package: NLP
2016-05-28T23:23:37.277941+00:00 shinyapps[104965]: 
2016-05-28T23:23:37.277944+00:00 shinyapps[104965]: Listening on http://0.0.0.0:39423
2016-05-28T23:25:47.044667+00:00 shinyapps[104965]: Server version: 0.4.3.1780
2016-05-28T23:25:47.044745+00:00 shinyapps[104965]: R version: 3.3.0
2016-05-28T23:25:47.044747+00:00 shinyapps[104965]: shiny version: 0.13.2
2016-05-28T23:25:47.044757+00:00 shinyapps[104965]: rmarkdown version: NA
2016-05-28T23:25:47.044759+00:00 shinyapps[104965]: knitr version: NA
2016-05-28T23:25:47.044760+00:00 shinyapps[104965]: jsonlite version: 0.9.19
2016-05-28T23:25:47.044761+00:00 shinyapps[104965]: RJSONIO version: NA
2016-05-28T23:25:47.044790+00:00 shinyapps[104965]: htmltools version: 0.3.5
2016-05-28T23:25:47.208398+00:00 shinyapps[104965]: Using jsonlite for JSON processing
2016-05-28T23:25:47.213754+00:00 shinyapps[104965]: 
2016-05-28T23:25:47.213757+00:00 shinyapps[104965]: Starting R with process ID: '47'
2016-05-28T23:25:47.235189+00:00 shinyapps[104965]: Loading required package: NLP
2016-05-28T23:26:16.163818+00:00 shinyapps[104965]: 
2016-05-28T23:26:16.163822+00:00 shinyapps[104965]: Listening on http://0.0.0.0:54120


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
global.R


#load packages
library(tm)
library(NLP)
library(stringr)


#read in data files: 
load('pred_app.RData')

# add prediction function : (bigram, trigram, and quadgram)
predict_bigram<- function(input) {
  if (sapply(gregexpr("\\S+", input), length)<=1){
    return("please enter more than one word")
  }else {
    input <- enc2native(input)
    input1<-tolower(input)
    input2<-str_replace_all(input1, pattern="[[:punct:]]","") #remove punctuations
    input3<-str_replace_all(input2, pattern="\\s+", " ") # replace whitespace with space
    input_clean<-removeNumbers(input3) 
    clean_tail<-tail(unlist(strsplit(input_clean, " ")), 1)
    start_with_term<-paste("^","\\b", clean_tail,"\\b", sep="")
    find<-df_bigram[grep(start_with_term, df_bigram$Word),]
    if (nrow(find)==0) {
      uni<-df_unigram[sample(nrow(df_unigram),nrow(df_unigram)/1000,replace=TRUE,prob=NULL),]
      uni<-head(uni)
      term<-as.character(uni$Word)
      return(term)
    }
    find<-df_bigram[grep(start_with_term, df_bigram$Word),]
    merge<-NULL
    find$Word<-as.character(find$Word)
    find_mat<-matrix(unlist(strsplit(find$Word, " ")), ncol=2, byrow=TRUE)
    find$pred<-find_mat[,2]
    find2<-df_unigram[grep(paste("^","\\b", clean_tail,"\\b", sep=""), df_unigram$Word),]
    if (nrow(find2)==0){
      warning("There is no prediction")
      return()
    }
    find2<-data.frame(find2)
    
    for (i in 1: nrow(find)) {
      new_find<-data.frame("pred"=find[[i,3]], "pred_prob"=find[[i,2]]/find2[[1,2]])
      merge<-rbind(merge, data.frame(new_find))
    }
    
    high_prob<-merge[order(-merge$pred_prob),]
    high_prob<-head(high_prob,10)
    possible_term<-as.character(high_prob$pred)
    return(possible_term)
  }}



predict_trigram<-function(input){
  if (sapply(gregexpr("\\S+", input), length)<=1){
    return("please enter more than one word")
  }else {
    input <- enc2native(input)
    input1<-tolower(input)
    input2<-str_replace_all(input1, pattern="[[:punct:]]","") 
    input3<-str_replace_all(input2, pattern="\\s+", " ") 
    input_clean<-removeNumbers(input3) 
    clean_tail<-tail(unlist(strsplit(input_clean, " ")), 2)
    words<-paste(clean_tail[1], clean_tail[2], sep=" ")
    start_with_term<-paste("^","\\b", words,"\\b", sep="")
    find<-df_trigram[grep(start_with_term, df_trigram$Word),]
    if (nrow(find)==0) {
      clean_tail2<-tail(unlist(strsplit(input_clean, " ")), 1)
      start_with_term2<-paste("^","\\b", clean_tail2,"\\b", sep="")
      find2<-df_bigram[grep(start_with_term2, df_bigram$Word),] 
      
      if (nrow(find2)==0){
        uni<-df_unigram[sample(nrow(df_unigram),nrow(df_unigram)/1000,replace=FALSE,prob=NULL),]
        uni<-head(uni)
        term<-as.character(uni$Word)
        return(term)
      }
      find2<-head(find2,10)
      find2$Word<-as.character(find2$Word)
      find_mat<-matrix(unlist(strsplit(find2$Word, " ")), ncol=2, byrow=TRUE)
      find2$pred<-find_mat[,2]
      terms<-as.character(find2$pred)
      return(terms)
      
    }
    merge<-NULL
    find$Word<-as.character(find$Word)
    find_mat<-matrix(unlist(strsplit(find$Word, " ")), ncol=3, byrow=TRUE)
    find$pred<-find_mat[,3]
    clean_tail3<-tail(unlist(strsplit(input_clean, " ")), 2)
    words2<-paste(clean_tail3[1], clean_tail3[2], sep=" ")
    find3<-df_bigram[grep(paste("^","\\b",words2,"\\b", "$",sep=""), df_bigram$Word),]
    
    for (i in 1: nrow(find)) {
      new_find<-data.frame("pred"=find[[i,3]], "pred_prob"=find[[i,2]]/find3[[1,2]])
      merge<-rbind(merge, data.frame(new_find))
    }
    high_prob<-merge[order(-merge$pred_prob),]
    high_prob<-head(high_prob,10)
    possible_term<-as.character(high_prob$pred)
    return(possible_term)
  }}



predict_quagram<-function(input){
  if (sapply(gregexpr("\\S+", input), length)<=1){
    return("please enter more than one word")
  }else {
    input <- enc2native(input)
    input1<-tolower(input)
    input2<-str_replace_all(input1, pattern="[[:punct:]]","") 
    input3<-str_replace_all(input2, pattern="\\s+", " ") 
    input_clean<-removeNumbers(input3) 
    clean_tail<-tail(unlist(strsplit(input_clean, " ")), 3)
    words<-paste(clean_tail[1], clean_tail[2],clean_tail[3], sep=" ")
    start_with_term<-paste("^","\\b",words,"\\b", sep="")
    find<-df_quagram[grep(start_with_term, df_quagram$Word),]
    if (nrow(find)==0) {
      clean_tail2<-tail(unlist(strsplit(input_clean, " ")), 2)
      words2<-paste(clean_tail2[1], clean_tail2[2], sep=" ")
      start_with_term2<-paste("^","\\b", words2,"\\b", sep="")
      find2<-df_trigram[grep(start_with_term2, df_trigram$Word),]
      
      if(nrow(find2)==0){
        clean_tail3<-tail(unlist(strsplit(input_clean, " ")), 1)
        start_with_term3<-paste("^","\\b", clean_tail3,"\\b", sep="")
        find3<-df_bigram[grep(start_with_term3, df_bigram$Word),] 
        
        if(nrow(find3)==0){
          uni<-df_unigram[sample(nrow(df_unigram),nrow(df_unigram)/1000,replace=FALSE,prob=NULL),]
          uni<-head(uni)
          single<-as.character(uni$Word)
          return(single)
        }
        find3<-head(find3,10)
        find3$Word<-as.character(find3$Word)
        find_mat3<-matrix(unlist(strsplit(find3$Word, " ")), ncol=2, byrow=TRUE)
        find3$pred<-find_mat3[,2]
        terms3<-as.character(find3$pred)
        return(terms3)
        
      }
      find2<-head(find2,10)
      find2$Word<-as.character(find2$Word)
      find_mat2<-matrix(unlist(strsplit(find2$Word, " ")), ncol=3, byrow=TRUE)
      find2$pred<-find_mat2[,3]
      term2<-as.character(find2$pred)
      return(term2)
    }
    
    merge<-NULL
    find$Word<-as.character(find$Word)
    find_mat<-matrix(unlist(strsplit(find$Word, " ")), ncol=4, byrow=TRUE)
    find$pred<-find_mat[,4]
    find4<-df_trigram[grep(paste("^","\\b", words,"\\b", "$",sep=""), df_trigram$Word),]
    
    
    for (i in 1: nrow(find)) {
      new_find<-data.frame("pred"=find[[i,3]], "pred_prob"=find[[i,2]]/find4[[1,2]])
      merge<-rbind(merge, data.frame(new_find))
    }
    high_prob<-merge[order(-merge$pred_prob),]
    high_prob<-head(high_prob,10)
    possible_term<-as.character(high_prob$pred)
    return(possible_term)
  }}


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
server.R

library(shiny)

#create a funtion that can use to identify radiobutton while do the prediction: 
nextword<-function(radiobutton, inputText,input){
  if (input$radiobutton=="bi"){
    predict_bigram(input$text)
  }else if(input$radiobutton=="tri"){
    predict_trigram(input$text)
  }else if(input$radiobutton=="qua"){
    predict_quagram(input$text)
  }
}

#Server-side application
shinyServer(function(input, output,session) {
  inputText<-isolate({input$text})
  radiobutton<-reactive({input$radiobutton})
  observeEvent(input$do, {
    output$textout<-renderTable({
    predictions<-nextword(radiobutton(),inputText,input)
    df<-data.frame(predictions)
    msg<-NULL
    for (i in 1:nrow(df)) {
      prediction<- df[i,1]
      msg<-rbind(msg, as.data.frame(prediction))
    }
    return(msg)
  }, quoted=FALSE)
  withProgress(message = 'Loading...', value = NULL, {
    Sys.sleep(.25)
    dat <- data.frame(x = numeric(0), y = numeric(0))
    withProgress(message = 'App is loading, please be patient', detail = "part 0", value = 0, {
      for (i in 1:10) {
        dat <- rbind(dat, data.frame(x = rnorm(1), y = rnorm(1)))
        incProgress(0.1, detail = paste(":", i*10,"%"))
        Sys.sleep(0.5)
      }
    })
    incProgress(0.5)
  })
  })
}) 
  

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ui.R

library(shiny)
library(shinythemes)

shinyUI(fluidPage(theme = shinytheme("cerulean"),
                  titlePanel("Text Prediction Prototype App"),
  navbarPage("",inverse = TRUE, collapsible = TRUE,
             tabPanel("N-gram prediction",
                      sidebarLayout(
                        sidebarPanel(
                          actionButton("do", "Load Application", icon("paper-plane"), 
                                       style="color: #2e6da4; background-color: #337ab7; border-color: #2e6da4"),
                          hr(),
                          textInput("text", label=h3("Text input:"),value = ""),
                          radioButtons("radiobutton", label=h3("N-Gram Predictor:"),
                                       choice=list( "Quadgram"="qua",
                                                   "Trigram"="tri",
                                                   "Bigram"="bi")),
                          br(),
                          helpText(h5("Instructions:")),
                          helpText("1. Load the application"),
                          helpText("2. Input a sentence"),
                          helpText("3. Select n-gram method"),
                          helpText("4. Wait for your results, they will show up on the right of your screen."),
                          helpText("5. Repeat steps 2-4"),
                          hr(),
                          helpText(h5("Examples:")),
                          helpText("Do you want some water"),
                          helpText("Are they going"),
                          helpText("Happy")
                        ),
                        
                        mainPanel(column(5,
                                 h2("Top recommendations:"),
                                 hr(),
                                 tableOutput("textout"),
                                 hr()
                                 
                                 
                          )
                        )
                      )
             ),
             
             tabPanel("About this app",
                      sidebarLayout(
                        sidebarPanel(
                          helpText(h5("Table of Contents")),
                          hr(),
                          helpText("1. NLP Background Info"),
                          helpText("2. Auto-fill Algorithm "),
                          helpText("3. References"),
                          hr(),
                          hr()
                          ),
                        mainPanel(
                          tabsetPanel(
                            tabPanel("Background",
                                     h3("What is NLP?"),
                                     hr(),
                                     p("Natural language processing (NLP) is a field of computer science, artificial intelligence, and computational linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human-computer interaction. Many challenges in NLP involve: natural language understanding, enabling computers to derive meaning from human or natural language input; and others involve natural language generation."),
                                     p(a("https://en.wikipedia.org/wiki/Natural_language_processing", href="https://en.wikipedia.org/wiki/Natural_language_processing"), ""),
                                     
                                     h3("What is n-gram?"),
                                     hr(),
                                     p("In the fields of computational linguistics and probability, an n-gram is a contiguous sequence of n items from a given sequence of text or speech. The items can be phonemes, syllables, letters, words or base pairs according to the application. The n-grams typically are collected from a text or speech corpus. When the items are words, n-grams may also be called shingles."),
                                     p(a("https://en.wikipedia.org/wiki/N-gram", href="https://en.wikipedia.org/wiki/N-gram"), ""),
                                     h3("Types of n-grams:"),
                                     hr(),
                                     p("An n-gram of size 1 is referred to as a", code("unigram"), "; size 2 is a", code("bigram"),"; size 3 is a", code("trigram"), ". Larger sizes are sometimes referred to by the value of n, e.g., ", code("four-gram"), code("five-gram"),  "and so on."),
                                     p(a("https://en.wikipedia.org/wiki/N-gram", href="https://en.wikipedia.org/wiki/N-gram"), ""),
                                     h3("Katz's back-off model is one model used to perform text prediction."),
                                     hr(),
                                     p("Katz back-off is a generative n-gram language model that estimates the conditional probability of a word given its history in the n-gram. It accomplishes this estimation by backing-off to models with smaller histories under certain conditions. By doing so, the model with the most reliable information about a given history is used to provide the better results."),
                                     p(a("https://en.wikipedia.org/wiki/Katz%27s_back-off_model", href="https://en.wikipedia.org/wiki/Katz%27s_back-off_model"), "")
                                     ),
                            tabPanel("Algorithm",
                                     h3("How this app was constructed"),
                                     hr(),
                                     p("At the foundation of this app is 1.0 GB of text that was compiled from US News, Blogs and Twitter. The text from these sources were went through standard NLP based text preparation methods: lower casing, punctuation removal, etc."),
                                     p("From there, the text was tokenized where each word is is formed into a token. The app utilizes four tokenizers. (unigram, bigram, trigram, and quadgram). In order to complete this work within the project timeline, a Microsoft Azure VM (4-core Xeon processor w/ 28GB of Ram) was secured and the program was run using the R parallel processing package."),
                                     p("Cnce the tokenizers completed processing, four tables were created with the token and frequency (count) for each row of data (each data)."),
                                     p("Finally, three prediction models were created. These models use the last two, three or four words at the end of the partial sentence entered by the user to predict the next word in the sentence"),
                                     hr(),
                                     h3("The model used in this app for word prediction"),
                                     hr(),
                                     p("Based on the program presented in The Practice of Programming (Kernighan and Pike, Addison-Wesley 1999).
                                       A Markov chain algorithm generates text by creating a statistical model of
                                       potential textual suffixes for a given prefix."),
                                     p(a("https://golang.org/doc/codewalk/markov/", href="https://golang.org/doc/codewalk/markov/"), ""),
                                     hr(),
                                     h3("A great visual representation of Markov Model's can be found here:"),
                                     hr(),
                                     p(a("http://setosa.io/ev/markov-chains/", href="http://setosa.io/ev/markov-chains/"), "")
                                     ),
                            tabPanel("References",
                                     h3("Reference links"),
                                     hr(),
                                     p(a("About the app creator", href="https://www.linkedin.com/in/jpolanco"), ""),
                                     p(a("App Milestone Report", href="http://www.rpubs.com/joelpolanco/177524"), ""),
                                     p(a("Handling and processing strings in R", href="http://gastonsanchez.com/Handling_and_Processing_Strings_in_R.pdf"), ""),
                                     p(a("CRAN NLP", href="http://cran.r-project.org/web/views/NaturalLanguageProcessing.html"), ""),
                                     p(a("About Markov Models", href="https://golang.org/doc/codewalk/markov/"), "")
                            )
                                     )
                          ))))))

--
You received this message because you are subscribed to the Google Groups "ShinyApps Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shinyapps-use...@googlegroups.com.
To post to this group, send email to shinyap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/shinyapps-users/8c5ea132-425f-4756-829c-fdd0580abf7a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Joel Polanco

unread,
May 29, 2016, 3:25:15 PM5/29/16
to ShinyApps Users, jpo...@gmail.com
Hi Tareef,

I just upgraded to the basic plan and increased the app size. The app is now running!

Thank You.

-JP
To unsubscribe from this group and stop receiving emails from it, send an email to shinyapps-users+unsubscribe@googlegroups.com.
To post to this group, send email to shinyapps-users@googlegroups.com.

Tareef Kawaf

unread,
May 29, 2016, 8:40:52 PM5/29/16
to Joel Polanco, ShinyApps Users
I am glad.  We hope to provide a better error message in the future.  
To unsubscribe from this group and stop receiving emails from it, send an email to shinyapps-use...@googlegroups.com.
To post to this group, send email to shinyap...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "ShinyApps Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shinyapps-use...@googlegroups.com.
To post to this group, send email to shinyap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/shinyapps-users/a1a377da-ed66-47c9-98c5-70f114bcbf1f%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages