Hello,
Help Needed! I've been troubleshooting all day. My app loads on the server and I am able to successfully load the application via my load button. Upon inputting a partial sentence into the text box the app crashes ... I get a "Disconnected from Server" error. The logs are no help.
File Size of dataframes being searched total ~351MB
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Logs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2016-05-28T23:18:19.890185+00:00 shinyapps[104965]: jsonlite version: 0.9.19
2016-05-28T23:18:19.890186+00:00 shinyapps[104965]: RJSONIO version: NA
2016-05-28T23:18:19.890195+00:00 shinyapps[104965]: htmltools version: 0.3.5
2016-05-28T23:18:20.075257+00:00 shinyapps[104965]: Using jsonlite for JSON processing
2016-05-28T23:18:20.081011+00:00 shinyapps[104965]:
2016-05-28T23:18:20.081013+00:00 shinyapps[104965]: Starting R with process ID: '12'
2016-05-28T23:18:20.103541+00:00 shinyapps[104965]: Loading required package: NLP
2016-05-28T23:18:49.178541+00:00 shinyapps[104965]:
2016-05-28T23:18:49.178544+00:00 shinyapps[104965]: Listening on http://0.0.0.0:41390
2016-05-28T23:19:25.455272+00:00 shinyapps[104965]: Server version: 0.4.3.1780
2016-05-28T23:19:25.455365+00:00 shinyapps[104965]: R version: 3.3.0
2016-05-28T23:19:25.455366+00:00 shinyapps[104965]: shiny version: 0.13.2
2016-05-28T23:19:25.455367+00:00 shinyapps[104965]: rmarkdown version: NA
2016-05-28T23:19:25.455388+00:00 shinyapps[104965]: knitr version: NA
2016-05-28T23:19:25.455389+00:00 shinyapps[104965]: jsonlite version: 0.9.19
2016-05-28T23:19:25.455391+00:00 shinyapps[104965]: htmltools version: 0.3.5
2016-05-28T23:19:25.623323+00:00 shinyapps[104965]: Starting R with process ID: '26'
2016-05-28T23:19:25.645527+00:00 shinyapps[104965]: Loading required package: NLP
2016-05-28T23:19:25.455390+00:00 shinyapps[104965]: RJSONIO version: NA
2016-05-28T23:19:25.617358+00:00 shinyapps[104965]: Using jsonlite for JSON processing
2016-05-28T23:19:25.623321+00:00 shinyapps[104965]:
2016-05-28T23:19:54.687756+00:00 shinyapps[104965]:
2016-05-28T23:19:54.687759+00:00 shinyapps[104965]: Listening on http://0.0.0.0:59129
2016-05-28T23:23:08.085859+00:00 shinyapps[104965]: Server version: 0.4.3.1780
2016-05-28T23:23:08.259866+00:00 shinyapps[104965]: Using jsonlite for JSON processing
2016-05-28T23:23:08.085933+00:00 shinyapps[104965]: R version: 3.3.0
2016-05-28T23:23:08.085944+00:00 shinyapps[104965]: shiny version: 0.13.2
2016-05-28T23:23:08.085945+00:00 shinyapps[104965]: rmarkdown version: NA
2016-05-28T23:23:08.266108+00:00 shinyapps[104965]: Starting R with process ID: '37'
2016-05-28T23:23:08.085946+00:00 shinyapps[104965]: knitr version: NA
2016-05-28T23:23:08.085948+00:00 shinyapps[104965]: jsonlite version: 0.9.19
2016-05-28T23:23:08.085978+00:00 shinyapps[104965]: RJSONIO version: NA
2016-05-28T23:23:08.086004+00:00 shinyapps[104965]: htmltools version: 0.3.5
2016-05-28T23:23:08.266106+00:00 shinyapps[104965]:
2016-05-28T23:23:08.288323+00:00 shinyapps[104965]: Loading required package: NLP
2016-05-28T23:23:37.277941+00:00 shinyapps[104965]:
2016-05-28T23:23:37.277944+00:00 shinyapps[104965]: Listening on http://0.0.0.0:39423
2016-05-28T23:25:47.044667+00:00 shinyapps[104965]: Server version: 0.4.3.1780
2016-05-28T23:25:47.044745+00:00 shinyapps[104965]: R version: 3.3.0
2016-05-28T23:25:47.044747+00:00 shinyapps[104965]: shiny version: 0.13.2
2016-05-28T23:25:47.044757+00:00 shinyapps[104965]: rmarkdown version: NA
2016-05-28T23:25:47.044759+00:00 shinyapps[104965]: knitr version: NA
2016-05-28T23:25:47.044760+00:00 shinyapps[104965]: jsonlite version: 0.9.19
2016-05-28T23:25:47.044761+00:00 shinyapps[104965]: RJSONIO version: NA
2016-05-28T23:25:47.044790+00:00 shinyapps[104965]: htmltools version: 0.3.5
2016-05-28T23:25:47.208398+00:00 shinyapps[104965]: Using jsonlite for JSON processing
2016-05-28T23:25:47.213754+00:00 shinyapps[104965]:
2016-05-28T23:25:47.213757+00:00 shinyapps[104965]: Starting R with process ID: '47'
2016-05-28T23:25:47.235189+00:00 shinyapps[104965]: Loading required package: NLP
2016-05-28T23:26:16.163818+00:00 shinyapps[104965]:
2016-05-28T23:26:16.163822+00:00 shinyapps[104965]: Listening on http://0.0.0.0:54120
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
global.R
#load packages
library(tm)
library(NLP)
library(stringr)
#read in data files:
load('pred_app.RData')
# add prediction function : (bigram, trigram, and quadgram)
predict_bigram<- function(input) {
if (sapply(gregexpr("\\S+", input), length)<=1){
return("please enter more than one word")
}else {
input <- enc2native(input)
input1<-tolower(input)
input2<-str_replace_all(input1, pattern="[[:punct:]]","") #remove punctuations
input3<-str_replace_all(input2, pattern="\\s+", " ") # replace whitespace with space
input_clean<-removeNumbers(input3)
clean_tail<-tail(unlist(strsplit(input_clean, " ")), 1)
start_with_term<-paste("^","\\b", clean_tail,"\\b", sep="")
find<-df_bigram[grep(start_with_term, df_bigram$Word),]
if (nrow(find)==0) {
uni<-df_unigram[sample(nrow(df_unigram),nrow(df_unigram)/1000,replace=TRUE,prob=NULL),]
uni<-head(uni)
term<-as.character(uni$Word)
return(term)
}
find<-df_bigram[grep(start_with_term, df_bigram$Word),]
merge<-NULL
find$Word<-as.character(find$Word)
find_mat<-matrix(unlist(strsplit(find$Word, " ")), ncol=2, byrow=TRUE)
find$pred<-find_mat[,2]
find2<-df_unigram[grep(paste("^","\\b", clean_tail,"\\b", sep=""), df_unigram$Word),]
if (nrow(find2)==0){
warning("There is no prediction")
return()
}
find2<-data.frame(find2)
for (i in 1: nrow(find)) {
new_find<-data.frame("pred"=find[[i,3]], "pred_prob"=find[[i,2]]/find2[[1,2]])
merge<-rbind(merge, data.frame(new_find))
}
high_prob<-merge[order(-merge$pred_prob),]
high_prob<-head(high_prob,10)
possible_term<-as.character(high_prob$pred)
return(possible_term)
}}
predict_trigram<-function(input){
if (sapply(gregexpr("\\S+", input), length)<=1){
return("please enter more than one word")
}else {
input <- enc2native(input)
input1<-tolower(input)
input2<-str_replace_all(input1, pattern="[[:punct:]]","")
input3<-str_replace_all(input2, pattern="\\s+", " ")
input_clean<-removeNumbers(input3)
clean_tail<-tail(unlist(strsplit(input_clean, " ")), 2)
words<-paste(clean_tail[1], clean_tail[2], sep=" ")
start_with_term<-paste("^","\\b", words,"\\b", sep="")
find<-df_trigram[grep(start_with_term, df_trigram$Word),]
if (nrow(find)==0) {
clean_tail2<-tail(unlist(strsplit(input_clean, " ")), 1)
start_with_term2<-paste("^","\\b", clean_tail2,"\\b", sep="")
find2<-df_bigram[grep(start_with_term2, df_bigram$Word),]
if (nrow(find2)==0){
uni<-df_unigram[sample(nrow(df_unigram),nrow(df_unigram)/1000,replace=FALSE,prob=NULL),]
uni<-head(uni)
term<-as.character(uni$Word)
return(term)
}
find2<-head(find2,10)
find2$Word<-as.character(find2$Word)
find_mat<-matrix(unlist(strsplit(find2$Word, " ")), ncol=2, byrow=TRUE)
find2$pred<-find_mat[,2]
terms<-as.character(find2$pred)
return(terms)
}
merge<-NULL
find$Word<-as.character(find$Word)
find_mat<-matrix(unlist(strsplit(find$Word, " ")), ncol=3, byrow=TRUE)
find$pred<-find_mat[,3]
clean_tail3<-tail(unlist(strsplit(input_clean, " ")), 2)
words2<-paste(clean_tail3[1], clean_tail3[2], sep=" ")
find3<-df_bigram[grep(paste("^","\\b",words2,"\\b", "$",sep=""), df_bigram$Word),]
for (i in 1: nrow(find)) {
new_find<-data.frame("pred"=find[[i,3]], "pred_prob"=find[[i,2]]/find3[[1,2]])
merge<-rbind(merge, data.frame(new_find))
}
high_prob<-merge[order(-merge$pred_prob),]
high_prob<-head(high_prob,10)
possible_term<-as.character(high_prob$pred)
return(possible_term)
}}
predict_quagram<-function(input){
if (sapply(gregexpr("\\S+", input), length)<=1){
return("please enter more than one word")
}else {
input <- enc2native(input)
input1<-tolower(input)
input2<-str_replace_all(input1, pattern="[[:punct:]]","")
input3<-str_replace_all(input2, pattern="\\s+", " ")
input_clean<-removeNumbers(input3)
clean_tail<-tail(unlist(strsplit(input_clean, " ")), 3)
words<-paste(clean_tail[1], clean_tail[2],clean_tail[3], sep=" ")
start_with_term<-paste("^","\\b",words,"\\b", sep="")
find<-df_quagram[grep(start_with_term, df_quagram$Word),]
if (nrow(find)==0) {
clean_tail2<-tail(unlist(strsplit(input_clean, " ")), 2)
words2<-paste(clean_tail2[1], clean_tail2[2], sep=" ")
start_with_term2<-paste("^","\\b", words2,"\\b", sep="")
find2<-df_trigram[grep(start_with_term2, df_trigram$Word),]
if(nrow(find2)==0){
clean_tail3<-tail(unlist(strsplit(input_clean, " ")), 1)
start_with_term3<-paste("^","\\b", clean_tail3,"\\b", sep="")
find3<-df_bigram[grep(start_with_term3, df_bigram$Word),]
if(nrow(find3)==0){
uni<-df_unigram[sample(nrow(df_unigram),nrow(df_unigram)/1000,replace=FALSE,prob=NULL),]
uni<-head(uni)
single<-as.character(uni$Word)
return(single)
}
find3<-head(find3,10)
find3$Word<-as.character(find3$Word)
find_mat3<-matrix(unlist(strsplit(find3$Word, " ")), ncol=2, byrow=TRUE)
find3$pred<-find_mat3[,2]
terms3<-as.character(find3$pred)
return(terms3)
}
find2<-head(find2,10)
find2$Word<-as.character(find2$Word)
find_mat2<-matrix(unlist(strsplit(find2$Word, " ")), ncol=3, byrow=TRUE)
find2$pred<-find_mat2[,3]
term2<-as.character(find2$pred)
return(term2)
}
merge<-NULL
find$Word<-as.character(find$Word)
find_mat<-matrix(unlist(strsplit(find$Word, " ")), ncol=4, byrow=TRUE)
find$pred<-find_mat[,4]
find4<-df_trigram[grep(paste("^","\\b", words,"\\b", "$",sep=""), df_trigram$Word),]
for (i in 1: nrow(find)) {
new_find<-data.frame("pred"=find[[i,3]], "pred_prob"=find[[i,2]]/find4[[1,2]])
merge<-rbind(merge, data.frame(new_find))
}
high_prob<-merge[order(-merge$pred_prob),]
high_prob<-head(high_prob,10)
possible_term<-as.character(high_prob$pred)
return(possible_term)
}}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
server.R
library(shiny)
#create a funtion that can use to identify radiobutton while do the prediction:
nextword<-function(radiobutton, inputText,input){
if (input$radiobutton=="bi"){
predict_bigram(input$text)
}else if(input$radiobutton=="tri"){
predict_trigram(input$text)
}else if(input$radiobutton=="qua"){
predict_quagram(input$text)
}
}
#Server-side application
shinyServer(function(input, output,session) {
inputText<-isolate({input$text})
radiobutton<-reactive({input$radiobutton})
observeEvent(input$do, {
output$textout<-renderTable({
predictions<-nextword(radiobutton(),inputText,input)
df<-data.frame(predictions)
msg<-NULL
for (i in 1:nrow(df)) {
prediction<- df[i,1]
msg<-rbind(msg, as.data.frame(prediction))
}
return(msg)
}, quoted=FALSE)
withProgress(message = 'Loading...', value = NULL, {
Sys.sleep(.25)
dat <- data.frame(x = numeric(0), y = numeric(0))
withProgress(message = 'App is loading, please be patient', detail = "part 0", value = 0, {
for (i in 1:10) {
dat <- rbind(dat, data.frame(x = rnorm(1), y = rnorm(1)))
incProgress(0.1, detail = paste(":", i*10,"%"))
Sys.sleep(0.5)
}
})
incProgress(0.5)
})
})
})
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ui.R
library(shiny)
library(shinythemes)
shinyUI(fluidPage(theme = shinytheme("cerulean"),
titlePanel("Text Prediction Prototype App"),
navbarPage("",inverse = TRUE, collapsible = TRUE,
tabPanel("N-gram prediction",
sidebarLayout(
sidebarPanel(
actionButton("do", "Load Application", icon("paper-plane"),
style="color: #2e6da4; background-color: #337ab7; border-color: #2e6da4"),
hr(),
textInput("text", label=h3("Text input:"),value = ""),
radioButtons("radiobutton", label=h3("N-Gram Predictor:"),
choice=list( "Quadgram"="qua",
"Trigram"="tri",
"Bigram"="bi")),
br(),
helpText(h5("Instructions:")),
helpText("1. Load the application"),
helpText("2. Input a sentence"),
helpText("3. Select n-gram method"),
helpText("4. Wait for your results, they will show up on the right of your screen."),
helpText("5. Repeat steps 2-4"),
hr(),
helpText(h5("Examples:")),
helpText("Do you want some water"),
helpText("Are they going"),
helpText("Happy")
),
mainPanel(column(5,
h2("Top recommendations:"),
hr(),
tableOutput("textout"),
hr()
)
)
)
),
tabPanel("About this app",
sidebarLayout(
sidebarPanel(
helpText(h5("Table of Contents")),
hr(),
helpText("1. NLP Background Info"),
helpText("2. Auto-fill Algorithm "),
helpText("3. References"),
hr(),
hr()
),
mainPanel(
tabsetPanel(
tabPanel("Background",
h3("What is NLP?"),
hr(),
p("Natural language processing (NLP) is a field of computer science, artificial intelligence, and computational linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human-computer interaction. Many challenges in NLP involve: natural language understanding, enabling computers to derive meaning from human or natural language input; and others involve natural language generation."),
h3("What is n-gram?"),
hr(),
p("In the fields of computational linguistics and probability, an n-gram is a contiguous sequence of n items from a given sequence of text or speech. The items can be phonemes, syllables, letters, words or base pairs according to the application. The n-grams typically are collected from a text or speech corpus. When the items are words, n-grams may also be called shingles."),
h3("Types of n-grams:"),
hr(),
p("An n-gram of size 1 is referred to as a", code("unigram"), "; size 2 is a", code("bigram"),"; size 3 is a", code("trigram"), ". Larger sizes are sometimes referred to by the value of n, e.g., ", code("four-gram"), code("five-gram"), "and so on."),
h3("Katz's back-off model is one model used to perform text prediction."),
hr(),
p("Katz back-off is a generative n-gram language model that estimates the conditional probability of a word given its history in the n-gram. It accomplishes this estimation by backing-off to models with smaller histories under certain conditions. By doing so, the model with the most reliable information about a given history is used to provide the better results."),
),
tabPanel("Algorithm",
h3("How this app was constructed"),
hr(),
p("At the foundation of this app is 1.0 GB of text that was compiled from US News, Blogs and Twitter. The text from these sources were went through standard NLP based text preparation methods: lower casing, punctuation removal, etc."),
p("From there, the text was tokenized where each word is is formed into a token. The app utilizes four tokenizers. (unigram, bigram, trigram, and quadgram). In order to complete this work within the project timeline, a Microsoft Azure VM (4-core Xeon processor w/ 28GB of Ram) was secured and the program was run using the R parallel processing package."),
p("Cnce the tokenizers completed processing, four tables were created with the token and frequency (count) for each row of data (each data)."),
p("Finally, three prediction models were created. These models use the last two, three or four words at the end of the partial sentence entered by the user to predict the next word in the sentence"),
hr(),
h3("The model used in this app for word prediction"),
hr(),
p("Based on the program presented in The Practice of Programming (Kernighan and Pike, Addison-Wesley 1999).
A Markov chain algorithm generates text by creating a statistical model of
potential textual suffixes for a given prefix."),
hr(),
h3("A great visual representation of Markov Model's can be found here:"),
hr(),
),
tabPanel("References",
h3("Reference links"),
hr(),
)
)
))))))