Invalid multibyte string error, as soon as App deployed

[Ali Kazmi]

Hi,

I am getting the following error as soon as I deploy my app:

'invalid multibyte string at '<97> as t'

So nothing happens when I launch my app, except that this error message is displayed. I researched about the error on web to resolve it on my own, but did not succeed.

I find it strange that I obtain this error without uploading any text file to my app (the purpose of my app is to perform basic analysis of text, so it contains textConnection and scan functions). I have also visually inspected my code for any strange looking characters, but there are none.

Any advice please? Thanks in advance!

[Andy Kipp]

Can you please send the full output from deplyoApp()?

Thanks

-Andy

--
You received this message because you are subscribed to the Google Groups "ShinyApps Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shinyapps-use...@googlegroups.com.
To post to this group, send email to shinyap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/shinyapps-users/da4504b8-f7ff-4453-9d96-79f7ada0cfd7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ali Kazmi]

Andy,

Thanks for getting back, again.

Apologies for not posting the output.

Preparing to deploy application...DONE
Uploading application bundle...DONE
Deploying application: 15099...
Waiting for task: 556377
  building: Building image: 29964
  building: Building package: gsl (1.9-10)
  building: Building package: stringr (0.6.2)
  building: Building package: xtable (1.7-3)
  building: Building package: fastcluster (1.1.13)
  building: Building package: RColorBrewer (1.0-5)
  building: Building package: proto (0.3-10)
  building: Building package: dichromat (2.0-0)
  building: Building package: caTools (1.17)
  building: Building package: digest (0.6.4)
  building: Building package: slam (0.1-32)
  building: Building package: NLP (0.1-3)
  building: Building package: htmltools (0.2.4)
  building: Building package: gtable (0.1.2)
  building: Building package: MASS (7.3-31)
  building: Building package: lattice (0.20-29)
  building: Building package: Matrix (1.1-3)
  building: Building package: colorspace (1.2-4)
  building: Building package: munsell (0.4.2)
  building: Building package: RCurl (1.95-4.1)
  building: Building package: modeltools (0.2-21)
  building: Building package: RJSONIO (1.2-0.2)
  building: Building package: igraph (0.7.1)
  building: Building package: Rcpp (0.11.2)
  building: Building package: plyr (1.8.1)
  building: Building package: reshape2 (1.4)
  building: Building package: scales (0.2.4)
  building: Building package: ggplot2 (1.0.0)
  building: Building package: httpuv (1.3.0)
  building: Building package: shiny (0.10.0)
  building: Building package: shinyIncubator (0.2.1)
  building: Building package: rJava (0.9-6)
  building: Building package: RWekajars (3.7.11-1)
  building: Building package: RWeka (0.4-23)
  building: Building package: tm (0.6)
  building: Building package: topicmodels (0.2-1)
  building: Pushing image: 29964
  deploying: Deploying 1 instance(s)
  rollforward: Activating new instances
  success: Removing old instances
Application successfully deployed to http://kazmialiarsalan.shinyapps.io/QualitativeAnalysisApp-R_v5

So, the deployment is successful, but when the app is launched, it reports the aforesaid error.

This can be checked from this link: http://kazmialiarsalan.shinyapps.io/QualitativeAnalysisApp-R_v5

[Ali Kazmi]

Andy,

Whilst I await the diagnosis of the problem, thank you for your time on this, and if you would require anything additional from my side, please let me know.

Thanks.

[Andy Kipp]

Ali,

Did you see my note about making sure your source and data files are UTF-8 encoded?

-Andy

To view this discussion on the web visit https://groups.google.com/d/msgid/shinyapps-users/e60fc7f2-a30d-4f79-b0ec-c189dd31a05d%40googlegroups.com.

[Ali Kazmi]

Yes, Andy.

Following your note, I used the encoding argument for certain functions I was using in my app (scan, textConnection) but it still produced the same error.

In my amateurish attempt to detect the source of error, I commented out all functions that would read text and re-deployed the app, but this did not resolve the error either.

If I run my app locally, it works fine...

[Andy Kipp]

Ali,

I have not looked at your source code extensively... but is it possible that the files referenced in https://github.com/noobuseR/Datasets are not UTF-8 encoded? 

-Andy

To view this discussion on the web visit https://groups.google.com/d/msgid/shinyapps-users/7fe2c265-bc9c-40cc-ad14-5eb61b097512%40googlegroups.com.

[Ali Kazmi]

Andy,

That is possible. But forgive me, I was under the impression that if we use the scan function in our App to read the files and use the fileEncoding() and encoding() options to specify "UTF-8" encoding, things should be fine. Is that not so?

And another reason I did not pay attention to the dataset files was because I thought if an error should occur, it must occur only upon reading the files, and not at the very start of the application. No?

Apologies, had I known answers to these questions, I may have saved you the headache of this problem :) But I will make the changes, see if I succeed, and report back to select your post as an answer.

Many thanks.

[Ali Kazmi]

Andy,

Thanks for bearing with me. I made the changes in the files referenced at the aforesaid URL to have them UTF-8 encoded. To make sure, then, I terminated my app and redeployed. The following are the functions I used to change the encoding to UTF-8:

This is the code that I use when reading in the referenced files:

txtFile<- scan(textConnection(myFile, encoding="UTF-8"),sep="\n",what="character",fileEncoding="UTF-8",encoding="UTF-8")
Encoding(txtFile)<- "UTF-8"

As I have options for users to enter text using keyboard too, I have used the following to make sure no encoding issues occur:

Encoding(inputOfAUser)<- "UTF-8"

Unfortunately this has no effect and the error still occurs:

ERROR: invalid multibyte string at '<97> as t'

[OL LQ]

Hi, this error stems mainly from the fact of not saving the scripts, or the files containing the R code, in UTF-8 encoding. This happens because usually this is not the default encoding of Windows Systems. Once you have saved the files in the wrong encoding, even after converting them to UTF-8 encoding, it is likely for the error you get to persist. This is because when you save the files in a different encoding, there may be characters that are not parsed properly, damaging the file. So, after converting your  files to UTF-8 encoding, you should look for strange characters, or characters that are not rendered properly, correct them, and everything should go fine. 

[Yihui Xie]

It seems you have taken down your app or moved it elsewhere? The URL https://kazmialiarsalan.shinyapps.io/QualitativeAnalysisApp-R_v5 returns the 404 error now. Can you show us the full source code of your app, including the files that you read into R? Thanks!

BTW, I do not quite understand why you had to use a complicated scan(textConnection(...)). It seems a simple readLines(..., encoding = "UTF-8") should just work.

Yihui

[Ali Kazmi]

OL LQ,

Thank you for your input.

I see. I have some files that I read into my App, depending upon user's selection. I had saved these files in UTF-8 encoding. And I don't think they are causing the problem, as a simpler version of my App is able to read the files properly. Given this, I did inspect my code files to see if they had any strange characters, but they too did not... So I'm confused as to what the source of error is...

Yihui,

Thank you for your contribution.
My apologies, I had taken down my app to try to find the source of error, but forgot to upload it back.

This is the link to my app: https://kazmialiarsalan.shinyapps.io/QualitativeAnalysisApp-R_v8/
The Server and UI files can be found at the links.
The text data sets that I am reading into my app can be found here.

An older, simpler version of my app that runs just fine can be found here. The difference between this and new app is that the older was constructed using older version of packages, and did not include the topicmodels package either. The older app was constructed a few months back.

Regarding the use of scan(textConnection()), I was using getURL() to read in raw text data from my GitHub, so once I had done
data<- getURL(...)
I couldn't use the readLines() function, and had to rely on a textConnection() first, after which scan() is possible.
I am amateurish with R :)

Thanks.

[Yihui Xie]

I opened one of the files that you tried to read using
RCurl::getURL():
https://raw.githubusercontent.com/noobuseR/Datasets/master/UAEexpatForum.txt,
and there were garbled characters in it. You can search for 0087 or <U
on that page (Ctrl + F). <U+xxxx> is a sign that your raw files were
not saved with the correct encoding (presumably UTF-8). However, I do
not think this is the reason for the error (<U+xxxx> is pure ASCII
after all). Since your app involves quite a few packages that are
unavailable in my system and server.R seems to be non-trivial for me
to read through, it may take a while for me to install all of them and
start diagnosing the problem. Before I really do that, could you try
this blind shot in the dark?

devtools::install_github("rstudio/shiny@v/0/10/1-rc")

Then re-deploy the app. We have tried hard to solve the Unicode
problem in shiny 0.10.1
(https://groups.google.com/forum/#!topic/shiny-discuss/kT4GAm_FAYU),
and I'm not sure if it will solve your problem by chance...

BTW, now I understand your rationale of scan(textConnection()), and I would use

data <- getURL(..., .encoding = 'UTF-8')
data <- unlist(strsplit(data, '\n'))

Meanwhile, I believe Encoding(x) <- 'UTF-8' should be very rarely
used, unless you are absolutely sure x is encoded in UTF-8 **and** R
marks its encoding as "unknown". In most cases, you should use iconv()
or enc2utf8() or enc2native() instead to do the conversion, instead of
forcibly assigning an encoding. When dealing with text processing, it
is best to convert non-native encodings to the system native encoding
in advance, and enc2native() can be very useful in this case. If you
do not live in the Windows world, you can simply ignore all I said
here :)

Regards,
Yihui
--
Yihui Xie <xiey...@gmail.com>
Web: http://yihui.name

[Ali Kazmi]

Yihui,

Many thanks for the tips! I included the .encoding argument for getURL() function and cut out the Encoding() function, as I am trapped in the world of Windows... I noticed now that even after using .encoding="UTF-8", the encoding is still 'unknown' for my text data. I am looking into this further to possibly save some headache for you and other members of ShinyApps team.

I also installed the latest version of shiny, but unfortunately it reports the same error:

ERROR: invalid multibyte string at '<97> as t'

Many thanks for the help so far! I'll keep trying myself too for any solutions...

[Yihui Xie]

Thanks for reporting back. Is it possible for you to make a minimal
reproducible example from this app? It will save us a lot of time from
understanding the irrelevant part. The part that is likely to be the
culprit is getURL(). If you do not want to do this, it is totally
fine. I'll jump in when I have got enough time. Thanks!

Regards,
Yihui
--
Yihui Xie <xiey...@gmail.com>
Web: http://yihui.name

[Ali Kazmi]

Sure, happy to be of any help, even if it be little. I will formulate minimal examples of my app to isolate the source of error (which, as you say, is seemingly getURL()) and report back.

[Ali Kazmi]

Hi Yihui,

I was able to spend some time on my App only recently, to figure out the source of error - 

invalid multibyte string at '<97> as t'

.

Before that, I downloaded the latest version of Shiny, as you suggested. However, it did not resolve the issue.

While trying to isolate the source of error, I found that getURL is most probably not behind the problem. The App reads in data properly. But there is another, new problem:

I am using a reactive input slider, for which I read in the corpus data from a reactive conductor, calculate the range for word count, and then generate a slider. The error I get is

no applicable method for 'meta' applied to an object of class "try-error"

It is strange, because the App works fine locally. To give an idea of how I am creating the reactive objects, the following is my code that gives an error from Cloud but works fine locally.

 preprocessedCorpus <- reactive ({
 if (input$preprocessBtn == 0)
 return()
 isolate ({
 originalCorpus <- initialCorpus()
 newCorpus <- cleanSweepCorpus(corpus=originalCorpus, useStopwords=input$stopwords, stem=input$stemming,
 removePunct=input$punctuation, removeNum=input$numbers,
 useSynonyms=input$customThes, initialWords=input$customThesInitial,
 replacementWords=input$customThesReplacement, useCustomStopwords=input$customStopword,
 customStopwords=input$cusStopwords)
 rm(originalCorpus)
 return(newCorpus)
 })
 })


 lowerFreqRangeClust <- reactive({
 corpus <- preprocessedCorpus()
 tdMat <- TermDocumentMatrix(corpus)
 rowTotals <- rowSums(as.matrix(tdMat))
 freqBounds <- range(rowTotals)
 rm(corpus,tdMat,rowTotals)
 return(freqBounds)
 })


 output$lowerFreqSliderClust <- renderUI({
 sliderInput("lowerFreqBoundClust", "Please set the Lower Bound for Word Frequency",
 min = round(lowerFreqRangeClust()[1]),max = round(lowerFreqRangeClust()[2]),
 value = round(lowerFreqRangeClust()[1]),step = NULL, ticks=TRUE)
 })

The CleanSweepCorpus() is a function that I have checked for any errors - it is fine. Perhaps I am setting the reactivity incorrectly?

The link to my App is: http://kazmialiarsalan.shinyapps.io/sampleApp2/

On Thursday, 24 July 2014 08:55:30 UTC+3, Yihui Xie wrote:

[Yihui Xie]

Okay, let's try to solve the new problem first. Every time you report
a problem, I'd appreciate it if you can point me to the source code as
well. I did not find sampleApp2 in your Github repo
https://github.com/noobuseR, although I managed to reproduce the error
in your app on ShinyApps.io.

After a blind google search, I saw this:
https://github.com/bstewart/stm/issues/1 It looks like it is still a
character encoding problem, but I cannot do anything without a minimal
reproducible example.

Regards,
Yihui
--
Yihui Xie <xiey...@gmail.com>
Web: http://yihui.name

[Ali Kazmi]

Oops, my sincere apologies for not sharing the code (and also for the short absence from this group because of which I couldn't respond to you timely.

I tried to identify the error on my own, hoping to prevent taking a lot of your time on this one. I think your guess was correct: the error is due to the stemDocument function in the tm package. When I use the function, no character vectors are returned at all. When I do not use it, everything is fine. I checked the SnowballC package, which seems to be the place from which tm package uses the stemmer.

It says: Elements of the vector are converted to UTF-8 encoding before the stemming is performed, and the returned elements are marked as such when they contain non-ASCII characters.

When I run the same function locally, however, things are fine. It is just a problem when I use runApp() or if I have deployed the application already.

Would you like to make any observations about this? 

Link to the app is: https://kazmialiarsalan.shinyapps.io/sampleApp/
Link to the code is: https://github.com/noobuseR/testApp

I am currently checking if the original problem (with the original app) is also similar.

Many thanks for your time.

[Ali Kazmi]

Just to clarify: when I wrote that your guess was probably correct, I meant to say that it does seem encoding is causing this problem.

[Amit Kohli]

My app: https://amit.shinyapps.io/WordCloudMaker/

My github: https://github.com/mexindian/Wordcloud.from.folder

Just to say I was also having a problem with my app working correctly locally but on shinyapps.io I would get the error:

no applicable method for 'meta' applied to an object of class "try-error"

After reading this discussion I removed the following from my server.R

if(input$stem) c <- tm_map(c, stemDocument)

and also on my ui.R where that could be selected. I was suspicious because this isn't called unless the checkbox is checked, and there was a problem as soon as I selected the file to upload... And indeed, this step was not sufficient, I still had the same error. 

The problem went away when I removed these two lines:

c <- tm_map(c, stripWhitespace)
c <- tm_map(c, removeNumbers)

Which were the two functions that were running immediately. So I think that rather than the problem being with the *stemming* function, the problem is with tm_map.

[Ali Kazmi]

Amit Kohli,

Thanks for sharing your solution. However, I think that tm_map itself may not be the problem, because in my app, the removeNumbers is not being used with the tm_map function, neither is the word stemming function. I do acknowledge that for stripping white spaces I used the tm_map function. This did not present any problem, still. And problem only occurs when I try to stem my words.

Link to the app is: https://kazmialiarsalan.shinyapps.io/sampleApp/
Link to the code is: https://github.com/noobuseR/testApp

Yihui,

In addition to the problem described above in a sample app (that stemming removes all character data - presumably this may be an encoding issue. But I do not know how to resolve this...), I have managed to create another short version of the app that produces the same error as faced originally:

This is the link to this new app: https://kazmialiarsalan.shinyapps.io/test/
This is the link to my code: https://github.com/noobuseR/testAppNew

We may focus on this original problem - I shared the other problem because 1. I think the two problems are related, and 2. I am hoping that it may give you some insight into the source of this error.

HTH

[Yihui Xie]

Hi Ali,

Thanks a lot for simplifying your app and showing the source code!
That is a great step forward. However, when I tried to run your app, I
got all sorts of problems, such as the missing introduction.html in
includeHTML("introduction.html"). Could you please further simplify it
to a minimal, self-contained, and reproducible example? There are
quite a few things that I think can be removed, such as the progress
bar from shinyIncubator. Are all of tm, fastcluster, gsl, and
topicmodels required to reproduce the problem? If not, please remove
as many add-on packages as you can. Similarly, are all UI components
absolutely needed?...

Regards,
Yihui
--
Yihui Xie <xiey...@gmail.com>
Web: http://yihui.name

[Ali Kazmi]

Hi Yihui,

Please accept my many thanks for your help! The problem has been solved!

You were correct in pointing out that the error (invalid multibyte string at '<97> as t) was due to inappropriate encoding.

However, it weren't the data sets that produced this problem, but (what I should have suspected much before) the about.html and introduction.html files! I was using an Em Dash and a 'ü' - these characters caused the problem.

I tried loading the HTML with the appropriate UTF-8 encoding, but the problem remained. For the time being, I simply removed the Em dash, and the character. Later, I will look into whether I was loading the encoding properly or not.

The functional application can be found at this link: https://kazmialiarsalan.shinyapps.io/QualitativeAnalysisApp-R/

Once again, thanks for teaching me how to test apps properly! I was being lazy, since I had been working on this app for quite some time. Not an excuse, though!

[Yihui Xie]

That is really great! I'm so glad that you problem has finally been
solved. I think I will have to patch the include* (such as
includeHTML) series of functions in shiny to make multi-byte
characters work there as well. Thanks!

Regards,
Yihui
--
Yihui Xie <xiey...@gmail.com>
Web: http://yihui.name