getting utf8 to display

[ignorant]

Hi there,

I am trying the new version that is supposed to take care of utf-8 issues but I am not able to get it working. For example, the following text does not display for me -

verbatimTextOutput("summary这里也可以用中文"),

(above copied from example - http://shiny.rstudio.com/gallery/unicode-characters.html

I am using the dev version of shiny. file ui.R confirms that this file is now UTF-8.

My OS is Ubuntu 4.8.2-19 with the following variable set -

LANG=en_US.UTF-8

My R session is - 

> sessionInfo()

R version 3.1.1 (2014-07-10)

Platform: x86_64-pc-linux-gnu (64-bit)

locale:

 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              

 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    

 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   

 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 

 [9] LC_ADDRESS=C               LC_TELEPHONE=C            

[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

I have the following in my global.R (not sure if necessary) - 

options(shiny.usecairo = FALSE)

options(encoding = 'UTF-8')

Any help would be appreciated.

[ignorant]

My console shows this message -

Error in gsub("&", "&", result, fixed = TRUE) : 

  input string 1 is invalid in this locale

[Yihui Xie]

No, do not use options(encoding = 'UTF-8') in global.R or anywhere. It
is a bad thing to do, which makes the assumption that everything is
UTF-8 encoded in your R session, and this is often not true.

Regards,
Yihui

[ignorant]

Thanks, this is definitely better. I can see a lot of text now that has utf-8, although my original verbatimTextOutput still doesn't work.

However, I cannot get tweets like this to display - https://twitter.com/sperez_21/status/467460288024174592

(random sample)

Maybe this is not utf-8 but it displays fine on my unix console but not through shiny.

If it cannot be displayed properly, is there any way to escape those characters?

[Yihui Xie]

Can you send a minimal example, along with your sessionInfo()?

Regards,
Yihui

[ignorant]

Hi Yihui,

I tried very hard to create a testcase. It works on my box but somehow I does not display properly on github. Here it is - 

https://github.com/sanealytics/shinyTestCase

I see the funny characters on my Ubuntu and Mac OSX boxes, but not on github. Nevertheless, I think you should be able to see them if you download the repository.

If you run testCaseBase.R, you will see little boxes instead of the funny characters, but the other text will show.

I have another case where I am fetching this from mongo, see testCaseMongo.R (needs to have local mongo and run load_testcase.sh previously).

In this case, the text does not display at all. I don't think this particular issue is with shiny but rather RMongo probably. I get back "unknown" encoding for that text that shiny does not know what to do with.  

> library(shiny)

> sessionInfo()

R version 3.1.1 (2014-07-10)

Platform: x86_64-pc-linux-gnu (64-bit)

locale:

 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              

 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    

 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   

 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 

 [9] LC_ADDRESS=C               LC_TELEPHONE=C            

[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:

[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:

[1] shiny_0.9.1.9006

loaded via a namespace (and not attached):

[1] bitops_1.0-6    caTools_1.17    digest_0.6.4    httpuv_1.3.0   

[5] Rcpp_0.11.2     RJSONIO_1.2-0.2 xtable_1.7-3   

But hopefully we can solve at least the first case. We should see those funny characters instead of boxes.

Thanks for looking into it.

[Yihui Xie]

The smileys were correctly shown when I ran your testCaseBase.R. If
they were shown as boxes, it is an indication of font problems. You
probably do not have the required font or did not configure the font
correctly.

Your shiny version is behind the CRAN version, so you'd better
update.packages(), although I do not think it is relevant in this
issue.

Regards,
Yihui