How to keep data fresh (real-time data)

[jerem...@validusrm.com]

Hi all,

As I am using shiny to design a dashboard contained real-time financial market data. I am wondering the best way to do it including

1. Where to store the data?
2. How to keep the data fresh automatically?

The way I am currently achieving this is to 

• re-download the data, and
• re-plot or re-build table.

The code is as follows: (some of the data information is hidden for business requirement) 

 output$table = renderGvis({

    invalidateLater(5000, session)

    

    DataWeb = "xxxxxxxxxx"     #which is a CSV file stored on Dropbox

    Dataset = repmis::source_data(DataWeb,

                                   sep = ",",

                                   header = TRUE)

    gvisTable(Dataset, options = list(width=800, height=150))

 

    })

It works but anyone knows better way to do it? As by my method, the whole table will disappear and appear again... It would be much better only the values on the table is updated.

Thank you. 

Jeremy

[Jeff Allen]

I'm not quite clear on your method, but there may be room for improvement.

a.) You may have better luck with the table disappearing by using something like renderDataTable or just renderTable; I'm not familiar with gvisTable, so I'm not sure about that.
b.) How are you pulling the file in from Dropbox? If you're reading it in from disk (i.e. this server is registered as a Dropbox client and syncs all the files to disk constantly), then that looks great to me. If you're pulling it in from the web every few seconds, I'd recommend exploring alternatives (such as setting up the server as a Dropbox client, assuming you're hosting your own Shiny Server). One outstanding issue with R/Shiny is that HTTP requests are "blocking", meaning that the entire R process is tied up as long as the request is outstanding. So if the server is slow to respond, or if it takes a couple of seconds to download the data, the process will be unresponsive to other incoming requests (like users changing input and expecting new graphs to be generated) during that request. In your case, if you're doing this every 5 seconds, your users will likely notice that the app isn't too responsive.

On the other hand, if it's a non-interactive dashboard, that may be OK. As the data will just update every 5 seconds and doesn't need to be interactive.

Unfortunately, streaming of new data into Shiny isn't an area we've explored much. There are no technical reasons why it would be any more difficult than any other system, we just don't have any official "best practices" just yet.

Jeff

[jerem...@validusrm.com]

Hi Jeff,

Thank you for getting back. As far as I am concerned, Shiny still does not have better solutions to feed in live data. The goal I aim to achieve is simple as the following website,

http://www.investing.com/quotes/single-currency-crosses

Where all the prices can be automatically updated. 

My design so far is as follows:

https://validusrm.shinyapps.io/Project/

We are using Shiny Hosting, which means we don't host our service. And all the data is synced on cloud instead of local computers. 

It will be really great to have more sources to know how to update the data live.

Please let me know if anything is unclear. Thank you.

[Roberto Carvalho]

Hi Jeremy! I know this post is a little bit old. But I would like to know if you found the solution for this problem. Also, could you provide a link to your project on Shiny and if possible your source code? I want to plot a graph instead of dashboard using similar approach. Thanks!  

[Tareef Kawaf]

I believe that the current state of the art is to use a reactivePoll.  There is a an example of a reactiveFileReader here, and you can read more generally about reactivePoll here. 

We have been thinking about other streaming use cases in the past, it would be interesting to understand the use case you are trying to support so we can make sure we address it when we go back to looking at the problem.

[Bárbara Borges]

You can always open a socket connection that is closed when the session ends. Example code (from Joe's CRAN dashboard app): https://github.com/rstudio/shiny-examples/blob/master/087-crandash/global.R