Using Django and R in a production environment?

[Derek]

Based on googling around this topic, it seems that using RPy2 is the most common way to interface with R from Python.  However all the discussions on this seem to centre around working in a desktop (single user) environment.

The one discussion I could find that deals with the issue of working with R "at scale" is this one - https://github.com/Sleepingwell/DjangoRpyDemo/blob/master/README.md#django-configuration  - which indicates problems with this approach; and suggests it might be able to be overcome via creating distinct processes dedicated to run a WSGI application (although this article does not give any steps on how to do this, or whether it would work in practice).

Another approach seems to be to use RPy2, with Twisted to enable multiple sessions: https://docs.google.com/presentation/d/11LJxej6jnbYKzJftpDudYFfVKjaB0BhOzrBSKaxJ2ME/edit#slide=id.p.

Yet another approach might be to use Rserve (http://www.rforge.net/Rserve/) and PyRserve (http://pythonhosted.org/pyRserve/manual.html), but the latter seems to currently be in beta.

Question is: does anyone have any practical experience actually using Django with R in a production environment (i.e dozens or hundreds of users doing high volume number crunching)?

Thanks
Derek

PS Yes, we do need R and not one of the Python-based alternatives, as R offers many routines simply not available in those as yet (also, the client needs to re-use, and create new, R scripts themselves)

[Per-Olof Åstrand]

I am not sure I understand your question, but is it really related to using specifically R? Could it be any kind of heavy number-crunching that needs to be done in the background by a scheduler/task manager? In that case, django-celery may be an option: http://docs.celeryproject.org/en/latest/index.html

Per-Olof

[Derek]

Thanks Per-Olof

No, it has more to do with the issue raised here:
https://github.com/Sleepingwell/DjangoRpyDemo/blob/master/README.md#django-configuration

Possibly Celery could solve that (?) but I really would like to hear from someone who actually has a production configuration set up and working.  Perhaps there are less people in the sciences using Django than I thought...

Derek

[Alex Mandel]

There is another large potential gotcha, R is very memory heavy.
I do think the route of using Celery or other job management tools makes
sense, especially if you can use R across multiple backend machines.
Would celery mean one rpy2 per celery? You don't really want all your
users using the same R session anyways.

Thanks,
Alex

On 04/23/2013 11:08 PM, Derek wrote:
> Thanks Per-Olof
>
> No, it has more to do with the issue raised here:
> https://github.com/Sleepingwell/DjangoRpyDemo/blob/master/README.md#django-configuration
>
> Possibly Celery could solve that (?) but I really would like to hear from
> someone who actually has a production configuration set up and working.
> Perhaps there are less people in the sciences using Django than I thought...
>
> Derek
>

[Derek]

Again, thanks!

But it still seems that no one is actually using R in a production environment themselves (which is a little surprising to me).

[Javier Guerra Giraldez]

On Mon, Apr 29, 2013 at 2:13 AM, Derek <game...@gmail.com> wrote:
> hat no one is actually using R in a production environment themselves (which
> is a little surprising to me).

well R itself is widely used in production... but the intersection
with Django is very small. (after all, if you write your own R
analysis routines, why have them run in a web system)

--
Javier

[Derek]

" if you write your own R  analysis routines, why have them run in a web system"

Two issues here:

1. We are writing R routines for someone else; who needs 'analysis on demand'

2. A web system means that multiple users in multiple locations can all readily access the same analysis routines (but with different data sets)

I agree that if you work on a desktop, with no need to access shared data or use the same analysis routines as anyone else, then a web interface makes no sense.

Basically we are trying to harness two disparate systems; each of which is very powerful in their own sphere, to create an application that is really useful for better and more efficient science.

[Nick Santos]

Is Django a hard and fast requirement? If so, is it just about integrating the routines? I think otherwise something like RStudio Server would work for you. I've deployed it previously and it was a pretty nice setup. It's based on Unix user accounts, so R processes run as if users were running them on a desktop, except not. You may be able to share code in this environment too, but I haven't played with it a ton.

http://www.rstudio.com/ide/docs/server/getting_started

-Nick

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Derek]

Thanks Nick; RStudio looks like a really good tool for development work.

The impression I get though, is that the server is designed for allowing interactive R sessions over the web; whereas for us Django is the primary requirement and calls to R should happen "behind the scenes". 

[Chanat Praserthdam]

Hi Derek,

Did you find the solution to your problem?
I ran to a similar issue.  I am wondering if RServe will work for me.

[Derek]

We have up on R altogether and switched to using pandas and related Python modules.

Our conclusion was that R is great for the desktop but not well suited to a dynamic web environment (happy to be proved wrong, of course).

[Mike Covington]

Not sure if this approach is useful for you, but the following is how we use R and Django together.

We use R to build Shiny apps. These are served on their default port 3838 (which users never actually go to directly). I've made a Django app to manage the Shiny apps and simply place them in an iframe.

Live site: http://symposium.plb.ucdavis.edu/apps/

Django App: https://github.com/mfcovington/djangocms-shiny-app

[Andrew Farrell]

One option is to use the conda package manager (you can get that here) to create an environment with 

`conda create --name django+r django=1.8`

activate it with

`source activate django+r`

And then install R packages with something like

`conda install --channel https://conda.anaconda.org/r r-data.table`

You can of also use conda to install pandas and, if you want a bunch of data science libraries at once, you can download the full Anaconda distribution.

Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/24469c91-e641-4361-be6d-9c90fe0f4bf0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.