Hand off time-consuming task to other thread/process/application/coroutine...?

[Maarten De Schrijver]

Hello,

I've written an application in Pyramid. In one particular view, I need to connect to a kind of CRM system via SOAP-XML (need to check if the user exists and if not add to this CRM system). It works fine, but, it is slow: the users web browser "hangs" (or rather: is busy) sometimes for over 10 seconds.

How could I best solve this? I'm thinking along the lines of "handing over" this particular task to another "application"? Or should I investigate Eventlet and coroutines?

Thanks in advance.

Maarten

[Jesaja Everling]

Hi Maarten,

I would suggest looking into using a task queue like Celery.
http://celeryproject.org/

There are also a few alternatives like
http://kr.github.com/beanstalkd/ and
http://nvie.com/posts/introducing-rq/, your you could roll your own
solution using Redis.
But Celery works really well for me so far.

Best Regards,

Jesaja Everling

> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To post to this group, send email to pylons-...@googlegroups.com.
> To unsubscribe from this group, send email to
> pylons-discus...@googlegroups.com.
> Visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Whit Morriss]

Couple other resque clones worth mention: pyres and my personal favorite retools.queue: http://retools.readthedocs.org/en/latest/api/queue.html#module-retools.queue

-whit


On Jan 28, 2013, at 5:53 AM, Jesaja Everling <jeve...@gmail.com>
wrote:

[Arndt Droullier]

Have you considered to run two instances of your pyramid application and simply route the slow running tasks to the one and the rest to the other? 

There would be 

- no need need to rewrite code, 

- no danger the main applications blocks and 

- for the slow running app webservers have reliable connection queing

You could even use a session cache for communication of both pyramid apps.

Arndt.

2013/1/28 Maarten De Schrijver <maartende...@gmail.com>

Maarten

--

You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To post to this group, send email to pylons-...@googlegroups.com.
To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
DV Electric / Arndt Droullier / Nive cms cms.nive.co

[Jonathan Vanasco]

I don't think you'd necessarily need to use celery/redis/etc for this.

No matter what you use to fetch the remote system data , you'll need
to implement a user pattern like this:

1. User loads the web page , which is 'blank' with a loading message
2. The web-page fires a javascript call to an API, which then returns
the relevant data and drops it on the page.

If you don't use an ajax solution like that... no matter what you do,
you'll still have people clicking 'reload' before the 10 seconds are
up.

My suggestion would be to add an animation like a "loading %" gif to
the screen. it can be complete BS -- you're not really showing the %
loaded, just that you're in a loading process. make it a 30s gif to
reach 100%.

Then have your system call an API endpoint, which returns your data in
a json packet. it can have the entire HTML you want to render via
mako/whatever templates. just return it within a json wrapper for
easy placement.

then you can look into:
- integrating a local cache to check if your 3rd party request was
asked / filled yet. this way , in case someone clicks 4 times, only
the first one generates a 3rd party call. the second and third say
'busy' and the 4th reload has the data from the 1st call.
- offloading onto other systems.

i'm a huge fan of using redis / celery / even custom twisted daemons
to do background processing and tasks that are essentially queued for
'later'. in your case though, your web content generation will 'hang'
on the third party no matter what you do. the best way to meet your
goals is to change the user experience, not your platforms.

[Maarten De Schrijver]

Celery seems interesting. Trying to get it to work, but getting some errors. I'll keep you posted.

Maarten

[Maarten De Schrijver]

@Arndt,

I can run multiple instances of my application but how exactly can I then "route the slow running tasks to the one and the rest to the other"?

Could you also elaborate on the session cache?

@Jonathan,

the response or fetched information returning from the "slow running task" (the connection to the SOAP-XML server) is not actually needed to complete the flow that the user is going through. It's actually an online contest (prize question):

- page 1 loads: user is asked for some data (name, address, ...) -> user clicks submit: data is not yet persisted but stored in the session

- page 2 loads: user is asked to answer a question and a tiebreaker (how many people will have participated by the end ...) -> user clicks submit: user's emailaddress is used to check if user already exists in the remote CRM (via SOAP), if not he is added, if yes, his CRM-id is fetched. Now, the user data along with his CRM-id and answers to both questions is persisted in an SQLite database.

Now, in my idea: wouldn't it be better, in the second step (page 2) to just immediately persist the user submitted data to the SQLite database to allow the user to continue the flow and to hand off the connection to the remote CRM to another app/process/Celery/whatever after which I can (if needed) update the user's data in de database?

Thanks already for all the responses!

Maarten

[Jonathan Vanasco]

On Jan 29, 3:26 am, Maarten De Schrijver

<maartendeschrij...@gmail.com> wrote:

> Now, in my idea: wouldn't it be better, in the second step (page 2) to
> just immediately persist the user submitted data to the SQLite database to
> allow the user to continue the flow and to hand off the connection to the
> remote CRM to another app/process/Celery/whatever after which I can (if
> needed) update the user's data in de database?

Yes, totally. I thought your app was going to be fetching data for
client display. Since there's no need to present this data, you can
totally just save it and process later. In your case, you could even
run a cron script or something else periodically - nothing you have is
time sensitive.

Having worked with a lot of contests before, I'd suggest keeping a
local cache of 'seen' email addresses. I'm always surprised at how
many times people will try to enter duplicates.

[Arndt Droullier]

2013/1/29 Maarten De Schrijver <maartende...@gmail.com>

@Arndt,

I can run multiple instances of my application but how exactly can I then "route the slow running tasks to the one and the rest to the other"?

Could you also elaborate on the session cache?

Hi,

for example, I use nginx to proxy requests to my pyramid application. The pyramid application runs twice on different ports. 

Now for the routing: if the slow running tasks use a url prefix (e.g. /tools) or unique path, I add this path to the nginx configuration

and let nginx handle the routing:

    location ~ ^/(tools) {

        proxy_set_header        Host $host;

        client_max_body_size    300m;

        proxy_pass              http://127.0.0.1:6011;

        proxy_redirect          off;

    }

    location ~ ^/(app) {

        proxy_set_header        Host $host;

        client_max_body_size    300m;

        proxy_pass              http://127.0.0.1:6012;

        proxy_redirect          off;

    }

Memcache for example is easy to set up and has pyramid binding. It runs outside of the pyramid process so you can

connect both applications to the same cache. Well I don't know whether or not it makes sense but you could use it to 

post progress reports and pick them up in another view and send feedback to the user.

Arndt.

--