Twisted Framework

[Vinodkumar Puliyadi]

Hello All,

Writing a program to call 100+ of API in every request and collect the data.

After researching and few suggestions decided to go with Twisted, want to know Twisted is good choice for such task?

Please any one with good Twisted learning resource let me know as Twisted documentation is very poor to understand.

Thanks and Regards,

Vinod Puliyadi

[Yousuf Fauzan]

This might be a bit late in the game, but why did you decide to go with Twisted?

--

Yousuf Fauzan

http://fauzism.com

http://codebunk.com

http://facebook.com/codebnk

http//texetra.com

--
--
_________________________________________________
Mumbai Python Users Group - http://www.mumpy.org/
Mailing Group - http://groups.google.com/group/mumpy/
Membership Management - http://groups.google.com/group/mumpy/subscribe/
 
---
You received this message because you are subscribed to the Google Groups "mumpy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mumpy+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Vinodkumar Puliyadi]

I am looking for some thing non blocking, asynchronous framework. i found that Twisted has these feature. 

[Dhananjay Nene]

So you want to use twisted to make lots of api calls concurrently (not having each of them blocking before you make another call). Is that correct?

----------------------------------------------------------------------------------------------------------------------------------
http://blog.dhananjaynene.com twitter: @dnene google plus: http://gplus.to/dhananjaynene

[Vinodkumar Puliyadi]

Yes  that's perfect.

[Devendra Rane]

Twisted is a good choice for the task and you are right, the documentation is bad.

For async stuff you should give gevent a try, its pretty well documented and works nicely with all other libraries. There are lot of implementations to make socket libraries async too based on gevent.

http://sdiehl.github.io/gevent-tutorial/ is a good place to start.

For your use cass, do have a look at this discussion :
http://stackoverflow.com/questions/15322701/gevent-pool-with-nested-web-requests

[Anand Doshi]

Hi Vinod,

Why not use node.js? It has a much better documentation and it will serve your purpose well.

Thanks,

Anand.

[Karan Ahuja]

yes node js looks tempting for this . 

erlang ?

[Vinodkumar Puliyadi]

@devendra

Gevent is also is in pipeline for research as i earlier also got suggestion by my friend.

@anand

Need to integrate in other python application so no chance for me to choose node.js as it is very powerful for this task.

[Vinodkumar Puliyadi]

@karan

Erlang was my first choice, 

after some researching if nothing will work in terms of 100+api and 1000s request i will definately switch to Erlang.

[Yousuf Fauzan]

I second gevent too. Twisted is a bit of a beast all around.

However, I think we are drifting from the main topic here. 

So what are the good resources for learning Twisted?

I am afraid I can't help you there as I just used the official documentation when I worked with Twisted, even though it was painful.

--

Yousuf

[Vineet Naik]

Taking a completely different approach - Would a message queue and multiple worker processes work for you? You said you need to integrate it with another application (web app?)

Vineet Naik

[Vinodkumar Puliyadi]

@vineet i did that using python multiprocessing package, diesel.io and parallelpython

with above it is taking 5-6 seconds for processing 25  urls, 10 - 15 concurrent calls used pools and workers.

i need to make it work in distributed architecture also , yes it will be web based.

Will take a benchmark of that also with message queuing .

[Devendra Rane]

While trying to build a large async application, we tried the following:

- Erlang : Painful to maintain because of the syntax and programming paradigm. You might be able to complete your task in less than 50 lines though. It will be fastest and will never fail. If you make Erlang talk to your python code (or any other application), Erlang will kill you.

- Python multiprocessing (and packages based on python) : They are easy to program and maintain, but they tend to become zombie if the python process runs for long intervals. Long running python codes always have weird Zombie issues, whoever says that they have debugged the memory leak is lying (gunicorn has a setting which makes the worker restart after max-requests has been handled because of similar reasons). The other problem is efficiency.

- Twisted : If you can figure out the documentation, good to go. It should be a good fit if you only want to do some API calls.

- Gevent : Simple and easy to maintain. Benchmarked to be damn good in Cpython or Pure python implementations.

- NodeJS : Simple JS code and good modules available for web requests. One word : "MultiCore"

We ended up using Gevent, Zeromq and NodeJS together. Scaling with zeromq queues across distributed machine/cores is peaceful. Python gives the much needed application logic sanity. NodeJS handles the async web operations (like scraping, api, xmpp etc). It could theoretically scale to as many cores limited by number of sockets and memory available.

[Yousuf Fauzan]

+1

(-1 for Erlang : Painful to maintain)

[Johnson Chetty]

On 22 May 2013 18:13, Devendra Rane <ran...@gmail.com> wrote:

While trying to build a large async application, we tried the following:

- Erlang : Painful to maintain because of the syntax and programming paradigm. You might be able to complete your task in less than 50 lines though. It will be fastest and will never fail. If you make Erlang talk to your python code (or any other application), Erlang will kill you.

- Python multiprocessing (and packages based on python) : They are easy to program and maintain, but they tend to become zombie if the python process runs for long intervals. Long running python codes always have weird Zombie issues, whoever says that they have debugged the memory leak is lying (gunicorn has a setting which makes the worker restart after max-requests has been handled because of similar reasons). The other problem is efficiency.

- Twisted : If you can figure out the documentation, good to go. It should be a good fit if you only want to do some API calls.

- Gevent : Simple and easy to maintain. Benchmarked to be damn good in Cpython or Pure python implementations.

- NodeJS : Simple JS code and good modules available for web requests. One word : "MultiCore"

We ended up using Gevent, Zeromq and NodeJS together. Scaling with zeromq queues across distributed machine/cores is peaceful. Python gives the much needed application logic sanity. NodeJS handles the async web operations (like scraping, api, xmpp etc). It could theoretically scale to as many cores limited by number of sockets and memory available.

This is really helpful stuff coming from someone who has been through the rigamarole of evaluation.

Thanks for this Devendra... Definitely something to keep in mind.

 

Cheers,

--
Regards,
Johnson Chetty

[Mukul]

Just to throw it out there, tornado, even though it is primarily a server, has a pretty good async http client and is easy to use.