qam vs callme
Jonas
I'm looking for an Python RPC framework with support for asynch
requests/responses, so I was quite happy to fin qam. So now I have a
couple of questions:
1. Is qam fit for production use? It doesn't have a lot of downloads
and I can't find any info about it being used in a live environment.
2. What is the status/plans for callme? When do you think it will
become more mature than qam=
Thanks
Christian Haintz
On 11/8/11 10:02, Jonas wrote:
> Hi,
> I'm looking for an Python RPC framework with support for asynch
> requests/responses, so I was quite happy to fin qam. So now I have a
> couple of questions:
> 1. Is qam fit for production use? It doesn't have a lot of downloads
> and I can't find any info about it being used in a live environment.
We are using QAM in the synchronous way in production 24/7 since October
2009 and hadn't a single issue. We use it to execute actions accross 15
servers over SSL via internet. Our system executes about 1.5 Millions of
RPC Calls with QAM per Year.
For our case it is mature, the point why we didn't update the status to
production/stable is because we didn't get feedback by other users of
QAM how it works for their use case. We either didn't get bug reports.
That's the reason we kept it in beta state.
The reason why we discontinue QAM is the complexity of the QAM system
design from a code perspective. The highly multi-threaded design is not
needed anymore with the new AMQP specification (at least for synchronous
calls). That's why we made a new architecture for callme which is more
light-weight (needs less threads, actually none on the proxy side)
Due to design considerations one of our base libraries QAM uses - carrot
- is discontinued. The successor of carrot (also written by Ask Solem)
is kombu which we are using in callme.
Callme only supports synchronous calls (at least at the moment) because
in our use case we decouple the asynchronous parts outside of our RPC
system which makes it more flexible for us. Our focus is on maturity
because in reasonable time we want to exchange our QAM production system
with a callme based system.
> 2. What is the status/plans for callme? When do you think it will
> become more mature than qam=
"More mature" is a good question. We didn't get any feedback that QAM is
not stable/mature. When our production system is running two years with
callme, then we can honestly say "callme is as mature as qam" :-)
The maturity depends a lot on the user feedback and issue reports.
It should be quite easy for other developers to provide possible fixes
and/or add functionality since callme is only about 500 lines of code,
is well documented and also the architecture is explained in the
documentation.
But without long-term tests the maturity cannot be measured. The
functionality which we provide in the versions we publish on pypi and
tag on github are stable from our perspective. Only the master version
on github might be a bit more unstable.
Cheers,
Christian
Jonas
Good to hear that you've been using qam in production for a while.
Since I want to use asynch integration, would you suggest going for
qam, or implement asynch support in callme?
Wouldn't it be possible to take most of the asynch handling code from
qam and extend callme, or do you think a complete rewrite is
necessary?
/ Jonas
On Aug 11, 2:31 pm, Christian Haintz <christian.hai...@gmail.com>
wrote:
Christian Haintz
> Thanks for the quick response!
> Good to hear that you've been using qam in production for a while.
> Since I want to use asynch integration, would you suggest going for
> qam, or implement asynch support in callme?
I wouldn't go with QAM as we don't maintain it any longer also due to
the discontinued carrot library we use in QAM. So if you use QAM and a
bug appears you need to fix it yourself.
I would say unless you maintain QAM yourself for your purposes, it will
be more future-safe to use callme.
> Wouldn't it be possible to take most of the asynch handling code from
> qam and extend callme, or do you think a complete rewrite is
> necessary?
I have not yet looked that up. IMHO the first question is: Is it useful
to provide asynchronous RPC calls? Or is it better to have solid
synchronous RPC library which acts like normal method calls and use
threading or multiproccesing in front of callme to handle this?
It is not the complexity why we didn't introduce this feature in callme,
it's the complexity we hide from the user and the implications it has
for the user when callbacks are executed in a separate thread or even
process (when using multiprocessing) and the user doesn't think of it
during developement/design.
I will try to give you a short snippet later how I would make a method
call (normal or RPC) asynchron using threading.
cheers
Jonas
Regarding implementing asynch within the framework or not:
It is mostly about scalability. With asynch support in the framework
you don't need to waste resources (i.e. threads waiting for responses)
since you can have a small number of response listeners serving a
large number of requesters.
IMHO it is also good to not try to hide the fact that a request is
remote, to avoid the problems you get when something you don't expect
with a local invocation occurs, e.g. the broker is down, or high
latencies.
Anyway, if you could provide a short intro on what it would take to
introduce asynchronicity into callme I'd be very glad.
/ Jonas
On Aug 11, 3:31 pm, Christian Haintz <christian.hai...@gmail.com>
wrote:
Christian Haintz
> Ok,
> Regarding implementing asynch within the framework or not:
> It is mostly about scalability. With asynch support in the framework
> you don't need to waste resources (i.e. threads waiting for responses)
> since you can have a small number of response listeners serving a
> large number of requesters.
That's true. But most probable you'll also need a high number of threads
which you have previously saved with a response listener, when doing the
callback execution via separate worker threads. The number of threads
don't decrease, if I understand your scenario the right way. But that's
a matter of design.
> IMHO it is also good to not try to hide the fact that a request is
> remote, to avoid the problems you get when something you don't expect
> with a local invocation occurs, e.g. the broker is down, or high
> latencies.
That's definitely important not to hide that fact.
> Anyway, if you could provide a short intro on what it would take to
> introduce asynchronicity into callme I'd be very glad.
It's a matter of design, there are many good ways: Some DIY ways which
also apply for callme are quickly described here on Stack Overflow
(http://stackoverflow.com/questions/1239035/asynchronous-method-call-in-python)
When you prefer libraries you could use eventlet, gevent, etc.
It depends on what you need. In case all suggestions doesn't fit and
feel too complex, maybe you'll feel better with just "plain" messaging
via amqp (just use kombu directly, it is already quite high-level). That
way you get it automatically asynchronous: send a task to the server and
fetch the answer when you have time to.
I don't know what you are trying to achieve but eventually celery
(http://celeryproject.org) might fit your needs as well.
Hth
Regards,
Christian
Jonas
> which you have previously saved with a response listener, when doing the
> callback execution via separate worker threads. The number of threads
> don't decrease, if I understand your scenario the right way. But that's
> a matter of design.
block to wait for something else) work as well. E.g. a proxy server
with non-blocking io on the incoming side and asynch callme on the
backend side would scale better with a low number of threads (a few to
handle incoming requests and a few to handle callbacks) than with
thousands of "synchronous" threads.
> It's a matter of design, there are many good ways: Some DIY ways which
> also apply for callme are quickly described here on Stack Overflow
> When you prefer libraries you could use eventlet, gevent, etc.
>
> It depends on what you need. In case all suggestions doesn't fit and
> feel too complex, maybe you'll feel better with just "plain" messaging
> via amqp (just use kombu directly, it is already quite high-level). That
> way you get it automatically asynchronous: send a task to the server and
> fetch the answer when you have time to.
>
> I don't know what you are trying to achieve but eventually celery
> (http://celeryproject.org) might fit your needs as well.
detail.
Regards / Jonas
Christian Haintz
>> That's true. But most probable you'll also need a high number of threads
>> which you have previously saved with a response listener, when doing the
>> callback execution via separate worker threads. The number of threads
>> don't decrease, if I understand your scenario the right way. But that's
>> a matter of design.
> Actually not, if the callbacks threads only do "local" (i.e. don't
> block to wait for something else) work as well. E.g. a proxy server
> with non-blocking io on the incoming side and asynch callme on the
> backend side would scale better with a low number of threads (a few to
> handle incoming requests and a few to handle callbacks) than with
> thousands of "synchronous" threads.
Thanks for making that clear. Yes, you are absolute right. We'll
consider this discussion for further development. As already mentioned,
it is not that hard to implement async support, but if we implement it
we want to provide easy-to-use, efficient and fast async calls. And that
needs a bit more investigation.
Regards,
Christian