Mixing circuits with other eventloop.

[Andrea Laspada]

Hello,
first of all thanks for this wonderful software.
I find it very useful and very close to what I was looking for.

Unfortunately, after really good start, I was confronted with a problem I can not solve.

I'm trying to create a component that is a producer and a consumer for AMPQ broker, using kombu as a library.

Unfortunately I can not figure out how to hang on to the hub eventloop exposed by kombu.
In particular, I'm trying to porting this code.
https://github.com/celery/kombu/blob/master/examples/experimental/async_consume.py

Obviously, creating a component that in the start:
1) sends a message (line 28)
2) run the hub (line 29)
freeze almost immediately.

there is a "simple" way to install circuits inside the kombu eventloop?
Or, is there a "generic way" to use in these cases?


p.s.
I find this software really, really nice and in my opinion really deserve a lot more of traction.
there's any plan to port over asyncio Python3? I know what J.Mills think about the uselfulness of this porting, but in my opinion "circuits" with his elengancy could be a really interesting platform, but need to "play nice" (or without to much hassle) with other library (which will use asyncio platform really soon).
Please, please, please reconsider this option. :)


Hello and again thank you so much!
Andrea

[James Mills]

Hi Andrea!

Short answer is to write an event source like pollers, timers and notify in the circuits library.

This basically involves writing a generate_events handler.

Long answer later when I'm not on a bus heading to work ☺

Cheers
James

--
You received this message because you are subscribed to the Google Groups "circuits" group.
To unsubscribe from this group and stop receiving emails from it, send an email to circuits-user...@googlegroups.com.
To post to this group, send email to circuit...@googlegroups.com.
Visit this group at http://groups.google.com/group/circuits-users.
For more options, visit https://groups.google.com/d/optout.

[Andrea Laspada]

Hello James,

following your suggestion I read some of your files: the most simple seems the "timer.py" code,

so I tried to use as model.

After few tries, the "most convincing component" is this: http://pastebin.com/5UhkzrTH

This script was tested with:

Pyhton 3.4.1

circuits 3.0.0

qpid-java-broker-0.28 as amqp broker.

At the moment I'm not able to reproduce the result of 

https://github.com/celery/kombu/blob/master/examples/experimental/async_consume.py

At least I wasn't able to reproduce the result in a clean way: my main problem seems  to be related 

to synchronization issue between Consumer object and the Hub object.

When I create the Consumer as in "send_receive", (substantially store a Consumer and wait for incoming messages),

I got no response.

Probably when Consumer get out of context, somethings goes wrong.

When, like in "alternate_send_receive", *in the same method* I create the Consumer object *and* call the hub.run_once, 

I'm able to read the message sent.

So, I'm a little bit lost again.

Maybe a more viable solution (avoiding twisted...) is to wrap the consumer in different 

component and make this run in some particular way?

Thanks

Andrea.

p.s.Good Talk :)

[James Mills]

One thing you _can_ do is thread the

3rd party library into a component.

It's not ideal though if you can vaoid it :)

Is this thing easy for me to setup and test locally

on a Linux desktop? I'd be curious to try this and

see if I can improve on what you've got.

What you've done in that pastebin is on the right track!

cheers

James

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au

[James Mills]

For example: https://bitbucket.org/prologic/autodock/src/6a5dabfc65f18a9ae50234ff5599adc4232eebee/listen_events.py?at=default

This wraps around the docker-py library

and runs the blocking docker I/O in a

separate thread and integrates the events

into circuits ones.

cheers

James

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au

[Andrea Laspada]

Hello James,
thanks for your time.

Installing something useful It's trivial:

for the amqp server:
download http://www.apache.org/dyn/closer.cgi/qpid/0.28/qpid-java-broker-0.28.tar.gz
untar
set the env var "QPID_WORK" to point to any directory (will be used to store internal db and settings)
launch qpid-server (you'll need a jvm also)


For the python part the dep are: kombu and circuits.


Wrapping a Consumer object as thread could be a way, but sound too hacky!
I was thinking to something similar to a "discrete component" which wrap a "kombu consumer".
This "discrete component" should be launched in a separate process withouth resorting on kobu eventloop.
However, it's just a vague idea.

really thanks for your help!

Andrea

[James Mills]

Will this be sufficient: https://registry.hub.docker.com/u/fedora/qpid/ ?

cheers

James

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au

[James Mills]

On Mon, Sep 1, 2014 at 7:39 PM, Andrea Laspada <alas...@gmail.com> wrote:

Wrapping a Consumer object as thread could be a way, but sound too hacky!
I was thinking to something similar to a "discrete component" which wrap a "kombu consumer".
This "discrete component" should be launched in a separate process withouth resorting on kobu eventloop.
However, it's just a vague idea.

If you wrap something in a separate process -- that's fine

but you'll then have to communicate with the rest of your

circuits app via either a circuits.Bridge or circuits.node

(which is also fine!).

If however you wrap it up so that it coperates asynchrnously

with the rest of circuits and it's component library then

the main thing you need to watch out for are "blocking calls".

Obviously in any event-driven system with asynchronous I/O

is that "blocking calls" block the main event loop so the less

blocking calls you have the more concurrent requests you can

handle :)

[Andrea Laspada]

Hi James,

the docker image  with qpid should be perfect.

Im my opinion wrapping in a separate process seems useful to limit the problems of blocking call: until I'll know better the internals of kombu (or the next library)  I'll rely on process separation to allow the "main component" which need to consume from ampq broker, to continue their work without beiing blocked by some wrong call. (and imho, using a bridge to route the amqp event into the rest of circuit, seems "the right thing to do")

Anyway, thanks for your time!

Andrea

[Andrea Laspada]

Hi James,

for some strange 

Hi James,

for some reason I posted the wrong script.

If you try, you get a very long list of exceptions raised by generate_events handler.

The right one was:http://pastebin.com/Q4rq7hZy

but, in the meantime, playing with kombu, probably I fond a solution: http://pastebin.com/d1pdCzbW

It's quite roughly, but working.

Andrea

[James Mills]

I haven't had a chance to test this out myself yet... Did you get it working in the final version?

--

[Andrea Laspada]

hello,
yes, the last one it's a good start and seems working right.

[James Mills]

That's grea tot hear! It looks right to me too!

Let me know if I can help in any other way(s).

cheers

James

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au

[Andrea Laspada]

hello james!

thanks for your supporting!
probably, the best thing you can do is to porting your eventloop on new asyncio python module! :) (sorry, I can't resist: circuits is a very neat framework and the idea of using this piece of cake, with several new projects without fighting with every eventloop i need, It's really appealling)

Anyway, I'll post on github my components.

Cheers,
Andrea

[James Mills]

We can't use or port the core of circuits to asyncio

as yet because we still support Python 2.x (specifically

2.6 and 2.7). When the time comes to drop Python 2.x support

(which probably won't be for quite some time), then we can look at that.

It is (right now) however possible to make circuits and

asyncio libraries/modules compatible in that asyncio applications

and libraries can run atop circuits just like Twisted ones can

in much the same way (at least I think so).

Also it's probably worth mentioning that unlike Twisted and AsyncIO (Python 3)

circuits event-loop and it's async I/O are separate things. circuits

is inherently a "component architecture" and that's it's main strnegth

and point of difference.

cheers

James

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au

[Andrea Laspada]

Hi James,

thanks for your reply.

I'm really glad to know that is possible to adapt circuits to asyncio and I'd like to try a few examples to see what I can get.

I know the reason behind your choice: supporting python2 and etc.etc.

And as daily dev-ops I perfectly know the effort to porting from a solution to another (and the delusion due to supporting another shiny disruptive library, after 10+ years of support for your baby)

but, please consider those point:

- Python 2.x is legacy: sure, there's a lot of libraries. Of the past. Please don't let circuits die with python2.x branch

- Python 2.x could be supported through "tulip" module

- with Python2.x you *must* develop a wrapper for every library with a event loop.

- Python 3.x isn't just "another version": is *THE* version used as base for every iteration of python itself. Our next python.

- with python 3 you adapt your eventloop to asyncio eventloop and nothing else.

- I'm really happy with the idea of component as found in circuits, but I don't see how "yield from" could be a problem for any of components (beyond the well know problems of concurrency). For example, in my opinion, "Kombu Wrapper" as components, would benefit from "yield from" almost every time I need to contact the broker: when declaring amqp entities or try to connect to check message, clearly passing the control to another circuits, would be useful.

- making circuits runnable on asyncio probably would allow to develop component capable of driving other asyncio adapted library , like pyzmq or tornado or NextGoodLibrary.

Anyway, James, please take my words as tangible sign of appreciation of your (and of your contributors) work for this usefull library.

Andrea

[James Mills]

I appreciate what you're saying so here's the thing:

a) We can't simply port to Python 3 at this stage because we need to support Python 2 still.

b) What we can do is:

Figurere out a way of writing a asyncio and tulip components that

could be either added to the circuits library or contributed as

separate packages/components as (for example) circuits.asyncio

and circuits.tulip

Because of circuits Component architecture -- I believe this is very

possible to do. It would merely (I hope) involve writing a new "poller"

around asyncio/tulip.

Perhaps you could try? :)

cheers

James

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au

[Andrea Laspada]

James,

your idea is to "wrap" the asyncio eventloop in a poller?

wouldn't modify the "manager" a more generic one?

Andrea

[James Mills]

The circuits.core.managers.Manager cannot really by adapted to Python 3's asyncio/tuilip

because asynio/tulip is more about Asynchronous I/O and Timed callbacks than an "event bus"

which is what's at circuits' core.

Does that make sense?

Writing a poller for asyncio/tulip will allow us to "run" and "integrate" modules/libraries/applications written for

asyncio/tulip into circuits and circuits applications in a clean way.

cheers

James

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au

[Andrea Laspada]

Ok James,
surely I don't grasp every single intricacy about developing an "event bus" upon asyncio, so I'll trust you.

I'll check the pollers code!

thanks

Andrea

[James Mills]

So to clarify:

AsyncIO/Tulip in Python 3 provide (AFAIK)

a clean consistent interface that's been agreed

upon for developing Asynchronous libraries/protocols/applications.

It also takes advantage of the new generator functionality

in Python 3 for flow control via "yield" and "yield from"

which makes coroutines really easy without having to write

callbacks and errbacks all over the place (Also AFAIK).

What it does not provide however is what circuits'

Component architecture is. A way to define and construct

loosely coupled components and relationships between

various parts of your application.

Async I/O is only one part.

In circuits all Async I/O is essentially just another Component in the system

(pollers) which can be swapped in and out at any time.

cheers

James

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au