Re: pypubsub question

50 views
Skip to first unread message

oliver

unread,
Apr 22, 2012, 10:24:57 AM4/22/12
to Jonathan Lettvin, PyPubSub
Hi Jonathan, glad that pubsub can be useful to you. So the Provider publishes a message and the listeners handle it, something like this: 

// provider: 
pub.sendMessage("newArray", array=ndarray)

is that right? and this call is from the main thread, while the N listeners are in N separate threads? (and N is unknown to the Provider?)

Oliver

On Sat, Apr 21, 2012 at 1:22 AM, Jonathan Lettvin <jlet...@gmail.com> wrote:
Hi Oliver,

I am really enjoying your code.
I have attached a test script that uses
numpy.memmap to communicate ndarrays
between Providers and Listeners.
I think it is largely well written except
I think I need to introduce some kind of lock.

Question:
How can I be certain all the Listeners have finished consuming the memmap file
before the Provider publishes a new one?

--
This e-mail is from Jonathan D. Lettvin, and may contain information that is confidential or privileged. If you are not the intended recipient, do not read, copy or distribute the e-mail or any attachments. Instead, please notify the sender and delete the e-mail and any attachments. Thank you.

Jonathan D. Lettvin

oliver

unread,
Apr 23, 2012, 10:14:33 AM4/23/12
to Jonathan Lettvin, PyPubSub


On Sun, Apr 22, 2012 at 11:16 AM, Jonathan Lettvin <jlet...@gmail.com> wrote:
Yes.  Almost.  I asked for help too early, and realized my error too late.

Basically, if I produced new arrays for every message or every memmap,
memory and disk would be consumed too rapidly.
So I am trying to figure out how to re-use arrays and memmaps.

I think I got what I needed from pubsub already, and I thank you for the simplicity of design.
Now I should get my own act together and use it properly.

 No problem. 

If you are interested in what I am doing, read on.

My project is a time-sequence image processing filter stack using unpublished algorithms.
The data I work on are largish (1e7-1e9 bytes).

Each filter has two state data planes (old and new).
Each time step causes two orthogonal actions:
  1. Each filter generates a new state by operations involving input data and old state.
  2. Each filter generates output by operations involving comparing old and new state.
During stage 1, the message data must not change but the states do.
During stage 2, the states must not change but the message data do.

I want to use messages to couple filters together into a stack
where a filter changes state only after all its server inputs are ready and
a filter generates its output only after all its client filters finish changing state.

I'm curious about this. Pubsub is designed to be used where there is data that has to be communicated without knowledge of the consumers (their type/class, how many, etc), and where consumers don't need to know the source of the data. Ie, fire and forget. 

Based on that, it seems that you want the filters to be "black boxes" and each one will listen to a certain type of message to receive data it should process, and each one will send a message with the new data, is that right? Can you clarify what you mean by "message data" vs "state"?

Oliver



Reply all
Reply to author
Forward
0 new messages