[Python-il] Looking for a Python idiom combining generators and serial input

[Omer Zak]

I am developing a Python application, which reads bytes from a serial
port (using pySerial).
Those bytes combine into packets by means of a non-trivial set of rules.

Using a chain of generators, I implemented the rules in a
straightforward way. A generator reads bytes from the serial port,
processes escape sequences and yields the resulting characters to
another generator. The next generator delimits packets and yields each
packet as a unicode string. The final generator parses the unicode
strings and yields the parse results for each frame as a data structure
(a Dict, whose elements are the various packet fields).

However, the ultimate input of the generators needs to be from the
serial port.

The question: do I have to use a separate thread for reading bytes from
the serial port (by means of the first generator in chain) and then use
some complicated mechanism for transferring the yielded data structures
to another thread?
The reads in such a case would be blocking reads.

Or is there another idiom for feeding data for processing by a chain of
generators, which can relieve me from having to use a separate thread?

--- Omer
--
Every good master plan involves building a time machine. Moshe Zadka
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html

_______________________________________________
Python-il mailing list
Pyth...@hamakor.org.il
http://hamakor.org.il/cgi-bin/mailman/listinfo/python-il

[Roman Yakovenko]

On Thu, Oct 1, 2009 at 9:10 PM, Omer Zak <w...@zak.co.il> wrote:
> I am developing a Python application, which reads bytes from a serial
> port (using pySerial).
> Those bytes combine into packets by means of a non-trivial set of rules.
>
> Using a chain of generators, I implemented the rules in a
> straightforward way.  A generator reads bytes from the serial port,
> processes escape sequences and yields the resulting characters to
> another generator.  The next generator delimits packets and yields each
> packet as a unicode string.  The final generator parses the unicode
> strings and yields the parse results for each frame as a data structure
> (a Dict, whose elements are the various packet fields).
>
> However, the ultimate input of the generators needs to be from the
> serial port.
>
> The question: do I have to use a separate thread for reading bytes from
> the serial port (by means of the first generator in chain) and then use
> some complicated mechanism for transferring the yielded data structures
> to another thread?

It depends. If you can effort single-thread & busy-wait - just do it
this way. It is easy to understand and support. If you don't, than
welcome to a wonderful world of asynchronous IO. In this case I
suggest you to read http://www.kegel.com/c10k.html article. Also it
talks about web servers, but the solutions are applicable to your
problem too.

> The reads in such a case would be blocking reads.
>
> Or is there another idiom for feeding data for processing by a chain of
> generators, which can relieve me from having to use a separate thread?

I am not sure whether you can completely avoid threads, but a good
async. package/library can simplify a lot of things. Boost.Asio ( C++
) does it very well.

HTH

--
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[Ori Peleg]

If you want to go async, and you're comfortable with Twisted, the framework supports serial port input.

If like Roman suggests blocking reads meet your needs, maybe you don't need a thread - the entire application just blocks.

If you do need to block in a different thread, communicate between threads using Python's thread-safe Queue: http://docs.python.org/library/queue.html

--
Check out my blog: http://orip.org

[Omer Zak]

I ended up implementing my serial I/O application using blocking input
(which feeds a chain of generators) and Queue, using pySerial and
threading.

Thanks to Ori Peleg and Roman Yakovenko for their suggestions.

On Tue, 2009-10-06 at 23:27 +0200, Ori Peleg wrote:
> If you want to go async, and you're comfortable with Twisted, the
> framework supports serial port input.
> If like Roman suggests blocking reads meet your needs, maybe you don't
> need a thread - the entire application just blocks.
> If you do need to block in a different thread, communicate between
> threads using Python's thread-safe
> Queue: http://docs.python.org/library/queue.html

--- Omer
--
The brain does not use addresses. www.werner-seyfried.com

My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html

_______________________________________________