pyserial and end-of-line specification

[F S]

I just started using Python and I am writing code to access my serial port using pyserial. I have no problem with unix based text coming in the stream using a LF (0x0A) record separator. I also am using unblocked IO. However I have some sensor devices that use the windows CRLF (0x0A,0x0D) record separators and also a 0x03 and 0x02 (stx,etx) framing so I need to change the EOL (end of line) specfier in order to get the pyserial readline to so this.

I read the doc page for pyserial and they allude to using TextIOWrapper: to accomplish this however the example is very unclear and I could not find better information on the IO page.

I would appreciate any advice on how to block the records using "x0Ax0D" and "x03".

Thanks
Fritz

[Andre Müller]

Just take a look into the documentation:
https://docs.python.org/3/library/io.html#io.TextIOWrapper

And in the example of Pyserial:
http://pyserial.readthedocs.io/en/latest/shortintro.html#eol

I think it shold be:
sio = io.TextIOWrapper(io.BufferedRWPair(ser, ser),
newline='yourline_ending')

But the documentation of Pytho says:
Warning BufferedRWPair does not attempt to synchronize accesses to its
underlying raw streams. You should not pass it the same object as reader
and writer; use BufferedRandom instead.


Maybe you should also try:

sio = io.TextIOWrapper(io.BufferedRandom(ser), newline='yourline_ending')

If it's readonly:
sio = io.TextIOWrapper(io.BufferedReader(ser), newline='yourline_ending')


I never tried it, but your question leads me to take a look into this cool
features of the io module.

Greetings Andre

[FS]

Thank you for your response Andre. I had tried some code like that in the document but it did not seem to work. However ever leaving my terminal for a time the code eventually wrote out the records so apparently there is some very deep buffering going on here. A little more searching on the web revealed the following:

https://stackoverflow.com/questions/10222788/line-buffered-serial-input

It is apparent that pySerial, or at least the documentation is falling short of my needs. It is very unclear what module in the layer is handling the buffering and newlines and so forth. Also unclear is whether the coupled python and OS is reading FIFO or LIFO--something important in quasi realtime scientific applications.
This is problematic since the serial port is still so ubiquitous to a lot of scientific instrumentation. I probably will patch up some byte oriented code for this or perhaps write the module in C.

Thanks again
Fritz

[Rob Gaddi]

Handling it .read(1) at a time is probably your best bet. Append them
into a bytearray and pull it all out when you're done.

It's a serial port; it's not like there is any degree of inefficiently
that you could write the code that will make it slow with respect to the
I/O. I write a LOT of serial instrument I/O and I've definitely had to
fall back to this plan. Your code gets a little long, you hide it all
in a function somewhere and never think on it again.

One paradigm I stick for ASCII serial is to have 3 functions:

def command(msg: str):
"""Sends a command, raises CommError if it doesn't get some
expected OK sort of thing."""

def query(msg: str):
"""Sends a commmand, returns a (trimmed) response line."""

def _communicate(msg: str):

The ugliest stuff, all the str->bytes->str stuff, the line-ending and
protocols, goes into _communicate. Query usually just calls
_communicate. Command slaps on whatever checks are needed. It feels a
bit heavy, but it leads to highly-usable code and makes it easy to
integrate logging, retries, integrating "*OPC?" handshakes, whatever
sort of things turn out to be necessary on a given device.

--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.

[FS]

Thanks Rob.
Yes I ended up with a read(1) and use a field count and a few other checks to make sure I don't get a partial record. Serial is the "best of times and worst of times". Sure beats dealing with USB enumeration, power hungry ethernet processors and a lot of other stuff. I can still "see" serial on my o'scope which is always nice and I don't see it going away any time soon at least in the laboratory.

Python has been a bit of a chore-seems like a lot of verison/rev inconsistencies. At any rate I am going to stick with it. I used PERL in the past but I covet the stats packages and a few other things I lost when I left matlab and I want to try for chemometrics work.

cheers
fritz