pymodbus and asyncio

[Mike Pagel]

Hi there,

I am interested in your opinion whether it at some point would make sense to port asynchronous pymodbus from Twisted to Python 3.4's standard library asyncio.

Background: I am developing an application which could be simplified as a web interface monitoring a bunch of modbus devices. I am using tornado and asynchronous pymodbus under Python 2.7, mainly because of the Twisted dependency that is not yet ported far enough to be used under Python 3. And I have to run all this on Windows.

With the advent of asyncio in 3.4 we now have lots of the Twisted features in form of a standard library. I don't know enough about the Twisted features being used my pymodbus, so I can't judge whether a port to asyncio is reasonable at all. Is it?

Thanks,

Mike

[Galen Collins]

A port absolutely makes sense. Not only would it remove a dependency, but it would be a good chance to get a fresh implementation to address a few bugs. Most all of the protocol code can be used, just the transport code needs to be rewritten.

--
You received this message because you are subscribed to the Google Groups "pymodbus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymodbus+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Pagel]

I just looked a bit at what and how pymodbus uses Twisted. It really seems to be well separated into the async packages, reusing common logic for the real protocol. Nice work. Also it looks like only basic features of Twisted were used that are indeed part of asyncio, in contrast to the long list of advanced googies where Twisted seems to distinguish itself.

I need to learn a bit about both libs first, at some point I'll probably start playing around, If I get anywhere useful for the public or find out about the one big bugger why this will never-ever work, I'll post it.

[Galen Collins]

Awesome, well have fun. I'll try to get around to a port when I get time if you don't beat me to it. Everything should work fine except RTU which you can work around. Basically just read as fast as possible and it should work well enough.

Galen

On Sep 24, 2014 9:29 AM, "Mike Pagel" <mikp...@googlemail.com> wrote:

I just looked a bit at what and how pymodbus uses Twisted. It really seems to be well separated into the async packages, reusing common logic for the real protocol. Nice work. Also it looks like only basic features of Twisted were used that are indeed part of asyncio, in contrast to the long list of advanced googies where Twisted seems to distinguish itself.

I need to learn a bit about both libs first, at some point I'll probably start playing around, If I get anywhere useful for the public or find out about the one big bugger why this will never-ever work, I'll post it.

--

[W. Martin Borgert]

On 2014-09-24 04:24, Mike Pagel wrote:
> I am interested in your opinion whether it at some point would make sense
> to port asynchronous pymodbus from Twisted to Python 3.4's standard library
> asyncio.

As a user of pymodbus I applaud this idea. Currently, I'm using
only the synchronous mode. Partly because I did not want to dig into
Twisted.

It would be still nice to maintain some compatibility with Python 2.
Maybe by using Trollius? It seems to be used by the OpenStack people:
http://techs.enovance.com/6562/asyncio-openstack-python3

Cheers

[Mike Pagel]

Hey, as christmas gets closer I finally got around to looking into this in a bit more detail. The work needs more testing, but I could talk to my modbus device like so:

import asyncio

from pymodbus.client.async2 import ModbusClientProtocol

loop = asyncio.get_event_loop()

@asyncio.coroutine

def process_tcp():

    transport_, protocol_ = yield from loop.create_connection(ModbusClientProtocol, "192.168.178.240", 502)

    result = yield from protocol_.read_coils(1)

    print("Result: %d" % result.bits[0])

    transport_.close()

loop.run_until_complete(process_tcp())

loop.close()

I still need to integrate and therefore debug it with my production application. The port so far was pretty straightforward and fairly minimal, but since I only need the asynchronous TCP client, that's all I tackled so far. Assuming this turns out to work reliable, I have two options:

• Move my port into my appplication code, as I did not need to open up anything from pymodbus any further, as all extension points were already well separated. Or:
• Refactor the two protocol classes (Twisted and Asyncio) such that all common code is moved into a common base class to get rid of redundancy.

I'd do the latter only, if you generally would consider integration of the ported TCP client protocol class. To be clear: I did not port the UDP client or the factory class, so only a third of the Twisted client async.py file is ported.

Please let me know. I'm happy to refactor and post a pull request if you like.

Mike

[Galen Collins]

Awesome; this is a great Xmas present. Any chance you have your work hosted so I can take a look? I would be very interested in making the 3.x branch focused on asycnio instead of twisted.

Galen

[Mike Pagel]

I'll try to upload the ported client code to Github today. Keep ya posted.

Mike

[Mike Pagel]

It's been pushed to Github. Please note that this is work in progress with respect to the final design. I'd like to get feedback and then possibly refactor. The main question is whether we want to allow coexistence of Twisted and asyncio. Personally, I'd like to keep the Twisted code around, as a reference as well as a possible implementation in the future. Also, such a design could even be backported to the python 2 branch and there use teh Trollius asyncio backport if desired.

But enough talking. Here is the client port for asyncio I created first without any framework abstraction in the new file async2.py. I simply replaced the protocol overloads and Deferred API. You can find it here:

https://github.com/moltob/pymodbus/blob/5c80161b9286a581a210a70dd9ee35051b92808d/pymodbus/client/async2.py

That would be the way to go if Twisted was dismissed. If not then we need an additional layer of abstraction. One way can be seen here:

https://github.com/moltob/pymodbus/tree/python3/pymodbus/client

The common logic has been extracted into a base class AsyncModbusClientMixin, which is using an abstract interface to communicate with the used asynchronous framework. Concrete implementations are then available in async_twisted.py and async_asyncio.py. The original file async.py is there for backwards compatibility only and points to the Twisted implementation.

What I don't like about this latter design is that the concrete asyncio class now has both versions of the protocol overloads, e.g. "connectionMade" from the common base class and "connection_made" from the asyncio protocol class. I'd rather use a delegator pattern here, but would like to hear your opinion. The Twisted implementation is in a way suffering from the diamond of death...

So questions are:

1. Do you agree to keep Twisted and Asyncio code in parallel?
2. If yes, do you agree to better refactor the use of common code into delegation, such that protocol classes do not have a mixed (multiple inherited) API?

I am happy to do the refactoring.

Also, with respect to the reconnecting client factory from Twisted: I will eventually need this for my application as well, so I could try offering a port here at some point as well.

Lemme know.

Mike

[Mike Pagel]

After thinking about how to do reconnection I tend to go with the delegator pattern even stronger. The design would look something like this:

This way no duplicate APIs are visible on the protocol classes, no diamond of death and a clean delegation and notification path. Does that work for you or do you think it's too bloated?

Mike

[Mike Pagel]

New thread of work emerging: Tried the full delegation approach, but then the whole plan with backwards compatibility is screwed, as the new AsyncModbusProtocol class needs to be explicitly asked for via the modus_protocol reference... Not compatible.

Therefore I reverted to the original design but with a twist: On your original mixin AsyncvModbusClientMixin I renamed the callback methods to carry a leading underscore (protected by convention). This way they are (at least semantically) removed from the public API and the concrete (and platform dependent) classes only expose the platform specific callbacks which then delegate to the protected methods of the common base class.

For the reconneting factory I am following the delegation path. Has just been pushed to my github. I will start writing tests and refactor somewhat to better match the Twisted factory API. But before that I'll merge this back to the Python 2 branch to see whether the abstraction is also working with Twisted.

Mike

[W. Martin Borgert]

On 2014-12-30 06:49, Mike Pagel wrote:
> New thread of work emerging: Tried the full delegation approach, but then
> the whole plan with backwards compatibility is screwed, as the new
> AsyncModbusProtocol class needs to be explicitly asked for via the
> modus_protocol reference... Not compatible.

While compatibility is always nice, I would not put unlimited
effort into it. As long as the differences are clearly
documented, I don't see a huge drawback. But I'm not yet using
async modbus, so I would need to change my code anyway.

Cheers

[Mike Pagel]

Tests for asyncio implementation have been pushed to Python 3 branch in my Github fork (https://github.com/moltob/pymodbus/tree/python3).

asyncio_test_helper.py: Some functions to support testing coroutines without integrating a real message loop. Works nicely with unitest.mock.

test_client_async_asyncio.py: Tests of callback delegation of protocol class and in particular the reconnecting TCP client logic.

The reconnecting client is used like this:

@asyncio.coroutine

def process_tcp2():

    client = ReconnectingAsyncioModbusTcpClient()

    yield from client.start("192.168.178.240")

After this it's trying to connect to the host, including retries on failure or connection loss, pretty much like the Twisted factory is doing it. The client has an accessor for the protocol instance, which can be asynchronously invoked from a coroutine as shown before:

result = yield from client.protocol.read_coils(1)

I'll now integrate this into my production application for real world testing. If it looks OK, I'll post a merge request.

Mike

[Mike Pagel]

Sure but at some point we may have multiple implementations (Twisted, asyncio, Trollius port, ...) as options in a single branch and allow people to choose the one they already use within the other parts of their application. In such a case you want to make it easy to switch and almost more important make it easy to maintain a single implementation (core).

But I feel the implementation pushed to Github so far is a fair choice in terms of compatibility versus super-clean design.

Mike

[Mads L]

Nice work with the pull request, just stumbled upon it on github :)

Any news in getting this merged, or does it await a larger rewrite?

I'll also try adding a UDP version of ReconnectingAsyncioModbusTcpClient. If this goes OK, who wants the pull request?

[Mike Pagel]

Hey Mads,

I don't think the asnycio stuff for Python 3 is still considered to be merged in the main project. However, feel free to send UDP asyncio extensions to my fork. I'd include it.

Mike

[Mads L]

fredag 16. oktober 2015 13.21.50 UTC+2 skrev Mike Pagel følgende:

Hey Mads,

I don't think the asnycio stuff for Python 3 is still considered to be merged in the main project. However, feel free to send UDP asyncio extensions to my fork. I'd include it.

Just sent a pull request your way :) I see quite a lot of commit messages came with the pull request, maybe I should've rebased or something...