Introduction to the Python Bindings

[William]

Hello All,

I'm a software engineer working with DNP3 for a power-electronics company.  I would like to use OpenDNP3 through it's Python bindings.  Unfortunately, all I can see learn with is the master-slave demo.  It's difficult enough to learn the library from the demo, without having to then find the corresponding Python functionality.  Can anyone provide more information?

If it helps, what I'm looking to do with the library is fairly limited.  I am only working with the master, having it connect to a slave and read/write some data points.

Thanks,
Will

[Jonathan Yu]

Hi Will,

On Wed, Jun 6, 2012 at 9:34 AM, William <semb...@gmail.com> wrote:

I'm a software engineer working with DNP3 for a power-electronics company.  I would like to use OpenDNP3 through it's Python bindings.  Unfortunately, all I can see learn with is the master-slave demo.  It's difficult enough to learn the library from the demo, without having to then find the corresponding Python functionality.  Can anyone provide more information?

These are very general hints since I haven't yet had a chance to play with opendnp3 myself :-(

Often, projects that are written in a given language (C++ in the case of OpenDNP3) have bindings to other languages automatically generated by some build tools. OpenDNP3 uses SWIG (I believe) to generate the bindings. These projects often do not have full documentation in the interfaces, nor even interfaces you might expect, since they have been automatically generated. For example, certain assumptions about design patterns and styles may exist in Python that do not exist in C++, which can make the interface a bit awkward to use.

What I have found useful in the past, and what I would suggest for you, is to look at the C++ code and corresponding documentation, and make the appropriate adjustments to get it into Python syntax -- so instead of #include <header>, you would import .. etc. Try It And See is a good strategy here - and once you get the hang of it, it might be easier for you to use.

(I'm speaking from past experience using wxWidgets' Perl bindings, where wxWidgets was originally written in C++ and thus follows those conventions. As the library itself was more mature than opendnp3, so too were the bindings, but nonetheless, reading the original C++ documentation helped make using the library make a lot more sense to me)

Cheers,

Jonathan

[Adam Crain]

That's exactly how it works. The Java/Python bindings are Swig
generated and are therefore 1:1 API mirrors of the C++.

Chris Verges got the Python bindings working with autotools. I
personally haven't played with them.

The .NET bindings are currently the only exception, they're hand
crafted with C++/CLI to get a little tighter integration and more
idiomatic interfaces.

-Adam

On Jun 6, 9:59 am, Jonathan Yu <jonathan.i...@gmail.com> wrote:
> Hi Will,
>

[William]

Thanks for the quick responses, guys.  Jonathan, I actually learned to use wxWidgets in the same way, only I was using its Python bindings.

The issue I think I'm running into is how much of the interface is exposed.  For example, I see in the C++ master demo (DemoMain.cpp) that one of the first things it does is create an EventLog object.  I search the SWIG-generated Python library (pyopendnp3.py), and can't find any mention of EventLog.  I search the OpenDNP3 source code and find that EventLog is defined in Log.h, which is part of the APL library.  As far as I know, there's no Python interface to APL.  So I'm left wondering if I actually need it, or how to code the master demo in Python without APL.

I'm sorry if I'm missing something here; I'm just starting on this.  Going through the documentation and finding the corresponding Python seems to be about the only thing I can do, though.  I'll post any results.

-Will

[William]

CORRECTION: the Python bindings do interface with APL to a point.  If I run Python in a terminal, import opendnp3, and type "opendnp3.LogEntry()", it outputs: <opendnp3.pyopendnp3.LogEntry; proxy of <Swig Object of type 'apl::LogEntry *' at 0xb736b5f0> >.

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[Christopher Verges]

Hi Will,

Regarding Adam's comment, while I got the Swig bindings created for Python, I also having played with them.  :-)

As I recall, another group member, Nichola, was working on a demo Python application, but I haven't seen any feedback on whether he got it working.  (Nichola, if you're reading this, any updates?)

Chris

[Nicola Zanella]

Hi Chris, unfortunately no, I have no news...
I moved to another project in the last 3 month, so I had to stop my attempt to write a simple python demo using your swing wrapper.
But what I remember is that the swing interface seems to export just a litte portion of all the C/C++ code... And also, at that time, I didn't find a simple
C++ demo to use as a starting point.

Anyway, I'm still interested in the python wrapper, so I will let you know when I am again on this project.
Thanks, Nicholas

2012/6/7 Christopher Verges <chris....@gmail.com>

[William]

I've gone through the Python bindings and created a list of all the classes available, describing them with information from the documentation.  Like Nicola said it's a very small portion of the C/C++ code, but hopefully it'll be enough to work with . . .

-Will

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[William]

This is a silly question . . . but is it possible that not enough of the interface is exposed to make a workable demo in Python?

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[Chris Verges]

Yes

[Chris Verges]

Yes, however highly unlikely.

On Jun 7, 2012, at 12:19, William <semb...@gmail.com> wrote:

[William]

I've run into a problem trying to re-code the following line in the master demo (DemoMain.cpp):

mgr.AddMaster(
    "tcpclient",           // port name
    "master",              // stack name
    LOG_LEVEL,             // log filter level
    app.GetDataObserver(), // callback for data processing
    stackConfig            // stack configuration
);

The first two arguments to AddMaster are easy enough.  The third argument (LOG_LEVEL) is supposed to be a FilterLevel enum.  Python doesn't have access to that enum, but I was able to "feed" one in anyways by creating a PhysLayerSettings object and using its LogLevel member (which is a FilterLevel enum) for the third argument.  Now I've hit an issue with the forth argument.  First, it sounds like it's a vital argument to me if it's the "callback for data processing."  The forth argument is supposed to be a pointer to an instance of the IDataObserver class.  This is an abstract class so I couldn't directly create it in my script by doing something like observer = IDataObserver().  (Python gives an error.)  So I created my own class in Python to inherit from it:

class Observer(IDataObserver):
    def __init__(self):
        # call the IDataObserver class's init function
        super(IDataObserver, self).__init__()

However, the AddMaster method rejected an instance of my class, because the instance was technically of the type ITransactable (which the IDataObserver inherits from).  I'm guessing this is so because IDataObserver is an abstract class.  So, I would like to create an instance of a class that inherits from IDataObserver.  (Possibilities include FlexibleDataObserver and QueueingFDO, QueueingFDO is what the master demo uses.)  But neither of those classes are available in Python...

I'm sorry if I'm missing something obvious here.  I'm learning C++ as I go and I'm just starting on this API, obviously.

-Will

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[Christopher Verges]

Hi William,

In your Observer class, are you implementing the required functions?  Specifically:

https://github.com/cverges/dnp3/blob/master/APL/DataInterfaces.h#L36

Look at the "pure virtual" functions that end in "= 0;"

https://github.com/cverges/dnp3/blob/master/APL/ITransactable.h#L49

Same as with this.

You should be able to use the C++ classes for FlexibleDataObserver, for example, as a guide.

https://github.com/cverges/dnp3/blob/master/APL/FlexibleDataObserver.h

https://github.com/cverges/dnp3/blob/master/APL/FlexibleDataObserver.cpp

Chris

[William]

Thanks for that point, Chris.  I'll try that.

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[William]

I was able to implement the virtual functions without difficulty.  One of the things things the FlexibleDataObserver does is use the SigLock class to lock and unlock for thread-safe communications.  I wasn't able to access the classes related to thread locking in Python, but I presume this is not an issue because I'm not using different threads.

I solved the problem where my Observer class was of the type ITransactable instead of IDataObserver; I was initializing the IDataObserver class incorrectly in Python.

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[Chris Verges]

Congrats!  So is the python master working now?

Chris

[William]

It should be set up correctly.  I need to track down how to make the process run from Python, and I need to find out how to send a read request for a point.  I have a micro-controller that implements DNP with which to test.  I'll keep posting about my progress....

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[Adam Crain]

Hi William,

Good progress. I recommend something that just prints to the console
in your IDataObserver to start. Check out this test program for
the .NET bindings:

https://github.com/gec/dnp3/blob/master/DotNet/DotNetMasterDemo/Program.cs

As far as asking the stack to do a read request, you can't. Instead
you define a polling schedule in the MasterStackConfig object. Right
now only 2 types of polls are supported:

1) Integrity Scans
2) Event polls

You basically setup a schedule and then the stack just runs. By
default, you'll get (3 or 5 sec?) integrity scans.

-Adam

[William]

Thanks for the heads-up Adam.  I'm glad I saw your post before I went looking for point-reading functionality.  I'm not familiar with DNP integrity scans, can those be configured to read specific points at certain time intervals?

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[Adam Crain]

Integrity scan means "Read everything". This is what all masters do
when they first connect. They usually also do this periodically to
ensure that the current state is never lost. Your device will
definitely support this.

Class Scans (class 1/2/3) are polls for event data i.e. "Read
changes". Your device will likely support this put you'll have to
refer to its documentation to see what classes the point(s) you're
interested in are assigned.

-Adam

[Chris Verges]

What would it take to support a by-request poll of a particular point/group? It seems like the Slave would already have to support this and there just needs some API exposed on the Master, right?

[William]

I've noticed that in the master demo the process is started by calling app.Run().  "App" is of the type class MasterDemoApp.  I looked through the classes MasterDemoApp inherits from, and the Run() method is inherited the class IOServiceThread.  Python doesn't have access to that, so I'm not sure how to start the process.

-Will

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[Chris Verges]

Use an infinite while or for loop for now.

[William]

Thanks . . . I should have looked at the .NET example before asking.

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[William]

I'm working on implementing the command_acceptor.AcceptCommand( . . . ), like in the .NET demo Adam linked.  Unlike in the .NET demo, the AccepCommand method expects four arguments: https://github.com/gec/dnp3/blob/master/APL/CommandInterfaces.h.  I'm having trouble finding out what information the functions wants.  I presume the AcceptCommand method is causing the stack to update.  Is there another way to do this?  Thanks.

On Wednesday, June 6, 2012 8:34:20 AM UTC-5, William wrote:

[Chris Verges]

Btw, use a sleep in the loop

[Adam Crain]

Hi William,

The function is asynchronous (non-blocking). You kick off a command
and then sometime later you get a callback on the IResponseAcceptor
you specified. The .NET bindings use futures to simplify this.

An explanation of the 4 arguments:

1) the actual command object (setpoint or binary output).
2) the index of the command
3) a sequence number (think correlation id) so you know which
operation.
4) The IResponseAcceptor interface where you'd like to be called back
when the operation is complete.

-Adam

[William]

@Chris, yes, good point.  I am using a sleep.

@Adam, thanks, that helps.  I created my own class which inherits from IResponseAcceptor and passed that as the last argument of AcceptCommand.  The execution dies due to a segmentation fault shortly after the call to AcceptCommand . . .

[Sam Hendley]

Make sure that your ResponseAcceptor object is not getting garbage collected.

[William]

It shouldn't be, because I still have a reference to it (I saw your post on the Java memory problem).  As in:

response = Response()
b = BinaryOutput(CC_PULSE)

 . . . other stuff . . .

command_acceptor.AcceptCommand(b, index, 0, response)

[William]

Here is a back trace on the seg fault:

#0  0x08062436 in PyObject_Call (func=<built-in method __new__ of type object at remote 0x82358a0>, arg=(<type at remote 0x8661274>,), kw=0x0)
    at ../Objects/abstract.c:2490
#1  0xb7af4a39 in SWIG_Python_NewShadowInstance (ptr=<value optimized out>, type=<value optimized out>, flags=0) at DNP3Java/PythonDNP3.cpp:2308
#2  SWIG_Python_NewPointerObj (ptr=<value optimized out>, type=<value optimized out>, flags=0) at DNP3Java/PythonDNP3.cpp:2419
#3  0xb7b21df3 in SwigDirector_IResponseAcceptor::AcceptResponse (this=0x82e5f38, arg0=..., aSequence=0) at DNP3Java/PythonDNP3.cpp:6814
#4  0xb798c5b9 in apl::CommandQueue::ExecuteCommand (this=0x863a0e8, apHandler=0xb74f6d68) at APL/CommandQueue.cpp:89
#5  0xb798c306 in apl::CommandQueue::RespondToCommand (this=0x863a0e8, aStatus=apl::CS_HARDWARE_ERROR) at APL/CommandQueue.cpp:61
#6  0xb7a1b383 in apl::dnp::Master::ProcessCommand (this=0x863a0d0, apTask=0x8651e80) at DNP3/Master.cpp:153
#7  0xb7a20c54 in boost::_mfi::mf1<void, apl::dnp::Master, apl::ITask*>::operator() (function_obj_ptr=..., a0=0x8651e80)
    at /usr/local/include/boost/bind/mem_fn_template.hpp:165
#8  operator()<boost::_mfi::mf1<void, apl::dnp::Master, apl::ITask*>, boost::_bi::list1<apl::ITask*&> > (function_obj_ptr=..., a0=0x8651e80)
    at /usr/local/include/boost/bind/bind.hpp:313
#9  operator()<apl::ITask*> (function_obj_ptr=..., a0=0x8651e80) at /usr/local/include/boost/bind/bind_template.hpp:32
#10 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, apl::dnp::Master, apl::ITask*>, boost::_bi::list2<boost::_bi::value<apl::dnp::Master*>, boost::arg<1> > >, void, apl::ITask*>::invoke (function_obj_ptr=..., a0=0x8651e80)
    at /usr/local/include/boost/function/function_template.hpp:153
#11 0xb7985f71 in boost::function1<void, apl::ITask*>::operator() (this=0x8651e80) at /usr/local/include/boost/function/function_template.hpp:1013
#12 apl::AsyncTaskBase::Dispatch (this=0x8651e80) at APL/AsyncTaskBase.cpp:83
#13 0xb7989933 in apl::AsyncTaskGroup::CheckState (this=0x8557688) at APL/AsyncTaskGroup.cpp:162
#14 0xb798593d in apl::AsyncTaskBase::Enable (this=0x8651e80) at APL/AsyncTaskBase.cpp:61

Notice the hardware error in line five.  Could the library be crashing if it fails to connect to my device?

[Sam Hendley]

Can you attach the PythonDNP3.cpp and .h files? I wouldn't expect it to be related to the fact its not connected.

[William]

Here they are.

[William]

Sorry I missed this.  Here's another piece of information which came before the back trace.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb74f7b70 (LWP 12152)]

0x08062436 in PyObject_Call (func=<built-in method __new__ of type object at remote 0x82358a0>, arg=(<type at remote 0x8661274>,), kw=0x0)
    at ../Objects/abstract.c:2490

2490    ../Objects/abstract.c: No such file or directory.
    in ../Objects/abstract.c

[Chris Verges]

Just as an interesting test, what happens if you wait until the connection is up before executing the command?

Chris

[William]

How do I wait until the connection is up?  I had it sleep for 20 seconds between adding the master to the stack and calling AcceptCommand, and it still crashed with the same error.

[Chris Verges]

Add about 2 min for safe measure.  There is a callback that notifies you when it is up ... Like OnStackChange or something.

[William]

I subclassed IStackObserver and added a callback to the stack.  However, the program is terminating with a Swig::DirectorMethodException when my callback is supposed to be called.  Here is some of the back trace.

Starting program: /usr/bin/python opendnp3test.py
[Thread debugging using libthread_db enabled]
[New Thread 0xb74f7b70 (LWP 28524)]
terminate called after throwing an instance of 'Swig::DirectorMethodException'

Program received signal SIGABRT, Aborted.
0xb7fe2430 in __kernel_vsyscall ()
(gdb) bt
#0  0xb7fe2430 in __kernel_vsyscall ()
#1  0xb7ca7651 in raise () from /lib/tls/i686/cmov/libc.so.6
#2  0xb7caaa82 in abort () from /lib/tls/i686/cmov/libc.so.6
#3  0xb766152f in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#4  0xb765f465 in ?? () from /usr/lib/libstdc++.so.6
#5  0xb765f4a2 in std::terminate() () from /usr/lib/libstdc++.so.6
#6  0xb765f5e1 in __cxa_throw () from /usr/lib/libstdc++.so.6
#7  0xb7b231e5 in Swig::DirectorMethodException::raise (this=0x8665878, aState=apl::dnp::SS_COMMS_DOWN) at DNP3Java/PythonDNP3.cpp:2994
#8  SwigDirector_IStackObserver::OnStateChange (this=0x8665878, aState=apl::dnp::SS_COMMS_DOWN) at DNP3Java/PythonDNP3.cpp:7148
#9  0xb7a1c44f in apl::dnp::Master::UpdateState (this=0x86389a0, aState=apl::dnp::SS_COMMS_DOWN) at DNP3/Master.cpp:113
#10 0xb7a1ccf8 in Master (this=0x86389a0, apLogger=0x865fc58, aCfg=..., apAppLayer=0x86388bc, apPublisher=0x82efbb8, apTaskGroup=0x85cc748, apTimerSrc=
    0x8519218, apTimeSrc=0xb7acacf8) at DNP3/Master.cpp:105
#11 0xb7a2133d in MasterStack (this=0x8638470, apLogger=0x85c7908, apTimerSrc=0x8519218, apPublisher=0x82efbb8, apTaskGroup=0x85cc748, arCfg=...)
    at DNP3/MasterStack.cpp:30
#12 0xb79f5210 in apl::dnp::AsyncStackManager::AddMaster (this=0x8519208, arPortName=..., arStackName=..., aLevel=apl::LEV_INFO, apPublisher=0x82efbb8,
    arCfg=...) at DNP3/AsyncStackManager.cpp:119
#13 0xb7a5b1b3 in apl::dnp::StackManager::AddMaster (this=0x85872e0, arPortName=..., arStackName=..., aLevel=apl::LEV_INFO, apPublisher=0x82efbb8, arCfg=...)
    at DNP3/StackManager.cpp:66
#14 0xb7b03044 in _wrap_StackManager_AddMaster (args=
    (<StackManager(this=<SwigPyObject at remote 0x85a72c0>) at remote 0x860356c>, 'tcpclient', 'master', 8, <DataObserver(this=<SwigPyObject at remote 0x85a7290>, newData=False) at remote 0x8600ccc>, <MasterStackConfig(this=<SwigPyObject at remote 0x85a72d8>) at remote 0x860532c>)) at DNP3Java/PythonDNP3.cpp:34031

[William]

Never mind that.  Of course it was failing when trying to call the callback; I had defined the callbacks incorrectly.  (The stupidest mistakes are the hardest to find.)  I fixed that, and fixed it for the other functions.  The stack state starts out as COMMS_DOWN, and doesn't change for three minutes.  Then the AcceptCommand method is called and the program seg faults as before.  I'm going to use Wireshark Network Analyzer to see what messages are being sent back and forth.

[William]

The master is sending out a connection establish request, and the slave device is responding that the destination is unreachable, the protocol is unreachable.

[William]

The device I'm using does not support DNP3 over TCP.  Only over UDP . . .

[William]

I'm sorry for so many posts in a row . . . but here is the final situation.  I ran an application on the port to convert TCP to UDP and vice-versa, so the computer and the device could talk to each other.  The master successfully sends out a class 0 poll, and the device responds.  Then the program seg faults.  Here is the output and some of the back trace.  The bold line is the output from my OnStackChange callback.  It looks like it was in the process of being called again with "COMMS_UP" when everything crashed.  Once this error is dealt with, I should have working Python master demo.

The stack state has changed: COMMS_DOWN

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread 0xb74f7b70 (LWP 15498)]
0x08064508 in PyObject_Call (callable=<instancemethod at remote 0xb7c1770c>, args=(0,)) at ../Objects/abstract.c:2490

2490    ../Objects/abstract.c: No such file or directory.
    in ../Objects/abstract.c

(gdb) bt
#0  0x08064508 in PyObject_Call (callable=<instancemethod at remote 0xb7c1770c>, args=(0,)) at ../Objects/abstract.c:2490
#1  call_function_tail (callable=<instancemethod at remote 0xb7c1770c>, args=(0,)) at ../Objects/abstract.c:2524
#2  0x080678f4 in PyObject_CallMethod (o=<StackObserver(this=<SwigPyObject at remote 0x85a7278>) at remote 0x8576d8c>, name=0xb7b6bd88 "OnStateChange",
    format=0xb7b6b199 "(O)") at ../Objects/abstract.c:2601
#3  0xb7b230bf in SwigDirector_IStackObserver::OnStateChange (this=0x8665878, aState=apl::dnp::SS_COMMS_UP) at DNP3Java/PythonDNP3.cpp:7143
#4  0xb7a1c44f in apl::dnp::Master::UpdateState (this=0x86389a0, aState=apl::dnp::SS_COMMS_UP) at DNP3/Master.cpp:113
#5  0xb7a1d69a in apl::dnp::Master::ProcessIIN (this=0x86389a0, arIIN=...) at DNP3/Master.cpp:122
#6  0xb7a1e1ce in apl::dnp::Master::OnFinalResponse (this=0x86389a0, arAPDU=...) at DNP3/Master.cpp:290
#7  0xb79ec35e in apl::dnp::AppLayerChannel::DoFinalResponse (this=0x8638930, arAPDU=...) at DNP3/AppLayerChannel.cpp:115
#8  0xb79ea954 in apl::dnp::ACS_Base::ProcessResponse (this=0xb7acada0, c=0x8638930, arAPDU=..., aExpectFIR=true) at DNP3/AppChannelStates.cpp:91
#9  0xb79eac73 in apl::dnp::ACS_WaitForFirstResponse::OnResponse (this=0xb7acada0, c=0x8638930, arAPDU=...) at DNP3/AppChannelStates.cpp:260
#10 0xb7a5a050 in apl::dnp::SolicitedChannel::OnResponse (this=0x8638930, arAPDU=...) at DNP3/SolicitedChannel.cpp:52
#11 0xb79f0565 in apl::dnp::AppLayer::OnResponse (this=0x86388b0, arCtrl=..., arAPDU=...) at DNP3/AppLayer.cpp:210

[Sam Hendley]

I wish I could help you more, you've done well to get through all of the issues you've had. Double check that your listeners are all not getting garbage collected. Also check the python threading model, I know in ruby you couldn't have a c++ thread call into ruby interpreter, all data had to buffered and then we let a ruby thread pull that data out. 

Sam

[William]

The instances of my classes are not being garbage collected, and the execution crashes whenever the library tries to call my Python callbacks (OnStackChange in the class which inherits from IStackObserver and _Start in the class which inherits from IDataObserver).  What I don't get is that it can successfully call the OnStackChange once, but crashes again when trying to call the same function.  Perhaps I've not implemented something that that function should be doing.  All it currently does is print a message.

CPython does have a GIL, Global Interpreter Lock, to prevent multi-threaded Python programs from damaging each other, because Python is not thread safe.  I don't think that would be causing the errors I'm getting, but I can't rule that out yet.

[Adam Crain]

Hi William,

At this point at might be worth getting a second pair of eyes on the entire demo program. Can you fork open-dnp3 and then put your python demo into the source tree?

I've written a tiny bit of python and maybe something will pop out at me.

-Adam

[William]

Adam,

Do I need to be a contributor to do that?  I also haven't used git before, so it could take me a little while to figure out.

Will

[Adam Crain]

No, anyone can fork a public project into their own namespace. You'd be committing to your copy of the project not GECs copy.

As to the Git question, I leave it up to you. If you can email me a zip of the python source that would be fine as well.

-Adam

[William]

I've emailed it to you.

[Adam Crain]

Btw, after a quick look, I agree that your objects shouldn't be getting collected, they're all still referenced on the stack... got to be something else funny going on here. I don't 100% trust swig. The autogenerated bindings for Java aren't nearly as good as the hand crafted ones for .NET b/c of memory management issues.

--

Adam Crain

Chief Platform Architect, GEC

Office: 919.836.9916

Cell:  919.428.1002

acr...@greenenergycorp.com

Green Energy Corp, enabling the smart grid of the future.   

www.greenenergycorp.com

www.totalgrid.org

[William]

I've worked with SWIG before, and I think it tends to do its own thing with memory.  When it comes to callbacks and such its own thing may not be the best thing.

I'm not sure if this is related, but the only real error I see in the back trace (aside from the seg fault notification) is "../Objects/abstract.c: No such file or directory,"

[Chris Verges]

I have no idea where that file is coming from.  It isn't an OpenDNP3 file...perhaps swig/python related?

[William]

I searched my entire file system for abstract.c and found nothing.  That could be the problem.  Either that the file is missing, or that it's looking for it in the first place.

[Chris Verges]

I wonder why a .c file is required at runtime...

[Chris Verges]

Looks like some kind of python binding:

http://www.velocityreviews.com/forums/t326095-c-swig-and-mpatrol.html

Maybe the autotools make for python swig isn't quite linked properly.  I can look further once I am back in the office next week.  Feel free to hack up a Makefile change if desired.  I can review any patches made.

Chris

On Jun 13, 2012, at 10:50, William <semb...@gmail.com> wrote:

[William]

I'll take a look at what I can do.

[William]

Just to put things in perspective, this is a fairly complicated thing the execution is choking on.  C++ is trying to call the "concrete" function of a Python class inheriting from a C++ abstract class wrapped with SWIG.

[William]

When I run autoreconf -i -f and then ./configure --with-python, part of the output from configure is "LIBS: -lpython2.6 -lpython2.6 -lpython2.6 -lpthread -lc".  I manually removed the duplicates from the Makefile, rebuilt, and the errors persisted as one would expect because the duplicate link commands should be ignored.  Yet their is a bit odd.

[William]

The issue outlined here seems to be very similar to what I'm running into.  http://stackoverflow.com/questions/6872701/python-to-c-from-deriv-to-base-to-deriv-again  Something might stick out to you, Chris, when you're back in the office.

[Chris Verges]

Actually, Adam may have the best insight on this one.  We do use pointers in the code, but I am not familiar enough with the oddities behind C++s use of them to know what strange things happen.  If we need to refactor the code to use boost::function callbacks or references or whatever, that's a bigger job.

[William]

I a correct in that the Python bindings are generated by the file DNP3Java/JavaDNP3.i?

[Chris Verges]

Just found this reference:

http://www.swig.org/Doc1.1/HTML/Python.html

Read the section "Implementing C callbacks in Python".  I think the swig bindings need to be updated specifically for Python.  (My original attempt used Java .i unmodified.)

Chris

On Jun 13, 2012, at 17:13, William <semb...@gmail.com> wrote:

[Chris Verges]

Yes

[William]

Implementing that is complicated by the fact the the callback is a virtual function...

[William]

I think the approach outlined in the document you linked, Chris, is roughly as follows:

1) Typemap a Python function to a Python object.
2) Create a C/C++ function (SpecialCallback) which calls the Python function passed to it via a pointer.  The function definition matches the prototype of the callback in the C++ class.  This function is invisible to Python
3) Add a function to the C++ class (SetPyCallback) which takes as an argument the Python function to call.  It sets SpecialCallback to be the class's callback, and sets a pointer in the C++ class to the Python function.

This then works during execution as follows:

1) Write the callback Python function (PyCallback).
2) Create an instance of the C++ class in Python.
3) Call SetPyCallback, passing to it PyCallback.  This ensures the C++ class has a pointer to PyCallBack, and sets SpecialCallback as the callback.
4) C++ code calls SpecialCallback, passing to it a pointer to PyCallback.  SpecialCallback calls PyCallback.

Now, the problem is that the callback function is virtual, so I don't think it's possible to create SpecialCallback.  I'm not very familiar with C++, but I believe it would have to be implemented in a class which inherits from the virtual class.  It would also be necessary to add pointers in the virtual classes to point to the Python callbacks.

So, would the "cleanest" way to do this would be to create C++ "wrapper classes" for the current virtual classes?  And then wrap those for Python?

[William]

EDIT: What I completely do not understand is that the C++ code manages to call the Python callback once before crashing.  It shouldn't be able to find it at all.

[William]

Is fixing the issue with the bindings something the development team will be able to/want to do in within a few weeks?  If not, I need the Python ability so I will go ahead and do it for my copy of the code.  If you guys have a specific approach in mind, I can use that, see how it works, and hopefully save you some work.

Thanks,
Will

[Christopher Verges]

Hi William,

I can't speak for everyone else, but I'm not planning to work on this anymore.  The Python bindings were a "nice-to-have" since no one had specifically requested it ... think of it as a way to gauge whether people were really interested in it.  However, I don't actually use Python, so have no personal incentive to spend the time required to create a proxy frontend for it.

That all being said, if you or someone else is willing to put in the time to code it up, I'm happy to provide a supporting role (a-la autotools support, advice, etc.)

I wonder if a "quick" solution might be found in something like JPype or Jython might be useful as a bridge, leveraging the Java Swig output in a Python interpreter.

Chris

[Adam Crain]

Echoing Chris's sentiment, these bindings are not on our priority list. The next binding for us that needs attention is the Java binding to fix threading/memory issues everyone is experiencing.

-Adam

[William]

Hello Adam and Chris,

Okay, that's fine.  I'll go ahead and see what I can do.  Thanks for all the help you guys have given and for you offer of support, Chris.

[William]

I've found the problem, and the Python master demo is now running without issue.  The bindings were created correctly, and the concept of Python classes extending C++ classes is sound (SWIG supports that), but Python is not thread-safe.  However, C++ was trying to call the callbacks from a separate thread.  All that is required to fix the issue is add the option -threads to the swig command.  This has more information http://matt.eifelle.com/2007/11/23/enabling-thread-support-in-swig-and-python/  Hopefully it will be smooth sailing from here on.  Should I email the demo to one of you?

Thanks,

[Chris Verges]

Nice!  Great sleuthing!  You can send a patch or pull a fork on github and issue a pull request.

[Christopher Verges]

Hi William,

I've pushed the -threads change up.

https://github.com/cverges/dnp3/commit/7a73a3362d46e1afa7eadac6f5d834dd2c32e508

Chris

[William]

Thank you, I'm glad I was able to find a fix.  How should I communicate the Python demo to the project?  Send a pull request, post it here, or email it to one of you?

[Chris Verges]

Feel free send it to me directly and I can add it to git.  We should probably get you familiar with github so you can do your own commits, too.

[Christopher Verges]

Hi all,

Thanks to William, we now have a demo of the Python bindings for OpenDNP3!  The demo is available in this commit:

https://github.com/cverges/dnp3/commit/32ba554f01e43ce058ac801d7d65af7a4914301a

Adam, I've issued a pull request via github for this plus all the other changes involved in making it work.  Please merge into gec/master when you have a chance.

William, great job!

Chris

[Aditya N]

Hi all,

This was amazing thread to read with so many ups and downs in writing python demo. At one point it looked everyone has given up but William's work is truly laudable. I am planning to use this, so can anyone please my question

My work on DNP3 is very very limited. I need to just create a master and perform integrity test to find all the connected slave devices in my network( Having prior no knowledge of slave devices). Can I use this demo and my purpose will be resolved? ( I didn't purchase a DNP3 device yet so want to make sure I have working code before doing it). The reason I ask this is when I run this python demo I see Binaries, Analogs, Counters, ControlStatus, Setpointstatus count instead of slave device information/count or any characterisation of slaves( or I suspect the code mentions count of 1 slave device points then should I modify something in the code if I have multiple slave devices?)

I did a similar discovery procedure with BACnet and Modbus where I used 70 and 120 property instances for bacnet slave device discovery and Function code 43 for Modbus slave devices identification. With those codes, I could get connect slave device vendor name and model name automatically. Just wondering if this discovery process is different in the case of DNP3 or can I do slight modifications in python demo file to get access to the slave devices information. Any suggestion really helps days/weeks of my time and I heartfully thank for help. I hope this is not a dead thread. Looking forward..


Thanks and Regards,
Aditya

[Daniel Evans]

Aditya,

Discovery is not a traditional feature of the DNP3 protocol. The protocol specification does have a relatively new set of objects for device information (group 0) but these objects are not implemented by this project. This is true of the C++ interface as well as the python bindings.

--
You received this message because you are subscribed to the Google Groups "open-dnp3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-dnp3+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Daniel Evans

Software Engineer

Green Energy Corp

o (919) 836-9916

dev...@greenenergycorp.com

 

Transforming the Smart Grid

www.greenenergycorp.com