Setting up file transfer using dnp3 library

[Christopher Verges]

Hi group,

I'm new to both DNP3 and the open source library at http://code.google.com/p/dnp3.  I'm looking for a little bit of mentoring and advice on using this DNP3 library to setup a file transfer.  What would be the best approach to use for this?

Thanks in advance,
Chris

[Adam Crain]

Hi Chris,

File transfer Group80 I think is not currently implemented in the
stack... it's a pretty rarely used feature. Most vendors implement
firmware upgrades, etc via proprietary tools and a parallel channel. I
have only seen a few devices that actually implement it. If you'd like
to implement it, I'd be happy to support your efforts.

At one point in time we had virtual terminal objects implemented which
is a more generic concept. It allows you to tunnel any stream oriented
(TCP) over DNP3. Pretty useful if you want to do FTP over a serial
port or the like. We tunneled ssh over 802.15.4 wireless.

Let me know more about your use case and I can provide a
recommendation. The user group at dnp.org would also have good
feedback on which feature is more appropriate.

-Adam

On Apr 14, 4:58 pm, Christopher Verges <chris.ver...@gmail.com> wrote:
> Hi group,
>

> I'm new to both DNP3 and the open source library athttp://code.google.com/p/dnp3.  I'm looking for a little bit of mentoring

[Christopher Verges]

Hi Adam,

Thanks so much for the quick reply.  The use case is a firmware upgrade, as you mention.  I'd prefer to use a standards-based approach if possible, so would be interested in identifying what would be involved in extending the DNP3 library to handle "File transfer Group80."

Doing some kind of tunnel-over-DNP3 like FTP/HTTP/SSH/SFTP would be an acceptable alternative for my immediate needs.  Does this feature still exist?

I'd love to join the UG at dnp.org, but their price tag is a bit too high at the moment.  I was happy to see an open source community exists for those of us without deep pockets.  :-)

Thanks again,
Chris

[Adam Crain]

Hi Chris,

If you have embedded Linux with ssh, I believe VTO is an easier route.
What it was designed for was tunnel proprietary protocols (like a
vendors programming tool) over DNP. It's actually *easier* to
implement then File Transfer because you just need to handle an opaque
stream instead of all the function codes and types.

Unfortunately, neither is part of our level 2 implementation. Both
features are either level 3 or 4. We had VTO implemented in an old low-
performance synchronous version of the stack, but didn't port it when
we refactored for the release. If you're interested in adding it, I
can further discuss what's involved with you.

You can't even join the discuss forum for free? That stinks! I'll ask
this question on your behalf tomorrow. I'm also on the dnp3 technical
committee.

-Adam

[Christopher Verges]

Hi Adam,

VTO does sound like a better idea.  My initial thoughts on this would be to actually stream HTTP over VTO (using a Linux TUN/TAP style device?) to enable file transfer capabilities.  I don't fully understand what you mean by level 2 vs 3 vs 4, but I am more than happy to aid in coding whatever might be required to support VTO.  Your thoughts on this are welcomed and appreciated!

Regarding dnp.org, much of it seems to be membership-based.  See the following link:  http://www.dnp.org/Modules/Store/Membership.aspx.  It's unfortunate that the discussion forum, email list, and protocol specs are not publicly available, as such could significantly aid in the adoption, proliferation, and interoperability of the protocol.  However, I have learned that business decisions don't always make sense to the engineer in me.  :-)

Thanks,
Chris

[Adam Crain]

Hi Chris,

Levels are just subsets of all the functionality the protocol offers.
This is how the protocol spec categorizes functionality to give
vendors a way to decide what's core and what's peripheral.

The biggest obstacle to implementing VTO is that I unfortunately we
can't just give you the specifications :(. I am in the process of
trying to work something out with dnp.org or turning totalgrid.org
into a legal entity that would allow us to distribute this material to
"members" for the purpose of helping with the protocol effort.

You'd really need a copy of the protocol spec to implement this
functionality. What's your timeline for needing this functionality?

In the meantime, to give you an idea of the work involved.... Instead
of the stack pushing sequence of measurements in the correct order, it
is pushing opaque blobs of bytes in the write order and ensuring
delivery).

The object type for VTO would be defined here, you'd have to
understand the object hierarchy and a bit about the byte structure:

https://github.com/gec/dnp3/blob/master/DNP3/Objects.h

You'd use a datastructure like this one for keeping track of the
stream blobs:

https://github.com/gec/dnp3/blob/master/DNP3/AsyncEventBuffers.h#L59

I will upload the dnp3 quick reference to the project page as that is
freely distributable but tells you a little about the structure of the
protocol.

-Adam

On Apr 14, 7:49 pm, Christopher Verges <chris.ver...@gmail.com> wrote:
> Hi Adam,
>

[Chris Verges]

Hi Adam,

I was able to get a hold of the DNP3 spec late last night, and that helped
tremendously to put some teeth in my understanding of the protocol. I
read up on the Virtual Terminal Objects section, and while am not nearly
an expert, have a beginning working knowledge.

I'll dive into the two links you provided and start working my way through
it. I'm sure more questions will come soon. :-)

Thanks!
Chris

[Chris Verges]

Hi Adam,

I've been perusing through the Objects.h file, and see the Group80Var1 and
Group[110-113]Var0 definitions around line 1019
(https://github.com/gec/dnp3/blob/master/DNP3/Objects.h#L1019). From the
DNP3 standard in section 5.2.2, am I to understand that these are the
relevant objects for the job? I believe that object group 112 is for
master -> outstation flow, and object group 113 is for outstation ->
master flow. Does the presence of these definitions, however, indicate
that VTO support might already exist? Or at least, do most of the
required components already exist?

The ideal implementation would be compatible with Section 5.2.4
"Discontinuous octet streams," as it would allow for maximum compatibility
with protocols riding on top of the VTO connection.

Any thoughts on the number of maximum octets per message as per Section
5.2.6? A customizable macro seems like the best option, so that different
implementations can simply override with their desired size.

Thanks,
Chris

On 4/15/11 11:51 AM, "Adam Crain" <jadam...@gmail.com> wrote:

[Sam Hendley]

Hi Chris, there are objects in the Objects.h file that we fully support but by having them there we are at least able to parse requests containing the objects and return a sensible error message. Supporting VTO requires a few components that I know have been removed from the implementation. Our first implementation was threaded and synchronous and used a number of threads for DNP3 connection and when we converted the stack to be fully asynchronous we didnt have time to port all of the VTO components to the boost asio framework.

There is one major component that needs to be written, a router between socket connections and VTO ports. When we had implemented VTO before we had a "router" process on both the master and slave side of the stack. Either the slave or master side could have listening ports (socket servers) or client ports (connecting to a listening service). When the side with the listening port got a new connection it send a VTO packet with size 0 which would prompt the other side to attempt a connection to the tunneled listening service. Whenever data was received locally by the router it was converted into the VTO data object, mapped to the correct VTO index and then sent using the same rules for DNP3 event data (usually unsolicited). When the stack on the other side gets VTO data it unpacks it, routes it to the correct connection and sends it out the local socket. We used a size 0 packet to indicate open/close events to the local socket though I don't believe this was in the specification. Without it you are forced to use some other out-of-band mechanism to indicate when a socket should be closed or opened (which is usually hard or impossible with arbitrary protocols). They may have updated this part of the spec. since we implemented VTO originally so there may be a better way to handle this.

Other work that would have to be recreated are the event buffering, configuration structs to configure VTO and hooks to get the data in and out. Porting the old implementation to the new framework is probably much easier that redoing the work from scratch, I'll talk to the original authors and see if we can send you the last version that had VTO support so you have a reference.

Sam

[Chris Verges]

Hi Sam,

Thanks for the explanation!  Any help you can give on identifying the original port work would be extremely appreciated.

I agree with you that a "router" component is needed, but have further questions about what it might look like.  In my mind, this "router" component is very similar to htun, a Linux application (http://linux.softpedia.com/get/System/Networking/HTun-14751.shtml).  htun uses two libraries external to itself — curl and sockets — to create an HTTP-based VPN tunnel from the client to the server.   The entire IPv4 packet received from the client socket side (outstation) would be encoded into an HTTP request and then sent over HTTP to the concentrator (master) on the other side.  The master would then decode the HTTP request and pass on the IPv4 packet to its final destination.  The actual application (VPN tunnel) is decoupled from the two underlying libraries (curl's HTTP implementation and Linux's network sockets implementation.)  A similar system (dnp3tun, if you will?) might provided similar benefits, whereby standard IP protocols like SSH and HTTP could be used across a DNP3 network backbone.  And Linux applications could use the standard TUN/TAP virtual networking device to quickly and easily add support to existing network daemons for DNP3 through this tunnel/router system.

A little bit of protocol framing might be required to ensure that fragmented packets are reassembled properly before being forwarded.  HTTP provided that implicitly in the htun example above, but we can develop that higher-level control protocol later.

Three questions that come to mind out of our discussion at this point:

1. Is the implementation of the socket-to-VTO router component something that is best done outside or inside of the DNP3 library?
2. If outside, then are there any special API hooks in the DNP3 library needed by the router?
3. Are there any "standards" (de facto or de jure) in this area that we should be keeping in mind, such as DNP3 addressing conventions, IP-to-DNP3 address mappings, etc.?

Thanks again!  I look forward to the pointer to the VTO-supported version of the library and any additional insights you and the rest of the mailing list have.

Chris

[Sam Hendley]

That sounds pretty neat, not something I would have thought to try. It may be the wrong way to attack the issue at first, fundamentally we need some sort of inter-process-communication to get the data and connect/disconnect signals into the DNP3 stack so it can be packaged as VTO and sent down the line. Then we have to use some IPC to get that data back out to appropriate place on the other side of the connection. I think sockets are the simplest and most useful, they are well supported on nearly every platform. The semantics are also very similar, its a bi-direction stream of data that can't instantly report sending failures but if the connection stays up the data will be received in order.

If we had socket-to-VTO support in the stack then creating the dnp3tun application wouldn't be very hard, it could connect to the one exposed socket on the local side, wrap the locally received data with some routing structure and then the remote dnp3tun router would unpack and send that data to the correct port. The VTO structure just doesn't have enough data to fully wrap the IP protocol, there is only 16 (32?) bits of addressing using the index, you need atleast 48 bits to encode an IP address:port combination.

I responded to your specific questions below:

On Mon, Apr 18, 2011 at 9:13 AM, Chris Verges <chris....@gmail.com> wrote:

Hi Sam,

Thanks for the explanation!  Any help you can give on identifying the original port work would be extremely appreciated.

I agree with you that a "router" component is needed, but have further questions about what it might look like.  In my mind, this "router" component is very similar to htun, a Linux application (http://linux.softpedia.com/get/System/Networking/HTun-14751.shtml).  htun uses two libraries external to itself — curl and sockets — to create an HTTP-based VPN tunnel from the client to the server.   The entire IPv4 packet received from the client socket side (outstation) would be encoded into an HTTP request and then sent over HTTP to the concentrator (master) on the other side.  The master would then decode the HTTP request and pass on the IPv4 packet to its final destination.  The actual application (VPN tunnel) is decoupled from the two underlying libraries (curl's HTTP implementation and Linux's network sockets implementation.)  A similar system (dnp3tun, if you will?) might provided similar benefits, whereby standard IP protocols like SSH and HTTP could be used across a DNP3 network backbone.  And Linux applications could use the standard TUN/TAP virtual networking device to quickly and easily add support to existing network daemons for DNP3 through this tunnel/router system.

A little bit of protocol framing might be required to ensure that fragmented packets are reassembled properly before being forwarded.  HTTP provided that implicitly in the htun example above, but we can develop that higher-level control protocol later.

Three questions that come to mind out of our discussion at this point:

1. Is the implementation of the socket-to-VTO router component something that is best done outside or inside of the DNP3 library?

I think inside, that way the stack is self contained and works to tunnel single ports on any device with IP port support (UDP could also work). 

1. If outside, then are there any special API hooks in the DNP3 library needed by the router?

If not using sockets we would need a different IPC communication method to shunt the data into and out of the stack. Could be done as well as sockets if necessary for performance reasons, though the nature of DNP3 is always going to be a bigger bottleneck than that.

 

1. Are there any "standards" (de facto or de jure) in this area that we should be keeping in mind, such as DNP3 addressing conventions, IP-to-DNP3 address mappings, etc.?

You need to keep in mind that DNP is usually deployed in hardened environments with regulatory and technical requirements to minimize access. VTO is not commonly implemented or enabled because of those risks, but when it is enabled, there is a very specific set of ports that are available for VTO. Probably the most common use case was hooking up VTO to a physical serial port for remote management. So the VTO index which we would think of as representing the port doesn't need to have any relation to the IP or serial port we are tunneling to. The mapping between IP and DNP3 indexes is therefore manually done and agreed upon when the device is commissioned.

[Chris Verges]

Hi Sam,

I apologize, it sounds like I started the conversation down a slightly separate thread.  The key first step is implementing VTO support in the DNP3 library.  Once we have that implemented as an atomic entity, we can then look at what to do as far as deploying/using it.

I'll put the horse blinders on and focus on the VTO implementation first.  :-)  Please let me know if you are able to find the pre-rewrite implementation of this!

Thanks,

[Chris Verges]

(PREFACE:  I HAVE NOT YET SUBMITTED THE CONTRIBUTOR LICENSE AGREEMENT.  AT THIS TIME, THE CODE SAMPLES CONTAINED IN THIS EMAIL ARE FOR EXAMPLE AND EDUCATIONAL PURPOSES ONLY.  A FINAL PATCH SET WILL BE SUBMITTED FOR INCLUSION IN THE PROJECT REPOSITORY ONCE THE CLA HAS BEEN SUBMITTED AND APPROVED.)

Hi Adam and Sam (and anyone else that jumps in),

I'm going through the older code (thanks for sending it!) to identify the components related to VTOs.  I started where Sam suggested, in EventBuffer.h with the InsertionOrderEventBuffer class.  In the current git trunk, AsyncEventBuffers.h defines a AsyncInsertionOrderedEventBuffer class.  I'm assuming that this is the proper class to extend/modify.

The key difference between InsertionOrderEventBuffer and AsyncInsertionOrderedEventBuffer seems to be a missing _Update() function override.  I made a first attempt at porting over these changes:

index 423ea82..aae0bd4 100644

--- a/DNP3/AsyncEventBuffers.h

+++ b/DNP3/AsyncEventBuffers.h

@@ -61,6 +61,11 @@ namespace apl { namespace dnp {

  public:

 

  AsyncInsertionOrderedEventBuffer(size_t aMaxEvents);

+

+ void _Update(const EventType& arEvent);

+ private:

+

+ int_32_t mNextEvent;

  };

 

 

@@ -76,7 +81,8 @@ namespace apl { namespace dnp {

 

  template <class EventType>

  AsyncInsertionOrderedEventBuffer<EventType> :: AsyncInsertionOrderedEventBuffer(size_t aMaxEvents) :

- AsyncEventBufferBase<EventType, InsertionOrderSet2< EventType > >(aMaxEvents)

+ AsyncEventBufferBase<EventType, InsertionOrderSet2< EventType > >(aMaxEvents),

+ mNextEvent(0)

  {}

 

  template <class EventType>

@@ -99,6 +105,26 @@ namespace apl { namespace dnp {

  }

  }

 

+ template <class EventType>

+ void AsyncInsertionOrderedEventBuffer<EventType> :: _Update(const EventType& arEvent)

+ {

+ if (this->mEventSet.size() == 0 && this->mSelectedEvents.size() == 0)

+ mNextEvent = 0;

+

+ /*

+ * When the object is first inserted, this flag will be 0, so we set it

+ * to the next value.  This allows us to guarantee that the set keeps

+ * the events in the correct order when we put the selected events

+ * back in the main buffer on failures.

+ */

+ if (arEvent.mLastEventValue == 0) {

+ const_cast<EventType&>(arEvent).mLastEventValue = ++mNextEvent;

+ }

+

+ this->mEventSet.insert(arEvent);

+ this->mCounter.IncrCount(arEvent.mClass);

+ }

+

 }} //end NS

 

 #endif

index fa8e0e8..5f24638 100644

--- a/DNP3/EventTypes.h

+++ b/DNP3/EventTypes.h

@@ -34,13 +34,15 @@ struct EventInfo : public PointInfoBase<T>

        EventInfo(const T& arValue, PointClass aClass, size_t aIndex) :

        PointInfoBase<T>(arValue, aClass, aIndex),

        mSequence(0),

-       mWritten(false)

+       mWritten(false),

+       mLastEventValue(0)

        {}

 

-       EventInfo() : mSequence(0), mWritten(false) {}

+       EventInfo() : mSequence(0), mWritten(false), mLastEventValue(0) {}

 

        size_t mSequence;       /// sequence number used by the event buffers to record insertion order

        bool mWritten;          /// true if the event has been written

+       size_t mLastEventValue; /// the last event value if using an insertion ordered buffer

 };

 

 typedef EventInfo<apl::Binary> BinaryEvent;

As an initial code review, how does this look as far as porting the event buffer code?

The DNP3Test/Linux_i686/release/dnp3test.exe suite passes with 298 test cases (no errors detected), but admittedly it passed before the changes as well.  Does this mean the changes ported from the older code were not needed?  Or that the test cases need to be extended to test something else?

Before I go through the VTOInterfaces, VTORouter and VTOToPhysicalLayerThread code, let's make sure the easy questions are wrapped up.  :-)  Looking forward to your feedback!

Thanks,

Chris

[sam.h...@gmail.com]

I don't know if anything was using that buffer in the current stack, there was a test case for the insertion order buffer in the code I sent. I would port that and see if it passes this test. Trying running the test suite with --help, it will show you the options you can use to run a single suite or test case.

Sam

On Apr 18, 2011 9:24pm, Chris Verges <chris....@gmail.com> wrote:
> (PREFACE:  I HAVE NOT YET SUBMITTED THE CONTRIBUTOR LICENSE AGREEMENT.  AT THIS TIME, THE CODE SAMPLES CONTAINED IN THIS EMAIL ARE FOR EXAMPLE AND EDUCATIONAL PURPOSES ONLY.  A FINAL PATCH SET WILL BE SUBMITTED FOR INCLUSION IN THE PROJECT REPOSITORY ONCE THE CLA HAS BEEN SUBMITTED AND APPROVED.)
>
>
> Hi Adam and Sam (and anyone else that jumps in),
>
>
> I'm going through the older code (thanks for sending it!) to identify the components related to VTOs.  I started where Sam suggested, in EventBuffer.h with the InsertionOrderEventBuffer class.  In the current git trunk, AsyncEventBuffers.h defines a AsyncInsertionOrderedEventBuffer class.  I'm assuming that this is the proper class to extend/modify.
>
>
> The key difference between InsertionOrderEventBuffer and AsyncInsertionOrderedEventBuffer seems to be a missing _Update() function override.  I made a first attempt at porting over these changes:
>
>
> index 423ea82..aae0bd4 100644
> --- a/DNP3/AsyncEventBuffers.h
> +++ b/DNP3/AsyncEventBuffers.h
> @@ -61,6 +61,11 @@ namespace apl { namespace dnp {
>   public:
>  
>   AsyncInsertionOrderedEventBuffer(size_t aMaxEvents);
> +
> + void _Update(const EventType& arEvent);
> + private:
> +
> + int_32_t mNextEvent;
>   };
>  
>  
> @@ -76,7 +81,8 @@ namespace apl { namespace dnp {
>  
>   template

>   AsyncInsertionOrderedEventBuffer :: AsyncInsertionOrderedEventBuffer(size_t aMaxEvents) :
> - AsyncEventBufferBaseEventType, InsertionOrderSet2 >(aMaxEvents)
> + AsyncEventBufferBaseEventType, InsertionOrderSet2 >(aMaxEvents),
> + mNextEvent(0)
>   {}
>  
>   template

> @@ -99,6 +105,26 @@ namespace apl { namespace dnp {
>   }
>   }
>  
> + template

> + void AsyncInsertionOrderedEventBuffer :: _Update(const EventType& arEvent)

> + {
> + if (this->mEventSet.size() == 0 && this->mSelectedEvents.size() == 0)
> + mNextEvent = 0;
> +
> + /*
> + * When the object is first inserted, this flag will be 0, so we set it
> + * to the next value.  This allows us to guarantee that the set keeps
> + * the events in the correct order when we put the selected events
> + * back in the main buffer on failures.
> + */
> + if (arEvent.mLastEventValue == 0) {

> + const_cast(arEvent).mLastEventValue = ++mNextEvent;

> + }
> +
> + this->mEventSet.insert(arEvent);
> + this->mCounter.IncrCount(arEvent.mClass);
> + }
> +
>  }} //end NS
>  
>  #endif
>
> index fa8e0e8..5f24638 100644
> --- a/DNP3/EventTypes.h
> +++ b/DNP3/EventTypes.h
> @@ -34,13 +34,15 @@ struct EventInfo : public PointInfoBase

>         EventInfo(const T& arValue, PointClass aClass, size_t aIndex) :

>         PointInfoBase(arValue, aClass, aIndex),

>         mSequence(0),
> -       mWritten(false)
> +       mWritten(false),
> +       mLastEventValue(0)
>         {}
>  
> -       EventInfo() : mSequence(0), mWritten(false) {}
> +       EventInfo() : mSequence(0), mWritten(false), mLastEventValue(0) {}
>  
>         size_t mSequence;       /// sequence number used by the event buffers to record insertion order
>         bool mWritten;          /// true if the event has been written
> +       size_t mLastEventValue; /// the last event value if using an insertion ordered buffer
>  };
>  

>  typedef EventInfo BinaryEvent;

[Chris Verges]

Hi Adam and Sam and all,

After further reviewing the test case for the insertion order buffer, I decided to rewrite a decent portion of the patch.  I've attached it here as a patch file for review.

I ported the ResetEventsProperlyOnFailure test case in TestEventBuffers.cpp.  However, I didn't fully understand the changes between the pre-APL code and current code to port the other test cases yet.

None of the VTORouter or VTOToPhysicalLayerThread code is ported yet.  I was hoping someone could explain what these older files do and how they interface to the external system before I begin this work.

Thanks,

Chris

[Adam Crain]

Hi Chris,

I'm reviewing this morning. If you're comfortable doing so in the
future, just fork the project on Github. That's the easiest way for us
to review changes.

thanks,
Adam

>  vto-20110419-1220.patch
> 39KViewDownload

[Chris Verges]

Hi Adam,

Done! I've never actually used Github, very cool interface. Fork is
available from https://github.com/cverges/dnp3.

Thanks,
Chris

[Adam Crain]

Great. I can see the fork, let me know when you've applied your patch
on top of it. If you need any help, let me know.

I am now reviewing the InsertionOrderedEventBuffer class as I believe
this is currently not being used and was originally taylored
specifically for VTO.


-AC

On Apr 21, 10:16 am, Chris Verges <chris.ver...@gmail.com> wrote:
> Hi Adam,
>
> Done!  I've never actually used Github, very cool interface.  Fork is

> available fromhttps://github.com/cverges/dnp3.
>
> Thanks,
> Chris

[Chris Verges]

Hi Adam,

Applied. I was having some Internet issues, so the email went out before
the commit. Apologies for the weird timing.

While you're reviewing, if real-time IM is easier, I'm on GTalk. Feel
free to reach out if needed.

Thanks for the help!

Chris

[Adam Crain]

Hi Chris,

This is almost identical to what we previously had, and I appreciate
the extra test case!

One thing I would have differently based on experience with the 1st
implementation is to keep VTO data as a separate, parallel interface.

I.E. removing update(VTOData&)... from IDataObserver/ChangeBuffer and
create a second parallel interface for writing data into the stack.
Change buffer could implement both interfaces our you can use another
class.

We can then add functions to AsyncStackManager to register VTO
interfaces to a configured/running stack:

AsyncStackManager {
IVTOWriter* RegisterVTOPort(const std::string& arStackName, byte_t
aVariation? , IVTOReader* apOnDataCallback);
}

Loose pseduo-code above, but does this idea make sense? Under the
hood, all of the changes to the stack internals make total sense, but
I would like to keep measurements as a seperate interface from VTO
data externally, expecially since you can be proxying multiple
distinct streams using different variations (ports).

Next steps:

1) You'll also have to add an event buffer of some sort to the Master
too since it's a bi-directional stream. The master will need to wake
up, read from the buffer, and do a write operation.

2) The VTOWriter* interface (or whatever you want to call it) may need
to throttle based on the space in the event buffer. Otherwise the
event buffer will overflow and you'll drop data. This could be
accomplished using a blocking write or by waking the router when
there's space and having it only read as much data from it's socket as
space exists in the event buffer.

3) We won't need any explicit thread synchronization between the
router and the stack if the router uses the same asychronous
"io_service" provided by AsyncStackManager. This also has the
advantage that we can write single threaded integration tests for the
whole system, end-to-end!


thanks!
Adam

On Apr 21, 10:23 am, Chris Verges <chris.ver...@gmail.com> wrote:
> Hi Adam,
>

> Applied.  I was having some Internet issues, so the email went out before
> the commit.  Apologies for the weird timing.
>
> While you're reviewing, if real-time IM is easier, I'm on GTalk.  Feel
> free to reach out if needed.
>
> Thanks for the help!
>
> Chris
>

[Chris Verges]

Hi Adam,

Very welcome! The VTO port that Sam sent was extremely helpful in doing
the rote work involved.

Regarding your suggestion, I like it. A basic skeleton might be helpful
in making sure that I understand the structure desired. I'll hack
something together and commit it to the fork later today.

By the way, is there a Developer API Reference that discusses the purposes
and relationships of all the various classes involved? I'm planning to
add a significant amount of doc work to the VTO classes involved in this
new effort, but want to make sure that this work is stylistically related
to anything that might already exist.

Thanks,
Chris

[Adam Crain]

Hi Chris,

I'll fork your changes and add skeleton interfaces that might serve
our purposes, but I'll do it tomorrow morning off the clock as we have
a holiday. We can then use the skeleton approach to converge on an API
for VTO.

We we have auto-generated docs produced by doxygen <rake document>
(you'll need to have doxygen on the path). Normally these show up on
the Hudson CI server, but the server is currently down. We badly need
to migrate this project over to our community EC2 node running
TeamCity.

We do not have a prose-like developer guide yet. Our best attempt at
this so far has been the example programs. We've used DocBook (http://
www.docbook.org/whatis) for another project with good results. It's
can render to HTML, pdf, etc. If we want to go down, it's easy to add
this as a task in rake. Any ideas on this are appreciated.

-Adam

On Apr 21, 11:28 am, Chris Verges <chris.ver...@gmail.com> wrote:
> Hi Adam,
>

> Very welcome!  The VTO port that Sam sent was extremely helpful in doing
> the rote work involved.
>
> Regarding your suggestion, I like it.  A basic skeleton might be helpful
> in making sure that I understand the structure desired.  I'll hack
> something together and commit it to the fork later today.
>
> By the way, is there a Developer API Reference that discusses the purposes
> and relationships of all the various classes involved?  I'm planning to
> add a significant amount of doc work to the VTO classes involved in this
> new effort, but want to make sure that this work is stylistically related
> to anything that might already exist.
>
> Thanks,
> Chris
>

[Chris Verges]

Hi Adam,

OK Step 1 should be done. I pushed updates to the fork that should remove
the VtoData portion from IDataObserver, ChangeBuffer, Database, Slave,
etc. This is in preparation for the second step of creating skeleton code
for the reader/writer handlers.

Also, I updated the doxygen config file to exclude the tinyxml and boost
source files, since they're outside the DNP3 code base. I wasn't sure if
these sources were included on purposes, but figured it was an easy thing
to revert. :-)

Thanks,
Chris

[Chris Verges]

Hi Adam,

Sounds good. I'm also on PTO, so good to hear that the DNP3 group is
apparently where all the geeks go when we're not supposed to be geeking
out. :-)

Thanks,
Chris

[Chris Verges]

Hi Adam,

I've been considering what API might be needed. As I understand the DNP3
protocol, VTO operates using Object Group 112 for master -> outstation and
Object Group 113 for outstation -> master. If that's the case, then will
we need to setup separate registration handlers?

Also, since the Virtual Channels feature of VTO will most likely involve
forking data streams into multiple directions, should we set it up so that
you register separate channels with separate handlers?

As you work on this more tomorrow, here's the code version of some of my
thoughts:

namespace apl { namespace dnp {
class AsyncStackManager {
public:

/**
* The virtual channel ID is defined when the application
* instantiates the apOnDataCallback object. A corresponding
* writer object will be created for that channel ID.
*/
IVtoMasterWriter* AddMasterVtoChannel(const std::string& arStackName,
IVtoMasterReader* apOnDataCallback,
size_t
reservedOctetCount = 0);
IVtoSlaveWriter* AddSlaveVtoChannel(const std::string& arStackName,

IVtoSlaveReader* apOnDataCallback,
size_t
reservedOctetCount = 0);
}

class IVtoHandler {
public:

IVtoHandler(uint_8_t channelId);
~IVtoHandler();

uint_8_t GetChannelId() { return channelId; }

private:

uint_8_t channelId;
}

class IVtoWriter : private IVtoHandler {
public:

void Send(const byte_t& data, size_t length);

size_t GetReservedOctetCount() { return reservedOctetCount; }
void SetReservedOctetCount(size_t count) { reservedOctetCount = count; }

private:

/**
* The reserved octet count is used as a dynamic throttle control
* mechanism. In any outbound DNP3 Application Layer message,
* a corresponding number of octet will be reserved for VTO-related
* data. Note that this reserved count corresponds to the VTO
* data only, and does not include the object header information.
*
* VTO is considered a "low-priority" message type, and could easily
* overwhelm the DNP3 application layer. Such an event could result
* in time-sensitive data (such as point values) being dropped or
* become useless. Barring the creation of a more generic Quality
* of Service (QoS) mechanism, this is a first attempt at that.
* The VTO data will be dropped or postponed if other data is
* in the queue, waiting to be sent. To ensure that SOME VTO data
* sneaks through, however, you can reserve a certain number of
* octets in the application layer for VTO.
*/
size_t reservedOctetCount;
}

class IVtoReader : private IVtoHandler {
public:

/**
* Blocks until data is received, then fills the data buffer and
* sets the length of the data received (up to maxLength). If
* block is set to false, the call does not block. The function
* returns the number of bytes stored into the data buffer.
*/
size_t Recv(const byte_t& data,
size_t maxLength,
bool block = true);
}

class IVtoMasterWriter : private IVtoWriter { ... }
class IVtoSlaveWriter : private IVtoWriter { ... }

class IVtoMasterReader : private IVtoReader { ... }
class IVtoSlaveReader : private IVtoReader { ... }
}

The existing RemoveStack() should be able to process the removal of the
VTO stacks. And I agree, using the io_service makes a lot of sense.

I'll merge the above code into my fork later today so that it's ready for
you tomorrow.

Looking forward to hearing your thoughts!

Thanks,
Chris

[Adam Crain]

Hi Chris,

Nice commit, great start. I forked and made some recommended changes.
Let me explain each commit:

https://github.com/jadamcrain/dnp3/commit/775991b7e57d01d9419c3878cc9fd92ed56b2dea

Just windows portability. Don't worry about this at all. I'm happy to
drag compatibility along since you're developing on Linux?
Just note the windows doesn't have <strings.h> so I used a macro for
bzero.

https://github.com/jadamcrain/dnp3/commit/61d17076e3e56f8268f1e2b38d1f729723562bc9

Moved the interfaces to DNP3 library since I think they're specific to
DNP3. We've actually used the data interfaces before for Modbus so
that's why that stuff is in APL (Async Protocol Library).

https://github.com/jadamcrain/dnp3/commit/3a1bf60f37acf83ff546202817c278871a1256c6

OK, so here's the big one. A few explanations.

1) I don't think that there needs to be a distinction between the
interfaces used for master/slave read/write. From the perspective of a
user it's just a bidirectional stream of bytes. I collapsed the
interfaces into 3 classes:

IVtoBase, IVtoWriter, and IVtoReader ->renamed-> IVtoCallbacks.

2) If you want to use io_service to run the router or write an
application using it, there can't be any blocking behavior because
there's only 1 thread. I added a method to IVtoCallbacks that
notifies when there's space available. The write function never
throws, but might might return a number less than the bytes requested.
When we write the router component, it can then use that callback to
determine how much to transfer and can use the operating system to
cache, only reading bytes when space is available to write.

https://github.com/jadamcrain/dnp3/commit/54f71d2c976902f1fe66d476a7001dd222a286d6

1) Added a remove function to disassociate a previously registered
IVtoCallbacks so we can start/stop dynamically

2) Added a pair of methods that raise the abstraction level for the
use case of redirecting VTO to one of our standard physical layers:

StartVtoRouter/StopVtoRouter.

The flow to configure a master would be the following:

1) Add the port for the master stack
2) Add the master
3) Add the port for Vto bridge (probably a TCPServer)
4) Start a VTO router using the port name in 3) and the master name in
2)

Connect to the TCP listener/port in 3) and the stack should be trying
to tunnel traffic the remote outstation.

Both sets of functions serve different use cases. If someone doesn't
want to use a canned physical layer, they can use the lower-level
interfaces you defined.


Thoughts?

Adam

[Chris Verges]

Hi Adam,

Great changes.  I've merged them into the fork.

Yes, I am developing on Linux.  I'm not as familiar with what's needed for Windows, but will try to keep it in mind as I go forward.  Thanks for agreeing to help prop me up in this area.  :-)

To start, I have a general question about the startup sequence that you described.  I get the following:

1. Add port
2. Add master or slave stack
• Stack gets associated to the port in Add[Master|Slave]()
3. Add VTO channel
• Channel gets associated to the stack in AddVtoChannel()
4. Start VTO router

The API proposed in the final commit seem to have some overlap a little, where ports and stacks are used in StartVtoRouter() that seem to have already been defined before AddVtoChannel() is called.  To enforce the API pattern, I've proposed a few changes.

Also, in IVtoWriter(), I noticed that Write() is virtual.  Is it intended that implementers will be extending this class?  It could have just been an omission during the code changes, so I wanted to check.

https://github.com/cverges/dnp3/commit/00828480f0a8ccbdec4ba7e897eed0f379156946

Consolidated API to enforce usage pattern.

The API should be called in the following order (example shown):

1. AsyncStackManager::AddSerial(portName)

2. AsyncStackManager::AddMaster(portName, stackName)

3. AsyncStackManager::AddVtoChannel(stackName, channelId)

4. StartVtoRouter(stackName)

The VtoRouter exists on the stack-level, which has already been

associated with a port (serial, TCP, etc.) As such, the only remaining

item is to define which virtual circuit ID will be used for the stack.

Then, of course, start the router.

https://github.com/cverges/dnp3/commit/4193097a7689be3485e9b6ff3bef1d178ad2175b

Please let me know your thoughts!

Thanks,

Chris

[Chris Verges]

Hi Adam,

I'm still working my way through the various classes in the DNP3 directory so that I know how to implement the VTO feature.  Here's what I have so far in the way of understanding:  (don't worry, it'll be short!)

class APDU

• This class represents a DNP3 Application Layer message.
• WriteIndexed() seems to be the proper one to use for sending Group112VarX and Group113VarX objects.

class ObjectBase

• Get() creates the Group112VarX and Group113VarX objects, used by class APDU.

class IVtoWriter, class VtoMasterWriter, and class VtoSlaveWriter

• This class writes a byte stream from the application of arbitrary length to an APDU instance.
• The stream is split into segments of at most 255 characters.  The minimum size can vary from 0 to aReservedOctetCount, as determined by the available space in the APDU instance.  (Unclear how to ensure that aReservedOctetCount is made available if a VTO transmission is pending.)
• Package each segment into a Group112VarX (if VtoMasterWriter) or Group113VarX (if VtoSlaveWriter) instance.

class IVtoCallbacks

• This class notifies the implementing application of newly received data.
• The application should then do whatever it needs with the data, though it should do it quickly, as this blocks other DNP3 functions from being processed.

class Master

• Master::ProcessDataResponse() seems to process a received class APDU instance.
• Various other functions like Master::OnUnsolResponse() seem to trigger other paths, unclear if they eventually converge back to Master::ProcessDataResponse().

class Slave

• Slave::Send() and Slave::SendUnsolicited() are used to tell class AppLayer to write the class APDU instance.
• Slave::CreateResponseContext() appears to create a class APDU instance, perhaps usable by class VtoSlaveWriter?  (Does this mean that Slave->Master communications will be VTO-packets-only — this is, the APDU instance will only hold VTO objects, no other types?)
• How does the class Slave instance receive VTO data?

OK, to be fair, a little longer than I first estimated.  :-)

Thanks!

Chris

From: Chris Verges <chris....@gmail.com>
Date: Fri, 22 Apr 2011 11:35:51 -0700
To: <open...@googlegroups.com>

[Adam Crain]

Hi Chris,

I think I missed the mark in my earlier explanations, so I'll take
another shot. As proposed, AddVtoChannel() and StartRouter() would
never be used *together*. They're two parallel ways of doing the same
thing at 2 different levels of abstraction. Let me elaborate:

1) If I understand your intent, AddVtoChannel()/RemoveVtoChannel() are
intended to provide access to a raw byte stream interface to a VTO
stream. Client code is responsible for handling the byte stream
itself! My assertion is that this is will be a less frequent use case
than a simple physical layer redirect like 2).

2) The intent of StartRouter()/StopRouter() is to provide a high-level
bridge/router from a predefined (by name) physical layer to a Vto
address on a particular stack (master or slave). The client code never
has to worry about handling the bytes themselves... A router object is
created an managed internally to do all of this work. No prior call to
AddVtoChannel() is required, the stack does all the work of setting it
up.

Yes. I made all of the interface functions pure-virtual ( = 0). I
intended that implementer would extend. This is the pattern we've used
throughout the stack, so I'd recommend it for consistency. We'd have
the interfaces, the production implementations, and mocks we'd us to
do unit testing.

Let me know if that makes sense.

Adam

[Chris Verges]

Hi Adam,

Umm, then I don't think I quite understood. My impression was that the
two worked together. So in the [Add|Remove]VtoChannel scenario, how do
the DNP3 data link through application layers work? And in the
[Start|Stop]Router scenario, how does the client code publish and receive
the VTO data?

Thanks,
Chris

[Adam Crain]

Hi Crhis,

Hi Chris,

In the [Add|Remove]VtoChannel scenario, client code uses the
interfaces (IVtoWriter/IVtoCallbacks) to interact with the stream and
the stack. This can all be done in memory within the same process.

In the [Start|Stop]Router scenario, the stack redirects the Vto stream
to the named physical layer. In this scenario, the client code doesn't
interact with the stream at all once it's started (think SSH
tunneling!). Here's an example using the later and the proposed API.

1) Create a port for the DNP3 connection using AddTCPClient() (connect
to IP REMOTE_ADDR on port 20000)
2) Create a master stack using AddMaster() and name the port in 1)
3) Create a port for the Vto tunnel using AddTCPServer() say port 22
4) Start up a router attached to the master and associated with the
port in 3)

What this does:
1) & 2) start up a dnp3 master talking to a remote outstation on port
20000 TCP
3) & 4) starts up a TCP listener on localhost:22 that is bound to a
Vto channelId on the master

When a process connects to localhost:22, the stream is redirected over
Vto to the outstation. If the outstation is configured to do something
with that Vto address (like redirect to localhost 22 with a TCP
Client), then we're able to tunnel ssh over DNP3.

Does that help at all? The [Start|Stop]Router functions are for
redirecting byte streams to any physical layer the stack supports
(TCP[Server|Client], Serial). The [Add|Remove]Channel functions
provide direct access to a stream so you can do something custom with
it besides redirection (i.e. directly handle it in your application
code).

Adam

On Apr 23, 2:07 pm, Chris Verges <chris.ver...@gmail.com> wrote:
> Hi Adam,
>

> Umm, then I don't think I quite understood.  My impression was that the
> two worked together.  So in the [Add|Remove]VtoChannel scenario, how do
> the DNP3 data link through application layers work?  And in the
> [Start|Stop]Router scenario, how does the client code publish and receive
> the VTO data?
>
> Thanks,
> Chris
>

> >>the...
>
> read more »

[Chris Verges]

Hi Adam,

So the equivalent would be as follows?

[Outstation] <---(tcp/ip)---- [Master 12,345]
[TCP 20,000] [ | ]
[ Router ]
[SSH Client] [ | ]
[TCP 23,456] ----(tcp/ip)---> [ Master 22 ]

I guess my initial thoughts were that the "router" functionality is
something completely separate from the library, so didn't expect to find
it embedded in the DNP3 directory. Maybe like a "DNP3Router" subproject.
But OK, this explanation has helped me to understand things better.

So StartVtoRouter() will create its own set of IVto[Writer|Callbacks] and
manage things, right? For actual masters and oustations that need to
interpret the VTO streams, they would make their own
Ivto[Writer|Callbacks] instances and go from there.

Does this interpretation jive with the library's actual usage?

Thanks,
Chris

[Adam Crain]

That's exactly it!

What file transfer protocol will you be using in your application to
upload new firmware?

I think that the redirect is such a common use case that I'd like to
embed so folks don't have to worry about the details. If you don't
need that functionality, I'm happy to implement it using the
IVtoWriter/IVtoCallbacks interfaces we defined.

thanks,
Adam

On Apr 23, 4:55 pm, Chris Verges <chris.ver...@gmail.com> wrote:
> Hi Adam,
>

> So the equivalent would be as follows?
>
>    [Outstation] <---(tcp/ip)---- [Master 12,345]
>    [TCP 20,000]                  [      |      ]
>                                  [    Router   ]
>    [SSH Client]                  [      |      ]
>    [TCP 23,456] ----(tcp/ip)---> [  Master 22  ]
>
> I guess my initial thoughts were that the "router" functionality is
> something completely separate from the library, so didn't expect to find
> it embedded in the DNP3 directory.  Maybe like a "DNP3Router" subproject.
> But OK, this explanation has helped me to understand things better.
>
> So StartVtoRouter() will create its own set of IVto[Writer|Callbacks] and
> manage things, right?  For actual masters and oustations that need to
> interpret the VTO streams, they would make their own
> Ivto[Writer|Callbacks] instances and go from there.
>
> Does this interpretation jive with the library's actual usage?
>
> Thanks,
> Chris
>

> >> >> >One thing I would have differently based on...
>
> read more »

[Chris Verges]

Hi Adam,

OK, I'll revert the changes in my fork either tomorrow or Monday. We
should be ready for another code skeleton review by Monday afternoon.
Thanks for your patience and help in explaining all this.

The exact protocol hasn't been selected yet. Obviously, support for the
standard set would be nice (IPv6 as a generic stack, HTTP, FTP, RSYNC,
etc.)

Regarding the master connecting with multiple outstations, I know we
discussed this in some depth on Friday. However, in thinking about it a
little more today, I may have confused myself again. :-) I'll keep
thinking about it and post something more succinct next week.

Thanks,
Chris

[Chris Verges]

Hi Adam,

Three commits pushed up to my fork:

https://github.com/cverges/dnp3/commit/d5704f3669b4b039b3ffb4c0963a18b63f13eb36

Added VtoMasterWriter and VtoSlaveWriter classes. We will need to create the appropriate instance during AsyncStackManager::AddVtoChannel() depending on the stack type defined by arStackName.

https://github.com/cverges/dnp3/commit/45cfbba8f28c8f32b5f6f462e305d796ef14f025

Fixed small syntax errors.

https://github.com/cverges/dnp3/commit/40770e612dc2e873fef07a1d03e242bcf724019b

Removed arPortName from AddVtoChannel() since this function terminates the VTO stream for a stack. Changed StopVtoRouter() declarations to operate based on the arStackName, not the arPortName.

On the last commit, I refactored StopVtoRouter() to operate based on the stack and channel ID rather than the daemon port.  This is open to change.  What I wasn't sure about was whether a port could be shared across stacks and virtual channel IDs.  We should allow the implementer to stop individual VTO virtual channels.  If this means the arPortName is a unique identifier for each virtual channel, effectively, then I'm OK changing back to that.  If the same arPortName can be use for multiple virtual channels, then we will need to specify something else.

Please take a look at the changes and let me know your thoughts.

Thanks,

Chris

[Adam Crain]

Hi Chris,

Answers below.

On Apr 22, 6:14 pm, Chris Verges <chris.ver...@gmail.com> wrote:
> Hi Adam,
>

> I'm still working my way through the various classes in the DNP3 directory
> so that I know how to implement the VTO feature.  Here's what I have so far
> in the way of understanding:  (don't worry, it'll be short!)
>
> class APDU

> * This class represents a DNP3 Application Layer message.
> * WriteIndexed() seems to be the proper one to use for sending Group112VarX
> and Group113VarX objects.

This is where my memory starts slipping. I just talked to Sam and he
recalls that the index encoded the "port" of the VTO object, and the
variation encoded the length (0-255). I believe that WriteIndexed()
is the correct function, but you'll have to do some unit testing here
and comparison to the protocol specification.

> class ObjectBase
> * Get() creates the Group112VarX and Group113VarX objects, used by class
> APDU.

I'm not sure if you need this function or not. Just use the existing
read/write handler for existing objects as a guide.

> class IVtoWriter, class VtoMasterWriter, and class VtoSlaveWriter

> * This class writes a byte stream from the application of arbitrary length
> to an APDU instance.
> * The stream is split into segments of at most 255 characters.  The minimum

> size can vary from 0 to aReservedOctetCount, as determined by the available
> space in the APDU instance.  (Unclear how to ensure that aReservedOctetCount
> is made available if a VTO transmission is pending.)

I think that aReservedOctetCount might be better as a Slave/Master
configuration parameter (Minimum amount from ANY VTO stream to put
into each APDU). It will be really hard to do this on a per stream
basis.

> * Package each segment into a Group112VarX (if VtoMasterWriter) or

> Group113VarX (if VtoSlaveWriter) instance.
> class IVtoCallbacks

> * This class notifies the implementing application of newly received data.
> * The application should then do whatever it needs with the data, though it

> should do it quickly, as this blocks other DNP3 functions from being
> processed.
> class Master

> * Master::ProcessDataResponse() seems to process a received class APDU
> instance.
> * Various other functions like Master::OnUnsolResponse() seem to trigger

> other paths, unclear if they eventually converge back to
> Master::ProcessDataResponse().

I believe that all of those functions return to ProcessDataResponse.
This is where you'll read the VTO objects out of the APDU and convert
them to array[byte].

> class Slave
> * Slave::Send() and Slave::SendUnsolicited() are used to tell class AppLayer

> to write the class APDU instance.

> * Slave::CreateResponseContext() appears to create a class APDU instance,

> perhaps usable by class VtoSlaveWriter?  (Does this mean that Slave->Master
> communications will be VTO-packets-only ‹ this is, the APDU instance will
> only hold VTO objects, no other types?)

No, the responses from the slave (either polled or unsolicted) will
contain a mixture of measurement data and VTO objects. The exact
mixture will be a function of:

1) VTO ReserverOctetCount
2) Events/VTO available
3) Maximum size of an APDU

> * How does the class Slave instance receive VTO data?

> OK, to be fair, a little longer than I first estimated.  :-)

Masters write VTO objects to the outstation using the FC_WRITE
function. You'd have to add a handler here:

https://github.com/gec/dnp3/blob/master/DNP3/Slave.cpp#L301

>
> Thanks!
> Chris
>

[Chris Verges]

Hi Adam,

Thanks for the response! Addition follow up inline as well ...

On 4/25/11 7:42 AM, "Adam Crain" <jadam...@gmail.com> wrote:

>On Apr 22, 6:14 pm, Chris Verges <chris.ver...@gmail.com> wrote:
>> Hi Adam,
>>
>> I'm still working my way through the various classes in the DNP3
>>directory
>> so that I know how to implement the VTO feature. Here's what I have so
>>far
>> in the way of understanding: (don't worry, it'll be short!)
>>
>> class APDU
>> * This class represents a DNP3 Application Layer message.
>> * WriteIndexed() seems to be the proper one to use for sending
>>Group112VarX
>> and Group113VarX objects.
>
>This is where my memory starts slipping. I just talked to Sam and he
>recalls that the index encoded the "port" of the VTO object, and the
>variation encoded the length (0-255). I believe that WriteIndexed()
>is the correct function, but you'll have to do some unit testing here
>and comparison to the protocol specification.

Let's get clear on terminology, since "port" seems to be overloaded
already. :-) As I understand it, a "port" in the GEC library is a
physical layer connection, such as a serial port or a TCP port. A
"virtual channel" in the DNP3 specification is similar to a TCP or UDP
port, allowing for multiplexing of the VTO mechanism. Does this match
with your understanding? If so, then I agree with your explanation if we
change "port" to "virtual channel."

>>class ObjectBase
>> * Get() creates the Group112VarX and Group113VarX objects, used by class
>> APDU.
>
>I'm not sure if you need this function or not. Just use the existing
>read/write handler for existing objects as a guide.

OK. Where can I find an example of this existing mechanism?

>> class IVtoWriter, class VtoMasterWriter, and class VtoSlaveWriter
>> * This class writes a byte stream from the application of arbitrary
>>length
>> to an APDU instance.
>> * The stream is split into segments of at most 255 characters. The
>>minimum
>> size can vary from 0 to aReservedOctetCount, as determined by the
>>available
>> space in the APDU instance. (Unclear how to ensure that
>>aReservedOctetCount
>> is made available if a VTO transmission is pending.)
>
>I think that aReservedOctetCount might be better as a Slave/Master
>configuration parameter (Minimum amount from ANY VTO stream to put
>into each APDU). It will be really hard to do this on a per stream
>basis.

Agreed. I will change the MasterConfig and SlaveConfig objects to reflect
this.

>>* Package each segment into a Group112VarX (if VtoMasterWriter) or
>> Group113VarX (if VtoSlaveWriter) instance.
>> class IVtoCallbacks
>> * This class notifies the implementing application of newly received
>>data.
>> * The application should then do whatever it needs with the data,
>>though it
>> should do it quickly, as this blocks other DNP3 functions from being
>> processed.
>> class Master
>> * Master::ProcessDataResponse() seems to process a received class APDU
>> instance.
>> * Various other functions like Master::OnUnsolResponse() seem to trigger
>> other paths, unclear if they eventually converge back to
>> Master::ProcessDataResponse().
>
>I believe that all of those functions return to ProcessDataResponse.
>This is where you'll read the VTO objects out of the APDU and convert
>them to array[byte].

OK. Master::ProcessDataResponse() will be the main change location, then.

>>class Slave
>> * Slave::Send() and Slave::SendUnsolicited() are used to tell class
>>AppLayer
>> to write the class APDU instance.
>> * Slave::CreateResponseContext() appears to create a class APDU
>>instance,
>> perhaps usable by class VtoSlaveWriter? (Does this mean that
>>Slave->Master
>> communications will be VTO-packets-only ‹ this is, the APDU instance
>>will
>> only hold VTO objects, no other types?)
>
>No, the responses from the slave (either polled or unsolicted) will
>contain a mixture of measurement data and VTO objects. The exact
>mixture will be a function of:
>
>1) VTO ReserverOctetCount
>2) Events/VTO available
>3) Maximum size of an APDU
>
>> * How does the class Slave instance receive VTO data?
>

>Masters write VTO objects to the outstation using the FC_WRITE
>function. You'd have to add a handler here:
>
>https://github.com/gec/dnp3/blob/master/DNP3/Slave.cpp#L301

Sounds good! So I'll need to add some case(MACRO_DNP_RADIX(112,0))
statements or the proper equivalent to the Slave::HandleWrite() function.

Thanks,
Chris

[Chris Verges]

https://github.com/cverges/dnp3/commit/fb724cb7b343ee0eeb7d57b452099fb2ce61
07a9

aReservedOctetCount moved to MasterConfig and SlaveConfig. The VTO
reserved octet count is now specified on the stack level for VTO as a
whole; each virtual channel will need to share this reserved count.

Chris

On 4/25/11 7:42 AM, "Adam Crain" <jadam...@gmail.com> wrote:

[Chris Verges]

Hi Adam,

Based on this and our IM conversation, I started looking at TestAPDU.cpp
and TestAPDUWriting.cpp to see what modifications to test cases would be
needed. Please see the following for further details.

In short, the current behavior of the APDU class on line 250
(https://github.com/cverges/dnp3/blob/master/DNP3/APDU.cpp#L250) is such
that only one VTO objects (either Groups 112 or 113) is allowed in a
single APDU message during the parsing operation. However, we can write a
DNP3 message such that multiple VTO packets can be sent.

What is the expected behavior? As best as I understand the DNP3 spec,
there is no limitation on the number of VTO objects allowed in a DNP3
application layer message. If that is the case, then it seems like
APDU.cpp needs to be modified to support this. Otherwise, is there
anything we can do on the writing-side to maintain consistency with the
APDU parser?

Thanks,
Chris

Relevant Commits:

https://github.com/cverges/dnp3/commit/8138a428227a2872aa062d8e04294813902a
8577

Added new test case VirtualTerminalWriteMultipleIndices to
TestAPDUWriting.cpp.

https://github.com/cverges/dnp3/commit/5547d13ee4d3106b80535e2707be59c7f478
a3ae

Added new test case VtoObjectBadWriteMultipleIndices and error code
ALERR_TOO_MANY_VARIABLE_OBJECTS_IN_HEADER.

[Chris Verges]

Hi Adam,

Thanks for confirming that this is an artificial limitation. I've removed
the count check and added a new test case VtoObjectWriteMultipleIndices to
TestAPDU.cpp.

https://github.com/cverges/dnp3/commit/5e9f75571099b843f07d96fd486e0ea3c46e
81b4

Chris

[Adam Crain]

I started another thread here because I was having trouble following
the length of the previous one.

1) I think we figured this out yesterday, but YES WriteIndexed is
correct and your additional tests look good.

2) We can add Group11XVar0 to static ObjectBase* ObjectBase::Get(int
aGroup, int aVariation), but it's largely irrelevant since we don't
have object types ( or need ) object types for all of the variations.
When the APDU is parsed, it will just throw a Var0 in as a placeholder
instead of a specific object, but the variation is recorded separately
so we can recover the length later when we read out the bytes.

-Adam

On Apr 25, 10:53 am, Chris Verges <chris.ver...@gmail.com> wrote:
> Hi Adam,
>

> Thanks for the response!  Addition follow up inline as well ...
>