Multiple outstations

[Raghavan Viswanathan]

Hi,

I am running the master-demo on one window and running a outstation in another window. I see all the data going from the outstation to master.

The outstation is on default port 20000.

How can I simulate like say 5 outstations ? Is that possible ?

I realise, we could like change the port to 21000, 22000...25000 and start 5 instances of outstation

Is that the right way ?. In reality do each of the dnp3 devices have a unique port assigned to them ?

What do you reckon is a smart way of simulating multiple outstations ?

Thanks.

Bye,

Raghavan V

[J Adam Crain]

On a single machine, I think the multiple port approach, (e.g. 20000, 20001, 20002) is best.

In a real system they're all on the same port, but have different IP addresses.

-Adam

[Nikola Bajić]

When instancing DNP3Manager for purpose of many connections, should I allocate one thread per each connection, using concurrency parameter? In my use case there can easily be more than 1000 connections.

[J Adam Crain]

No, never allocate more threads than a small multiple (1-2) of the actual parallelism (# of cores) on your machine. All of the I/O in opendnp3 is non-blocking, so multiple threads is really only for the purpose of capturing multi-core performance, not for executing multiple sessions as all of the sessions would behave just fine on a single thread.

This is captured in the docs here:

https://www.automatak.com/opendnp3/docs/guide/current/api/architecture/

-Adam

[jithend...@gmail.com]

Adam,

What if I have 1000s of outstations running in a single machine ? Incase each outstation has to have an unique port number as you suggest, does it mean I have to create separate channels for each outstation ? Or can I have all outstations listening on the same channel (i.e same ip and port) ?

What is the best approach ? Anyway each outstation will be addressed by unique local address (incase of same ip and port). 

Thanks,

Jithendar

On Tuesday, June 28, 2016 at 12:07:50 AM UTC-4, Raghavan Viswanathan wrote:

[Adam Crain]

One outstation, lots of addresses:

Pros: Uses only a single listener

Cons: Master can't communicate with each outstation simultaneously.

Multiple outstations on different channels:

Pros: Master can communicate with each outstation in parallel

Cons: Requires the usage of multiple ports

[jithend...@gmail.com]

I am trying simulate 1000's masters and outstations on different ports on the same machine. This configuration works fine for upto 10 outstations. But when I increase it to 100 outstaions, I face following issues :

1) The TCP connection takes upto 3 minutes to get established (with 3 or 4 O/Ss still trying to connect) ? Is this expected ?

2) I am assuming the reason for task 1 being slow is because as soon as the connection is established it also starts handshaking tasks like disabling/enabling usolicited, clear restart and enabling unsolicted. I have configured master and outstaion only to disable unsolicted on the startup. I am not sure why others tasks are executing (restart and enabling unsolicted).

3) I am also seeing a bunch of  following warnings when I try to send some commands:

Task was explicitly rejected via response with error IIN bit(s): Disable Unsolicited 

WARN    master99 - Link status request - response timeout

WARN - Response with bad sequence: 0

WARN - Not expecting a response, sequence: 0

Header: 0 Index: 1 State: INIT Status: UNDEFINED Received command result w/ summary: FAILURE_START_TIMEOUT

Header: 0 Index: 1 State: INIT Status: UNDEFINED Received command result w/ summary: FAILURE_RESPONSE_TIMEOUT

Even though I see few of these (1 and 2) in case of 10 outstations, all the commands sent from master are succesfuuly received at the outstation. The problem is when I exceed more than 10 outstaions. It doesn't write to any of the outstations and I receieve a bunch of timeout failure response.

Here's my master configuration (in c++) :

asiodnp3::DNP3Manager& getManager() {

static asiodnp3::DNP3Manager *manager = new asiodnp3::DNP3Manager(3, asiodnp3::ConsoleLogger::Create());

return *manager;

}

public:

//Constructor

PVMaster(std::string outstationName, int port, int addr) {

auto channel = getManager().AddTCPClient("tcpclient"+ outstationName, FILTERS, ChannelRetry::Default(), "127.0.0.1", "0.0.0.0", port, PrintingChannelListener::Create());

MasterStackConfig stackConfig;

stackConfig.master.responseTimeout = TimeDuration::Minutes(5);

stackConfig.master.taskStartTimeout = TimeDuration::Minutes(5);

stackConfig.master.disableUnsolOnStartup = true;

stackConfig.master.startupIntegrityClassMask = ClassField::None();

stackConfig.link.LocalAddr = addr;

stackConfig.link.RemoteAddr = addr;

master = channel->AddMaster("master"+outstationName, PrintingSOEHandler::Create(), asiodnp3::DefaultMasterApplication::Create(), stackConfig);

master->Enable();

}

public :

void scanRange(){

master->ScanRange(GroupVariationID(30, 5), 8, 14);

}

void classScan() {

master->AddClassScan(ClassField::CLASS_1, TimeDuration::Minutes(1));

}

void setPowerFactor(float pf) {

AnalogOutputFloat32 crob(pf);

master->SelectAndOperate(crob, 17, PrintingCommandCallback::Get());

ControlRelayOutputBlock crob1(ControlCode::LATCH_ON);

master->SelectAndOperate(crob1, 2, PrintingCommandCallback::Get());

}

//Other functions like restart, disable unsolicited and CROB commands

Here's my outstation configuration (in c#):

static IDNP3Manager m_manager = DNP3ManagerFactory.CreateManager(Environment.ProcessorCount);

public static DEROutstation createOutstation(string ipAddress, ushort port, ushort localAddress, ushort remoteAddress, ICommandHandlerCallback callbacks, string id)

        {

            TimeSpan startStop = TimeSpan.FromSeconds(5.0);

            IChannel channel = m_manager.AddTCPServer(id, LogLevels.ALL, startStop, startStop, ipAddress, port);

            OutstationStackConfig config = new OutstationStackConfig();

            config.link.localAddr = localAddress; //Starting from 101

            config.link.remoteAddr = remoteAddress; //Starting from 101

            //config.link.keepAliveTimeoutMs = 3600000;

            config.databaseTemplate = new DatabaseTemplate(76, 1, 74, 4, 0, 46, 917, 11);

            //config.slave.disableUnsol = false;

            //config.outstation.config.allowUnsolicited = true;

            config.outstation.config.allowUnsolicited = false; 

            config.outstation.config.unsolClassMask = ClassField.AllClasses;

            

            //config.outstation.config.selectTimeout = TimeSpan.FromMinutes(5);

            //config.outstation.config.unsolClassMask = ClassField.AllEventClasses;

            // config.outstation.buffer = new EventBufferConfig(10);

            

            for (int index = 0; index < config.databaseTemplate.analogOutputStatii.Count; index++)

            {

                config.databaseTemplate.analogOutputStatii[index].clazz = PointClass.Class0;

                config.databaseTemplate.analogOutputStatii[index].staticVariation = StaticAnalogOutputStatusVariation.Group40Var3;

            }

            //Other database configs

            .............

                       ............

            // DEROutstation derOutstation = new DEROutstation(name);

            DERCommandHandler commandHandler = new DERCommandHandler(callbacks);

            IOutstation outstation = channel.AddOutstation(id, commandHandler, DefaultOutstationApplication.Instance, config);

            outstation.Enable();

            DEROutstation derOutstation = new DEROutstation(id, outstation, commandHandler, callbacks, channel, config, port, ipAddress);

            return derOutstation;

        }

I use loops to create master stations with port starting from 20000 and local and remote address from 101 (same as outstation). Here's a part of the code:

std::cout << "Enter the number of outstations :";

int num=0;

std::cin >> num;

vector<PVMaster> PVMasters;

PVMasters.reserve(num);

std::cout << "Creating outstations....."<< std::endl;

for (int i = 0; i < num; i++) {

PVMasters.emplace_back(std::to_string(i), 20000 + i, 101 + i);

}

std::cout << "Outstations created....."<< std::endl;

switch (cmd)

{

case('a'):

for (int i = 0; i < num; i++) {

PVMasters[i].scanRange();

}

break;

case('d'):

for (int i = 0; i < num; i++) {

PVMasters[i].disableUnsolicted();

}

break;

case('r'): 

for (int i = 0; i < num; i++) {

PVMasters[i].restart();

}

break;

case('x'):

// C++ destructor on DNP3Manager cleans everything up for you

return 0;

case('p'):

{

std::cout << "Enter a powerfactor: ";

float pf = 0.0;

std::cin >> pf;

for (int i = 0; i < num; i++) {

PVMasters[i].setPowerFactor(pf);

}

}

break;

//Other cases

}

I am pretty sure there's nothing wrong with the link addressing. Please note the above config works fine with few outstations and CROB commands are received succesfully. It cannot handle more than 100 outstations. (In my case I have to simulate 1000's of outstations eventually)

I am also seing enabling/disabling unsolicited and clear restart tasks being repeated for all masters/outstations since it is failing everytime and I am assuming that's causing the traffic and doesn't execute any CROB commands later.

I have tried setting the responseTimeout and taskStartTimeout upto 5 minutes but it still doesn't seem to work. I am using a personal machine to run the program. Do you think the performance will improve in a server machine with large no. of outstations? 

I am very new to DNP3 & the library. Thanks in advance for any inputs you have.

[Adam Crain]

Do any of your callbacks block or delay?

I see that you have three threads allocated to your thread-pool. That's fine, but if any callbacks are taking time to complete it could delay the other sessions.

-Adam

[jithend...@gmail.com]

Adam,

 

I started debugging to find which part delayed the process. I realized the createOutstation function(above) in itself is slow in adding all the outstations.  So I made a stopwatch for each line of code in this method. The line which slowed down the process was outstation.Enable();

 

This doesn’t say much so I tracked down the parameters we pass to enable the outstation like config, commandhandler and channel. As I was playing with various parameters, I switched the loglevel parameter in the channel set to “ALL”. When I switched it to NONE or NORMAL, all the outstations were added almost instataneously.

 

IChannel channel = m_manager.AddTCPServer(id, LogLevels.ALL, minRetryDelay, maxRetryDelay, ipAddress, port);

 

I further tested with simple CROB commands for 100 outstations. It was pretty quick as well.

Thanks for your feedback.

[Adam Crain]

Yeah, the debug level logging is VERY verbose. =)

Glad that you have things tuned up now.

[Jim Scavuzzo]

Hi Adam:

Would serial behave the same way in terms of having multiple outstations communicating via a single channel / COM port?

I tried this and it doesn't seem to be communicating in  parallel...

5 outstations communicating,  in parallel,  though a single channel / COM port (via rs485 to rs232), utilizing 1 channel that has 5 different Local Link Addresses and 5 different Remote Link Addresses, a unique pair of Link Addresses for each outstation.

Each of the 5 Masters were created with its own instance of the ManagerFactory and linked accordingly to each outstation.

[Adam Crain]

This configuration is known as "multi-drop" where multiple outstations share common physical transmit/receive lines.

Some rules must be obeyed so that this physical resource is shared a way that doesn't cause contention:

1) The master must wait for responses to requests from one outstation, before making a request to another outstation. Otherwise, the two outstations might both transmit at the same time leading to garbage on the line.

2) Outstations typically should have their unsolicited reporting mode disabled. While the specification has some things in for detecting and avoiding collisions, I've never seen anyone actually use it. The best way to operate in a  multi-drop configuration is to let the master be 100% in control of what communication is happening. This works well is so called "half-duplex RS485" where there's only a single communication line that switches between RX and TX.

Adam