OPC replacement for 3000 tags per second?

[Owen David]

Hi all,

We currently use Kepserver to poll data from an AB compactlogix PLC. The data is then made available via OPC where it is recorded to a database. We handle 280 tags at 10hz so 2,800 tags per second.

Is libplctag a good candidate for replacing Kepserver at this data rate?

I would be using the python wrapper as the database is written in python.

 

[Jody Koplo]

I'm sure Kyle will weigh in on performance, but that seems reasonable to me in my testing.

I've been working on the C# wrapper and from my limited research I suspect most OPC-UA servers use the OPC Foundation's reference implementation (which seems to have been written with a lot of help from Microsoft employees). It's here: https://github.com/OPCFoundation/UA-.NETStandard

There are also some other OPC-UA servers around like this one for Python: https://github.com/FreeOpcUa/opcua-asyncio

It's kind of been a "back of the shelf" idea to see what it would take to smash libplctag and an OPC-UA server implementation together to create a more complete offering. That might be a route to go if you don't want to change your codebase downstream of the OPC-UA server.

I'm curious why you want to move from Kepware? Licensing? Support?

Jody

--
You received this message because you are subscribed to the Google Groups "libplctag" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libplctag+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/libplctag/5f6ed18a-de30-461a-bf12-c6d846cab666n%40googlegroups.com.

[Kyle]

There are many factors in performance.   The library will happily pack as many requests as it can into a 4002 byte packet if it can negotiate one.   There is roughly 10 bytes overhead per packet plus the size of the tag name so getting about 200 tags per request is reasonable if the tag names are not extremely long.   In general 3000 tags per second sounds feasible.  I have run tests that have hit rates far greater than that but those were reading single DINTs from an array.

The main factors I see in performance are:

• The type of the network module.  An L81 CPU with a gigabit controller will smash anything else with very low latency and high throughput.  If the rest of your network can keep up, this will make a huge difference.   If that is not possible, an EN2T is better than an ENBT.   If you are talking to a PLC/5, you can go get a cup of coffee while you wait for the packets to come back.   If you have any ENBTs, just throw them away and get a better network module.  Really.
  
• The network itself can be a big factor.  A lot of industrial networks are held together with spit and baling wire and it shows when you look at packet loss and retransmission counts.   When the network is healthy with zero packet loss, everything is great.  When you start losing packets, the whole thing degrades really badly.   And it does so far more quickly than you might thing.  My suspicion is the network stack is not very good.  Measure your latency.
• Network module load.   Check the load on your network modules.   In my experience if it is over 25-40% CPU, you are probably gaining latency and possibly losing packets just because of the load.   If you have a lot of traffic/requests, this load can build up without any obvious notice.
• The total number of requests.  This is really only a factor when you get high latency from other causes and have so many tags that they do not fit into one or two request packets _and_ that the responses do not fit into one or two packets.
  
• Oddly, the actual size of the tags is not as much of a factor.   It basically costs the same to read 10 DINTS as to read 1.   Obviously 10 will take up a lot more room in the response packet and thus reduce the ability of the library to pack requests into single packets.
  

I am not sure why, but at least in my experience, industrial networks tend to be in fairly poor health.  I have done a lot of projects to fix up PLC control and monitoring where the underlying network health was actually the problem.   Packet loss, route loops, duplicate DHCP controllers, ARP storms etc. all seem to be far more common than you would think.  The fact that all the main network modules except the newest ones built into the L80 series are 100-Mbit and woefully underpowered does not help.  

One way to test is to use the async_stress.c example program.  Modify it to read 300 tags and pass it a typical tag string.   It will try to read 300 tags as fast as possible over and over.  It will log how many milliseconds it took per iteration.   Hit ^C to stop it.   There really is no substitute for testing on your own network.  If you are seeing response times of more than 100ms, you are not going to read your tags at 10hz.   On a direct network to a L80 CPU, I see response times in the low single digit milliseconds.

One thing to keep in mind is that the lowliest PC, even a RaspberryPI, can flood the network module of any PLC easily.   You really need to pay attention to your network module load.  I think that a lot of PLC programmers do not monitor this and tend to flood the network far beyond its carrying capacity.   Using produced/consumed tags can do great things to your network health in some cases.

All that was a very long winded way of saying, "maybe" :-)

Best,

Kyle

[twos...@googlemail.com]

Hello Owen,

If it helps, here are some figures I have for a Logix5575 cpu... approx 50k tags 10sec which seems pretty linear at 22k at 5sec to a OPC-UA server.

None of the data was optimized, so thats individual tags rather than arrays.

I feel the bottleneck was the PLC, messing around with the CPU timeslice and scheduling code within the PLC helped but the PLC was running some really heavy code (1200 - 800ms scan rate).

Kind regards,

Twoshrubs

[Kyle]

Thanks for the numbers, twoshrubs!

In our tests and in prodution, we see that the network activity is independent of the CPU.   We can tell this because we see partially updated arrays and UDTs as if the data was grabbed while the PLC was in mid-scan.   The network module seems to make most of the difference in our experience.   An ENBT will be overrun easily.  And EN2T is definitely better and a full 1Gbps link from a L80 CPU is best.

All that said, I am sure that the CPU load has some sort of impact on the response, particularly if it is using the backplane bus.

From your numbers, it looks like about 4000 second?   That's a fairly good rate!  I will do some tests to see what I can get with 4000 DINTs being read at a time.

Best,

Kyle

[Kyle]

Here is what I see:

./async_stress 8000 'protocol=ab-eip&gateway=10.206.1.40&path=1,4&plc=ControlLogix&elem_count=1&name=TestDINTArray'
Hit ^C to terminate the test.
Creation of 8000 tags took 996ms.
Read of 8000 tags took 902ms.
Read of 8000 tags took 877ms.
Read of 8000 tags took 907ms.
Read of 8000 tags took 879ms.
Read of 8000 tags took 891ms.
Read of 8000 tags took 889ms.
Read of 8000 tags took 889ms.
Read of 8000 tags took 890ms.
Read of 8000 tags took 1011ms.
Read of 8000 tags took 897ms.
Read of 8000 tags took 887ms.
Read of 8000 tags took 896ms.
Read of 8000 tags took 920ms.
Read of 8000 tags took 892ms.
Read of 8000 tags took 888ms.
Read of 8000 tags took 900ms.
Read of 8000 tags took 949ms.
^CRead of 8000 tags took 757ms.
Program terminated!

This is from my laptop on WiFi to a powerline Ethernet network (max about 6MBps) to a 100Mbps switch to a ControlLogix L80 CPU (1Gbps port).   It reads 8000 tags that are single DINT from an array, `TestDINTArray`.   As you can see, that performs about one full read per second.   The program is not doing anything with the data, so if your program does something there will be overhead.

The network is not great.  Here is a quick ping test so you can see how much variation in latency there is:

ping 10.206.1.40
PING 10.206.1.40 (10.206.1.40) 56(84) bytes of data.
64 bytes from 10.206.1.40: icmp_seq=1 ttl=64 time=13.4 ms
64 bytes from 10.206.1.40: icmp_seq=2 ttl=64 time=8.86 ms
64 bytes from 10.206.1.40: icmp_seq=3 ttl=64 time=9.71 ms
64 bytes from 10.206.1.40: icmp_seq=4 ttl=64 time=8.17 ms
64 bytes from 10.206.1.40: icmp_seq=5 ttl=64 time=9.82 ms
64 bytes from 10.206.1.40: icmp_seq=6 ttl=64 time=6.23 ms
64 bytes from 10.206.1.40: icmp_seq=7 ttl=64 time=8.64 ms
64 bytes from 10.206.1.40: icmp_seq=8 ttl=64 time=10.9 ms
64 bytes from 10.206.1.40: icmp_seq=9 ttl=64 time=19.2 ms
64 bytes from 10.206.1.40: icmp_seq=10 ttl=64 time=9.07 ms
64 bytes from 10.206.1.40: icmp_seq=11 ttl=64 time=8.19 ms
64 bytes from 10.206.1.40: icmp_seq=12 ttl=64 time=8.85 ms
64 bytes from 10.206.1.40: icmp_seq=13 ttl=64 time=13.2 ms
64 bytes from 10.206.1.40: icmp_seq=14 ttl=64 time=9.29 ms
64 bytes from 10.206.1.40: icmp_seq=15 ttl=64 time=7.80 ms
64 bytes from 10.206.1.40: icmp_seq=16 ttl=64 time=9.19 ms
^C
--- 10.206.1.40 ping statistics ---
16 packets transmitted, 16 received, 0% packet loss, time 15024ms
rtt min/avg/max/mdev = 6.236/10.046/19.252/2.967 ms

As you can see

[Kyle]

Oops, hit Send accidentally.

As you can see from the ping time, there is a lot of variation in latency and the ping times are, frankly, pretty bad compared to a fully wired network.   Normally I would expect sub-millisecond response time.

Doing a bit of napkin math, the maximum we could get here is about 100 request/response pairs per second since our ping time is about 9-10ms on average.  So the library is getting 8000 individual tag reads in per second with only 100 request/response pairs (at most).

Note that there are three major factors that will impact performance of a real application:

1. The packet size supported by your PLC.
   
2. The latency of the network.
3. Any processing your application does to the tags takes some time (though usually in the microsecond range).

Your platform will also be a factor.   Linux tends to do rather well with heavily threaded apps.  Windows a little less so (though to Microsoft's credit, they have made major strides in this) and macOS is fairly bad.

The above is from the async_stress.c application (prerelease version) which I recently rewrote to take the number of tags to test as an argument as well as factoring out some of the status checking code.   So this uses a single thread and the capability of the library to pack requests.  With that PLC, it will use 4000-byte request packets.

Here is a quick sample against a ControlLogix L55 with an ENBT network module:

./async_stress 800 'protocol=ab-eip&gateway=10.206.1.39&path=1,0&plc=ControlLogix&elem_count=1&name=TestDINTArray'

Hit ^C to terminate the test.

Creation of 800 tags took 967ms.
Read of 800 tags took 806ms.
Read of 800 tags took 890ms.
Read of 800 tags took 857ms.
Read of 800 tags took 899ms.
Read of 800 tags took 872ms.
Read of 800 tags took 905ms.
Read of 800 tags took 989ms.
Read of 800 tags took 868ms.
Read of 800 tags took 892ms.
Read of 800 tags took 885ms.
Read of 800 tags took 847ms.
Read of 800 tags took 878ms.
Read of 800 tags took 868ms.
^CRead of 800 tags took 39ms.
Program terminated!

I have a hard coded timeout of 1 second in the program, so I could only create about 800 tags before hitting that limit.  Note that it takes just shy of one second merely to create the tags (they are done async).   When created, each tag does a hidden read once before the creation process is considered done.  This is to make sure that you start out with valid data if you want to immediately write the tag and to get the tag's AB data type.  So creation will be a little longer than a plain read.

Here is the ping data for the ENBT:

ping 10.206.1.39
PING 10.206.1.39 (10.206.1.39) 56(84) bytes of data.
64 bytes from 10.206.1.39: icmp_seq=1 ttl=64 time=12.3 ms
64 bytes from 10.206.1.39: icmp_seq=2 ttl=64 time=10.7 ms
64 bytes from 10.206.1.39: icmp_seq=3 ttl=64 time=8.94 ms
64 bytes from 10.206.1.39: icmp_seq=4 ttl=64 time=8.59 ms
64 bytes from 10.206.1.39: icmp_seq=5 ttl=64 time=8.85 ms
64 bytes from 10.206.1.39: icmp_seq=6 ttl=64 time=8.93 ms
64 bytes from 10.206.1.39: icmp_seq=7 ttl=64 time=7.86 ms
64 bytes from 10.206.1.39: icmp_seq=8 ttl=64 time=13.9 ms
64 bytes from 10.206.1.39: icmp_seq=9 ttl=64 time=9.40 ms
64 bytes from 10.206.1.39: icmp_seq=10 ttl=64 time=9.12 ms
64 bytes from 10.206.1.39: icmp_seq=11 ttl=64 time=9.27 ms
^C
--- 10.206.1.39 ping statistics ---
11 packets transmitted, 11 received, 0% packet loss, time 10014ms
rtt min/avg/max/mdev = 7.864/9.824/13.954/1.742 ms

As you can see, the data shows that the latency is about the same.   Same bad variation.  The WiFi and powerline Ethernet contribute most of that.   The wired network is not great (ten year old equipment) but much, much better than that!

TL;DR - the biggest factor is the kind of network module you have (and the firmware level).  If you have newer equipment that can negotiate a 4000-byte packet, you will get almost 10x the performance.  Which makes some sense on my network as we are latency-bound to about 100 request/response pairs of packets.  The older hardware will only negotiate a 500-byte packet.   So about 8x in size to the newer system with its 4000-byte packet.   At least on my network, the packet size is definitely the most important factor given the poor latency.

These tests are probably about best and worst case on my network.

As always, YMMV!

Thanks, Owen and twoshrubs for making me work on async_test and running these tests.   I tend to not use my own performance numbers because my network is so bad.   Given the difference in networks, I am not sure that you can really do a fair comparison to the OPC-UA numbers that twoshrubs posted.   At least we can say that they are in the same order of magnitude, but probably not a lot beyond that.

Best,

Kyle

[Kyle]

I have not tracked OPC UA lately.  It looks like it now supports a binary protocol which will help, but according to Wikipedia, the spec alone is 1250 pages!   It would be surprising if libplctag cannot outperform an OPC UA implementation as it gets rid of the entire server in the middle between your application and the PLC.  The latency due to the encoding and decoding and request handling of the server component of libplctag is zero because there is no server :-)

When I first looked at OPC UA, I thought it was dead, but there was a release of the spec in 2017, so I guess it is still alive.   It is far better than the older OPC code, but still has a significant amount of overhead.  The XML/SOAP version had crazy overhead.

OPC and the bad quality and poor licensing options were what drove me to write libplctag in the first place :-)  I started it after I had to write a bunch of code to work around memory leaks in the latest stack we were using and had found a document that showed some of the binary packets.  Then I found TuxEIP and started to use that.

Honestly, 4-5k tags per second is really good considering how much overhead OPC UA has. That is a well tuned stack if you are getting that kind of read rate!

Best,

Kyle

[twos...@googlemail.com]

Hi Kyle,

Yes, I'm limited by hardware as we have to use old firmware due to Rockwell not updating the redundancy modules for such a long time.

I linked libplctag up with the Open62541 library to create an OPC-UA server that runs on Linux rather than the stock OPC-UA provided by OPC foundation.

I'm new to C++ (I'm a PLC guy) so its a lot of it is guess work on my part and probably done a lot of things wrong, but it works ;)

The OPC Foundation like their paperwork :(

Yes, I agree.. personally I can see libraries like yours replacing the need for kepware/OPC servers for stand alone applications. The issue is with DCS/SCADA vendors, as they generally only support thing like Kepware or OPC.

You could even get past the point of not needing PLC's as most I/O is remote, but that would be bad for people like me :D

[Kyle]

Wow!   That sounds really useful!  Is that something you could share?   I would be more than happy to set up another project in the libplctag organization on GitHub for you.  I am sure that a lot of people would be interested!   Would support for reading in UDT definitions be as helpful to you as to Owen?

I think PLCs will be around for a while.   PCs just are not that reliable. Neither the programming languages nor the hardware.  Perhaps solid hardware and programming languages like Rust will finally move everyone away from PLCs.   There is fairly decent Rust support for ARM-based small IoT chips now.

In my experience, adding things like OPC decreases reliability while increasing interoperability considerably.   Perhaps when there is sufficient intelligence in each sensor to all speak a common language (for instance some common protocol over MQTT) then we will really have a solid foundation to build on.

Best,

Kyle

[Jody Koplo]

We're way off-topic, but these are fun things to speculate on...

A co-worker actually found (publicly) an interesting presentation on the evolution of the control platforms in use at some Amazon warehouses in Asia. They basically detailed their transition from Allen-Bradley and OPC down to running a software based PLC and Greengrass (their edge node) all on a Raspberry Pi. They took the 'cattle-not-pets' route and every deployment was automated. If a Raspberry Pi died they merely replaced it and all the code/configuration for the station was pushed down from cloud storage. Ditto for software changes - they'd go through integration testing and then get automatically rolled out en masse to the Pis. Of course, the scale they deal with makes that worth it.

As for a 'common protocol over MQTT' - you're basically describing Sparkplug. I didn't realize this until recently, but the origins of MQTT date back to 1999 (before IoT was a buzzy thing) for low-bandwidth SCADA systems for oil pipelines over satellite uplinks. MQTT was literally invented as an alternative to traditional OPC in order to do report by exception instead of polling. It's just taken 20 years for it to catch on...

Jody

To view this discussion on the web visit https://groups.google.com/d/msgid/libplctag/af44ae95-556a-4638-9837-3aae4c21812fn%40googlegroups.com.

[Kyle]

Heh, yeah Amazon is doing its own thing.   Big difference for Amazon: they have a very strong culture of CI/CD and "cattle vs. pets."   Most industrial locations do not (almost the opposite).   So even if you have 20 machines that are all supposed to be the same, within five or ten years, they all differ from each other somewhere.   Different thermocouples were in the shop when one went out on machine #6.   An old analog input card had to be replaced on machine #15.   Machine #3 suffered a large mechanical failure and when it was rebuild, they used a different motor and controller...  You get the idea.   For each of those you need machine-specific tweaks.  

Management never wants to replace working parts just to keep all the machines the same.   And if you extend the PLC code such that the same code works on all machines, you have spaghetti after you are done.

Hopefully prices will drop and keeping all the machines the same will become more feasible.

But back to the numbers...

Twoshrubs, were your numbers with the Open62541 library plus libplctag or was this a commercial OPC stack?   If you are using the older firmware and hardware (all you need is version 21 or greater for PLC firmware and version...  I can't remember the version for the EN2T modules, somewhere I had an ENBT that would work with 4k packets).   Maybe it was version 24?   I think it was version 21.

But use of older firmware and Ethernet modules is a very good point.   On those systems, using symbol instance addressing would probably help a lot.   That is a very strong argument for implementing the whole tag listing part as at least an add-on library, and probably the UDT stuff too.   At the very least, I need to implement symbol instance addressing support.  

I added issue #200 for tracking this.  I will add issue for UDT support later as that is not directly a performance requirement.

[Kyle]

Hi Owen,

I am currently a bit under water at work and still trying to get the new Java wrapper squared away so that it builds JAR and AAR files and sets up JCenter correctly.   I have decided that I will find a way to support the following:

1. Accessing AB tags via symbol IDs.
2. Getting UDT information.
3. Determining whether to reload tag information.

Each of these is going to be a separate piece.  I will do #1 first and then probably #3 as listing out tags and using the IDs does not make a lot of sense unless you can determine when you need to reload the tag.

At least at first, this will be done in a way similar to current tag listing.   The library will support the parts it must and the rest will be at the user level.   This will keep the library small and fast but still support additional features.

Best,

Kyle

[Jody Koplo]

Kyle-

For my purposes, I'm most excited about number 2. We often write one application and use it on various AB controllers/programs/machines by dynamically listing all the tags (including UDTs) on the machine and building up logging/UI/etc based on the contents. I don't care so much about detecting when a change in tags occurs. If you change the AB code, you're just required to refresh or relaunch the HMI app. It's hard to know what the performance improvement of the symbol ID stuff will be, but that could be a nice bump. And dynamically knowing when to reload the tag info would be a great feature to pull it all together, but not strictly as necessary (the current commercial offering we use doesn't include this feature).

This is of course from my own perspective and with the usage pattern that I support.

Jody

To view this discussion on the web visit https://groups.google.com/d/msgid/libplctag/0b405854-29aa-4382-a134-0371a4054358n%40googlegroups.com.

[Kyle]

Hi Jody,

Unfortunately, #2 is going to be non-trivial unless I really make application code or wrappers work for it.   To get UDTs there are actually two calls you need to do for each UDT (AB just could not make things easy!).   The first one gets information/metadata about the number of entries (fields) and their sizes.   The next one gets that data.   Note that interpreting the fields is also... interesting.   You need to ignore fields that start with "ZZZZZZZZZZ" for instance.  AB did not use special flags or types or anything that could be easily checked programmatically.  You have to interpret the field names.   That is all going to be on the application code.   Then you have things like overlapping offsets and knowing the types of fields (from the metadata in the first call) to tell how to handle bit/bool fields.

For example, if you have two bit fields in your UDT they are packed into a dummy SINT field at the beginning of the UDT.   The offsets of both fields are in bytes and will be the same, zero (0).   The dummy field will have a type of SINT and a name that starts with "ZZZZZZZ..."  You have to look at the type metadata of the fields to see that the type is 0xC1 for bit/bool, and the remaining byte (types are two bytes) contains the bit offset in the lower three bits.    So to figure out which bit is that field, you need to look in multiple places.

For now what I will do for these is probably (subject to change as I try to implement it):

1. For this one, take tags with a special name or format.   For instance 'name=@tagID:3af56.myField[42].anotherField'.   The leading '@tagID:3af56' would be the top level tag symbol ID.   You would list the tags out and get the mapping from your tag, e.g. 'myTag' to the ID '3af56'.   I will update list_tags.c to show how to do that.  The data is there already.

2.This, I am still not sure.   Maybe a special tag like '@udt:820af2'.   It will need to get the metadata about the UDT and then the data about the UDT.  I can batch that up into one tag, but that is starting to be a lot of extra code in just the ControlLogix path.

3. This is going to need a special tag as well.  Not sure about what that would be maybe '@tagVersion' or something.   From what I understand, you can just treat the result as a byte string and look for changes.   There are certain errors that need to be handled appropriately and very differently from other calls.

After some consideration, I will not have raw CIP support.   Why?   The security implications are horrible.  Any time you have a tag name exposed to user input, a maliciously crafted name can destroy the PLC program or the PLC run mode, nearly anything.  I had thought to use this as a way to quickly prototype, but the more I thought about the security problems, the less I liked the idea.  It would make things easier to prototype, but, wow, would the the black hats have fun with that!

Because of depth of the code changes, I am going to do some refactoring first to split out how the tags work for AB.   The Modbus code ended up drastically simpler in overall structure and does zero allocation during normal operation.  I want to get that for AB as well.  It will help keep all these special tag operations separate from the main code for reading and writing tags.

I will go as fast as I can, but right now my priority is getting libplctag4j into good enough shape that I can get it up on JCenter (the equivalent of Nuget) for both plain Java and Android use.   With work taking 12-16 hours a day right now, that only leaves some weekend time.   And I need most of that to spend with my family as they are peeved already at my lack of time.   What I am trying to say is that this is probably not going to go very quickly.  Sorry about that!

Best,

Kyle

[Jody Koplo]

No worries on timing, this all helps a lot.

It sounds like the code is getting more and more AB specific (really clogix specific). Would it be better to split it into separate libraries? Or is it possible to handle some of this stuff at the higher-language level? It seems unlikely that someone writing C is going to be doing dynamic queries of UDT definitions...

I'd offer to help on the C code, but you don't want my ham-fists in there. If there's something that can be done to explore in C# I might be able to do that.

For now, I might just go the alternate route of parsing an l5x file to get all the tag definitions.

Jody

To view this discussion on the web visit https://groups.google.com/d/msgid/libplctag/1bad99a2-9103-46e9-a6a9-ee7cd34ca0fan%40googlegroups.com.