Multithreading support

[adl...@gmail.com]

Hi,
I'm discovering the library and hence have some questions regarding multithreading support in the lib :)

1) What does 'multithreading' actually refers to in case of open62541 library - does it mean that it is allowed to created multiple, parallel instances of functions like:

threadmain()
 {

UA_ServerConfig config = UA_ServerConfig_standard;
UA_ServerNetworkLayer nl = UA_ServerNetworkLayerTCP(UA_ConnectionConfig_standard, 16664);
config.networkLayers = &nl;
config.networkLayersSize = 1;
UA_Server *server = UA_Server_new(config);
UA_Server_run(server, &running);
UA_Server_delete(server);
nl.deleteMembers(&nl);

}

or does it mean it is allowed to manipulate server from multiple threads e. g. adding/removing variable nodes like:

(...)

UA_StatusCode retval = UA_Server_addVariableNode(server, myIntegerNodeId, parentNodeId,
                                                     parentReferenceNodeId, myIntegerName,
                                                     UA_NODEID_NULL, attr, NULL, NULL);
(...)

2) Am I getting this right that multithreading is not available when using amalgamated library version?

3) And in more general is amalgamated library functionally limited in any way compared to header + *lib. config? What would you recommend to use: amalgamated lib version or classic header + *.lib configuration?

Thanks,
Adam

[Julius Pfrommer]

Hey Adam,

1) With open62541, you can run several OPC UA servers at once. Also in parallel threads.
But with a normal build, no two threads should operate on the same server at the same time.
Note the UA_Server_run_iterate method that can be used to "manually" drive each server's main loop.

With the UA_ENABLE_MULTITHREADING compile flag, multiple threads can interact with a server concurrently.
But this feature currently works only on Linux. And it must be considered as experimental. It works. But we didn't have the resources to do significant testing / verification so far.

2) 3) You get the same features from the amalgamated version as from a "normal" library build.
We recommend to use whichever is easier to integrate. The header + lib configuration is a bit easier to navigate as the different topics are contained in their respective header file.

Best regards,
Julius

[adl...@gmail.com]

Hi Julius,
thanks for your response, It is all clear. And I found your tip with UA_Server_run_iterate() really useful!

I've tried both methods: amalgamated source and I also built the current master branch using VS2015. It works but only with static link library (open62541.lib), using fully static linking (open62541-static.lib) produces linker unresolved external errors - I'll try to investigate it later anyway. I'm curious what the reason is.

Just been wondering, what do you think would be more efficient solution in case we have lots of client accessing server at the same time:
1)  to provide just a single instance of the server in a single thread and accept all client at the same tcp port

or

2)  to run multiple instances of the server in parallel threads at multiple tcp ports and to scatter the clients among these ports?

Thanks,
Adam

Thanks,
Adam

[Julius Pfrommer]

Hey Adam,

my laptop does 20k read-requests/sec with a single-threaded server. Looping with a fixed request over the decode/process/encode method gives more than a million requests/sec.
Hence, most of the time is spent in the network layer of the operating system and the context-switch between userland/kernel.

Having many servers on a different port each is an option, albeit a bit ugly.
To further increase the speed for a single server you'd have to write some code.

1) Use architecture-specific networking APIs instead of POSIX, such as epoll/kqueue/IO Completion Ports [1].
That should give a huge boost in throughput. Especially when you need to keep many sockets open at the same time.
You can take the existing /plugins/ua_network_tcp.c as a starting point.

2) Improve multithreading support of open62541. It's all there. Incoming messages are received from a single thread and distributed to workers with a central dispatch queue.
But I'd like to see unit tests for all the edge cases and testing with a sanitizer [2] before recommending productive use.

Best regards,
Julius

[1] https://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx
[2] http://clang.llvm.org/docs/ThreadSanitizer.html

[adl...@gmail.com]

This sounds great. Looks like the design is quite efficient by default :)

BTW: I followed your tip regarding UA_Server_run_iterate() and implemented a simple demo that runs the server in parallel thread. I used VS2015(update 2) + Win10 + latest amalgamated open62541 version. Unfortunately, I encountered a strange behaviour - it works as expected when compiled as release and it is not possible to connect to server when it's compiled with Debug settings. I'm testing the connection with UaExpert and it looks like this:




-> getting error: "Could not connect to server: BadTimeout"

Anyway, exactly the same code works fine when compiled as release. I get some minor warnings from open62541.c compilation and also some linker warnings poping up only in debug compilation:
1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_Server_new imported in function "private: bool __thiscall OPCConnection::OpcUaServer::initialize(void)" (?initialize@OpcUaServer@OPCConnection@@AAE_NXZ)
1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_Server_delete imported in function "private: void __thiscall OPCConnection::OpcUaServer::uninitialize(void)" (?uninitialize@OpcUaServer@OPCConnection@@AAEXXZ)
1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_Server_run_startup imported in function "private: bool __thiscall OPCConnection::OpcUaServer::initialize(void)" (?initialize@OpcUaServer@OPCConnection@@AAE_NXZ)
1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_Server_run_iterate imported in function "private: void __thiscall OPCConnection::OpcUaServer::serverThread_main(void)" (?serverThread_main@OpcUaServer@OPCConnection@@AAEXXZ)
1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_ServerNetworkLayerTCP imported in function "private: bool __thiscall OPCConnection::OpcUaServer::initialize(void)" (?initialize@OpcUaServer@OPCConnection@@AAE_NXZ)
1>OpcUaServer.obj : warning LNK4049: locally defined symbol _UA_ServerConfig_standard imported

Not sure if it has anything in common. Do you have any idea what could be the reason for such strange behaviour? I shared the full VS solution at: https://github.com/adliner/OPCServerDemo it contains a simple reproduction. Just open the solution and build debug/release, I tested 32bit only. I'm not sure how to investigate it further.


Thanks,
Adam

[adl...@gmail.com]

Ok, 

sorry, my bad. I did not fill UA_ConnectionConfig structure correctly and hence these issues in Debug builds. I used the default 'UA_ConnectionConfig_standard' and now it works in parallel threads as well. 

So, things that still concern me are these warnings I get when using amalgamated source:

1>  open62541.c

1>c:\users\adam\downloads\opcserverdemo-master\opc\open62541.c(21702): warning C4996: 'inet_ntoa': Use inet_ntop() or InetNtop() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings

1>  c:\program files (x86)\windows kits\8.1\include\um\winsock2.h(1868): note: see declaration of 'inet_ntoa'

1>c:\users\adam\downloads\opcserverdemo-master\opc\open62541.c(22002): warning C4996: 'gethostbyname': Use getaddrinfo() or GetAddrInfoW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings

1>  c:\program files (x86)\windows kits\8.1\include\um\winsock2.h(2238): note: see declaration of 'gethostbyname'

1>c:\users\adam\downloads\opcserverdemo-master\opc\open62541.c(22284): warning C4146: unary minus operator applied to unsigned type, result still unsigned

1>c:\users\adam\downloads\opcserverdemo-master\opc\open62541.c(22313): warning C4146: unary minus operator applied to unsigned type, result still unsigned

1>     Creating library C:\Users\Adam\Downloads\OPCServerDemo-master\Debug\OPC.lib and object C:\Users\Adam\Downloads\OPCServerDemo-master\Debug\OPC.exp

1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_Server_new imported in function "private: bool __thiscall OPCConnection::OpcUaServer::initialize(void)" (?initialize@OpcUaServer@OPCConnection@@AAE_NXZ)

1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_Server_delete imported in function "private: void __thiscall OPCConnection::OpcUaServer::uninitialize(void)" (?uninitialize@OpcUaServer@OPCConnection@@AAEXXZ)

1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_Server_run_startup imported in function "private: bool __thiscall OPCConnection::OpcUaServer::initialize(void)" (?initialize@OpcUaServer@OPCConnection@@AAE_NXZ)

1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_Server_run_iterate imported in function "private: void __thiscall OPCConnection::OpcUaServer::serverThread_main(void)" (?serverThread_main@OpcUaServer@OPCConnection@@AAEXXZ)

1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_Server_run_shutdown imported in function "private: void __thiscall OPCConnection::OpcUaServer::uninitialize(void)" (?uninitialize@OpcUaServer@OPCConnection@@AAEXXZ)

1>OpcUaServer.obj : warning LNK4217: locally defined symbol _UA_ServerNetworkLayerTCP imported in function "private: bool __thiscall OPCConnection::OpcUaServer::initialize(void)" (?initialize@OpcUaServer@OPCConnection@@AAE_NXZ)

1>OpcUaServer.obj : warning LNK4049: locally defined symbol _UA_ConnectionConfig_standard imported

1>OpcUaServer.obj : warning LNK4049: locally defined symbol _UA_ServerConfig_standard imported

are they expected? Though it doesn't seem to influence functional operation I don't like warnings messing around. Winsock warnings are clear I think, regarding the linker warnings - I could have eliminate them by removing UA_EXPORT (__declspec(dllexport)) from function headers in open62541.h. Are they really needed in case of using amalgamated source file?

Thanks,

Adam