Orthanc Threading/Connection Handling

[Chris Hafey]

I am writing a Node.JS application that integrates with Orthanc.  I have noticed that my requests will sometimes fail with ECONNREFUSED.  This seems to happen when many simultaneous requests are made to Orthanc.  Has anyone else run into this problem before?  Has Orthanc been designed and tested for high numbers of concurrent requests?  

[Sébastien Jodogne]

Hello,

Please would you kindly send a small code sample (preferably in Python or Node) that would allow us to reproduce your problem?

Sébastien-

[Chris Hafey]

I dug into this a bit more and discovered that Orthanc was not supporting HTTP Persistent Connections:

http://en.wikipedia.org/wiki/HTTP_persistent_connection

This means that each HTTP request was establishing a new socket connection which is a huge hit to performance.  The mongoose web server does support HTTP persistent connections but the default configuration surprisingly does not enable it.  To enable it, you need to set the configuration setting "enable_keep_alive" to "yes".  I added this setting and the performance of my application improved significantly.  Before I made this change, I was getting very inconsistent request timings - sometimes a request would complete in 13ms, other times in 300ms.  I made the change to line 787 of MongooseServer.cpp - I can send a patch if needed but it was quite simple:

      const char *options[] = {
        "listening_ports", port.c_str(),
        ssl_ ? "ssl_certificate" : NULL,
        certificate_.c_str(),
        "enable_keep_alive" , "yes",
        NULL
      };

I am going to hold off on sending the Node.JS script to reproduce the ECONNREFUSED problem unless I see it happen with persistent http connections enabled.  

Chris

[Chris Hafey]

Oh and just to add a bit more information for those interested.  Orthanc uses the Mongoose C++ library which by default spawns 10 threads to handle incoming HTTP requests and the listening socket is setup with a backlog of 100.  

Chris

[Chris Hafey]

Oops, my quoted code won't work if there is no ssl certificate, do this instead:

      const char *options[] = {
        "listening_ports", port.c_str(), 


        "enable_keep_alive" , "yes",


        ssl_ ? "ssl_certificate" : NULL, certificate_.c_str(),


        NULL
      };


[Sébastien Jodogne]

Hi,

I have just pushed in the mainline a large refactoring of the HTTP framework of Orthanc to correctly handle HTTP Keep Alive. Indeed, previously, the "Content-Length" HTTP Header was not always properly set, which could corrupt the HTTP transfers.

There is now an experimental option in the configuration file that is called "KeepAlive". It defaults to "false". Turn it to "true" to enable keep-alive support. All my integration tests pass with this option enabled.

Unfortunately, when I enable keep-alive, I do not notice any performance improvement with my integration tests, even over a network connection.

Chris, could you kindly play with this option to make sure that it solves your performance problem?

Regards,

Sébastien-

PS 1: I had to upgrade from Mongoose 3.1 to Mongoose 3.8 to solve a timeout error that was only appearing with keep-alive enabled. For reference, here is the command line I use to simulate a high load on Orthanc [1]:

# ab -k -c 50 -n 10000 http://localhost:8042/instances/8e289db9-0e1437e1-3ecf395f-d8aae463-f4bb49fe

PS 2: Besides setting "enable_keep_alive" to "yes", I also had to set the HTTP header "Connection: keep-alive" in the answer [2,3].

[1] http://stackoverflow.com/a/12732410/881731

[2] http://blog.lolyco.com/sean/2009/11/25/ab-apache-bench-hanging-with-k-keep-alive-switch/

[3] http://stackoverflow.com/a/7038068/881731

[Chris Hafey]

Hi Sébastien,

  Nice work, I just tried out the latest and it appears to be working fine - no performance issue when KeepAlive = true (the performance issue returns if I set it to false).  The performance issue I observed is on Windows 7 x64 when connecting to Orthanc from google Chrome.  Interestingly, there is no performance issue when I connect to the same Orthanc server from Mac OS X chrome so it may be specific to Windows.  I am curious to hear if you can reproduce it.

Chris

[Sébastien Jodogne]

Hi Chris,

To be sure, did you have a look at the "Orthanc Explorer is slow under Windows on the localhost" on the Orthanc troubleshooting section?

https://code.google.com/p/orthanc/wiki/FAQ#Troubleshooting

Perhaps your performance problem is related to IPv6.

Cheers,

Sébastien-

[Chris Hafey]

You are right, this is the same issue.  Switching from using 'localhost' to 127.0.0.1 fixes the performance even with KeepAlive=false.  You may see a performance improvement when using KeepAlive=true with SSL connections since it would avoid the need for SSL handshakes.

Chris

[Sébastien Jodogne]

OK, fine! This sounds good news.

Anyway, you remark allowed me to improve many things in the way Orthanc handles the HTTP answers.

Please also note that I will certainly have to change the Mongoose back-end to another embedded Web server, as its newly chosen GPLv2 license is incompatible with the GPLv3+ license [1]. I will presumably keep using Mongoose 3.8 (since it is the last version released under an MIT license), but if some bug occurs and an upgrade of Mongoose is necessary, I will have to switch to another backend to cope with the fact that the company behind Mongoose does not allow GPLv2+ licensing (at least for the time being [2]).

Cheers,

Sébastien-

[1] https://groups.google.com/d/msg/mongoose-users/aafbOnHonkI/09MXDIy4ZcoJ

[2] https://groups.google.com/d/msg/mongoose-users/aafbOnHonkI/nG0wxOfIXNQJ