Per-connection threading model

[Ivo Manca]

Hi,

In the past, there have been various discussions on the mongoose mailing list about the threading model currently used.

In the current implementation, there is always one thread available per connection, which will have scaling issues on systems with a large number of persistent connections.

Now that browsers and clients are moving more and more to a large number of concurrent connections (6 to 8 seems to be the default), this issue becomes more exposed.

When the client is idle for a longer period it will cause a large number of threads to become idle, while at the same time the number of connections will reach the limit as specified by 'num_threads', causing new connections not to be served.

For the end-user the system will seem to be unresponsive, yet the server might have a very low load.

The only solution currently would be to disable keep_alive or increase the num_threads configuration parameter to a high enough value to serve all expected number of concurrent clients.

Both have a major draw back. Disabling keep_alive introduces a significant overhead for the non-idle connections, increasing the thread amount introduces a whole variety of performance/ scalability‎ challenges.

Browsing around the mailing lists,I found that Ger Hobbelt had already created a clone where he did some major effort in mitigating this issue:

See: https://github.com/GerHobbelt/mongoose

This seems to be the solution for my current problem, yet I prefer to stick to the master branch of the webserver. 

Since I am now unable to upgrade to a newer version of mongoose due to the license problems, I was hoping to be able to switch to civetweb; yet this performance aspect is very important for this decision.

Is a change in the threading model something that could be placed on the roadmap, is it already being looked into or is the current implementation good enough for the (current) projects goals?

---

Ivo

[Thomas Davis]

This conversation is a good start.  This project is what people make of it.  I would suggest either writing a feature ticket on it on github, or forking the project and changing it and then pushing back to main branch (via a pull request).  If you do the later, please wait until next week because the repo is being fixed to restore the mongoose history in it.

I will be interested to see if there are any thoughts on the thread model change from the group.  When I have time this week, I will take a closer look at the proposal.

Cheers!

Thomas

--
Sourceforge
https://sourceforge.net/projects/civetweb/
---
You received this message because you are subscribed to the Google Groups "civetweb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to civetweb+u...@googlegroups.com.
To post to this group, send email to cive...@googlegroups.com.
Visit this group at http://groups.google.com/group/civetweb.
To view this discussion on the web visit https://groups.google.com/d/msgid/civetweb/b5bbc25a-3b14-4c07-86f2-dd4fb4126794%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[No Face Press]

I started scanning that repository and I did not see any threading model changes.  I did see some pthread API additions which could be what he was talking about or it could have been a later merge from mongoose.  But I did not see where there would be any change in connection allocation strategy.

So, lets start at the beginning.  You already described the use case.  How would you like civetweb to behave when the maximum number of connection is reached?

Thomas 

[Ivo Manca]

The first commit where he made changes to the way mongoose handles the threads vs connection can be found at:

https://github.com/GerHobbelt/mongoose/commit/d76e6f13b3c1e93cb7cea0eb13ae253d2df5c45f

This also introduces a connection handling model which would suffice for me.

For me, the real problem lies in the threads being idle and not serving any connection 

Instead of binding the connections to a fixed thread and leaving that thread idle until either new data appears or the connection is closed, this change will group the idle connections and select them when any activity has been noticed.

This would ultimately allow civetweb to handle more persistent connections than the number of threads configured.

Another valuable addition would be to set a timeout on the idle connections, making sure they will get dropped and the socket can be reused. 

This combination will allow me to reduce the number of threads by a huge factor, while still being able to serve a large amount of simultaneous clients, without adding the complexity of creating / removing new threads on the fly.

I am not sure how stable / finished the fork of Ger Hobbelt is.

@Ger Hobbelt: would you like to jump in this discussion and share your thoughts? 

---

Ivo

[Ger Hobbelt]

[EDIT: sorry for possible double-post; indeed I was not subscribed to the civetweb ML before this message :-(  Unfortunately I have missed what came before in this thread.]

Subject: Re: Per-connection threading model

Cc: cive...@googlegroups.com

L.S.,

mongoose/civetweb is currently low on my radar (24h per day and a startup getting into gear and all that jazz ;-) )

I'll do this off the top of my head, so a few details may be wrong but the general gist of my repo is:

status: experimental (as usual ;-) ). It has been tested but some parts lightly or even 'superficially'.

The few things which have seen quite a bit of testing (and actual use) are

- the lockup fixes (it was a very hairy situation, sometimes extremely hard to reproduce, but I feel my code is robust now in that regard - while it is quite a larger chunk of the mongoose/civetweb server code than what Sergey did/merged from others. When that issue (#347 IIRC) was a 'hot topic' I stuck with my code because I couldn't get his to perform a close predictably and consistently under all circumstances where my flavor worked). Scenarios like flaky DSL modems combined with a series of old and new browsers/browser-versions which support persistent connections led to some surprises.

- the change in the threading model works and for me at least it works very well. The thread model there for the original mongoose is simple: set it up with N threads, take N+1 clients, each of which open a persistent (or otherwise long-open) connection and the one client per thread approach of mongoose denies client N+1 any access to the server. For a reasonable amount of threads (< 1000; it's not just Windows which fares better with fewer threads to manage) this is very risky as even embedded servers will see multiple clients at a time when they serve data and the viz is done client-side, for example. Browser X keeps connection for several seconds while loading and building a visual, thus denying browser/client Y access once the threadcount has been reached.
Hence I redid the code such that inactive (not transmitting, not receiving, i.e. no data I/O pending) connections are pushed back onto the queue, freeing the thread(s) to pick up waiting connections and/or other connections which report I/O activity.

Word of caution: I never took the time to merge in Sergey's work re Lua integration as I did not need it at the time and afterwards my focus had to move to frontend programming so mongoose got onto the backburner for me.

As I coded a generalized approach to hooking into the I/O processing loop, suitable for custom protocols and stuff like socketIO alike, which approach is quite different from Sergey's, you may have trouble merging with his latest, but it is certainly doable.

Side note / OFF TOPIC:

I did create a version/build for Windows where you could drop a file or directory on a window pane and the mongoose would serve that file/directory tree on the spot. This mongoose flavor was created as a tool for a book I was involved in: that way we could easily offer the JavaScript examples in there as ready-to-go-play, at least for Windows boxes.

The point of this bit is that that commit/branch has seen some serious testing and may thus serve as a reference. There has been done very little to nil work on mongoose from my part after I used it for that purpose.
I intend to revisit (and repurpose) mongoose/civetweb again in the near future but given the amount of frontend work in the pipeline reckon a revival in months, rather than weeks.

Mgt summary:

threadmodel rewrite is pretty solid
I vouch for my 'connection close' handling code, particularly the seemingly 'hairy bits' in there.
somewhat tough to merge the latest as big difference due to generic I/O protocol hooking code vs. Sergey's socketIO
no Lua integration so some merge/mix work to be done there

P.S.: I missed the start of this discussion as a search in my gmail did not uncover the starting email or any other with the same subject line as this one. I saw the civetweb 'happening' when Sergey announced the license change, but did I miss a notice where it says that civetweb has a new mailing list, or have I been predictably ignorant in turning on 'watch' for civetweb in github?

[EDIT: can answer this myself: missed the new mailing list! ]

Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   g...@hobbelt.com
mobile: +31-6-11 120 978
--------------------------------------------------

[Ger Hobbelt]

Grmbl. Typo:

On Tue, Oct 8, 2013 at 2:06 PM, Ger Hobbelt <g...@hobbelt.com> wrote:

The thread model there for the original mongoose is simple:

--> The *THREAT* model there for the original mongoose is simple:

(Even when you don't worry about 'attacks', it's an issue: ("it" ≡ "1 thread reserved per client, fully served & no interruptions until done") for example, I had quite a bit of trouble with the mix Safari 5/Win + mongoose/Win before switching to the new model as Safari5 kept some connections open for a very long time. Hit [F5] refresh a few times and you were already dead in the water.

One can argue that the new threading model is a 'threadpool' variant, also there's no-one stopping you now from serving everyone through a single thread which is handy for RAM-strapped embedded machines; at least, for me, data I/O performance has been excellent since then.)

[Thomas Davis]

Ok, so now I understand what is going on a bit better.  This is a thread pool model being applied, instead of a thread per connection.  I think a good argument can be made for that.  However, it could be equally fruitful to look at how connections are managed as well.  I like the idea of timing out threads in a thread pool.  That a pretty clever twist to just capping the desired reserve size.

In any event, I think this would be to large of a change for 1.5 which I am trying to close out now.  Perhaps, we can discuss this a primary feature for 1.6.  It is just too much to through in there without testing it well on all platforms.  One of the nice things about the way it is now, is that it is simple with minimal synchronization requirements.  That is not to say, we should not improve it.  I want to do it cleanly where it is obvious to the code reader what is going on, so it can be maintained over the long run.

Thomas

--

Sourceforge
https://sourceforge.net/projects/civetweb/
---
You received this message because you are subscribed to the Google Groups "civetweb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to civetweb+u...@googlegroups.com.
To post to this group, send email to cive...@googlegroups.com.
Visit this group at http://groups.google.com/group/civetweb.

To view this discussion on the web visit https://groups.google.com/d/msgid/civetweb/CAFP60foLv2DC1%3DfNczAfF_w_07iT_KBsDFddit-R6riPxwUj%2Bg%40mail.gmail.com.

[Ivo Manca]

Thank you, Ger, for shining your light on the status of your changes. 

@Thomas: I do agree that this change should not be treated lightly, so wrapping up the 1.5 version before starting to work on this, sounds like a sane idea.

It is good to hear that the idea is being liked. Would it help if I would create a feature request on github to better track this, or is that not so relevant for you?

FYI: the main targets where I am running (for now: )  mongoose on, is an armv6l target running linux (2.6.31 still) and regular PCs running both Linux and Windows for development / testing.

I'll check to see if I can move my fairly outdated mongoose version (3.3) towards the to-be-released 1.5 version of civitweb to facilitate testing and hopefully development.

[Thomas Davis]

Yes, both of those would be excellent.

Thank you!

Thomas

[bel]

Am Dienstag, 8. Oktober 2013 14:06:12 UTC+2 schrieb Ger Hobbelt:

threadmodel rewrite is pretty solid
I vouch for my 'connection close' handling code, particularly the seemingly 'hairy bits' in there.
somewhat tough to merge the latest as big difference due to generic I/O protocol hooking code vs. Sergey's socketIO
no Lua integration so some merge/mix work to be done there

@Ger Hobbelt:
Do yo mean the connection close handling topics we worked on in spring 2012 (I made some tests and some first fixes for keep-alive, you finally solved the it for Win32 with the DisconnectEx, but not all our fixes got integrated in mainline mongoose).
I already fixed some of the remaining keep-alive issues here. My old "ajax" test for connection-close (with minor/insignificant changes) currently works for >95% on the same Win32 PC, which is much better than it was for mongoose in 2012, yet it is not perfect.

The Lua integration merge does not seem too difficult, and the Lua scripts I had in the old tests (via CGI) still work. However websockets may require some additional attention.


Regarding the thread model:
Is this something you can really do efficiently in an OS independent way? Basically you need only one thread (per CPU core) if you do all I/O with an async IO model, but this is quite different in different operating systems, sometimes even different between different versions of the same OS.