fast C HTTP server library comparison & wishlist
1,048 views
Skip to first unread message
Solar Designer
Feb 16, 2016, 5:54:10 PM2/16/16
to onio...@coralbits.com, Mark Ellzey, Christian Grothoff, David Moreno Montero, Sergey Lyubka
Hi,
Trying to choose an embeddable HTTP server library for a project, and
also considering writing my own special-purpose code, I came up with
the following comparison of libonion vs. other C libraries that include
high-performance HTTP support and are currently maintained.
Licenses:
libevhtp+libevent - 3-clause BSD
libmicrohttpd - LGPL 2.1
libonion - Apache 2 (except for some examples) or GPLv2+
mongoose - GPLv2 (and commercial)
Build environment:
libevhtp+libevent - cmake+autotools
libmicrohttpd - autotools
libonion - cmake
mongoose - none (one large file, like SQLite)
Code size ("text" as reported by the size(1) command on the library or
on a tiny sample program if statically linked, on Scientific Linux 6.6
on x86_64):
libevhtp+libevent - ~500 KB, or ~200 KB without unicode.c.o and reg*.c.o
libmicrohttpd - ~100 KB default, ~55 KB with most ./configure --disable-*
libonion - ~100 KB with most ONION_USE_* set to false
mongoose - ~100 KB including JSON-RPC
For the smaller builds of libmicrohttpd and libonion, I kept threads
support enabled, but disabled pretty much everything else that could be
disabled without patching the code. It looks like libmicrohttpd wins
this test. Maybe there's more code in libonion to disable (make into
compile-time options) - I haven't checked yet.
Built-in JSON support:
libevhtp+libevent - none
libmicrohttpd - none
libonion - JSON builtin, JSON-RPC in Apache 2 licensed example
mongoose - JSON-RPC builtin (simple JSON parser not exported?)
All of this is for current versions on GitHub or in recent release
tarballs as of a few days ago.
Maybe someone else will find this useful. I'd appreciate corrections.
It is very likely that I overlooked something.
On a related note, I found the list of alternate implementations on the
libmicrohttpd homepage very helpful. That's classy. Thanks.
My wishlist:
A processes (pre-fork) + [e]poll mode, like nginx has. Processes have
pros and cons vs. threads: more reliable, faster malloc/free (no lock
contention risk), but OTOH slower context switches (if running process
count exceeds number of logical CPUs). I would likely prefer this mode,
but all four libraries appear to be missing it.
Ability to accept not only HTTP, but also raw TCP connections, and
handle them in application code along with the library-handled HTTP.
Such as for implementing JSON-RPC directly over TCP, while also having
it over TCP+HTTP, and without having to manage an own/separate
threads/processes pool. Do any of the four have this? I found no such
examples with any of them.
Easily and cleanly embeddable into an application's source tree, while
also allowing easy updates to new upstream versions. mongoose almost
achieves this, but at the expense of sacrificing meaningful separation
into multiple translation units within the library itself. I think we
don't have to pay this price. We could have multiple files (10 or so?),
in a subdirectory, which are also easy to list in a project's Makefile.
Maybe I'd do that for libonion, freeing it from cmake, but then updating
to new upstream versions would be harder. Do I really have to bite the
cmake or/and autotools bullet for something as simple as accepting HTTP?
I'd prefer a more permissive license like 2-clause BSD or MIT. But I
guess I'll have to settle on Apache 2 or such. mongoose' use of GPLv2
is understandable - need to make money - but is otherwise a disadvantage
(even for a commercial project that could pay, and even when publishing
any source code changes is not a problem and would be planned anyway; we
just don't want to put our time into something that we would not always
be able to reuse in other projects).
Optional JSON from the same upstream is a plus, ideally exported both as
a generic JSON parser and as JSON-RPC support. Looks like only libonion
sort of delivers both (but the code might not be production quality).
Ability to exclude more of the functionality - for example, to include
only the POST method (and not compile in code for the rest). I am
concerned not so much about code size per se, as I am about attack
surface, and about ease of code reviews (not having to determine if some
compiled-in code is actually dead code in a given case, but to know
reliably that it's not compiled in).
On a related note, David's use of Coverity for libonion is commendable,
but it looks abandoned since 2014, and many "defects" (even if false
positives) remained unfixed back then.
Mark's use of Coverity for libevhtp is also commendable... and looks
abandoned since May 10, 2015. It shows "48,919 Lines of Code Analyzed",
only "4 Total defects" and "0 Outstanding" - I guess it means that
everything detected by Coverity before (which must have been many more
"defects") had been eliminated prior to that run. That's impressive.
But we don't know how many new "defects" may have appeared in the 9
months that passed. Also, I haven't looked into whether libevent has
been subjected to similar static analysis or not (although being
initially written by Niels Provos speaks in its favor, given Niels'
other work), and accepting TCP connections isn't as much risk as parsing
HTTP and JSON.
I don't give a lot of weight to the Coverity results for my
decision-making, but it shows whether the maintainers care, and there
are few other somewhat-meaningful metrics I could use before having
spent time to analyze and try to use the code myself.
Why am I posting this to the onion mailing list specifically? I find it
likely that libonion wins for me, although not by a large margin (and
there's a lot that I dislike about it). This is not a final decision
yet. I might as well end up reverting to writing special-purpose code
from scratch.
Thanks,
Alexander
Trying to choose an embeddable HTTP server library for a project, and
also considering writing my own special-purpose code, I came up with
the following comparison of libonion vs. other C libraries that include
high-performance HTTP support and are currently maintained.
Licenses:
libevhtp+libevent - 3-clause BSD
libmicrohttpd - LGPL 2.1
libonion - Apache 2 (except for some examples) or GPLv2+
mongoose - GPLv2 (and commercial)
Build environment:
libevhtp+libevent - cmake+autotools
libmicrohttpd - autotools
libonion - cmake
mongoose - none (one large file, like SQLite)
Code size ("text" as reported by the size(1) command on the library or
on a tiny sample program if statically linked, on Scientific Linux 6.6
on x86_64):
libevhtp+libevent - ~500 KB, or ~200 KB without unicode.c.o and reg*.c.o
libmicrohttpd - ~100 KB default, ~55 KB with most ./configure --disable-*
libonion - ~100 KB with most ONION_USE_* set to false
mongoose - ~100 KB including JSON-RPC
For the smaller builds of libmicrohttpd and libonion, I kept threads
support enabled, but disabled pretty much everything else that could be
disabled without patching the code. It looks like libmicrohttpd wins
this test. Maybe there's more code in libonion to disable (make into
compile-time options) - I haven't checked yet.
Built-in JSON support:
libevhtp+libevent - none
libmicrohttpd - none
libonion - JSON builtin, JSON-RPC in Apache 2 licensed example
mongoose - JSON-RPC builtin (simple JSON parser not exported?)
All of this is for current versions on GitHub or in recent release
tarballs as of a few days ago.
Maybe someone else will find this useful. I'd appreciate corrections.
It is very likely that I overlooked something.
On a related note, I found the list of alternate implementations on the
libmicrohttpd homepage very helpful. That's classy. Thanks.
My wishlist:
A processes (pre-fork) + [e]poll mode, like nginx has. Processes have
pros and cons vs. threads: more reliable, faster malloc/free (no lock
contention risk), but OTOH slower context switches (if running process
count exceeds number of logical CPUs). I would likely prefer this mode,
but all four libraries appear to be missing it.
Ability to accept not only HTTP, but also raw TCP connections, and
handle them in application code along with the library-handled HTTP.
Such as for implementing JSON-RPC directly over TCP, while also having
it over TCP+HTTP, and without having to manage an own/separate
threads/processes pool. Do any of the four have this? I found no such
examples with any of them.
Easily and cleanly embeddable into an application's source tree, while
also allowing easy updates to new upstream versions. mongoose almost
achieves this, but at the expense of sacrificing meaningful separation
into multiple translation units within the library itself. I think we
don't have to pay this price. We could have multiple files (10 or so?),
in a subdirectory, which are also easy to list in a project's Makefile.
Maybe I'd do that for libonion, freeing it from cmake, but then updating
to new upstream versions would be harder. Do I really have to bite the
cmake or/and autotools bullet for something as simple as accepting HTTP?
I'd prefer a more permissive license like 2-clause BSD or MIT. But I
guess I'll have to settle on Apache 2 or such. mongoose' use of GPLv2
is understandable - need to make money - but is otherwise a disadvantage
(even for a commercial project that could pay, and even when publishing
any source code changes is not a problem and would be planned anyway; we
just don't want to put our time into something that we would not always
be able to reuse in other projects).
Optional JSON from the same upstream is a plus, ideally exported both as
a generic JSON parser and as JSON-RPC support. Looks like only libonion
sort of delivers both (but the code might not be production quality).
Ability to exclude more of the functionality - for example, to include
only the POST method (and not compile in code for the rest). I am
concerned not so much about code size per se, as I am about attack
surface, and about ease of code reviews (not having to determine if some
compiled-in code is actually dead code in a given case, but to know
reliably that it's not compiled in).
On a related note, David's use of Coverity for libonion is commendable,
but it looks abandoned since 2014, and many "defects" (even if false
positives) remained unfixed back then.
Mark's use of Coverity for libevhtp is also commendable... and looks
abandoned since May 10, 2015. It shows "48,919 Lines of Code Analyzed",
only "4 Total defects" and "0 Outstanding" - I guess it means that
everything detected by Coverity before (which must have been many more
"defects") had been eliminated prior to that run. That's impressive.
But we don't know how many new "defects" may have appeared in the 9
months that passed. Also, I haven't looked into whether libevent has
been subjected to similar static analysis or not (although being
initially written by Niels Provos speaks in its favor, given Niels'
other work), and accepting TCP connections isn't as much risk as parsing
HTTP and JSON.
I don't give a lot of weight to the Coverity results for my
decision-making, but it shows whether the maintainers care, and there
are few other somewhat-meaningful metrics I could use before having
spent time to analyze and try to use the code myself.
Why am I posting this to the onion mailing list specifically? I find it
likely that libonion wins for me, although not by a large margin (and
there's a lot that I dislike about it). This is not a final decision
yet. I might as well end up reverting to writing special-purpose code
from scratch.
Thanks,
Alexander
David Moreno Montero
Feb 17, 2016, 5:34:21 AM2/17/16
to Solar Designer, Sergey Lyubka, Mark Ellzey, onio...@coralbits.com, Christian Grothoff
Thanks a lot for the comparison and wishliat, it's very nice to hear what people expect from the library, as sometimes it just fits some requirements and there are some blind spot easy to fulfill, but just unknown.
I think the following ideas are possible within onion:
1. process + epoll. I also though about it, but it could be used only on some specific projects. Anyway as would be a flag it's entirely doable.
2. Create a TCP server for json RPC. It would need to create a new listen point type and protocol. It's easy, but definitely needs some onion internals knowledge. Just know it knoews about http and https, but it's ready for other as http2, quic, or any other. It could also not map to http semantics (URL, headers) and only use the onion epoll, which I think it's your idea.
3. To just create a makefile with onion (and don't use cmake), you can copy the required files and just compile them. There is a build option that concatenates all required files into one, in this case for optimisation, but you could use this one big file as the source for easy makefile compilation. With some extra makefile, or scripts, it's easy to keep it up to date with upstream.
If you need more help using onion, please write to me, or at the list.
Also I'm a freelancer and love working on onion, so if you need professional services, please contact me privately.
Thanks again for the comparison.
Sergey Lyubka
Feb 17, 2016, 9:32:55 AM2/17/16
to Solar Designer, onio...@coralbits.com, Mark Ellzey, Christian Grothoff, David Moreno Montero
Thanks for sending that comparison, it's quite helpful.
As original author of Mongoose server, I'd like to make a couple of points:
1. Binary size. There is a lot of functionality enabled in Mongoose by default, which can be trimmed down with compilation flags. For example, this compilation line trims mongoose binary to about 70k (run in cloned mongoose repo), on x86_64 MacOS:
make -C mongoose/examples/simplest_web_server clean all CFLAGS_EXTRA="-Os -DMG_DISABLE_MQTT -DMG_DISABLE_JSON_RPC -DMG_DISABLE_HTTP_DIGEST_AUTH -DMG_DISABLE_CGI -DMG_DISABLE_DAV -DMG_DISABLE_DNS -DMG_DISABLE_HEXDUMP -DMG_DISABLE_SSI -DMG_DISABLE_RESOLVER" && strip mongoose/examples/simplest_web_server/simplest_web_server && ls -l mongoose/examples/simplest_web_server/
-rwxr-xr-x 1 lsm staff 68052 Feb 17 14:21 simplest_web_server
2. Mongoose on Linux is single threaded, can use epoll for IO multiplexing.3. Mongoose is not just a web server. It's a swiss army knife, currently having support for HTTP, Websocket, MQTT, CoAP, DNS, JSON-RPC, plain TCP, plain UDP protocols. Also, for each protocol, it implements both client and server functionality.
4. "mongoose almost achieves this, but at the expense of sacrificing meaningful separation into multiple translation units within the library itself" - Internally, our development repo has multiple files (just like SQLite), but we amalgamate sources for the production use. This makes embedding easy, which is our ultimate goal.
Mongoose license is GPLv2/commercial, cause we have a company that backs up support & development, and it's trusted by small and big businesses, our clients include NASA, Intel, Qualcomm, Samsung, HP, and many more. </marketing>.
Again, thanks for a useful review.
--
Sergey Lyubka Founder, CTO Cesanta Software
www.cesanta.com sergey...@cesanta.com +353 86 043 45 19
Solar Designer
Feb 19, 2016, 11:22:14 PM2/19/16
to Sergey Lyubka, onio...@coralbits.com, Mark Ellzey, Christian Grothoff, David Moreno Montero
On Wed, Feb 17, 2016 at 02:32:54PM +0000, Sergey Lyubka wrote:
> Thanks for sending that comparison, it's quite helpful.
> As original author of Mongoose server, I'd like to make a couple of points:
Thank you!
> Thanks for sending that comparison, it's quite helpful.
> As original author of Mongoose server, I'd like to make a couple of points:
> 1. Binary size. There is a lot of functionality enabled in Mongoose by
> default, which can be trimmed down with compilation flags. For example,
> this compilation line trims mongoose binary to about 70k (run in cloned
> mongoose repo), on x86_64 MacOS:
>
> -DMG_DISABLE_MQTT -DMG_DISABLE_JSON_RPC -DMG_DISABLE_HTTP_DIGEST_AUTH
> -DMG_DISABLE_CGI -DMG_DISABLE_DAV -DMG_DISABLE_DNS -DMG_DISABLE_HEXDUMP
> -DMG_DISABLE_SSI -DMG_DISABLE_RESOLVER" && **strip
> -DMG_DISABLE_CGI -DMG_DISABLE_DAV -DMG_DISABLE_DNS -DMG_DISABLE_HEXDUMP
> mongoose/examples/simplest_web_server/simplest_web_server && ls -l
> mongoose/examples/simplest_web_server/*
>
> *-rwxr-xr-x 1 lsm staff 68052 Feb 17 14:21 simplest_web_server*
That's nice. FWIW, my ~100 KB figure for libonion was based on running
size(1) on the .so file. I've since tried linking the examples/jsonrpc
program (including JSON-RPC) against the .a file, and for that I also
got 68 KB for "text". The binary size is slightly larger than 70 KB,
but overall it appears the different libraries here are of similar size,
probably except for libevhtp.
> 2. Mongoose on Linux is single threaded, can use epoll for IO multiplexing.
several other single-threaded embeddable HTTP server libraries.
> 3. Mongoose is not just a web server. It's a swiss army knife, currently
> plain UDP protocols*. Also, for each protocol, it implements *both client
> and server functionality*.
Yes, I noticed you had another project called Fossa, which you've since
merged into Mongoose. I didn't realize it included client functionality.
> 4. "mongoose almost achieves this, but at the expense of sacrificing
> meaningful separation into multiple translation units within the library
> itself" - Internally, our development repo has multiple files (just like
> SQLite), but we amalgamate sources for the production use. This makes
> embedding easy, which is our ultimate goal.
> Mongoose license is GPLv2/commercial, cause we have a company that backs up
> support & development, and it's trusted by small and big businesses, our
> clients include NASA, Intel, Qualcomm, Samsung, HP, and many more.
> </marketing>.
> > On a related note, David's use of Coverity for libonion is commendable,
> > but it looks abandoned since 2014, and many "defects" (even if false
> > positives) remained unfixed back then.
> >
> > Mark's use of Coverity for libevhtp is also commendable... and looks
> > abandoned since May 10, 2015. It shows "48,919 Lines of Code Analyzed",
> > only "4 Total defects" and "0 Outstanding" - I guess it means that
> > everything detected by Coverity before (which must have been many more
> > "defects") had been eliminated prior to that run. That's impressive.
> > But we don't know how many new "defects" may have appeared in the 9
> > months that passed. Also, I haven't looked into whether libevent has
> > been subjected to similar static analysis or not (although being
> > initially written by Niels Provos speaks in its favor, given Niels'
> > other work), and accepting TCP connections isn't as much risk as parsing
> > HTTP and JSON.
been subjected to Coverity, and in fact libmicrohttpd scores best -
last checked in 2016, with zero outstanding "defects".
https://scan.coverity.com/projects/5084 libevhtp
https://scan.coverity.com/projects/1815 onion
https://scan.coverity.com/projects/gnu-libmicrohttpd
https://scan.coverity.com/projects/kazan417-mongoose
> > I don't give a lot of weight to the Coverity results for my
> > decision-making, but it shows whether the maintainers care, and there
> > are few other somewhat-meaningful metrics I could use before having
> > spent time to analyze and try to use the code myself.
Reply all
Reply to author
Forward
0 new messages