examples of multi-threaded HTTP server (POST requests, constant file GET-ing)

[Basile Starynkevitch]

Hello all,

I am working (in static analysis of C or C++ code) on GCC MELT (a lispy domain specific language to customize the GCC compiler) and I am trying to make some Web interface to it (some MELT monitor, as I call it). As soon (that might be never!) as something is working, I will release it as free software, probably GPLv3 (or perhaps LGPLv3) licensed.

So the dream would be to have a long-running Linux (free software) program which I call the MELT monitor, with a web interface, communicating with MELT enhanced GCC compilation processes (using some JSON protocol), and interacting with users to show them their C++ source code with some annotations or inferred properties. So I need an HTTP C++11 server library, with:

1. a few web clients, typically a dozen of browsers (for a team of a dozen developers using my hypothetical MELT monitor) i.e. a few dozen of browsers tab, i.e. less than a hundred of not very active connection. In particular, scalability to thousands of HTTP connections is not a concern to me.
   
2. C++11 compatible for Linux (and I love playing tricks like generating C++ code at runtime, forking a compilation of that emitted C++ code into a shared object, then dlopen-ing these runtime generated plugins; MELT is doing that and I am ease with such ideas.).
   
3. multi-thread friendly (that could even mean a rule like all HTTP processing happens only in the main thread, but I need to understand what to do)

Hence, I need to understand how to handle POST HTTP requests (mostly Ajax) and how to deliver thru a GET HTTP request some content given by a static file (such as a local copy of jquery.min.js, or some CSS file, etc...) or a static string on the HTTP server side.

BTW, I am a bit familiar with libonion/ (a C library for HTTP server). But its C++11 support is still very weak.

Are there examples of HTTP server code above hello_world_server.html explaining how to handle some POST and GET request? Perhaps a simple thing with an old Web POST <form> using some constant CSS style (in a separate file)?

Are there guidelines for choosing asynchronous versus synchronous servers?

My apologies for asking such a simple questions!

Thanks for reading, regards.

--
Basile Starynkevitch (France)     http://starynkevitch.net/Basile/

[Basile Starynkevitch]

For reference, I also asked on StackOverflow and I am really after the equivalent, in C++11 for cpp-netlib 0.11.x, the equivalent of the simple examples/post/ of libonion (but in idomatic C++11).

Regards.

[Dean Michael Berris]

Thanks Basile (sorry for the delay, I'm based in Sydney, and timezones are hard) -- quick points as I'm prepping for travel to the US this weekend:

- See https://github.com/cpp-netlib/cpp-netlib/pull/468 to find an example of how to do this.

- I suggest the documentation (http://cpp-netlib.org/0.11.1/index.html), more specifically the part on the HTTP server (http://cpp-netlib.org/0.11.1/reference/http_server.html)

- I've always wanted to put an example of how to do POST handling, and I just might do it on the plane from Sydney to Los Angeles.

If you manage to pull something off, I encourage you to definitely send me a pull request on an example. I would be very happy to review it.

Cheers

--
You received this message because you are subscribed to the Google Groups "The C++ Network Library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpp-netlib+...@googlegroups.com.
To post to this group, send email to cpp-n...@googlegroups.com.
Visit this group at http://groups.google.com/group/cpp-netlib.
For more options, visit https://groups.google.com/d/optout.

[Basile Starynkevitch]

On Saturday, March 21, 2015 at 1:52:57 AM UTC+1, Dean Michael Berris wrote:

Thanks Basile (sorry for the delay, I'm based in Sydney, and timezones are hard) -- quick points as I'm prepping for travel to the US this weekend:

- See https://github.com/cpp-netlib/cpp-netlib/pull/468 to find an example of how to do this.

Thanks for that URL, but I find that example completely opaque and non-understandable for a newbie like me.

To be specific, examples/post of libonion is doing the following:

It starts a web server on e.g. http://localhost:8080/ and when you GET that URL (that is, you issue an HTTP GET request from your browser) the program answers the following content of text/html MIME type

<!doctype html>
<html>
<head>
 <title>Simple post example</title>
</head>
<body>
Write something:
<form method='POST' action='data'>
<input type='text' name='text'>
<input type='submit' name='send' value='do'>
</form>
</body>
</html>

The next HTTP request (done from the user's browser) is of course a POST request for http://localhost:8080/data with a body of MIME type application/x-www-form-urlencoded and with content
text=Basile+Starynkevitch&send=do

In my wildest dreams I would want the example to answer the following text/html content:

<!doctype html>
<html>
<head><title>hello</title></head>
<body>
<h1>hello</h1>
Hello, <i>Basile Starynkevitch</i>
</body></html>

Unfortunately, after spending two hours trying various things, I am not able to make something as simple as that working with ccp-netlib. I really believe that you should have more than one single hello_world_server.html in your documentation. It is simply not enough.

BTW, standard C++ <iostream> is also internally a complex template library, but every tutorial gives the std::cout << "hello world" << std::endl; example then the int i; std::cin >> i; example, and the equivalent is missing in cpp-netlib. It would be really nice if the documentation contained a few graduately increasing examples (not just the single hello world one) about HTTP server library in cpp-netlib.

If any cpp-netlib guru could take a few minutes to give such a simple example (which I did not found on the web with Google!), I would be very grateful.

BTW, the stackoverflow Accessing request headers in cpp-netlib http server question stays unanswered since October 2013, and that is IMHO significant. It looks that very few people
are actually using cpp-netlib for HTTP server side.

Regards.

[Dean Michael Berris]

First, thanks for your feedback Basile -- it's been a while since I've gotten new user feedback this detailed!

Next, I don't trawl StackOverflow as often as I used to anymore (but I really should do that again). Thanks for bringing those questions up!

Next, please find my response in-line (I also responded to your question on stack overflow, along with the other question you linked to).

On Sun, Mar 22, 2015 at 12:09 PM Basile Starynkevitch <bas...@starynkevitch.net> wrote:

On Saturday, March 21, 2015 at 1:52:57 AM UTC+1, Dean Michael Berris wrote:

Thanks Basile (sorry for the delay, I'm based in Sydney, and timezones are hard) -- quick points as I'm prepping for travel to the US this weekend:

- See https://github.com/cpp-netlib/cpp-netlib/pull/468 to find an example of how to do this.

Thanks for that URL, but I find that example completely opaque and non-understandable for a newbie like me.

Sorry about that -- that's written using the asynchronous server implementation, which is actually a little less straight-forward than the synchronous version implementation.

 

To be specific, examples/post of libonion is doing the following:

It starts a web server on e.g. http://localhost:8080/ and when you GET that URL (that is, you issue an HTTP GET request from your browser) the program answers the following content of text/html MIME type

<!doctype html>
<html>
<head>
 <title>Simple post example</title>
</head>
<body>
Write something:
<form method='POST' action='data'>
<input type='text' name='text'>
<input type='submit' name='send' value='do'>
</form>
</body>
</html>

That's easy in cpp-netlib. Too easy actually:

struct handler {

  void operator()(server::request const& req, server::response& res) {

    if (req.destination == "/") {

      server::response::stock_reply("text/html", "The HTML goes here");

      return;

    }

  // ... continue below ...

 

The next HTTP request (done from the user's browser) is of course a POST request for http://localhost:8080/data with a body of MIME type application/x-www-form-urlencoded and with content
text=Basile+Starynkevitch&send=do

In my wildest dreams I would want the example to answer the following text/html content:

<!doctype html>
<html>
<head><title>hello</title></head>
<body>
<h1>hello</h1>
Hello, <i>Basile Starynkevitch</i>
</body></html>

  // ... continue from my example...

    if (req.destination == "/data" && req.method == "POST") {

      // look up the content-type header

      std::string content_type, content_length;

      for (const auto& header : req.headers) {

        if (header.name == "Content-Type") content_type = header.value;

        if (header.name == "Content-Length") content_length = header.value;

        if (!content_type.empty() && !content_length.empty()) break;

      }

      if (content_type.empty() || content_length.empty()) return server::stock_reply(server::response::status_type::bad_request);

      // maybe do some better validation before proceeding...

      std::string filled_in_html;

      // do whatever you want to filled_in_html;

      return server::response::stock_reply("text/html", filled_in_html);

    }

    return server::response::stock_reply(server::response::status_type::not_found);

  }

  void log(const string&) { ... };

};

 

Unfortunately, after spending two hours trying various things, I am not able to make something as simple as that working with ccp-netlib. I really believe that you should have more than one single hello_world_server.html in your documentation. It is simply not enough.

Sorry about that. Yes, we should fix that.

Do you have any time to probably contribute something to the documentation? Perhaps, which questions you'd like answered, and what kind of example would be helpful for people that's compelling and easy to build upon?

If not your direct contribution, would you mind if I ask you to review some pull requests to the main repo, so that we can iterate on something that's helpful to you and potentially to others finding the library a little opaque?

 

BTW, standard C++ <iostream> is also internally a complex template library, but every tutorial gives the std::cout << "hello world" << std::endl; example then the int i; std::cin >> i; example, and the equivalent is missing in cpp-netlib. It would be really nice if the documentation contained a few graduately increasing examples (not just the single hello world one) about HTTP server library in cpp-netlib.

If any cpp-netlib guru could take a few minutes to give such a simple example (which I did not found on the web with Google!), I would be very grateful.

I hope this helps. :)

 

BTW, the stackoverflow Accessing request headers in cpp-netlib http server question stays unanswered since October 2013, and that is IMHO significant. It looks that very few people
are actually using cpp-netlib for HTTP server side.

I know people are using cpp-netlib on the server side, based on the evidence on the mailing list archives. There's been discussion on this for a while already, and unfortunately I haven't been able to devote enough time to it (the documentation) recently.

Maybe that can change now that I'm traveling and may have a little more time to deal with some "20%" time. ;)

Cheers

[Basile Starynkevitch]

On Mon, Mar 23, 2015 at 04:56:15PM +0000, Dean Michael Berris wrote:
> First, thanks for your feedback Basile -- it's been a while since I've
> gotten new user feedback this detailed!
>

> Do you have any time to probably contribute something to the documentation?

Not really directly, because I strongly believe that to contribute some documentation
I need to understand what I am talking about, which is not the case today
for cpp-netlib.

> Perhaps, which questions you'd like answered, and what kind of example
> would be helpful for people that's compelling and easy to build upon?
>
> If not your direct contribution, would you mind if I ask you to review some
> pull requests to the main repo, so that we can iterate on something that's
> helpful to you and potentially to others finding the library a little
> opaque?
>

I'll be delighted to help by reviewing some pull requests
(but I won't spend a lot of time on that).


Currently (but that might change a lot), I gave up using cpp-netlib
(primarily because of the lack of documentation), and I am (up to the
moment) investigating the use of POCO library version 1.6 see
http://pocoproject.org/ ; I was able to write a tiny Web server in a
couple of minutes (because the documentation is IMHO well enough
written).

Previously, I was using libonion (in C99, not C++11) and I even have
contributed some code patches to it. Unfortunately, its C++11 wrapping
layer is currently buggy to the point of being unusable.

I am taking care to wrap my use of web server libraries, so in
principle it should be easy to switch.

BTW, I have an open-minded question for cpp-netlib community? Do you
have any cpp-netlib specific coding rules that I might try to validate
with my tool http://gcc-melt.org/ ?

[Basile Starynkevitch]

On Monday, March 23, 2015 at 5:56:18 PM UTC+1, Dean Michael Berris wrote:

First, thanks for your feedback Basile -- it's been a while since I've gotten new user feedback this detailed!

Sorry, but the HTML form had an <input type='text" name='text'> and I still don't understand where in your code are you accessing that?

Ideally I was dreaming of req.get_POST_argument("text") or something the like.

Thanks for your time answering to my naive questions.

Cheers

[Dean Michael Berris]

You're going to have to parse the incoming body of the request to do that. The server doesn't do any pre-parsing for you, because there's other kinds of data that can come through in a POST request.

--

[Basile Starynkevitch]

On Wed, Mar 25, 2015 at 08:31:53PM +0000, 'Dean Michael Berris' via The C++ Network Library wrote:
> You're going to have to parse the incoming body of the request to do that.
> The server doesn't do any pre-parsing for you, because there's other kinds
> of data that can come through in a POST request.
>

I know that, but I believe that HTTP server libraries in C++
are in particular useful for an embedded web server
(either embedded in a device, like most home or small office printers
or wifi routers toda, or embedded in a bigger software, like e.g. CUPS)

In both cases, simple GET or POST requests with a small content
(fitting into memory) are common, and POST requests in
application/x-www-form-urlencoded MIME types are really common
(and they are also used in many AJAX requests).

So I would suggest to have convenience functions and classes
for these common cases (which have simple support both in POCO
and in libonion -for C-).

I don't believe that most usages of cpp-netlib would be to
write Apache, Lighttpd, or Facebook or Google like things.
Obviously, it is a guess and I could be very wrong.

Cheers
--
Basile Starynkevitch http://starynkevitch.net/Basile/

[Dean Michael Berris]

On Wed, Mar 25, 2015 at 1:57 PM Basile Starynkevitch <bas...@starynkevitch.net> wrote:

On Wed, Mar 25, 2015 at 08:31:53PM +0000, 'Dean Michael Berris' via The C++ Network Library wrote:
> You're going to have to parse the incoming body of the request to do that.
> The server doesn't do any pre-parsing for you, because there's other kinds
> of data that can come through in a POST request.
>


I know that, but I believe that HTTP server libraries in C++
are in particular useful for an embedded web server
(either embedded in a device, like most home or small office printers
or wifi routers toda, or embedded in a bigger software, like e.g. CUPS)

Sure.

 

In both cases, simple GET or POST requests with a small content
(fitting into memory) are common, and POST requests in
application/x-www-form-urlencoded MIME types are really common
(and they are also used in many AJAX requests).

The second part of your statement isn't so true. People have used cpp-netlib embedded to POST binary data from the client and handle binary data from POST requests on the server side. REST-ful interfaces don't bother with application/x-www-form-urlencoded when you POST data that's basically JSON.

 

So I would suggest to have convenience functions and classes
for these common cases (which have simple support both in POCO
and in libonion -for C-).

I would love to have those hosted in something like boost/network/http/utils/ :)

If you can write them, I would certainly merge them in. :D

 

I don't believe that most usages of cpp-netlib would be to
write Apache, Lighttpd, or Facebook or Google like things.
Obviously, it is a guess and I could be very wrong.

That's fine, but cpp-netlib isn't meant to operate at the "web framework" level. It's meant to provide an HTTP server and client, in the most raw form possible to enable all the kinds of things users would want to do with them.

If you're writing a CMS or something that's web-related, you can build that on top of cpp-netlib. It also allows you to build really novel things that use HTTP as the transport mechanism.

It could be easier, yes. But unfortunately I haven't had too much time to devote to making this true. :(

Cheers