Accessing the body of a request

[Tori Baker]

Hello,

We run a test which mocks out a webserver and we have embedded the mongoose webserver into the test to do this.  We moved away from the original mongoose server because of its inability to access the body of an incoming request. However, we're now likely going back to this server or something similar. Does civetweb offer a way to read the body of an incoming request? I couldn't find any documentation or examples on this specifically, and I would really like to know if this is possible before swapping out the web server.  Any documentation and examples on this would be greatly appreciated.

Thanks!

Tori

[bel]

Hello,

Yes, it is possible to read the HTTP body data.

There are multiple ways to do this, depending on your exact needs.

If you to write some C/C++ code to read the body data, to store/process it somehow in a C/C++ program, the embedded_c example would be the best starting point.

If you want to use the pre-built server executable, you could add a Lua or CGI script to read and process the body data - reflect it to the client, store it to a file, calculate a checksum, decode it from JSON or XML and compare elements ... - then the test directory contains some examples.

If you tell me a little what you want to do with the body, I could give you a hint where to look exactly.

[Tori Baker]

Thanks! I am trying to read the entire body of an incoming request into a string in C/C++ so that it can be cast to a protocol buffer in C++.

--
Sourceforge
https://sourceforge.net/projects/civetweb/
---
You received this message because you are subscribed to the Google Groups "civetweb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to civetweb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/civetweb/1ad7de94-f5b3-4c72-9ff6-d133db387862n%40googlegroups.com.

[bel]

The full API of CivetWeb is available in C, so I usually recommend using the C API, not the C++ API.

A "POST" handler reading the entire body data of a request can be found here:

https://github.com/civetweb/civetweb/blob/c2918246bcbe87a627fe46b955a3ce9e2d465554/examples/embedded_c/embedded_c.c#L562

The handler function is registered using

mg_set_request_handler(ctx, "/postresponse", PostResponser, 0);

--> All requests to http://127.0.0.1:8884/postresponse are handled by the function PostResponder. It essentially reflects the message body back to the client

The relevant calls to read a message body in a handler function:

r = mg_read(conn, buf, sizeof(buf));
while (r > 0) {
   r_total += r;

   // could use str.append(buf); in C++ here and feed the result to the protobuf library
   r = mg_read(conn, buf, sizeof(buf)); 

}

For completeness: The equivalent code using the C++ wrapper is here

https://github.com/civetweb/civetweb/blob/master/examples/embedded_cpp/embedded_cpp.cpp#L161

Another alternative might be to use Lua for the same:

https://github.com/civetweb/civetweb/blob/master/test/page6.lua

There is also a protobuf library in Lua (https://github.com/djungelorm/protobuf-lua). Sometimes using a scripting language is more convenient, but you have to decide this on your own what's most appropriate for your architecture. Just want to mention this possibility.

[Tori Baker]

Hi,

I have implemented the above, but it appears that I'm not getting the entirety of the request body.  I've run the same code with another webserver and get a much longer request body.  What I'm getting appears to just be the first 11 bytes of the request and then it drops the rest.  ri->content_length is set to 11 and using mg_read as you did above also only reads the first 11 bytes and leaves out the rest.  I believe it's possible that there is a nullptr in the middle of the request body, but I really need to get all of it. Is there another way to read the entire request? 

Thanks,

Tori

To view this discussion on the web visit https://groups.google.com/d/msgid/civetweb/7c3f7eb2-ab0e-4011-b7a6-31bac85eb584n%40googlegroups.com.

[bel]

> I have implemented the above ...

What alternative did you use ... I explained different options - which one did you go for?

> I've run the same code with another webserver and get a much longer request body. 

You mean the same web client code?

Can you reproduce this with a simple client like curl/wget, or do you need a specific client application for this test?

Do you always get this result, or is it sporadic?

HTTP or HTTPS?

localhost or remote?

Server on Windows or Linux or something else?

> ri->content_length is set to 11

Then I'm pretty sure the client did explicitly set 11 as a content length. If it does not send it, the value will be -1 (or 0 .. need to check ... but not 11), and you must read until you don't get any more data.

> I believe it's possible that there is a nullptr in the middle of the request body

You mean a 0 character? A 0 character in the message body is definitely not a problem for the CivetWeb webserver. Internally it does not handle the body as a null-terminated string, but as as binary (blob) - all characters are allowed.

> Is there another way to read the entire request? 

I can post a full example ... probably tomorrow.

Do you prefer Windows or Linux? (It does not matter for the server, but probably for a test script).

[bel]

Good to hear it's working for you now.

Here is the example, in case you (or sombody who finds this thread later) is interested in a reduced C example (smaller as compared to the embedded_c example).

It is built on Linux using the command:

gcc examples/demo.c src/civetweb.c -Iinclude -DNO_SSL -DNO_FILES -lpthread -odemo

run using 

./demo

and tested with curl using

curl -d mydata=here http://localhost:8081/post-me
--> Got 11 bytes of POST body data (expected 11 bytes)

curl -d @demo.c http://localhost:8081/post-me

--> Got 2078 bytes of POST body data (expected 2078 bytes)

curl -d @some-big-file.tar http://localhost:8081/post-me
--> Got 454561699 bytes of POST body data (expected 454561699 bytes)

And this is the source (demo.c):

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#include <unistd.h> // or #include <windows.h> for Windows

#include "civetweb.h"

int with_https = 0;


int post_handler(struct mg_connection *conn, void *cbdata)
{
    const struct mg_request_info *request_info = mg_get_request_info(conn);
    if ((request_info == NULL) || (request_info->request_method == NULL)) {
        mg_send_http_error(conn, 500, "Internal Error"); // will not happen
        return 500;
    }
    if (0!=strcmp(request_info->request_method, "POST")) {
        mg_send_http_error(conn, 405, "Only POST allowed");
        return 405;
    }

    long long expected_len = request_info->content_length;
    long long read_len = 0;

    while (1) {
        char buffer[1024];
        int rnow = mg_read(conn, buffer, sizeof(buffer));
        if (rnow <= 0) {
            /* No more data. Connection closed --> rnow == 0, Connection error --> rnow < 0 */
            break;
        } else {
            /* Append rnow bytes from buffer to .. wherever you want to put the post data */
            read_len += rnow;
        }
    }

    char response[1024];
    int len = sprintf(response, "Got %lli bytes of POST body data (expected %lli bytes)\n", read_len, expected_len);
    mg_send_http_ok(conn, "text/plain", len);
    mg_write(conn, response, len);

    return 200;
}


static void init_context(const struct mg_context *ctx)
{
    mg_set_request_handler(ctx, "/post-me", post_handler, NULL);
}


static int log_message(const struct mg_connection *conn, const char *message)
{
    fprintf(stderr, "Error: %s\n", message);
    return 0;
}


volatile int running;


int main(int argc, char *argv[])
{
    /* Init CivetWeb library once for every process */
    mg_init_library(0 /* HTTP only: 0, HTTP+HTTPS: MG_FEATURES_TLS */);

    struct mg_callbacks callbacks;
    memset(&callbacks, 0, sizeof(callbacks));
    callbacks.init_context = init_context;
    callbacks.log_message = log_message;

    const char *configuration_options[] = {"listening_ports", "8081", NULL, NULL};

    struct mg_context *    ctx = mg_start(&callbacks, NULL, &configuration_options);
    if (!ctx) {
        /* Error */
        mg_exit_library();
        return 1;
    }

    running = 1;
    while (running) {
        sleep(1); // or Sleep(1000) for Windows
    }

    mg_stop(ctx);
    mg_exit_library();

    return 0;
}

[Tori Baker]

Hi, I believe I may have been looking at the wrong request :S. It looks to be working now! Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/civetweb/fdea4c29-cb15-4846-84f9-f8f362161275n%40googlegroups.com.