How to use multi-thread server?

[tumbler]

If I use server in single-threaded mode it works fine.
WebServers.createWebServer(8080);
But if I try it with multi-threaded executors, like that:
WebServers.createWebServer(Executors.newFixedThreadPool(2), 8080);
Then I have exceptions both on client and server sides. On the server
I sync by connection object, but it doesn't help.

[Aslak Hellesøy]

On Thursday, 10 January 2013 at 16:59, tumbler wrote:

If I use server in single-threaded mode it works fine.

WebServers.createWebServer(8080);

But if I try it with multi-threaded executors, like that:

WebServers.createWebServer(Executors.newFixedThreadPool(2), 8080);

Then I have exceptions both on client and server sides.

Exception stack traces would be handy.

Aslak

[tumbler]

On 10 янв, 19:34, Aslak Hellesøy <aslak.helle...@gmail.com> wrote:
> Exception stack traces would be handy.
>
> Aslak

I'll try describe my problem in detail.
On the server side I create and start WebSocketServer with my handler
like that:

1 public void onOpen(WebSocketConnection connection) {
2 ConnectionWS conn = new ConnectionWS(connection);
3 connection.data("user_connection", conn);
4 }
5
6 public void onMessage(WebSocketConnection connection, String msg)
{
7 ConnectionWS conn = (ConnectionWS)
connection.data("user_connection");
8 submitToExecutor(conn, new WorkTask(uc, msg));
9 }

Also I have a load testing suite with my domain objects.
That environment generates bunch of clients and emulates their
activities in parallel.
I use JettyWebSocketClient to connect my server, but implementation
doesn't matter.

So, in case of single-threaded Webbit server my test suite finished
successfully.
But I think single thread to all clients' events may be bottleneck for
1000+ active clients.
That's why I try use some ThreadPool and create multi-thread Webbit:
WebServers.createWebServer(Executors.newFixedThreadPool(2), 8080);

That one change(+1 thread) completely breaks test suite.
On the server side I have casual NullPointerException because in line
#7 conn is null. Even if I do synchronize by WebSocketConnection
connection in all methods. More over if I sync all listener methods
this error may happens!

On the client side(my load tests) sometimes I have error when connect
to the server:
java.net.ProtocolException: Bad Sec-WebSocket-Accept.

So, how I should use Webbit if I want more than one thread?

[Joe Walnes]

Hi Tumbler,

Webbit is designed to be a single threaded non-blocking server. All the assumptions inside this code are for this model. In short, you should only pass in a single threaded executor to Webbit.

However, your handlers are free to communicate with other threads, for example if you are doing computationally expensive tasks or need to interact with an IO library that does not have a non-blocking interface. So long as you follow these rules:

* Only ever pass a single threaded executor into the Webbit constructor.

* Perform all your interactions with Webbit (including reading requests and writing responses) on this main executor.

If you find you are doing this a lot and it's really inconvenient, you may be better of using a classic thread-per-request model, and would have more success with a typical Servlet container. Async non-blocking models only make sense for some situations.

[tumbler]

Thanks for explanation, Joe. Then I think assumption about only single-
threaded executor should be added to the constructor javadoc or
somewhere else.
After that I have one question. How important send the responses to
WebSocketConnection in the same event thread?
I mean not just call connection.send(message) from others threads,
but do:
connection.execute(new Runnable() {
@Override
public void run() {
connection.send(message);
}
});
I'm asking because until didn't meet any issue with that.

[Joe Walnes]

On Fri, Jan 11, 2013 at 4:32 AM, tumbler <and...@gmail.com> wrote:

Thanks for explanation, Joe. Then I think assumption about only single-
threaded executor should be added to the constructor javadoc or
somewhere else.

I agree.

 

After that I have one question. How important send the responses to
WebSocketConnection in the same event thread?
I mean not just call  connection.send(message) from others threads,
but do:
connection.execute(new Runnable() {
  @Override
  public void run() {
    connection.send(message);
  }
});
I'm asking because until didn't meet any issue with that.

WebSocketConnection.send() should always be called from the main event thread (i.e. the executor code you have above is correct). It may work at the moment from other threads, but I cannot guarantee that in the future.

[Luke Hutchison]

On Thursday, January 10, 2013 11:17:44 AM UTC-8, Joe Walnes wrote:

Webbit is designed to be a single threaded non-blocking server. All the assumptions inside this code are for this model. In short, you should only pass in a single threaded executor to Webbit.

However, your handlers are free to communicate with other threads, for example if you are doing computationally expensive tasks or need to interact with an IO library that does not have a non-blocking interface. So long as you follow these rules:

* Only ever pass a single threaded executor into the Webbit constructor.

* Perform all your interactions with Webbit (including reading requests and writing responses) on this main executor.

All my requests hit MongoDB, which is relatively quick at responding, but I don't want to hold up other requests (e.g. requests for static content) because the single Webbit executor thread is blocking on MongoDB. Therefore, for each request that comes in, I want to farm off response generation to worker threads, and have them then pass the responses back to the main Webbit thread when they are done. However, I don't see how that can be done, since I assume that the HttpResponse object goes away when the HttpHandler.handleHttpRequest() method exits (which it will do right after sending the response generation task to another thread in this async scenario).

Can you please give an example of how to correctly handle large numbers of concurrent blocking requests in Webbit using worker threads that send responses back to the main thread when they're done?

[Joe Walnes]

The handleHttpRequest() method does not finish the request on exit. That has to be done explicitly using HttpResponse.end(). So what you described will work.

The one thing to watch out for is that you should only ever work with the HttpResponse from the main Webbit thread. So when your background thread retrieves a result you should called HttpControl.execute(Runnable) to get back onto the main thread.

--
You received this message because you are subscribed to the Google Groups "Webbit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to webbit+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Luke Hutchison]

Thanks for explaining how to have the worker thread call HttpControl.execute(Runnable) to get back onto the main thread followed by writing to the HttpResponse from there. It seems to work fine.

This does complicate the design though, because now I need a wrapper object to pass back all the HttpResponse fields to the main thread (content type, cookies, status code, content, other response headers). This duplicates the methods of HttpResponse. Is it safe to call all the HttpResponse methods except for end() or error() from worker threads, assuming that only one worker thread will ever be handling one request?  It would be much simpler to only have to run end() or error() back on the main thread once the response is fully composed. (Chunked responses would still need to call HttpControl.execute(Runnable) for each new chunk, once chunked encoding is supported.)

[Joe Walnes]

I would recommend against it. Although right now it would actually work, there's no guarantee that future versions of Webbit will uphold this contract.

Though it does make me think that we should add a helper to Webbit to make it easy to package up these values and pass around.

On Tue, Apr 9, 2013 at 5:41 AM, Luke Hutchison <luke....@gmail.com> wrote:

Thanks for explaining how to have the worker thread call HttpControl.execute(Runnable) to get back onto the main thread followed by writing to the HttpResponse from there. It seems to work fine.

This does complicate the design though, because now I need a wrapper object to pass back all the HttpResponse fields to the main thread (content type, cookies, status code, content, other response headers). This duplicates the methods of HttpResponse. Is it safe to call all the HttpResponse methods except for end() or error() from worker threads, assuming that only one worker thread will ever be handling one request?  It would be much simpler to only have to run end() or error() back on the main thread once the response is fully composed. (Chunked responses would still need to call HttpControl.execute(Runnable) for each new chunk, once chunked encoding is supported.)

[Luke Hutchison]

I don't know how a nontrivial data-backed website or web service could be built that doesn't block on most HTTP requests, because most HTTP requests will need to hit a database. If all your database collections can fit in HashMaps in memory, then maybe you can do all data accesses on the main thread (and you can manually persist changes to the HashMaps to disk), but otherwise you need to hit a database, and it's a bit of a pain to have to create a Runnable that is sent off to a worker thread and then create another Runnable to go back to the main thread. I know that the callback-that-calls-a-callback model is what Node.js and similar follow today, but it makes your code unnecessarily complicated, especially in Java.

In my code, I created a helper class that spawns the worker thread, collects all the HttpResponse-style parameters of the result and then calls back to the main thread. But I do think that something like this should provided in Webbit itself, because the need to query a database or access a file in an HTTP handler is pretty much universal, especially in the "stateless HTTP server" model, which punts all state to the database (and to session-in-client browser cookies) for scalability.

You already have the async worker logic in place for IOWorkers; it would be nice to genericize this and create some sort of BlockingHttpHandler or something that is automatically wrapped in the async worker logic for you.

--
You received this message because you are subscribed to a topic in the Google Groups "Webbit" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/webbit/Vr_MN9CwNsA/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to webbit+un...@googlegroups.com.

[peter hausel]

On Wednesday, April 10, 2013 3:39:20 AM UTC-4, Luke Hutchison wrote:

I don't know how a nontrivial data-backed website or web service could be built that doesn't block on most HTTP requests, because most HTTP requests will need to hit a database. If all your database collections can fit in HashMaps in memory, then maybe you can do all data accesses on the main thread (and you can manually persist changes to the HashMaps to disk), but otherwise you need to hit a database, and it's a bit of a pain to have to create a Runnable that is sent off to a worker thread and then create another Runnable to go back to the main thread. I know that the callback-that-calls-a-callback model is what Node.js and similar follow today, but it makes your code unnecessarily complicated, especially in Java.

In my code, I created a helper class that spawns the worker thread, collects all the HttpResponse-style parameters of the result and then calls back to the main thread. But I do think that something like this should provided in Webbit itself, because the need to query a database or access a file in an HTTP handler is pretty much universal, especially in the "stateless HTTP server" model, which punts all state to the database (and to session-in-client browser cookies) for scalability.

You already have the async worker logic in place for IOWorkers; it would be nice to genericize this and create some sort of BlockingHttpHandler or something that is automatically wrapped in the async worker logic for you.

The trouble I see with this is that it would be very cumbersome to implement a generic user facing API without closure+lightweight SAM creation support that would allow you to carry the context while managing the thread switching (or at least, the end result would not look much better than it is today). Java 8 will fix both of these problems, until then one can use one of the alternative JVM languages with Webbit or create use case specific control-flows. 

[Luke Hutchison]

On Wed, Apr 10, 2013 at 7:37 AM, peter hausel <peter....@gmail.com> wrote:

The trouble I see with this is that it would be very cumbersome to implement a generic user facing API without closure+lightweight SAM creation support that would allow you to carry the context while managing the thread switching (or at least, the end result would not look much better than it is today). Java 8 will fix both of these problems, until then one can use one of the alternative JVM languages with Webbit or create use case specific control-flows. 

I don't really understand what you mean here: what problems are you describing, and how will features in Java 8 fix these problems?

Here's an example of my asynchronous handler code. This class is detected on the classpath and automatically added to the server on startup by virtue of extending RestHandler, and the @Get annotation enables parameter parsing using Webbit Easyremote. The parameters "name" and "password" are then bound to name and pw respectively using the @Param annotation. The html() method creates a Response object, which is the class that wraps up response status, content, content type etc.:

@Get("/login/{name}/{password}")

public class LoginHandler extends RestHandler {

    public Response handleRequest(HttpRequest req, @Param("name") String name, @Param("password") String pw) {

        User user = User.authenticate(name, pw);

        if (user != null) {

            String pageHtml = Page.render("Login", Page.text("Successfully logged in));

            Session cookie = new Session(user);

            return html(pageHtml) //

                    .cookie(cookie, user.getAuthTokenExpires());

        } else {

            return html(Page.render("Login failed", Page.text("Username/password incorrect"))) //

                    .status(Response.UNAUTHORIZED)

                    .deleteCookie(Session.COOKIE_NAME);

        }

    }

}

The following is off-topic, but you can also add a parameter of type User to the handleRequest method, and the authentication is performed for you by the RestHandler superclass (handleRequest will never be called unless the user is already authenticated):

@Get("/profile/{profileid}")

public class ProfileHandler extends RestHandler {

    public Response handleRequest(HttpRequest req, @Param("profileid") String profileid, User user) {

        return html(Page.render("Login", Page.text("Looking at profile for " + profileid

                    + "; user is logged in and verified as " + user.getEmail())));

    }

}

The RestHandler superclass that all handlers extend implements HttpHandler, and does a bunch of introspection in the constructor to find the handleRequest method and check the annotations, then in handleHttpRequest it does the following to invoke the subclass method handleRequest:

// [...in method handleHttpRequest:...]

final RestHandler handler = this;

Gribbit.threadPoolExecutor.submit(new Runnable() {

@Override

public void run() {

// Invoke subclass method handleRequest() with extra params extracted from URL. Method can block.

Response response;

try {

response = (Response) handleRequestMethod.invoke(handler, args);

} catch (Exception e) {

response = new Response().status(Response.INTERNAL_SERVER_ERROR).contentText("Error processing this request");

}

// Once request is complete, schedule writing of result back to response on the main Webbit server thread.

final Response responseCopy = response;

ctl.execute(new Runnable() {

@Override

public void run() {

// Send response back on 'res'

responseCopy.sendResponse(res, ctl);

}

});

}

});

Note the "response = new Response()..." line, this is what the html() method does internally to create a Response object in the above examples.

Anyway, it's a fairly simple system that works well for what I need. The only problem is it doesn't support chunked responses as-is, because the handleRequest method is not passed the HttpResponse object to prevent handleRequest doing anything with the response directly, since it is not the main Webbit thread. To solve this problem, I may have to pass in a wrapper for HttpResponse that is able to call ctl.execute(new Runnable(){}) for each chunk of response, but that waits until handleRequest exits to send a non-chunked response.

Does this look useful to anybody? I'll probably put up a github repo soon with all this code. (For reference, my tiny Web framework that builds on Webbit is called "Gribbit".)

[peter hausel]

On Wednesday, April 10, 2013 1:07:19 PM UTC-4, Luke Hutchison wrote:

On Wed, Apr 10, 2013 at 7:37 AM, peter hausel <peter....@gmail.com> wrote:

The trouble I see with this is that it would be very cumbersome to implement a generic user facing API without closure+lightweight SAM creation support that would allow you to carry the context while managing the thread switching (or at least, the end result would not look much better than it is today). Java 8 will fix both of these problems, until then one can use one of the alternative JVM languages with Webbit or create use case specific control-flows. 

I don't really understand what you mean here: what problems are you describing, and how will features in Java 8 fix these problems?

Java 8 will help you to generalize a snippet like this[1] so you could write

public class MyHandler extends WorkerHandler {

  @Override

  public void handleHttpRequest(HttpRequest request, HttpResponse response, HttpControl control) {

      

      worker<BlockingTaskResult>(() -> {arbitraryBlockingTask}, (blockingtaskResult) -> control.execute(()-> { response.write(blockingtaskResult.toString()}; response.end();) );

  }

}

I like your solution but this is what I was referring as use case specific control flow (i.e. it looks to me that your async blocking IO handler implementation has many non core webbit specific ideas in it)

Just my 2c.

Peter

[1]Gribbit.threadPoolExecutor.submit(new Runnable() {

[Vijay Nathani]

I am unable to get this to work. I am getting multiple WebbitException: cannot send more responses than requests on ....

The code used to run the background long running task is mentioned below.

public class BackgroundRunner { //Singleton

    private ExecutorService threadPool = Executors.newFixedThreadPool(5);

    public void execute(final ComputeJSON backgroundTask, final HttpResponse res, final HttpControl ctl) {

        threadPool.execute(new ComputeJSONRunner(backgroundTask, res, ctl));

    }

}

interface ComputeJSON {

    String task();

}

class ComputeJSONRunner implements Runnable {

    private final ComputeJSON taskToBeRun;

    private final HttpResponse res;

    private final HttpControl ctl;

    public ComputeJSONRunner(ComputeJSON taskToBeRun, HttpResponse res, HttpControl ctl) {

        this.taskToBeRun = taskToBeRun;

        this.res = res;

        this.ctl = ctl;

    }

    public void run() {

        final String out = taskToBeRun.task();

        ctl.execute(new Runnable() {

            public void run() {

                res.header("Content-type", "application/json").content(out).end();

            }

        });

    }

}

Help appreciated.