How not to lose messages on event bus?

[Štefan Miklošovič]

Hello,

I want to write vert.x verticle which listens to some TCP port and once an HTTP request comes, it should delegate this request to some worker verticle. Hence I can accept a lot of connections concurrently.

If I understand that correctly, I have to register some handler and when HTTP request comes, I send that message to the event bus and eventually some worker will pick that message and does some work on it and eventually sends some message back to the original sender.

I read a lot about vert.x and there is constantly one remark mentioned that it is possible that you can lose messages sent to event bus when e.g. node on which that event bus is running suddenly fails. In my problem, I can not afford the fact that some of the message could be potentially lost. I need all of them.

What bugs me particularly is the fact what happens with messages between the time they are sent to event bus and time they are delivered by event bus to some worker verticle. When I have 100 connections, that mean I will have 100 messages on event bus and when I have 20 workers, there will be 80 messages pending to be processed with now-busy worker.

When I "turn off" event bus, client who sent HTTP request waits for response but he will never get one because event bus fails to deliver them to the workers. Could be this problem somehow overriden?

Thanks

[Štefan Miklošovič]

It would be enough for me to sent e.g. status code 500 back to the original sender of HTTP request.

[Jordan Halterman]

Are you saying you don't want messages to be queued? Vert.x is an asynchronous framework, so nothing happens synchronously. There is absolutely nothing wrong with 80 messages waiting in a queue while 20 are being processed. Even using a synchronous framework where only one message is handled at a time, there is still potential for a client to send a request and never receive a response if the server dies while handling that one request and before handling whatever other requests the client has sent since. That's just the nature of clients and servers. Aside from the fact that queuing is not an internal server error (500), there's nothing fundamentally wrong with it. Messages are not going to be lost if you're, for instance, receiving a request and forwarding it to a worker within the same JVM. However, they could be lost when traveling over the network.

If your goal is to always have the server send some response to the client then that goal may be unattainable. And sending 500 errors for queued messages will only result in an incredibly unstable server from the client's perspective.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Štefan Miklošovič]

Thanks for your valuable insight, maybe I need to describe what I am trying to do

1. I create 1 verticle, in start() I create HttpServer and listen to some port.

2. From now on, clients can connect, so they do and as they reach HttpServer, I send body in that request to the event loop

3. Then I want to process that message which was sent to the event loop when I recieved the HTTP request

4. That message is handled by some worker verticle, I can have e.g. 20 worker verticles, these are blocking, they write something somewhere and I am waiting in that worker verticle until it is done

5. Once worker verticle is done, handler which I registered to event bus when I send that message to the bus is triggered and it that handler I do request.response().end()

6. step 5 effectively closes HTTP connection with the client.

Now question:

1) Is my approach valid?

2) How does this scale? HttpServer accepts connections and send them to the event loop instantly so my memory is being occupied more and more over time. Meanwhile, worker verticles handles these messages I publish to the event bus and eventually client is not responded until my worker verticle processes the message.

3) I wonder what happen when I accept connections more quickly then worker verticles are able to process them, taking them from event bus

4) Is there some mechanism which would say that I can put to event bus only in case my worker verticle is free to handle it hence to be sure that I will not run out of memory merely because I accepted connections to quickly?

If i understand it right, for this problem there is is Pipe abstraction but it works only in ReadStream <-> WriteStream scenario but I am sending these messages to event bus ...

[Jochen Mader]

First of all: You have to understand that Vert.x is event loop based.

Events will always be queued. There is no other way of doing it.

Your question centers around getting some kind of flow control.

There is no flow control in Vert.x

You can implement it by exchanging messages between Producer-Verticles and Consumer-Verticles (google for back pressure).

But I honestly think you won't need it. Pump the messages out, attach a timeout and if things don't get processed on time resend. That works for most scenarios.

--

Jochen Mader | Lead IT Consultant

codecentric AG | Elsenheimerstr. 55a | 80687 München | Deutschland
tel: +49 89 215486633 | fax: +49 89 215486699 | mobil: +49 152 51862390
www.codecentric.de | blog.codecentric.de | www.meettheexperts.de | www.more4fi.de

Sitz der Gesellschaft: Düsseldorf | HRB 63043 | Amtsgericht Düsseldorf
Vorstand: Michael Hochgürtel . Mirko Novakovic . Rainer Vehns
Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus Jäger . Jürgen Schütz

[Stefan Miklosovic]

Is the attempt to introduce some flow control into vert.x by myself
fundamentally flawed and why should I avoid to implement something
like that in the first place? Is it only about the "this is not how we
want you to use it"?

I was thinking about that "out of memory / producer faster then
consumer" problem and very naive approach would be to measure the
length of each message received from the client and send this size
down to the event loop to some "counter" verticle which only purpose
would be to track how much memory I have already consumed. Once worker
verticle is done with the message taken from event loop, a message to
that counter would be sent again which would decrease the amout of
occupied memory. Hence I would response with 500 in case I can not put
any other message to the event loop becase I would overflow the
capacity of the memory.

This would be also beneficial to the scenario where I want to undeploy
the verticle (stop it) but I do not want to do so unless all messages
sitting at the event loop are processed. Hence I could send an HTTP
request to "stop" the receiving of clients requests by calling some
endpoint. All further requests would receive some non-200 code. Then
once all messages are processed, I would be free to kill the instance
without any message lost.

> You received this message because you are subscribed to a topic in the
> Google Groups "vert.x" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/vertx/ZybXBAJ9CYI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> vertx+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Stefan Miklosovic

[Julien Viet]

Vert.x does implement flow control for Buffer based event stream.

-- 
Julien Viet
www.julienviet.com

[Jochen Mader]

On the event bus? I was aware of using the ReadStream and WriteStream for IO bound operations but he is talking about event bus interaction.

[Ronoel Júnior]

There are a few solutions, one is to use circuit breaker (see https://vertx.io/docs/vertx-circuit-breaker/java/). But if you don't have capacity to consume all messages just in time I recommend you to use a message broker like Kafka.

On Sat, Sep 28, 2019 at 8:33 AM Ashish Lamba <ashish...@gmail.com> wrote:

Hi have you got any solutions for this problem, I'm also Tring to implement same. 

--

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a05f24b7-4051-4fad-a1a3-5d0838e53161%40googlegroups.com.

--

Ronoel Júnior