Undeploying Verticles Gracefully

[Jonathan Willis]

When a verticle is undeployed are current requests to that verticle killed off? or do they finish in the background while the verticle is taken down? Is that pending request moved to a different verticle? I'm not seeing anything in the docs that specifically talks about this.

[Julien Viet]

On 27 Aug 2018, at 21:46, Jonathan Willis <quicksi...@gmail.com> wrote:

When a verticle is undeployed are current requests to that verticle killed off? or do they finish in the background while the verticle is taken down?

when you undeploy a verticle, the corresponding http server (and all services created by the verticles) are stopped, this will likely stop some inflight requests.

Is that pending request moved to a different verticle? I'm not seeing anything in the docs that specifically talks about this.

no it cannot be.

if you want to handle this, you can have your verticle process a message (like an event bus message) to stop processing new requests and send you a response when all inflight requests have been processed, you can do that using a boolean flag check in the server request handler and a counter you increment / decrement according to request begin / end. 

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/246dd1ca-252c-42d5-b64e-8e36983960c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Willis]

Does it matter that it is a worker verticle?

So just to clarify a couple examples:

1 - a) A request comes in

     b) Soon to be removed worker verticle grabs the request   

     c) parent verticle undeploys that worker verticle

     d) request was lost in flight
2 - a) A request comes in

     b) parent verticle undeploys that worker verticle

     c) request is picked up by any other remaining worker verticle of the same type

3 - a) A request comes in 

     b) Soon to be removed worker verticle grabs the request

     c) Worker verticle subscribes to an observable to continue processing the request

     d) parent verticle undeploys that worker verticle

     e) that observable being processed is killed in flight

Are these accurate descriptions of what is going on?

[Jonathan Willis]

And how to you prevent future requests from being processed by a verticle to inform when the current verticle is ready to be shutdown?

[Julien Viet]

Hi,

unfortunately, that's not actually possible I think, in this case you will have to perform an HTTP redirection when you are in the shutting down phase in the server.

Can you tell what is your actual use case ? I think there were some enquiries in the past to do something similar already.

Julien

On 28 Aug 2018, at 00:04, Jonathan Willis <quicksi...@gmail.com> wrote:

And how to you prevent future requests from being processed by a verticle to inform when the current verticle is ready to be shutdown?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/8a38842b-d695-4c9e-94d5-a4e925e9fe46%40googlegroups.com.

[Jonathan Willis]

So this is mainly for auto scaling of verticles.  We have worker verticles that process a queue of tasks and the queue gets low enough we just want to scale down our workers (slowly) to one worker verticle. When the load gets high enough, spin up more worker verticles.

[Julien Viet]

this is a feature that has been asked several times

I believe we should provide a graceful shutdown of the Vert.x HttpServer at least, i.e  a shutdown method with a callback that stop to process incoming request and leave them to other verticles and calls back when all inflight requests have been processed.

it sounds reasonable to have this for 3.6

Julien

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f5bf3021-f2f0-4203-89cd-dce8891fcd13%40googlegroups.com.

[Jonathan Willis]

Now our worker verticle doesn't use/need an http server, it just receives messages over the event bus. Would that feature also be included?

[Julien Viet]

event bus is actually different (and it's a good news for you): you can unsubscribe and continue to process inflight messages.

with your use case you can unregister the event bus consumer and undeploy the verticle when all inflight messages have been replied to.

Julien

On 28 Aug 2018, at 21:07, Jonathan Willis <quicksi...@gmail.com> wrote:

Now our worker verticle doesn't use/need an http server, it just receives messages over the event bus. Would that feature also be included?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/6d90807d-9aba-4ea5-962d-175bffa65c9c%40googlegroups.com.

[Ali Sabzevari]

Hi Julien,

Did you implement this feature in 3.6? I couldn't find anything related in the docs. If not, it would be great if you could refer to an issue so that I can follow up on that later.

On Tuesday, 28 August 2018 19:57:29 UTC+2, Julien Viet wrote:

this is a feature that has been asked several times

I believe we should provide a graceful shutdown of the Vert.x HttpServer at least, i.e  a shutdown method with a callback that stop to process incoming request and leave them to other verticles and calls back when all inflight requests have been processed.

it sounds reasonable to have this for 3.6

Julien

On 28 Aug 2018, at 17:50, Jonathan Willis <quicksi...@gmail.com> wrote:

So this is mainly for auto scaling of verticles.  We have worker verticles that process a queue of tasks and the queue gets low enough we just want to scale down our workers (slowly) to one worker verticle. When the load gets high enough, spin up more worker verticles.

On Tuesday, August 28, 2018 at 1:01:47 AM UTC-6, Julien Viet wrote:

Hi,

unfortunately, that's not actually possible I think, in this case you will have to perform an HTTP redirection when you are in the shutting down phase in the server.

Can you tell what is your actual use case ? I think there were some enquiries in the past to do something similar already.

Julien

On 28 Aug 2018, at 00:04, Jonathan Willis <quicksi...@gmail.com> wrote:

And how to you prevent future requests from being processed by a verticle to inform when the current verticle is ready to be shutdown?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/8a38842b-d695-4c9e-94d5-a4e925e9fe46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ve...@googlegroups.com.

[Swaroop Nagaraju]

Hi  Julien,

I couldn't find this vertx documentation. We are using vertx 3.8.5. Can you provide documentation?

Regards,

Swaroop

[Julien Viet]

Hi this is a very old conversation.

what are you looking for exactly ?

Julien

> Disclaimer: "The information contained in this electronic communication and/or in any of the attached files is confidential and proprietary information of InfoCredit Services Private Limited and is intended solely for the individual(s) or entity to which it is addressed. It may also contain legally privileged information. Any review, retransmission, dissemination, printing, copying or other use of, or taking any action in reliance on the contents of this information by a person(s) or entities other than the intended recipient is unauthorized, strictly prohibited and may even be unlawful. If you have received this electronic communication or any part thereof in error, please notify us immediately by reply email and permanently delete all copies of this electronic communication and any attachments thereof from your system(s). The contents of this electronic communication do not necessarily represent the views or policies of InfoCredit Services Private Limited with respect to the subject matter hereof."

>
> --
> You received this message because you are subscribed to the Google Groups "vert.x" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/7acec1f2-470a-49d3-a828-923514f2af59n%40googlegroups.com.

[Swaroop Nagaraju]

Hi Julien,

We are using vertx with zookeeper service discovery with auto-scaling, There is no issue with scale-out, whenever scale-in happening, we are doing graceful shutdown i.e unpublish all record before vertical shutdown. What will happen to the in-flight requests from the event bus in that shutting down application?

event bus is actually different (and it's good news for you): you can unsubscribe and continue to process inflight messages.

with your use case you can unregister the event bus consumer and undeploy the verticle when all inflight messages have been replied to.

Julien

You have mentioned this in the same thread, but we couldn't find any related document for this, Can you please help me with this.

Regards,

Swaroop.

[Dan O'Reilly]

I am interested in this statement that Julien made back in 2018:

>  I believe we should provide a graceful shutdown of the Vert.x HttpServer at least, i.e  a shutdown method with a callback that stop to process incoming request and leave them to other verticles and calls back when all inflight requests have been processed.

I don't think this was ever implemented, was it? Is that still something that could end up on the roadmap?

I would be interested in this feature for a use-case I have: I start with an HttpServer that is bound to 0.0.0.0. For security reasons, after some client-initiated configuration occurs, I need to change the HttpServer configuration so that it is only bound to one specific IP address, instead of 0.0.0.0. An ideal solution would be if I could do it this, assuming I start with one HttpServer bound on 0.0.0.0:80:

1. Spin up a new HttpServer that is bound to 1.2.3.4:80, without it conflicting with the existing server on 0.0.0.0:80. At this point, requests being sent to 1.2.3.4:80 could be round-robinned between 0.0.0.0 and 1.2.3.4.

2. Once the new server is up, put the server bound to 0.0.0.0 into a "shutting down" state where it no longer accepts new requests, but does not close out in-flight requests. At this point all requests to 1.2.3.4:80 would be routed to the new HttpServer; no more round-robinning. Requests to any other IP addresses on port 80 that covered by the 0.0.0.0 binding would be rejected.

3. Once all in-flight requests to the 0.0.0.0 server are complete, it shuts down completely.

Obviously the unusual round-robin behavior I'm looking for is a different topic, and maybe isn't feasible/desirable to implement in Vert.x. But the behavior I'm looking for in #2, where the server rejects new requests, but keeps in-flight requests open, I think is exactly what Julien is describing above. Even having just that would make my life easier. Without the round-robin functionality, I'd just have to have a window where I reject all new requests (while waiting for in-flight requests to finish during the "shutting down" state), and a very brief window where requests just get lost (while the HttpServer restarts, once all in-flight requests complete).

[Asher Tarnopolski]

any way to workaround this by unregistering all routers on 0.0.0.0 and waiting until 

all of the processed requests' responses are returned?

although i am not sure if having multiple http servers bound on the same 

ip range is possible and if yes - if the requests will round robin between them

[Dan O'Reilly]

Yes, removing all routes from the 0.0.0.0 server would be one way to get the "shutting down" behavior. The other I was considering having the first registered Route check a "shuttingDown" property, and if its set, reject all requests. It would also add handlers for keeping track of the number of in-flight requests, and could initiate the final shutdown of the server once the final in-flight request is complete.

Vert.x currently supports having multiple HttpServers listening to the exact same IP/Port, and will transparently round-robin between them. So two servers listening on 1.2.3.4:80 will round-robin today. Listening on 0.0.0.0 is an interesting special case, because it could technically be shared with any server listening on any network interface on the system, though I'm not sure this is really the kind of use-case the Vert.x devs had in mind when they added this feature. I think it was really aimed at making it easy to run multiple instances of Verticles that start HttpServers, which would have the exact same configuration (and therefore listen on the exact same IP).

I guess another possibility here is to leave the 0.0.0.0 HttpServer running until the whole JVM is restarted, but reject all requests that weren't send to the 1.2.3.4 IP address from clients. I'm actually not sure if that can be detected reliably, or if rejecting requests on "unnecessary" IPs, rather than not listening on them at all, is good enough to meet the security requirement we have.

[Martin Halliday]

Hi, another 18 months went by. Any signs of graceful shutdown of HttpServer yet?

Specifically I want to finish in-flight requests before closing the network connection. Right now, when SIGTERM is received, HttpServer immediately closes the network connections and all responses from in-flight requests are dropped.