Deploying new Verticle refers to previous Verticle instance

[Imran Hasan]

Here's a simplified version of a couple classes that I have, along with some debug printlns:

public class VerticleMain {
    public static void main(String[] args) {
       Vertx vertx = Vertx.vertx();
       HttpServer server = vertx.createHttpServer();
       Router router = Router.router(vertx);
       router.get("/v1/connect/:id").handler(ctx -> {
           String id = ctx.request().getParam("id");
           WebsocketVerticle wsVerticle = new WebsocketVerticle(router, id);
           vertx.deployVerticle(wsVerticle, result -> {
               System.out.println("deployed verticle " + result.result());          
               ctx.response().end("success");
           }          
       }
       router.route("/static/*").handler(StaticHandler.create());
       server.requestHandler(router::accept).listen(4100);
}

public class WebsocketVerticle {

    private Router router;
    private String id;

    public WebsocketVerticle(Router router, String id) {
        this.router = router;
        this.id = id;

    }
    public void start() {
        router.route("/live/" + id + "/*").handler(sockHandler());
    }

    public SockJSHandler sockHandler() {
        SockJSHandler handler = SockJSHandler.create(vertx).socketHandler(socket -> {
            sock.endHandler(ctx -> {
                System.out.println("hash code outside handler: " + this.hashCode());
                vertx.undeploy(deploymentID(), res -> {
                    System.out.println("attempting to undeploy verticle " + deploymentID());
                    System.out.println("undeployment successful: " + res.succeeded());
                }
            });
        });
        System.out.println("hash code outside handler: " + this.hashCode());
        return handler;
    }
}

I do the following:

1. Make a GET request to /v1/connect/10, which deploys a verticle. The following is printed:

hash code outside handler: 1258000200
deployed verticle c6692d24-9061-4b5b-bd2e-2d0be36e3fcc

2. My application makes a GET request to /live/10, which calls my SockJSHandler. The following is printed:

hash code inside handler: 1258000200

3. Disconnect my client application from my server. The following is printed:

attempting to undeploy verticle c6692d24-9061-4b5b-bd2e-2d0be36e3fcc

undeployment successful: true

So far, so good. I deployed a new verticle instance with the hash code 1258000200, then undeployed it.

4. Make another GET request to /v1/connect/10, which deploys a new verticle. The following is printed:

hash code outside handler: 1738849029
deployed verticle d96abbb22-c7e7-481c-9285-2dc66893cbe8

We know that there's a new verticle that got created here, because this one has a different hash code from the previous one, as well as a different deployment ID. But...

5. Reconnect my application, which makes a GET request to /live/10, calling my SockJSHandler. The following is printed:

hash code inside handler: 1258000200

6. Disconnect my client application from my server. The following is printed:

attempting to undeploy verticle c6692d24-9061-4b5b-bd2e-2d0be36e3fcc
undeployment successful: false

So to recap, I created and deployed a verticle, then undeployed it. I then created and deployed a new verticle, which had a different hash code and deployment ID from the previous one. However, when I made a new request to the same URL, it took me to the previous (now undeployed) verticle instance. Based on the fact that the route handler printed out the first verticle's hash code, and the end handler printed the first verticle's deployment ID, it seems that the route handler is calling the first verticle's SockJSHandler, even after it's been undeployed and a new one has been created.

My guess is that this has to do with how handlers are attached to a route, and replaced (or not replaced) when a verticle is undeployed and then redeployed. I've tried things like:

router.route("/live/" + id + "/*").remove();

But that doesn't seem to make any difference (I've tried putting it in the end handler and the stop() method). I'd appreciate any help anybody might be able to offer!

[Imran Hasan]

Slight typo...the println inside of sock.endHandler() should say "hash code inside handler." The output that I gave is correct, but I copied down the wrong line there :)

[Jez P]

I'm not sure you should share Router instances between vertices. Presumably you're doing this because want to share common handlers. However, all those handlers you're sharing will - I think - be run on the context of the verticle which creates the router, not the context of the verticle you pass the router into. This means they run on its event loop thread, which may not be the same as the one you passed the router into, which means you are losing the threading guarantees of vert.x. If you use a router per verticle instance, as I think you should, then a side effect would be that your problem is removed anyway.

Why do you feel the need to share the router? Presumably there's a reason for that choice.

[Imran Hasan]

Hmmm, I was under the impression that the Router had to be shared across a single application. I need my entire service to be listening on the same port (4100) due to the way that the client application (which I don't control) was constructed. I don't think it's possible to have multiple Routers listening on the same port, is it?

[Imran Hasan]

And just to clarify a bit more, I do need a Router in my VertxMain, because the URL that my WebsocketVerticle listens on depends on the ID that the client initially connects with. So if the solution were related to Router usage, I'd either need to re-use the Router that I initially create as I'm attempting to do now, or somehow use two together (and I'm not sure if that's possible).

[Julien Viet]

Hi,

I confirm that a router should not be shared between verticles.

Julien

> On Aug 5, 2017, at 7:18 AM, Jez P <mr.n...@gmail.com> wrote:
>
> I'm not sure you should share Router instances between vertices. Presumably you're doing this because want to share common handlers. However, all those handlers you're sharing will - I think - be run on the context of the verticle which creates the router, not the context of the verticle you pass the router into. This means they run on its event loop thread, which may not be the same as the one you passed the router into, which means you are losing the threading guarantees of vert.x. If you use a router per verticle instance, as I think you should, then a side effect would be that your problem is removed anyway.
>
> Why do you feel the need to share the router? Presumably there's a reason for that choice.
>

> --
> You received this message because you are subscribed to the Google Groups "vert.x" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
> Visit this group at https://groups.google.com/group/vertx.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/ee8589bb-6003-4cf3-a9af-866824f70049%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Imran Hasan]

So is that problem that even when I undeploy the verticle, the Router will remain and any future requests made to the same URL when the verticle is re-deployed will go to the original verticle's handler? i.e. I set the route handler for the URL when I deploy verticle #1, then undeploy verticle #1 and deploy verticle #2 and attempt to set a new route handler for the same URL. But since I'm sharing a Router instance, the route handler itself isn't actually changing.

In that case, is there any way for me to have two different Routers (one for the initial "/v1/connect" GET request and one for the "/live" GET request) that both listen on the same port? Or would I need to find some workaround?

[Julien Viet]

a way to handle this is to use the event bus: you deploy a controller that uses the event bus to communicate with the verticles that encapsulate the functional part of the application.

Julien

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/137ec0cb-3471-4acc-8829-23d94e96b5e7%40googlegroups.com.

[JavaDave]

Imran,

I'm going to guess at what you're doing here, because it looks similar to something I'd done recently. It looks as though you are expecting some arbitrary number of users to use your web application, and that it will be a "single-page" application relying on web socket communication. I'm also guessing that your users will be authenticated somehow and will have a unique ID. Your plan is to deploy a separate WebsocketVerticle instance for every user, and that WSV is effectively stateful and handles only that user.

If I'm completely off base, feel free to ignore this post :)  -- except that I'd recommend looking into the use of SubRouters as a possible replacement for sharing routers across verticles.

Assuming I'm correct, though, I think that approach is wrong. You do not want an instance of a verticle per user; that's going to challenge even vert.x's ability to scale well. Not to draw comparisons with Java EE, but that would be like creating a new Servlet per user. Instead, you should add a single route to your main router:

router.route("/live/:id/*")

and let your single socket handler handle those requests. If you want to segregate that logic into another class/verticle, rather than stuffing all of your routing logic into your VerticleMain class, which is a good idea if you expect your app to grow in functionality, then you can use SubRouters (http://vertx.io/docs/vertx-web/java/#_sub_routers).

Obviously I don't know the nature of your app, but it's definitely possible to disambiguate between user IDs when dealing with Web Sockets. For example, with what I did I was primarily concerned with pushing information to users' browsers, and the info was specific to a given user (i.e. the info I pushed to user A should not be pushed to user B). The event bus made this easy. Without going into too many details here, using the web socket event bus bridge (http://vertx.io/docs/vertx-web/java/#_sockjs_event_bus_bridge). Note the BridgeOptions I'd created below:

        BridgeOptions bo = new BridgeOptions()...

            .addOutboundPermitted(new PermittedOptions().setAddressRegex("service.ui-taskitem-[a-f0-9\\-]+"));

The regex's suffix represents a client's unique UUID. This allows me to push a customized message to the event bus, which the WS EB bridge will push only to the client who should receive the message.

On Friday, August 4, 2017 at 4:06:17 PM UTC-7, Imran Hasan wrote:

[Imran Hasan]

Julien, I don't fully understand what you mean. I can use my event bus to communicate with my WebsocketVerticles, but where do the Routers come into play? Am I wrong in saying that, when I eventually need to create my 2nd Router within my WebsocketVerticle that listens at "/live", it is not possible for it to also exist on port 4100? Maybe the event bus could provide a workaround for that, but I can't think of anything right now. Am I maybe just not thinking about this the right way?

JavaDave, you're right, my application is more or less similar to what you said. The only difference is that it's not one verticle per user, but rather one verticle per "meeting" between users (each meeting has its own ID, and any number of users can connect to it). This is useful because each meeting has its own state and it's nice to encapsulate it into a single class. I'm not too concerned about scaling, because I ran some load testing scripts and everything seemed to be fine, even with thousands of deployed verticles.

So it is definitely preferable if I can have separate verticle instances for each connected ID. I did give subrouters a shot (register a new router on the main router after every verticle is deployed), but I seemed to run into the same issue, which seems to make sense because the same Router is still being re-used.

[Imran Hasan]

I also did a bit more testing on my own to see what the problem is. In my WebsocketVerticle, I replaced my handler with the following:

router.route("/live/" + id + "/*").handler(context -> {
   System.out.println("hash code: " + hashCode());
});

I connected a client, disconnected it, and then reconnected. I got this output:

hash code: 939440626
hash code: 939440626

So once again, this implies that we call the same verticle's handler twice, even after the first one is undeployed.

Then I added another line to my handler:

router.route("/live/" + id + "/*").handler(context -> {
   System.out.println("hash code: " + hashCode());
   context.next();
});

This printed out:

hash code: 682078496

hash code: 682078496

hash code: 2145675319

So we call the same verticle's handler twice, but when I call context.next(), I call the new verticle's handler. So the handlers are getting registered correctly, but the problem is that they seem to be getting "stacked" on top of each other, i.e. my 2nd verticle handler is the 2nd handler in the list, my 3rd one is the 3rd one in the list, etc. Theoretically, I could just keep on calling context.next() and reach the correct handler, but that's obviously not ideal.

If there's some way to unregister a handler (and I can't seem to find a way to do that), then I think that would effectively take care of this shared Router issue. But again, I'm not sure if this is the right way to approach the problem.

[JavaDave]

One thing I realized as I looked back over the code. I think that the discussion of whether the WebsocketVerticle is being undeployed or not is a bit of a red herring. What matters is that you've added a handler to the router in WV's start() method, and you never remove it. So even if the WV is undeployed, it doesn't matter; the router is still configured to send requests of "/live/" + id + "/*" to  that first SockJSHandler. 

I'd wager that that is the core of the problem.

How to solve it... I don't know if you can selectively remove a single route from a router; there is a clear() method, but then that would affect all clients currently connected to your app. So you will need a layer in between. I could think of various different options, but why not make use of the Event Bus? You could deploy and underlay verticles as you currently plan to do, but you would have a single handler listening on router.route("/live/:id/*") that sends a request-response message on the Event Bus to a channel like "ws-reg-" + id  . Your WebsocketVerticle instances can listen to the same channel (e.g. 

vert.eventBus().consumer("ws-reg-" + id, msg -> {

  ...

  msg.reply("Data to send back to client #" + id);

})

). 

When it's time to undeploy the WV, you do so and then unregister from the Event Bus (which, according to the vert.x docs, happens automatically anyway).

[Imran Hasan]

I think I get what you're saying, but doesn't the problem remain the same? If there's a single handler listening on router.route("/live/:id/*"), then won't any future requests to that URL go to the first registered handler?

I tried switching my code up a bit so it looked like this inside VerticleMain:

router.get("/v1/connect/:id").handler(context -> {
   String id = context.request().getParam("id");

   WebsocketVerticle wsVerticle = new WebsocketVerticle(router, id);
   vertx.deployVerticle(wsVerticle, result -> {

       String deploymentID = result.result();
       System.out.println("deployed verticle " + deploymentID);          
       ctx.response().end("success");
       router.route("/live/" + meetingID + "/*").handler(ctx -> {
            System.out.println("deployment id: " + deploymentID);
            vertx.eventBus().publish("ws.register." + deploymentID, "test");
       });
   });
});

But I still end up running into the same problem. We're making repeated calls to "/live/10" (or whatever ID the client uses) after undeploying and then redeploying. So when the second connection occurs and we access "/live/10" for the second time, we still end up calling the first handler because that was what got registered first. My output ends up looking like:

deployed verticle 7341c0b7-328a-4255-bda8-111c910b3284
deployment id: 7341c0b7-328a-4255-bda8-111c910b3284
deployed verticle 94ce1033-218e-4fd5-a043-35acaa7cbeb0
deployment id: 7341c0b7-328a-4255-bda8-111c910b3284

I could be misunderstanding your suggestion, but I think the fundamental problem is the existence of multiple handlers on the same route. I'm not totally sure how I can manipulate the Event Bus to get around that, because I will at some point have to have a Router listening at "/live/10" on port 4100, and that handler should update to reflect the new state of a new Verticle when it ends up getting deployed.

[Imran Hasan]

Basically it seems to be the case that any and all information defined inside of a handler is effectively permanent. So if I define a WebsocketVerticle outside my "/live/:id" handler and then call a method on that WebsocketVerticle, I'll end up calling the same verticle instance's method every time, even after I've undeployed it and deployed a new one. I think I just need some way to differentiate between WebsocketVerticle instances inside of my route handler, but the issue of multiple handlers/routes existing simultaneously (and not being removed when the verticle is undeployed) seems to complicate that.

[Imran Hasan]

Actually...nope, you're right. Registering an EventBus address at "ws.register." + id (rather than "ws.register." + deploymentId) does work. I'm unregistering the address in my Verticle stop() method as you suggested, and then when I send a message to the event bus, I am in fact referencing a different verticle instance every time I undeploy and then redeploy. Sweet!

Now it's just an issue of figuring out how I can get the RoutingContext from VerticleMain to my WebsocketVerticle's SockJSHandler...

[JavaDave]

Glad to hear that worked (and also glad to hear that I was at least being somewhat clear)

Getting the RoutingContext to the SockJSHandler within the WSV might be difficult/impossible. Basically you're not going to be able to pass it by reference over the EventBus. Theoretically you could serialize it (e.g. by writing it out as JSON, or writing your own codec) and pass that as the message body, and in that case you query it within the WSV, but you wouldn't be able to modify the ultimate http response in any way.

One thought is whether you actually do need to send the RoutingContext. Can instead VerticleMain's handler pull out any information that the WSV needs and pass that in the Event Bus message (and have WSV pass any information that VerticleMain needs in its Event Bus response message)?

Alternatively, can you refactor a bit further so that you have a single SockJSHandler within VerticleMain? You can still deploy/undeploy instances of WebSocketVerticle on a per-conversation basis; the WSV's job would be to maintain state and calculate how to handle requests, and then pass that information back over the Event Bus for the SockJSHandler to handle?

I do something similar in one of my projects, with the difference being that I don't handle HTTP requests once the WebSocket connection has been made; it's all reactive, listening to a RabbitMQ instance. But it could help give you ideas. Let me know if you're interested in seeing any code.

[Imran Hasan]

So it turns out there's actually a very, very simple solution which I somehow managed to overlook. 

In WebsocketVerticle:

private Route route;

...

public void start() {
   route = router.route("/live/" + id + "/*");
   route.handler(sockHandler());
}

public void stop() {
   route.remove();
}

And that's it. Now every time a new WSV gets deployed, the handler refers to the current instance rather than the previous one.

I mentioned in the OP that I was previously trying to do  router.route("/live/" + id + "/*").remove(); but it turns out that that creates a new Route instance at that URL and then removes that, rather than removing the Route living at that URL from the Router. So the Route was just never getting removed. But maintaining a reference to the current Route allows you to remove it later, and that solves this problem.

I really appreciate all of the support!