Is RedisClient thread-safe?

[Shiva Ramagopal]

Hi,

I've been successful in using the read, write and multi capabilities of RedisClient. When testing my application with a single instance of a verticle (the application has only one verticle as of now) all goes well.

However when I increase the number of verticle instances to two, a number of weird errors start cropping up. The main issue is that multi transactions fail intermittently. Sometimes a simple read operation returns "QUEUED" as the result.

I'm wondering if I'm doing something basically wrong in sharing a RedisClient across different verticle instances. Is this correct? Is RedisClient thread-safe and can be safely shared between multiple verticle instances?

Would appreciate any kind of help/advice/pointers on this. Without multiple verticle instances my application cannot scale to use all cores.

Thanks,
Shiva

[Blake]

Is the RedisClient static?

[Blake]

Disregard last question. If you're sharing it across verticles guess it doesn't matter if it's static, because it would work similarly. You generally don't want to share clients between verticles because they each run on their own context and you get the benefit of biased locking when they are not shared across different threads/contexts. Just looked at the client code and it runs via pipelining - you can set how many handlers will be queued to wait in the RedisOptions, but if you've only got one client then this is going to bottleneck pretty fast.  Because each verticle runs on it's own context you should create a client on each verticle, which will setup a new queue for each client on each verticle.

[Shiva Ramagopal]

Hi Blake,

Thanks for the explanation. It matches the behavior that I observed - I start sending traffic to the client, I don't immediately see any errors, but after a few seconds the errors start coming up all over the place.

I have a few follow up questions -

1. I already have multiple RedisClient instances to connect to the individual Redis instances. How do I create and access these clients on a per verticle-instance? Can I instead create a larger pool of RedisClient instances and share it among the verticle instances so as to not overwhelm a single RedisClient instance?

2. Is there a mechanism available to limit the rate of requests coming in to a verticle? Since the RedisClient doesn't reject calls but fails when the call rate goes beyond what it can handle, I'm thinking of throttling the traffic at the verticle end and return a HTTP 500 when it exceeds a safe limit.

3. Is there an API that I can call on RedisClient or RedisOptions to increase the queue length of handlers? You had mentioned that RedisOptions has an API but I couldn't find it. I'm on version 3.6.3.

Thanks,

Shiva

[Blake]

Hey Shiva,

1. I would create the clients on the verticle itself. So call RedisClient.create(vertx) in your verticle's start method and then pass it down it to other parts of the verticle as necessary. If you need to dynamically retrieve the connection credentials/options then i would pass those to the verticle through the verticle config when deploying.

2. I'm not sure there's a mechanism to do this. You could look at circuit breaking, but I don't think that will throttle.

3. It actually looks like new RedisOptions().setMaxWaitingHandlers() will be a new option in the next release, you may be able to use it now if you use a snapshot of the next version. I didn't notice that this was a new addition.

Blake

[Paulo Lopes]

QUEUED is a reply from the server itself (not the client). This means that probably you're sharing the connection (which is not a good idea for redis, let each verticle have it's own) and somewhere in your code you execute a blocking command for example a redis transaction.

[Shiva Ramagopal]

Hi Paulo and Blake,

Creating a new connection for every access (read/write) to Redis increases my application's latency beyond acceptable limits. I'd like some suggestions on how to create a pool of RedisClients that can be shared across (1) Multiple instances of the same verticle and (2) possibly across different verticles.

In my application, reads from Redis and transactions are blocking calls, because I want to keep things simple at this point of time as I'm introducing Vertx to my team. If its not a good idea to share clients across multiple instances of a verticle, I'm thinking to create a pool of clients keyed by the Vertx instance (not sure how I should do that) into a concurrent hashmap. Can this work?

Thanks,

Shiva

[Blake]

We didn't say to create a new client for every __access__, but a client for each __verticle__ (each verticle will make many requests on its own RedisClient).

When you start the verticle, create the client, and pass that client around WITHIN the verticle (not to other verticles).

[Shiva Ramagopal]

Blake,

I got what you said. One of the problems is that the methods where Redis is accessed is several levels deep from the handler that receives the request, which makes it a bit tedious to pass the client down. Its more convenient to access it from a client factory that can take in a vertx instance as key and return the corresponding client. That is what I'm trying to figure out how.

Thanks for pointers, really appreciate it!

[Blake]

Ah, gotcha. Are you passing the vertx instance down to that level too though? If so, you can still create it there if you need (just not every access). We don't know your design, but you may have to rework your design in general. I feel like you're going to have this problem with most frameworks/platforms anyway; sooner or later you're going to need some client in multiple places and you can either recreate it, pass it down, or as you said write a map or something to retrieve the correct one.

If i'm not mistaken it's important that you use the same client on the verticle it was created from or else the client will be running (some parts at least) on a different context. So you could use a map for that, but that means you're going to have to pass down the information that states what verticle (whether that be a string for the verticle ID, or vertx itself) you're on anyway. Why not pass the client down if so?

[Shiva Ramagopal]

Hi Blake,

My approach to ensure that the clients are created and used in the same verticle instance is to create a ClientFactory class and pass the vertx context to it; and then pass the factory around wherever necessary.

I've just finished the code changes - and I must say that the factory is passed just about everywhere since Redis (and other external data sources) are accessed from several classes.

In my opinion, this limitation of not sharing clients across multiple verticle instances causes unnecessary complexity. I'm a lesser fan of Vertx now, in spite of its async awesomeness. In the interest of the users of Vertx, I think a tutorial can be created to educate/enlighten users about this limitation and suggest a design approach.

Its clear that the code to access external data sources must be separated out from the core logic. It seems that the only clean way to do this is to create a separate verticle class to do the data access and the main verticle should communicate the data to be read/written to it via the event bus.

Is this the way to go? Are there better ways possible?

Thanks,

Shiva

[Blake]

Hey Shiva,

Not sharing clients across verticles is what makes vertx more modular and performant too, though.

As previously discussed, you could create some type of static, concurrent, map and create a redis client per context and then retrieve the redis client for whatever context you're on. However, you don't really want to be accessing the map every time you want to use the client - it would be better to get the client from the map when you create the class if it isn't transient.

I really just haven't run into your issue to where it was a problem. This is just more of a dependency injection issue yeah? Whenever i've had a similar design as yours I would create the RedisClient (or whatever) at the earliest point possible (or maybe the earliest point it was needed) and then pass it down if some lower level class needed it, it would be passed in.

You have this issue in all types of design. Where you need some "data" in multiple places, so you need to have some type of central place to access it or you need to have it at a higher level and pass it in. Depending on what the "data" is determines which approach is better.

I'd definitely be up for hearing some constructive criticism on what i've said here, because it will only help me design things better in the future. Just at the moment I haven't really run into a situation where this was a huge problem, but would love to hear more on different ways people handle it.

Blake

[Matt Baldree]

Or just you dagger2

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/17ed3ae3-c3b6-4821-98df-ab200ea78803%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shiva Ramagopal]

I've succeeded in ensuring that RedisClient:s are created and used within each verticle instance, and I'm seeing a jump in the throughput.

I've used the old-fashioned way of creating the client in the start() method and passing it around in the constructor of objects. I haven't used DI (dagger2 or guice) because it wasn't clear to me as to how to bind a __specific__ instance of the RedisClient created in the verticle to objects within it.

While the immediate pressure is off (the weird errors have disappeared and I'm seeing a jump in throughput), I'm more interested in exploring the best design approach for this problem. Surely there must be other vertx users who have faced this before or are bound to, and the community can benefit from design recommendations.

DI can be a solution if someone can help me figure out how to bind a specific instance of an implementation to an interface in each verticle. I'm sort of leaning towards a more service-like design where a separate verticle type (e.g RedisService, or even something like DataAccesssService) handles the read/write of data to external data sources and the consumers pass messages to it over the event bus instead of direct method calls. One advantage I see with this design is that its straightforward to scale the service by simply increasing the number of verticle instances.

Any further thoughts?

Or just you dagger2

To unsubscribe from this group and stop receiving emails from it, send an email to ve...@googlegroups.com.

[Bipul Karnani]

Using vertx.getOrCreateContext().put("redisConnection", connection); in start method of the verticle? 

Since context is tied to a verticle instance, we can access redisConnection by vertx.currentContext().get("redisConnection") in all classes where RedisService is required.