Connection Pooling in Jedis

[Ramu MS]

Hi,

I started to use JedisPool for my multi threaded application.

When I configure the pool with the resources, say 128, Jedis actually
opens 128 connections to Redis immediately after initialization, even
though no body has requested any Resource from JedisPool.

- Why don't we handle Lazy initialization in case of Connection
Pooling?

- Any specific reason why we have gone for a own implementation of the
JedisPool?
I have used apache commons pooling in other scenarios.
It worked pretty well and gives lot of other configurations like max
active connections, max idle, eviction time etc.

Just curious to know the reasons for this?

One side effect of this was seen when I tried to increase this number
to 1024(I know this is not a good number, but just wanted to try) and
tried to do some IO operations, like list files on directory. Which
started to fail, indicating an IO error.(Because File descriptors
available for any process in Linux was configured to be 1024, which
were occupied by Jedis Connections)

Regards
Ramu

[Jonathan Leibiusky]

Hi!

On Tue, Oct 26, 2010 at 3:49 AM, Ramu MS <ramu....@gmail.com> wrote:

Hi,

I started to use JedisPool for my multi threaded application.

When I configure the pool with the resources, say 128, Jedis actually
opens 128 connections to Redis immediately after initialization, even
though no body has requested any Resource from JedisPool.

- Why don't we handle Lazy initialization in case of Connection
Pooling?

This is because I wanted to make sure that when you ask for a connection you will get one right away, without waiting to open it, etc.
 

- Any specific reason why we have gone for a own implementation of the
JedisPool?
 I have used apache commons pooling in other scenarios.
 It worked pretty well and gives lot of other configurations like max
active connections, max idle, eviction time etc.

Just curious to know the reasons for this?

I just had this code and wanted to try it. And I didn't know about the apache connection pool. But I'm willing to change for something better. If you think it make sense to use the apache connection pool, lets go and try it! :)
 

One side effect of this was seen when I tried to increase this number
to 1024(I know this is not a good number, but just wanted to try) and
tried to do some IO operations, like list files on directory. Which
started to fail, indicating an IO error.(Because File descriptors
available for any process in Linux was configured to be 1024, which
were occupied by Jedis Connections)

Yeah. It is a big number. I would say that a normal app will never have so many connections. But I understand you point. We can try apache and see how it goes.
 

Regards
Ramu

[Jeff Liu]

Yes, I think it's better to maintain a dynamic pool. We don't always
need a full pool, but we need to handle the peak times. About the
commons-pool, the process of reclaiming unused objects seems to be too
complex.

[Joe Fawzy]

hi
-1 for apache commons
i think we do not need to add unneccessery dependency as we have a
working code base
thanks
Joe

On Oct 26, 2:54 pm, Jonathan Leibiusky <ionat...@gmail.com> wrote:
> Hi!
>

[Jonathan Leibiusky]

It is probably something perfect for a contrib project. I will create a repository for that and prepare everything so this can be pushed there.

[Ramu M S]

I already have implemented this for Redis Connections, (Just waiting for approval from my Company to publish this to Jedis Community)
Commons-pool has been implemented for other connections like FTP/SFTP/DB and these are already in production from quite long time and working very well for us.

Regarding dependency, In my honest opinion I too agree its better to have as less as possible(0 is perfect) but also something which is difficult to avoid completely in Java Projects.

I had forked the Jedis branch and added the code related to pooling in a separate package. As told earlier, waiting for approval to push these changes.

@Jonathan, If you think its best suited as a separate project, please let me know the location, where I need to add the code.

Regards,
Ramu

--
Thanks and Regards
Ramu MS
"Excellency is rarely found, more rarely valued"

[Ted Pennings]

We could make commons-pool an optional dependency in maven ("<optional>true</optional>"). Jedis would work perfectly fine then without it. It would then be up to the user to declare the dependency manually (or manually provide it on the classpath), if, and only if, the user wished to use pooling. (JVMs load classes lazily, so it wouldn't be a problem if the user didn't use pooling)

It might be worth providing a second, JedisCommonsPool implementation that is more "enterprisey" to address the shortcomings enumerated in this thread. Users could then choose which kind of pooling implementation they wanted.

Just a modest suggestion

[Eric Hauser]

Commons Pool is Apache 2 licensed. If there is a concern about adding
the dependency, why not just include the relevant classes and change
the package name? That code is pretty much the de facto for
connection pooling, so I don't see why one wouldn't want to rely on
it.

--
Eric Hauser
http://erichauser.net/