gremlin driver question

[Ramzi Oueslati]

Hi

Looking (again) at the code of ConnectionPool, I found out that basically when we want to borrow a connection, the least used connection is selected. Then if the number borrowed exceeds the max simultaneous usage (and the pool max size is not reached) then we open a new connection. Sounds good to me.

However I also understand that if the number borrowed exceeds the available in-process then we wait for a connection to be available. By default, the max in-process is 4 and the max simultaneous usage is 16. It seems those default values prevent from considering a new connection, doesn't it ? It seems we would wait for an available connection before we could consider a new connection. Am I missing something ?

I read the comment block about the "busy-ness" but still I'm a bit confused by the max number of in-process requests and the max simultaneous usage.

I'm trying to understand why the pool size seems to be stuck at min pool size. If someone could give some details about this, I would be very grateful.

Best regards,
Ramzi

[Ramzi Oueslati]

Hi

Maybe I should rephrase my question.

Am I wrong if I say that if "maxSimultaneousUsagePerConnection" is greater than "maxInProcess" then there is no chance for "borrowed" to reach "maxSimultaneousUsagePerConnection". Therefore the pool size is stuck at "minPoolSize".

Best regards,
Ramzi

[Stephen Mallette]

i've only done a cursory review on this at this point but between your explanation and what i see in the code, the defaults do appear to look "wrong". It's almost like they should be reversed? Is that all that is amiss here or is there something more fundamentally wrong from your perspective?

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/fb3722df-6c17-48fe-b19e-8b1b7a371962%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ramzi Oueslati]

I would not say "fundamentally wrong" but It's true I'm quite puzzled by the use of both maxInProcess and maxSimultaneousUsagePerConnection. Guess I miss the global picture.
I don't have any suggestion though.

[Stephen Mallette]

I don't get why the maxInProcess is bound by the number of times the connection is "borrowed".  I dug through old commits and realized that perhaps in refactoring somewhere something got messed up - not sure.....but I'm glad you noticed. It's also pointing out that it would be nice if there were metrics on the driver that could give some insight into what its doing. These internals are a bit opaque. Created this ticket for future thought:

https://issues.apache.org/jira/browse/TINKERPOP-1353

As for this problem, I think I have a fix that makes sense. I'm basically going to replace this line:

https://github.com/apache/tinkerpop/blob/5395aaeb057e4c163233aa8ace2a2975ea827fe5/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/ConnectionPool.java#L159

with:

if (borrowed >= maxSimultaneousUsagePerConnection && leastUsedConn.availableInProcess() == 0) {

that way we only force a "wait" if we've maxed out the number of processes trying to ship stuff over a single Connection AND if in-flight requests for that connection are also exceeded. When I make that change, the pool seems to grow and shrink more nicely.

Ticket is here:

https://issues.apache.org/jira/browse/TINKERPOP-1352

with a pull request for review following shortly.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/b48596b3-5115-4035-98c4-431d0de5600b%40googlegroups.com.

[Stephen Mallette]

Ramzi, if you have a moment it would be great if you could try out the change and comment on the pull request:

https://github.com/apache/tinkerpop/pull/352

Thanks again for pointing this out. I also fixed another weird lurking bug in Gremlin Server in the process where a self-imposed pause in writing from Gremlin Server to a overwhelmed client would sometimes cause an really weird FastNoSuchElementException - took a while to track that one down.