Problem with asadmin monitoring - thread-pool.currentthreadcount-count not right

[Dave Hubbard]

Hi

We have been doing some performance tests through our Payara servers. 

In trying to monitor performance/behaviour we have been watching thread-pool counts.   We have aligned a thread-pool within payara with our database - all requests expected to perform a single JDBC look up via DB Connection pool.

With a setting of:

       <thread-pool name="http-thread-pool" min-thread-pool-size="10" max-thread-pool-size="25"></thread-pool>

We never see "server.network.http-listener-1.thread-pool.currentthreadcount-count" go above 15 while the count of sessions actually in our database is at 25.

We have done further tests, with aggressive timeouts on our database pool (to understand behaviour) and see that the 25 is current - in use sessions.  (obviously these numbers go up and down with varing load)

So my problem is the number "15" looks wrong it seems to be the number minus the initial minimum value of 10.

To prove that we set the following:

        <thread-pool name="http-thread-pool" min-thread-pool-size="0" max-thread-pool-size="25"></thread-pool>

with this the number returned from asadmin monitor was correct ("25") but "mirroring" (within load fluctations) the database pool size.

So as a work-around we are now setting the min = "0" - recognising that there will be a small initial impact to requests arriving having to add a thread to the pool - in the scheme of things probably minimal, but obviously not the intent of being able to set a minimum.

We are running payara 4.11.162.

Thanks

[Ondrej Mihályi]

Hi Dave,

You are using a very old version 4.1.1.162 - the version 4.1.1.164 added some fixes to the thread pool monitoring, see https://github.com/payara/Payara/pull/1123

It's always best to test with the latest version because every new version contains lots of bug fixes and improvements.

Ondro

--
You received this message because you are subscribed to the Google Groups "Payara Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum+unsubscribe@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/86cbcdbe-ea7e-40e8-961b-207d057b015f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Hubbard]

Ondro hi

That's it. 

We had 4.1.2.172 scheduled for next patch but had to defer it recently - I will use my work-around until we get that in.

Thanks
Dave

On Thursday, September 14, 2017 at 2:15:58 PM UTC+1, Ondro Mihályi wrote:

Hi Dave,

You are using a very old version 4.1.1.162 - the version 4.1.1.164 added some fixes to the thread pool monitoring, see https://github.com/payara/Payara/pull/1123

It's always best to test with the latest version because every new version contains lots of bug fixes and improvements.

Ondro

2017-09-14 15:02 GMT+02:00 Dave Hubbard <hubbard...@gmail.com>:

Hi

We have been doing some performance tests through our Payara servers. 

In trying to monitor performance/behaviour we have been watching thread-pool counts.   We have aligned a thread-pool within payara with our database - all requests expected to perform a single JDBC look up via DB Connection pool.

With a setting of:

       <thread-pool name="http-thread-pool" min-thread-pool-size="10" max-thread-pool-size="25"></thread-pool>

We never see "server.network.http-listener-1.thread-pool.currentthreadcount-count" go above 15 while the count of sessions actually in our database is at 25.

We have done further tests, with aggressive timeouts on our database pool (to understand behaviour) and see that the 25 is current - in use sessions.  (obviously these numbers go up and down with varing load)

So my problem is the number "15" looks wrong it seems to be the number minus the initial minimum value of 10.

To prove that we set the following:

        <thread-pool name="http-thread-pool" min-thread-pool-size="0" max-thread-pool-size="25"></thread-pool>

with this the number returned from asadmin monitor was correct ("25") but "mirroring" (within load fluctations) the database pool size.

So as a work-around we are now setting the min = "0" - recognising that there will be a small initial impact to requests arriving having to add a thread to the pool - in the scheme of things probably minimal, but obviously not the intent of being able to set a minimum.

We are running payara 4.11.162.

Thanks

--
You received this message because you are subscribed to the Google Groups "Payara Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum...@googlegroups.com.

[Dave Hubbard]

Just one observation with my "work-around" (setting min pool size to 0) with 4.1.1.162

In a load test all the first wave of requests failed (with HHTP Response 503) - with a min size of 10 they don't

I confess I had assumed they would queue and payara would start opening threads to service them, but that didn't happen so potentially not a "safe" option to use - as this behaviour is likely to be seen after a restart (or potentially after quiet period when count returns to minimum. 

I will try and retest with latest version (but for that the work-around is not required).