java.lang.OutOfMemoryError: unable to create new native thread

[redouane]

Hi everyone, 

I'm running Selenium Grid v 4.1.0 on Windows system composed of 9 nodes (Windows). I'm actually having a Grid error each day (after 20~30h of execution) that say that Grid can't no more to create thread  "java.lang.OutOfMemoryError: unable to create new native thread".

This error stop Grid to working and accept no more request. For me to resolve this error temporarily, I have put on a cron that kill java session of grid and create another one each day (24h).

So I would like if someone had already faced this error before ?

On windows system, I think that we can't increase number of thread, unless on linux system easliy with ulimit cmd

Thank you,

Have a good day,

[⇜Krishnan Mahadevan⇝]

I don't think you are facing a problem with creating threads. It looks like your JVM is running out of memory.

Have you tried increasing the Heap space via -Xxms and -Xxmx args  (Min and Max heap) ?

It would help if you could basically share the commands that you are using to start the hub and node.

If you are running the hub in the simple mode I think it basically clubs multiple capabilities into it.

You could try running it in the distributed mode as explained here: https://www.selenium.dev/documentation/grid/getting_started/#distributed

But I would definitely start off with adding some additional Heap memory to the Hub JVM and see if that helps.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/77ecd617-2dfb-4cd1-8a81-f8bc90506951n%40googlegroups.com.

[redouane]

Hi, 

thanks for your response. I appreciate that,

In fact I'm already using  -Xxms and -Xxmx args to run the hub and nodes with this cmd on a windows batch file ".bat" :

Hub :

java -Xms512m -Xmx1600m -jar ../selenium-server-4.1.0.jar hub --config config.toml --log "C:\selenium-4.1.0\HUB\hub.log"

Node :

start java -Xms512m -Xmx1600m -jar ../selenium-server-%SELENIUM_VERSION%.jar node --config config.toml --log "C:\selenium-4.1.0\Chrome\node.log"

And I don't think it comes from the Heap space,  because when the Grid stop working, I have noticed that it takes only 300m of memory in task manager.

Also I have read in stackoverflow this answer : "This is not a memory problem even though the exception name highly suggests so, but an operating system resource problem. You are running out of native threads, i.e. how many threads the operating system will allow your JVM to use."

https://stackoverflow.com/questions/16789288/java-lang-outofmemoryerror-unable-to-create-new-native-thread

So for me using  -Xxms and -Xxmx args, doesn't work.

I remain at your disposal for any further information

Thanks,

[⇜Krishnan Mahadevan⇝]

Thanks for adding that additional context.

It re-iterates my hunch now. The Hub is basically coupling a lot of other distributed components inside it and I am not very conversant with the Grid4 distributed architecture. 

Quoting the documentation

A Hub is the union of the following components:

• Router
• Distributor
• Session Map
• New Session Queue
• Event Bus

In Grid3 I know that the Hub merely plays the role of a dispatcher of requests to the appropriate nodes for new sessions or existing sessions.

The Grid4 does the same stuff in a different way.

I briefly dug in the codebase and I realised that Grid4 now seems to be using Netty under the hoods. Earlier it was Jetty. There was one JVM argument that would let you control the number of threads in the Jetty world. In Netty that JVM argument is "-Dio.netty.eventLoopThreads". By default it is n*2 (Where n is the number of processors on your machine)

Now with respect to each of the components that are part of the same JVM:

• The New Session Queue is basically just an executor service with a thread pool size of 1.
• The distributor also is using just an executor service with a thread pool size of 1.
• The event bus is using zero mq under the hoods and I wasn't able to figure out what its thread utilisation is.

So I am going to reiterate what I mentioned earlier. To be able to decipher as to which of the Hub component is basically ending up with thread starvation, you might want to run the Hub in the distributed mode (i.e., instead of having just 1 JVM for the Hub, it would now comprise of 5 JVMs 1 each for each of the above mentioned components). This will also give you better visibility into what is causing the thread starvation and you can accordingly take this forward. 

Hope that helps

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/2150ba12-d937-43c6-8be6-424bb4144eb2n%40googlegroups.com.

[redouane]

Thank you for this explication. I appreciate that,

In fact maybe when I had putted  -Xmx1600m maybe this is not enough for the Hub, because this week when I have run the hub without -Xxms and -Xxmx args I have noticed that in one time it reaches to 3G memory when I have launched too many heavy tests on it.

I well keep it running without -Xxms and -Xxmx args  and see if it'll causes this error again or not.

Have a good day,