Why ThreadLocal is bad idea for parallelisation? - "Simon Stewart — Scaling Selenium"

[miklas]

Hello,

I have now opportunity to write some tests and testing framework in new project completely from scratch. I'm thinking about the best structure of testing framework for tests written in Java with TestNG. I would like to have options for parallel execution in this framework

In almost every example of java parallel frameworks with TestNG which I've found there is WebDriver instance stored in ThreadLocal variables.

On the other hand I've watched some time ago "Simon Stewart — Scaling Selenium" video on YouTube. This is a video with Simon Stewart speaking on Heisenbug from 2018. He told that "Threadlocals are evil" and should be avoided. (https://youtu.be/y1pSkqIJvAw?t=1477).

Simon didn't show any examples of wrong usages of ThreadLocals and I am curious if he means to not use ThreadLocals at all or just that they can be used wrongly and you should be aware of how you use it. Simon is a great authority when it comes to selenium, so he made me really confused.

Does anyone knows what he exactly means and why "ThreadLocals are evil"? I cannot find any answer to this in the internet.

[⇜Krishnan Mahadevan⇝]

Here's my take on this.

I guess what Simon was hinting at is that, if someone uses ThreadLocal without truly understanding how to work with it, they would expose their tests to unpredictable behavior and as you might already be aware of, concurrency issues are the hardest to debug because they are difficult to recreate.

The basic notion of ThreadLocal is that any data member that is stored in a thread local is available as a copy to all the threads.

Now some of the things that I have encountered in my experiences with ThreadLocals are as below which are easy to miss and hard to debug.

1. What happens to the child threads that your original threads spawn and those child threads try to access the threadLocal data from their parent threads (i.e., you invoke threadLocal.get() from your child thread). I believe the child threads won't have access to the data unless and until you are using an InheritableThreadLocal 
2. What happens if you are using a threadpool (say ExecutorService for e.g.,) and you miss clearing up the data associated with a particular thread. In this case, the task that is running on a  re-used Thread would end up getting the data that was associated with that Thread in the earlier execution. So if you don't explicitly clear up the data, you are setting up yourself for some really long hours of debugging.
3. Some folks define a ThreadLocal along with a defaultValue() implementation. Now depending on your use-case these may or may not be desired behavior you want, because may again experience some undesired behavior.

These are some of the things that springs to my mind when someone says "ThreadLocals are evil" similar to how people say "Singletons are evil".

Hope that helps!

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/32467f69-3282-4fad-b73b-446646b6cbe2n%40googlegroups.com.

[miklas]

Thanks Krishnan for such detailed response.

So as far as I understand, keeping WebDriver instance as ThreadLocal in parallel test execution is fine but we should remember about things like:

1. To avoid first situation which you described, we should remember that our tests will fail if during the test child thread will be spawn and this child thread will refer to driver kept in ThreadLocal.

2. Our test should clean driver from threadlocal after test method execution and setup driver before test method execution to avoid reusing driver from earlier execution.

I've found even probably yours old blog post about "Parallel WebDriver executions using TestNG" (https://rationaleemotions.wordpress.com/2013/07/31/parallel-webdriver-executions-using-testng/). I know that it is from 2013, and we have already newer versions of Java and TestNG, but do you think that this logic is still valid for parallel execution in Java with TestNG or you will propose other way to do it?

[⇜Krishnan Mahadevan⇝]

Yes, you pretty much summarised it well.

The contents of that blog, the code and the logic are pretty much relevant even today as well because not much has changed from then to now, except for the APIs in WebDriver perhaps when it comes to instantiating the Capabilities of a webdriver etc.

All said and done, I went on to build a small little library which manages stuff for you and lets you automatically spawn/clean-up a browser. Please check https://github.com/RationaleEmotions/autospawn and see if that helps.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/821bbf40-f1d1-43b5-8383-654dd03d8cf0n%40googlegroups.com.

[Andrei Solntsev]

I am also curious.

I do use ThreadLocals in work projects. Selenide (a testing library based on Selenium) also holds WebDrivers in ThreadLocals. No issues detected. 

Andrei Solntsev

ср, 25 мая 2022 г. в 14:26, miklas <mikla.k...@gmail.com>:

--