Running Selenium tests in parallel using rspec

[Tucker McKnight]

This question isn't specifically about rspec; however, I'm hoping that someone in this group has experience with this, since I haven't been able to find much discussion about doing it.

tl;dr version: When running Selenium tests in parallel, do you have one instance of your rails application getting hammered on by all the threads, or multiple instances with one rails app instance per thread?

I'm trying to run many Selenium tests in parallel -- about 200 at a time. I'm using rspec and selenium-webdriver to run these Selenium tests. To run them in parallel, I'm using the parallel_tests gem (https://github.com/grosser/parallel_tests), which first creates many copies of your development database, and then calls rspec in many threads, with each thread using its own database, assigning different spec files to each thread.

Right now, when using that to run Selenium tests, multiple instances of the rails server get started up. When you start rspec, it starts the application using "rails s", which uses some available local port, and then the Selenium tests on that thread use that instance of the rails application. When parallel_tests calls rspec many times, "rails s" is getting called many times, having many local instances of the server running on as many ports, with each having a database specific to that server.





This becomes a problem when trying to run many, many tests at once. Calling "rails s" too many times takes a while. Unfortunately, we do need each test thread to have its own database, since we rely on having each test run inside of a Postgres transaction. But I'm wondering if it would be better to have all threads use the same rails server, while still having their requests directed to the appropriate database. For example, using a subdomain, like test1.app.local, and then test2.app.local. That way, they're all hitting just one rails server, but they're still using separate databases.




I wanted to ask if anyone has tried this before, and if you've run into any issues when doing that. I haven't been able to find many others talking about this same approach online, so I'm wondering if that's for a reason.


[Jon Rowe]

Hi Tucker

It depends on your web server, if you’re using a threaded web server such as puma, you’ll find you can probably get away with one boot, and then many workers, although you might have to do some trickery to reconfigure the database.

The main reason people create multiple databases for parallelised tests tends to be conflicting data amongst your tests, and ease of reset in between each test. 

It’s quite rare to be able to use transactions to handle this as servers often exist outside the transactions thread, and thus don’t share data, if you are going down that route you may find you need a server per thread, so you can stay inside your transaction.

Cheers

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk

On 29 January 2019 at 22:13, Tucker McKnight wrote:

This question isn't specifically about rspec; however, I'm hoping that someone in this group has experience with this, since I haven't been able to find much discussion about doing it.

tl;dr version: When running Selenium tests in parallel, do you have one instance of your rails application getting hammered on by all the threads, or multiple instances with one rails app instance per thread?

I'm trying to run many Selenium tests in parallel -- about 200 at a time. I'm using rspec and selenium-webdriver to run these Selenium tests. To run them in parallel, I'm using the parallel_tests gem (github.com/grosser/parallel_tests), which first creates many copies of your development database, and then calls rspec in many threads, with each thread using its own database, assigning different spec files to each thread.

[Tucker McKnight]

Hi Jon,

Thanks for the reply. We're using Puma, so I expect that the server will be able to handle all of the traffic from the concurrent tests. And you're right about why we're using multiple databases: our tests aren't written to avoid conflicts. We're relying on multiple databases to avoid conflicts.

Could you go into more detail as to why using one server would be an issue with transactions? Each rspec thread exists outside of the server thread, but since the rspec threads are the ones that start and stop transactions, I don't understand why having only one server would be an issue. One database would be blocked on a transaction, but when the server receives another request on a different subdomain, it will change the ActiveRecord config to use a different database, and that different database would not be blocked by a transaction.

I could be misunderstanding how this process works, as I'm not really an expert in ActiveRecord or database transactions. Does what I'm saying make sense?

[Jon Rowe]

Hi Tucker

I was attempting to highlight the problem with using transactions from within your RSpec processes. The server won’t be able to see the contents of the transactions you’ve started from the RSpec thread unless they are committed, or the server shares the thread with RSpec. This is why traditionally when testing with capybara people have had to use truncation based database cleaning.

Cheers

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk