Selenium Grid performance

[krisztián forrai]

Hi,

Until now, I have successfully tested the great Selenium Webdriver on my local machine, but now, I would like to move on and use Selenium Grid.

From my first expressions, It seems that when running a full browser (IE: Firefox), it takes a lot of resources (mainly CPU).

My questions are:

• On a "typcial" PC configuration (e.q. i7, 2.5 Ghz, 4 cores, 8 Gm RAM) what is the number of browser it is cabable of runnig? Is is somewhere between like 5-10 or maybe 20-30? (at most of the times I need to run javascript, so HtmlUnit will not do :-()
• Is there a way to reuse the RemoteWebdriver, so when opening a page we don't have to wait for the time it takes to start is?
• Are there any best practices I should follow (disabling save as dialogs, using xvbf, settings some things up in an FF profile, etc.)?

My main goal is to use the grid for crawling (that is not for automated testing); is Selenium grid the right tool for me?

Thanks,

krisy

[Oscar Rieken]

On Sun, May 18, 2014 at 11:11 AM, krisztián forrai <forraik...@gmail.com> wrote:

Hi,

Until now, I have successfully tested the great Selenium Webdriver on my local machine, but now, I would like to move on and use Selenium Grid.

From my first expressions, It seems that when running a full browser (IE: Firefox), it takes a lot of resources (mainly CPU).

My questions are:

• On a "typcial" PC configuration (e.q. i7, 2.5 Ghz, 4 cores, 8 Gm RAM) what is the number of browser it is cabable of runnig? Is is somewhere between like 5-10 or maybe 20-30? (at most of the times I need to run javascript, so HtmlUnit will not do :-()

for IE thinking 5

~10 using chrome or firefox

you will have to see how your app/boxes perform

I would make the hub as beefy as possible

my config looks something like 

4 core 8gb of ram on the nodes (off the top of my head I think windows and linux are the same)

8 core 32gb of ram on the grid hub 

last time I checked running at about 60 nodes our 2700 tests take about 30 min, running full blast we are right around 12 min.

 

• Is there a way to reuse the RemoteWebdriver, so when opening a page we don't have to wait for the time it takes to start is?

meh I dont like reusing for tests. each test imo should control its own life 

• Are there any best practices I should follow (disabling save as dialogs, using xvbf, settings some things up in an FF profile, etc.)?

My main goal is to use the grid for crawling (that is not for automated testing); is Selenium grid the right tool for me?

Thanks,

krisy

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/e552e587-6b7b-4303-8f8b-c73458d67eca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dmitri T]

Hi Krisy,

If you don't care about rendering and just need simulate load or fetch data from the website I would recommend switching to Apache JMeter. It can act pretty much like a real browser, however resourcewise it will give a huge benefit, especially in non-GUI mode. On your i7 with 8Gb of Ram you can spawn 300-500 concurrent threads. 

If you still need to continue with Selenium there is a PhantomJS driver which is headless Webkit engine and it appears to be the least resource consuming option. 

Hope this helps.  

[krisztián forrai]

Hi,

Dmitri:

Rendering is not required, "only" running javascript - so I guess Phantom JS will be perfect!

(JMeter also looks nice, but I think it can not handle javascript as fully as PhantomJS)

Oscar:

Great, just the numbers I was needed! :-)

Is Selenium the right tools for me for crawling the web (if I want to crawl javascript-based pages)?

Thanks,

krisy