Windows & Mac cloud providers for Selenium Grid

[Zach]

Sorry if this information is out there and I just couldn't find it –

I'm exploring the feasibility of setting up Selenium Grid with enough nodes to run tests against about a dozen browsers (ideally up to 4 instances of each concurrently). 

Do most people who do this buy physical Mac/Windows computers to set up locally as nodes, or do they rent remote Mac/Win machines (or Linux + VirtualBox) from cloud providers?

If the latter, what are some good options?

Thanks!

[Krishnan Mahadevan]

Zach,

Except for MAC, you can actually leverage virtual box and ISO images for both Windows and Linux needs. 

For windows : https://dev.modern.ie/tools/vms/windows/

The problem starts off only when dealing with a MAC. Even if you do end up buying a MAC, OSX officially allows you to ONLY spin off 2 OSX VMs on a MAC. So that basically means you effectively get only 3 OSX machines for your automation needs.

Depending upon what your automation needs are, you have to weight out the pros and cons.

I have had some past experiences with using SauceLabs and haven't had any hiccups. But the company I worked for, was an enterprise level customer with SauceLabs.

You can explore BrowserStack as well, if you decide to take the cloud route.

If your automation needs doesn't mandate the need of OSX, you are better off taking the VM route.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/ede69516-fd18-40cd-9949-d7b8789f0f36%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Collin]

You could have a look at Docker to set up a local grid. I have some slides from a workshop I did at the LTG Workshops earlier this year, they may help point you in the right direction:

https://www.icloud.com/keynote/AwBWCAESECgk3XXEFAk8LTMxcfIPn4EaKpE3cmLBL0AzMGIjk33OEdpGwh-FDKlvfQVcPwXEHmb8WgiJjeQBO7cpaAMCUCAQEEINQW4DNmgf1jozjpQztcd3IetKKF-xg-zhJJtoJeY-D2#Hooking_Docker_Into_Selenium.key

There is also a chapter on Docker in my book which tells you how to set up a Selenium Grid using Docker:

http://www.amazon.co.uk/Mastering-Selenium-WebDriver-Mark-Collin/dp/1784394351

If you want to connect to a pre-configured grid you can't really go too far wrong looking at Sauce Labs:

https://saucelabs.com/

There are other companies offering a similar service, (but IMHO Sauce Labs are the leaders) have a look at:

https://testingbot.com/

https://www.browserstack.com/

The other options are virtual machines or physical boxes. 

I personally think Docker is the best quick solution, it's definitely worth giving Sauce Labs (or Testing Bot/BrowserStack) a go as they all will let you have a basic account that you can try for free.  Trying them all out will give you a feel for the advantages/disadvantages of each option which will help you make a better informed decision.

Hope this helps.

[David]

Docker won't help in the Mac and Windows case though right Mark? With Docker, you can test Firefox and Chrome on Linux. Though  one can try to equate that as good enough to extrapolate for Firefox and Chrome on Windows (and Mac). But you currently can't dockerize Windows so no IE on Docker. You can't dockerize Mac either, so no Safari on Docker. You can just run Docker on Windows and Mac, but the Docker containers are all Linux images.

[Mark Collin]

Docker is coming to Windows:

https://msdn.microsoft.com/en-us/virtualization/windowscontainers/quick_start/manage_docker

My understanding is that you won't be able to test legacy windows browsers (In other words IE) because the versions of Windows running docker won't support the old versions of IE.  You may be lucky and get IE11 as well as Edge.  There should be no reason why you cannot test using Chrome and Firefox on WIndows.

Realistically though, the chances of things not working functionally in FireFox on Windows but working in Firefox on Linux are pretty slim (The same goes for chrome).

You can also have a hybrid where the Grid Hub and firefox/chrome nodes are running in docker, but nodes running windows and OSX are not.  Weather it is suitable or not depends upon your requirements.  Does the OP need Windows nodes?

[MWQA]

+ 1 for Sauce Labs.  Simple to use, integrates with Jenkins nicely

[Vikram]

Hi Mark,

Can you please share any blog link for Docker + Grid setup , as the presentation instructions seems to be old now

Thanks in advance.

Regards,
Vikram

[Mark Collin]

Once you have installed docker the following commands from the presentation should work:

docker run -d -p 4444:4444 --name selenium-hub selenium/hub:2.45.0

docker run -d --link selenium-hub:hub selenium/node-chrome:2.45.0

docker run -d --link selenium-hub:hub selenium/node-firefox:2.45.0

You should then have a selenium grid running on your local machine which you can access by going to:

http://<YOUR_IP_ADDRESS>:4444/grid/console

Looking at https://github.com/SeleniumHQ/docker-selenium the latest version is 2.47.1 so to bring your grid bang up to date change the docker commands to:

docker run -d -p 4444:4444 --name selenium-hub selenium/hub:2.47.1

docker run -d --link selenium-hub:hub selenium/node-chrome:2.47.1

docker run -d --link selenium-hub:hub selenium/node-firefox:2.47.1

[Vikram]

Hi Mark,

Good morning , thanks again for detailed info.

I could get Docker setup up and running , has got 2 nodes.

Now am facing new issue , which is only 1 node is being used and other one is idle.

My assumption is hub will distribute all the test cases to nodes based on its availability.



Do you have any github repo which I can look at and see if that works fine with Grid setup. ( my end goal is to run test cases on as many nodes and finish automation asap )

Thanks in advance.

Best Regards,
Vikram

[Mark Collin]

Your selenium implementation will need to support multiple threads.  Grid will allocate all concurrent requests it receives (if it has available nodes) but if your selenium implementation only passes one test over to it at a time it will only run one test at a time.

I have a Selenium Maven template that supports multiple threads, it should give you a good starting point:

https://github.com/Ardesco/Selenium-Maven-Template

[Vikram]

Thanks Mark for quick info , I will try it out and update here.

Regards,
Vikram

[Mark Collin]

Just got shown this today which has some potential as well:

https://spoon.net/docs/sandboxes/selenium-sandbox

Looks like it could integrate with a Docker based Grid with a bit of work to give you Windows and Linux containers right now.

[Vikram]

Hi Mark,

Quick update on this , while running test cases on Docker node , I'm constantly getting remote driver time out issues.

The implicit and explicit values set for local setup and not getting applied to remote driver.

Not sure how to handle this scenario ? Did you face similar problem with Docker setup, can you please clarify ?

Thanks in advance.

Regards,
Vikram

[Mark Collin]

It's not something I have come across.  

Can you provide an example?

[Mark Collin]

You are quitting after a test yes?  Something like this:

@AfterMethod
public static void closeConnectionToGrid() throws Exception {
    driver.quit();
}

If you don't quit your driver instance the grid won't know you have finished and will sit there waiting for another command until it times out.  Also if you try to use more threads than there are nodes available the Grid will probably return an error saying there are no nodes available.  Your implementation that connects to the Grid will need to handle this problem.

[Krishnan Mahadevan]

​Mark,

>>>> Also if you try to use more threads than there are nodes available the Grid will probably return an error saying there are no nodes available.  Your implementation that connects to the Grid will need to handle this problem.

I believe in this case, the Grid will basically house the request in its queue until a node becomes available. I dont think the grid will error out stating no nodes are available. The only time that happens is when the requested capabilities by the test dont match with any of the actual capababilities available with any of the nodes (or) if the grid has no node [ Empty pool ]

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--

You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/ab75b1ed-f673-4455-b492-e8ce47280014%40googlegroups.com.

[Vikram]

Hi Mark,

I could run https://github.com/Ardesco/Selenium-Maven-Template successfully on Docker setup.

But still need to solve couple of challenges

1. Above project only ran on single docker node even when I had 2 nodes

How to distribute test cases across all available nodes ?

I ran test cases with command mvn clean verify -DSeleniumGridURL=http://....  -Dremote=true -Dplatform=LINUX -Dbrowser=chrome


2. I will look into my project now , why is it failing as per your answer 3:27 PM . Thanks for this hint

Best Regards,
Vikram

[Krishnan Mahadevan]

Vikram,

Try bumping up the number of threads via the JVM argument : -Dthreads=2

and see if that helps.

This is documented in https://github.com/Ardesco/Selenium-Maven-Template#anything-else

You could also try going through the pom file to understand what other JVM arguments exist [ Basically everything within the <properties> tag can be altered via JVM arguments ]

​

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--

You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/f98fdb0c-8d12-4709-b71d-649c1aa603ba%40googlegroups.com.

[Mark Collin]

I'm by no means an expert on the Grid so it's quite possible it will wait until something is free.  I would expect there to be a time-out hidden away somewhere though to prevent things waiting indefinitely.  Something to investigate when I get some free time :)

There is the possibility of a time out at both ends though.

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-users+unsubscribe@googlegroups.com.
To post to this group, send email to selenium-users@googlegroups.com.

[Vikram]

Thanks Mark and Krishnan.

Adding one more parameter -Dthreads = 2 , could distribute the test cases to 2 nodes.


Anyone who wants to do POC with Docker + Grid , I suggest try this github repo.


@Mark , still debugging issue in my project , will update here once I find the problem.

Best Regards,
Vikram

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.

[Vivek Soundararajan]

Hi,

     I am trying to connect the docker selenium remote node to running docker hub, But it's not connecting, It shows the connection refused exception, Even I can connect my remote node to hub, above screen shot shows the error. please help me.

[Mark Collin]

Can you show us the command you are using to start up the node and the hub?

[Vivek Soundararajan]

Hi Mark Collin, here is my hub & node start up commands
Hub running command

         docker run -d -p 4444:4444 --name my_hub selenium/hub:2.47.1

Remote node running command

         docker run -d -p 5558:5555 -e REMOTE_HOST="http://172.21.113.202:4444" -e HUB_PORT_4444_TCP_ADDR="172.21.113.202" -e HUB_PORT_4444_TCP_PORT="4444" --name firefox selenium/node-firefox:2.47.1

Thanks in advance

[Mark Collin]

Looks like you aren't linking the nodes to the hub.  Each docker container is running in an isolated namespace that means it cannot see anything unless you explicitly open up a link between the isolated namespaces.  To do this you need to link the containers together.  Try the following:

docker run -d -p 4444:4444 --name my_hub selenium/hub:2.47.1

docker run -d --link my_hub:hub selenium/node-firefox:2.47.1

[Vivek Soundararajan]

do you mean to run both hub and node will run in same machine? If it is, the scenario is failure,  and I want to run hub in my machine and the nodes are run in remote machine.

[David]

Why do you want to run hub & node on different machines? In terms of docker, each container itself is a separate "machine". So linking them as Mark suggested is simply having two machines remotely talking to each other within the "network" of the docker host, although behind the scenes the docker host that runs these containers is on your single machine of physical hardware, but in terms of virtualization they are already remote. Because docker uses less resources, this is sufficient. You only need to run each container on separate docker hosts if you scale up your grid/tests to be very high volume that a local deployment can not handle, and since no one has really mentioned this for docker selenium (as far as I know), I would imagine no one has really hit that limit/need yet.

It gets more complicated when hub and node containers are not on the same docker host (e.g. hub on docker host A, node on docker host B), because the linking step gets more complicated. You might want to look at these:

https://github.com/elgalu/docker-selenium

(read over the sections on "-e SELENIUM_HUB_HOST=docker.host \", "-e SELENIUM_NODE_HOST=docker.host \", "Note SELENIUM_HUB_HOST and SELENIUM_NODE_HOST represent a network firewall config challenge when running on different machines and should be changed to the proper host names or IP addresses of those.")

or to try to keep things simple on the Grid management side (i.e. treat all remote docker hosts that run containers as a single local machine), you then may need to complicate your Docker deployment more to make the docker networking simpler between containers, with these:

https://docs.docker.com/swarm/

http://kubernetes.io

[Mark Collin]

David got there first :)

You could also have a look at CoreOS that has been around for a while

https://coreos.com/using-coreos/containers/

The problem with running docker container across a cluster of machines is not the same problem as getting docker containers to talk to each other :)

[Sasi kumar]

Small clarification, i thought selenium-docker is by default for grid environment. Don't we usually use grid for distributed setup? .I think he is talking about selenium docker here!

[David]

Yes Sasi, Selenium docker can be used in grid environment. But you have to be aware of how Docker networking works (to maximize that capability). So for using Selenium Docker beyond simple basic setup, one has to be "knowledgeable" with Docker and Selenium not just mostly Selenium with a little Docker (or the I'll follow whatever is in the readme/docs and assume things work like "magic").

Perhaps someone in the Selenium (or Docker) user community should write up more complex examples of using Selenium Docker in the really "distributed" remote grid mode some folks here are thinking of. Until then, you are all on your own trying to set up this mode of operation.

And to make things worse, be aware that Docker on Mac (and Windows) adds a bit more complexity than Docker on Linux. I myself am having trouble port forwarding docker containers to docker to localhost/Mac to be able to connect to the docker container from my Mac (e.g. use browser on Mac to connect to website running in docker container). So you may hit issues here if you try to do something like open up a FF/Chrome browser Selenium instance graphically on the Mac. if things aren't configured correctly on the boot2docker/etc. side, it may fail.

Also bear in mind one of the benefits of Selenium docker isn't necessarily for grid use but rather being able to deploy a super small virtual machine preconfigured with Selenium and browser (FF and Chrome) to use for testing/debugging on your own dev box (or in some test environment). No need to fuss with installs, upgrades, configuration, etc. or having to use/share your browsers on your local machine for testing.

[Sasi kumar]

David thanks for the detailed reply. I will try from my side and update here. 

Incase any one already tried ,please reply so that  everyone can benefit. 

[Vikram]

Hi Sasi , David ,

I am trying to setup Docker + grid setup since last one weeks.

currently struck with below 2 issues

https://github.com/SeleniumHQ/docker-selenium/issues/112

https://github.com/SeleniumHQ/docker-selenium/issues/111

I have single hub and 3 nodes ( all 4 are docker containers )

In case you know any work around for above issues , please let me know.

Thanks & Regards,
Vikram

[Sasi kumar]

All this gird ,nodes in same machine or in remote machines?

[Vikram]

Hi Sasi,

Each of the grid and hub are running in one of the individual Docker containers ( on Mac mini )

Thanks,
Vikram

[Vikram]

Hi All,

I have found the root causes and solution as well. please find below info , will be useful to others who goes through this thread.

Root causes

1. I was overloading Mac mini to run 4 containers without seeing how many actually it can support
2. check memory and CPU usage by node/s and HUB with docker stats $(docker ps -aq)
3. Keep Activity Monitor open and see how much CPU is utilized by your machine

DO NOT OVERLOAD MACHINE , or else you will see random errors as I had faced.

Solution:
1. Now allocate proper memory and CPU per your webapp & selenium scripts need basis

docker run -ti -m 150M --memory-swap 300M --cpu-shares=104 -d -p 4444:4444 --name selenium-hub -e GRID_BROWSER_TIMEOUT=15000 selenium/hub:2.48.2

docker run -ti -m 750M --memory-swap 900M --cpu-shares=460 -d --link selenium-hub:hub -v /dev/shm:/dev/shm selenium/node-chrome:2.48.2

With my almost month long "Docker + Grid" experiments , I will definitely suggest to go for this approach instead of costly commercial cloud based services.

Best Regards,
Vikram

[David]

Thanks for sharing your insights Vikram.

I would say that for max Selenium Docker performance, one should run it on Linux host rather than a Mac (or Windows) host, better to go native virtualization. On Mac/Windows, you go through an extra VirtualBox VM layer that eats up extra memory, disk, and CPU, even though it is supposedly minimal. Plus an additional layer of network routing from Docker to the VirtualBox VM to Mac/Windows. I would think you could fit more (if only slightly more) Docker containers on Linux than Mac.

So develop and test the config on Mac is ok, but final deploy for use should be on Linux Docker setup if wanting to maximize resources.

[Vikram]

Hi Davi,

At last I got a powerful cloud Linux VM and trying to replicate docker setup on cloud VM.

I'm not able to solve issue of;  opening grid console from any machine within network.

I got the grid container ip address but can't access http://xxx.xx.x.x:4444/grid/console from any other machine.

Can you please share details how to expose Docker Grid container so outside world ?

Thanks & Regards,
Vikram

[Vikram]

Hi David, others,

After 2 weeks of hard work ; at last could get Linux VM ( Azure ) + Docker + Jenkins setup up and running; to run web automation automation scripts in parallel.

Just to summarize my setup

1. Developer creates nightly build
2. this kicks off Jenkins job which destroys existing Docker containers , create new once and start web automation in parallel
3. After automation run is over , results email is sent to stack holders.

If somebody has any improvement suggestions , please let me know.

I feel for now this is most optimal web automation solution in cloud.

Regards,
Vikram

[Siraj Syed]

Hi Vikram, 

Can you send me the jenkisn jobs or any link for referenec