Selenium Grid Hub dynamic control

[Dmitry Makhno]

Hi, (Grid newbie alert)

I have an idea to implement simple dynamic WebDriver Grid.
For this I'd like to consume webhooks from Grid Hub, OR from time to time request for status.

I already have one-button Grid Setup, Scale-Up, Scale-Down in aws.

What I'd like to get:
If Grid Hub has long queue of WebDrivers for N minutes, Scale-Up executed according to queue size... Short queue +1 node, Long queue M nodes.
If Grid Hub is not used for K mins, meaning no browser session started than Scale-Down, leaving only one Node.

It would be nice in future, get specific WebDrivers seeking capabilities, for example for IR start specific scale-up-Win, for Chrome/FF common scale-up.

Can someone push me where to dig, which interfaces to implement. Or any appropriate articles?

Thanks in advance,
Dmytro

[François Reynaud]

Hi Dmytro,

That would be a good new feature I think. I started to work on an json api for that.
code : https://github.com/freynaud/selenium/commit/afeecf32123a164d9856ea9d9ca70917a8e04839
you can download the jar with the changes here :  https://s3-eu-west-1.amazonaws.com/selenium-freynaud/selenium-server-standalone-2.31.0.jar or build yourself. ( ./go release )
Those changes give you access to http://hub:port/grid/api/hub/queue
That will return a json object. The capabilities inside that object is a list of all the capabilities currently waiting for a node.

Would that solve your problem ? If it does, there is still some work to be done.At least allow some filtering to avoid returning big objects ( firefox profiles ) in the new service.

thanks,
François

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/SrtyDB5VkqsJ.
For more options, visit https://groups.google.com/groups/opt_out.

[Dmitry Makhno]

Hi François,

Seems this is exactly what i need, I will have a chance to play with this not early than weekend.
Do you know any info where I can read about the rest of /grid/api/ requests, except of code itself... I was looking for json api for several months, and almost was assured that it is not in box and have to be made from scratch and your answer is very positive surprise.

P.S. This post initially were created from seleniumCamp, I planned to meet you there personally. I hope that your relatives are better now.

Best wishes,
Dmytro

[François Reynaud]

cool. The grid APIs are not documented. Writing new one doesn't take much time :when you see something you need and doesn't seem there, the easiest is to ask for it to be created or submit a patch.

You can also create and register your own servlets.If they extends registryBased servlet, they have access to the registry and should be able to see everything and make a web service out of it.

For the example we're talking about, it wouldn't have worked because this getter was missing :

https://github.com/freynaud/selenium/commit/afeecf32123a164d9856ea9d9ca70917a8e04839#L1R324

The documentation has always been an issue. One possible way would be to have /grid/api/help sumarizing them all, but that takes time and this isn't a very sexy task :) 

To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/5xUelmtnt5cJ.

[Krishnan Mahadevan]

Francois,

let me see if I can do something on the documentation here and *hopefully* come up with my very first pull request to the selenium codebase :)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

[Dmitry Makhno]

François,

Got it to play. I can use it as is. Awesome.

Looking forward to see it in drop.

However few things:

I don't know is it possible to get or not is some timestams:

a) for capability - when it was requested.

b) for queue - last update (keep the same if sessions are not moving)
This can simplify monitoring, and keep appropriate behavior for cases:

- test tool created too much threads, but they are planned to be consumed fast enough. - scale up is not effective.

- if queue was not update for long period, assume this may only happen if no sessions (however specific timeouts may create this situation too) - scale down to min, no need to monitor manually time range for no sessions.

Also looking into existing api, I assume it would be nice also to get opened(used) capabilities vs available.

Summarizing seems all following info might be useful in future via api:

- started sessions(capabilities)

- avaliable capabilities (overall for hub), as console shows per node.

- avaliable connections (nodes*NodeMaxSession - startedSessions)

- queue of capabilities

- session timestamps

To understand code better, I'll try to make api request similar to "get api/hub/capability?browser=firefox"

answers number:

>0 there are free connections

=0 all connections are busy 

<0 there are same capabilities in queue

response success=false, if capability is unknown == none of registered nodes have it.

Thanks again, you really gave me a push forward.

  Dmytro

[Vijay Kumar]

Hi Dmytro,

Can you make your dynamic WebDriver Grid code open source?

I am interested in that otherwise I have to code myself everything you already did.

Thanks

Vijay