A way to manipulate node system conditions after each test on grid

[Eugene Shragovich]

Hi all,

I am looking for a way I can ensure certain things happen after each test on a Selenium Grid node. For example that a certain process is not running and if it does, kill it. 

I know how to perform the check and actually kill the process but I want to know what is the best way to do on Selenium Grid? Perhaps I don't need to reinvent the wheel and there is some way I can use existing Grid infrastructure to determine when a test has finished on a node programatically?

Many thanks,

Eugene S

[Krishnan Mahadevan]

Eugene,

 

How do you want this to work ?

 

1. Should it happen irrespective of what test finishes i.e., lets say you don’t care about the test, you merely want to know when the user calls driver.quit() which signals a terminate session message to the hub, so that you can run some sort of clean-up at the node.

 

If this is what you want, then you should be building your own custom proxy and embed your logic of clean-up as part of afterSession() so that it gets executed after driver.quit() is executed but before the session is released by the Grid. You need to ensure that you don’t trigger any exceptions here. Please refer to this article that I wrote up on grid to help you understand how to work with custom proxies.

 

(or)

 

2. Should it happen with respect to only certain specific tests i.e., lets say you may have been running some particular tests, for which you would like the screen-recording to be done and would like to stop it only for those tests that requested it.

 

If this is what you want, then

First you should build a servlet which when invoked can do the cleanup for you. You just inject this servlet into the node. To learn how to do this, please refer to this article of mine.

just before calling driver.quit() you should do the following :

• Obtain the IP and port address of the machine to which your test was routed to. You can learn how to do this by referring to this blog post of mine, or just by using this library of mine.

           Now after you call driver.quit(), trigger a http operation (GET/POST) to the servlet that you added to the node, by making use of the IP and port address that you obtained from above.

 

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/51bf13f1-923c-4a32-b8b3-3b5ec9fb2cec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eugene S]

Hi Krishnan,

Thank you so much for the information! It is super useful and I learnt a lot from your articles and now have a much better idea of what I need to do.

I had a look at the SeLion project that you refer to and I'd like to implement some of the ideas I saw there in my local grid. I followed the example in your article to create the ListProxiesServlet and got it working fine but I am thinking about integrating some more functionality into the existing grid console. For example, I'd like to add restart button next to each node on the console page view or a form that will allow me to push node configuration changes. What would be the approach to something like this?

From looking at how SeLion works, it seems that it wraps the original grid and starts it internally. I'd like to avoid that if possible and just create another servlet that I can plug into existing grid hub/node.

Thanks again!

Eugene

[Krishnan Mahadevan]

Eugene,

 

Am glad that my articles on the Grid helped you out ☺ That was the intent behind creating ‘em ☺

 

>>> I followed the example in your article to create the ListProxiesServlet and got it working fine but I am thinking about integrating some more functionality into the existing grid console. For example, I'd like to add restart button next to each node on the console page view or a form that will allow me to push node configuration changes. What would be the approach to something like this?

 

For getting this done, you would need the following:

 

1. A Custom servlet injected at the hub end, which can serve as your new console. The other option is basically to resort to what I call as CLASSPATH overriding, wherein you create the same class with the same package name in your project, and change its behavior. That way, when the Grid tries to wire up the console servlet, it would wire up your version instead of the default one. That way you don’t need an additional servlet.
2. For building the “On-demand restart” functionality, you would need to first construct a self healing grid which would have the following traits:
1. A custom proxy, which has the ability to know that a user intends to restart a node (Your console could trigger this as a flag to your proxy).
2. Once the proxy gets this signal, it should prevent new sessions from being routed to the actual node, but should wait for all the existing tests to run to completion (or) timeout.
3. Your node should expose a custom servlet (lets call it as the shutdown servlet) which when invoked will shutdown the node by invoking “System.exit()”
4. Your node is being watched by a watchdog kind of setup, which would basically spin off the java node, if it finds it to be down. That way your node can come back on its own.
3. For pushing in config changes also, you would need to do a similar kind of setup, because the node reads configuration only at startup which it uses to fire up the registration request. So the only way of forcing a node to use a new configuration is to restart it.

 

Sometime back, I blogged about the salient features of a self healing grid in this blog post of mine : https://rationaleemotions.wordpress.com/2013/01/28/building-a-self-maintaining-grid-environment/

 

The self-healing capabilities were added to the SeLion grid, when I worked on it a long time back. So you may want to start off with the SeLion grid (so that you get the self healing capabilities out of the box) and add your customizations on top of it. Even if you don’t use the SeLion Grid’s startup/shutdown mechanism, I would strongly urge you to leverage its proxy (which contains this capability of stopping new sessions, waiting for sessions to run to completion, triggering shutdown, the watchdog capability etc.,)

 

Hope that helps!

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/97f9fae1-8eea-4a05-b2b4-5e1b148d36e2%40googlegroups.com.

[Eugene S]

Hi Krishnan and thanks again for you elaborate explanations. It is very much appreciated.

>>> A Custom servlet injected at the hub end, which can serve as your new console.

I assume that such servlet will need to return an html page similar to original grid console but with my amendments (buttons and such..). Does that mean that effectively I will need to recreate the console page from scratch? Is this why you suggest the other option with "classpath overriding"? 

>>>The other option is basically to resort to what I call as CLASSPATH overriding

If I understand correctly, in this case too, I will be able to reuse the original console page and amend it according to my requirements?

And if I create another class with similar names, how do I make sure that mine is being used and not the original one? If I have both of them in my classpath, won't they have a name clash?

Thanks!

Eugene

[Krishnan Mahadevan]

Eugene,

 

Comments inline.

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <seleniu...@googlegroups.com> on behalf of Eugene S <shrag...@gmail.com>
Reply-To: <seleniu...@googlegroups.com>
Date: Monday, December 11, 2017 at 7:23 AM
To: Selenium Users <seleniu...@googlegroups.com>
Subject: Re: [selenium-users] A way to manipulate node system conditions after each test on grid

 

Hi Krishnan and thanks again for you elaborate explanations. It is very much appreciated.

 

>>> A Custom servlet injected at the hub end, which can serve as your new console.

I assume that such servlet will need to return an html page similar to original grid console but with my amendments (buttons and such..). Does that mean that effectively I will need to recreate the console page from scratch? Is this why you suggest the other option with "classpath overriding"? 

 

[Krishnan] Yes you would need to recreate the console page from scratch if you would need your new functionality plus the existing functionality of the Grid’s inbuilt console page. Yes that is why I was suggesting the “classpath overriding” approach.

 

 

>>>The other option is basically to resort to what I call as CLASSPATH overriding

If I understand correctly, in this case too, I will be able to reuse the original console page and amend it according to my requirements?

And if I create another class with similar names, how do I make sure that mine is being used and not the original one? If I have both of them in my classpath, won't they have a name clash?

 

[Krishnan] I was wondering when you would ask this question ☺ Well, the idea here is that you put your jar ahead of selenium’s jar. Java basically runs through your classpath in a sequential manner, loading all classes in the current jar before it moves on to the next one. So lets say you have your overridden version of the console residing in “myconsole.jar”, then you would be starting the Hub using the java command : java -cp myconsole.jar:selenium-server-standalone-3.8.1.jar org.openqa.grid.selenium.GridLauncherV3 -role hub

As you can see, the class part is avoided, by ensuring that your version of the class is found by Java first, ahead of what resides in the Selenium jar. Java doesnot reload the same class again, if it finds two or more variants of the same class. Its always “first come, first serve” basis. If you want to get past all of ths confusion then you can easily build your own uber jar (jar of jars) and then use that. This uber jar would contain your overridden console and would shadow out the selenium provided console.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/f06c89b6-1eda-4356-851d-2f6f1cae6312%40googlegroups.com.

[Eugene S]

Thanks again Krishnan, as always - super helpful!

I was going through the SeLion code trying to understand how can I utilize parts of it  for my purposes as you suggested and I have few questions I hope you might be able to help me with:

1. As far as I understand, the SeLion grid standalone jar is designed to be the main entry point so that we do not call the SeLion jar and Selenium standalone server jar separately from command line like we do for a single servlet (like this: java -cp some-servlet.jar;selenium-server-standalone=X.X.X.jar). Instead, SeLion starts Selenium internally by itself. Is that so?
   
   
2. If the above is correct, should I follow the same principle for my purposes or keep a separate jar as in your examples with a single servlet?(when you include the servlet jar and selenium server standalone jar in command line) In such case, how should I go if I have multiple servlets? Just include all of them in command line command (like this: java -cp servlet1.jar;servlet2.jar;...;servletN.jar;selenium-standalone-server-X.X.X.jar) or there is a preferred way?
   
   

Thanks in advance!

Eugene S

...

[⇜Krishnan Mahadevan⇝]

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Tue, Dec 12, 2017 at 12:33 PM, Eugene S <shrag...@gmail.com> wrote:

Thanks again Krishnan, as always - super helpful!

I was going through the SeLion code trying to understand how can I utilize parts of it  for my purposes as you suggested and I have few questions I hope you might be able to help me with:

1. As far as I understand, the SeLion grid standalone jar is designed to be the main entry point so that we do not call the SeLion jar and Selenium standalone server jar separately from command line like we do for a single servlet (like this: java -cp some-servlet.jar;selenium-server-standalone=X.X.X.jar). Instead, SeLion starts Selenium internally by itself. Is that so?
   

​[Krishnan ] Yes that is correct.​

1. 
   
2. If the above is correct, should I follow the same principle for my purposes or keep a separate jar as in your examples with a single servlet?(when you include the servlet jar and selenium server standalone jar in command line) In such case, how should I go if I have multiple servlets? Just include all of them in command line command (like this: java -cp servlet1.jar;servlet2.jar;...;servletN.jar;selenium-standalone-server-X.X.X.jar) or there is a preferred way?
   

​[Krishnan] Either ways is fine, but I personally prefer building an UBER jar (a jar of jars) so that I can manage everything with just one jar and dont have to worry about juggling with a multiple set of jars.  Irrespective of whatever approach you use, the command line parsing should be done by the GridLauncherV3 (you should be delegating to it either directly (as SeLion grid does) or indirectly (by you not providing any main() method of your own, but falling back to what Selenium provides). With respect to the separate jar part, you can basically put in all of your customizations into it (by that I mean a custom proxy, a custom prioritizer, a custom renderer [this piece is still not allowed by selenium and I have an open PR for that ​], custom hub/node servlets etc., ) It doesn't have to be one jar per one class.

 

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-users+unsubscribe@googlegroups.com.
To post to this group, send email to selenium-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/e9f637d6-c8c3-4d64-940e-86faf8d21480%40googlegroups.com.

[Eugene S]

Hi Krishnan and thank you.

I am not sure I understand how SeLion is invoking the original Selenium runner (GridLauncherV3)? I can see that it definitely does that but not sure how. When we launch SeLion, we just do 

java -jar SeLion-Grid-1.2.0-jar-with-dependencies, we do not specify anything else. So in the MANIFEST points to JarSpawner as the main thread but I can't find the place in code where the underlying Selenium server is being invoked.

Could you pleas point me to the right direction?

Thanks,

Eugene

...

[⇜Krishnan Mahadevan⇝]

Eugene,

Its kind of hard for me to call out the entire traceroute. But the main method of JarSpawner() is where all the action begins. A bunch of parsing happens to identify the SeLion specific params that were provided and for the rest, we basically pass it on to the Selenium uber jar (which we spin off as a new Java process using apache watchdog) as command line arguments.

To help you figure out the call, what I would suggest is to add a dependency to the SeLion grid to your project, create a main() method from wherein you basically invoke

JarSpawner.main(new String[] {"-role", "hub"})

​

and then start tracing this call. This should tell you the chain of calls as they happen. Its not that complicated to trace. Its just hard for me to explain the call chain on an email :)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-users+unsubscribe@googlegroups.com.
To post to this group, send email to selenium-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/b32024fb-d283-47d0-9401-4db49a5e0f8e%40googlegroups.com.

[Eugene S]

Yes, of course. I understand. I will continue from here.

Just would like to say thank you again for your elaborate answers here and the valuable information in your blog.

It has been a great help and is very much appreciated.

Eugene

...