Integration with Selenium Grid

[Kevin Menard]

Hi,

Is anyone out there working with the browsermob proxy on Selenium
Grid? It seems there's a bit of a chicken-and-egg situation whereby
you don't know what what machine a browser is launched on until after
it's launched, but you need to provide the browser proxy config when
starting the browser. I'm thinking this could be achieved with a
custom node proxy (in the grid sense) that would create the browsermob
proxy coincidentally with launching the browser. But if anyone else
has cracked this nut, I'd love to hear about it.

Thanks,
Kevin

[Graham Abell]

I'm in the exact same situation!

I was thinking about just using a common BMP for the nodes, though it's not the solution I'd prefer.

I'd be interested in anyone elses approach too or more info on what you're planning. 

[Kevin Menard]

Hi Graham,

Thanks for confirming that I'm not crazy. I guess I'll tackle this
one. I've been wanting a good real-world use case for my grid
workshop at this year's SeleniumConf, so I can kill two birds with one
stone.

The basic idea, however, is that you can configure the proxy class
used for each node in your grid. By default it's
org.openqa.grid.selenium.proxy.DefaultRemoteProxy. This class
basically serves as the interface between the grid hub and the node
and it has a series of callbacks that can be used. Notably there's a
beforeSession and afterSession. My initial thinking is the desired
proxy configuration would be provided as a capability, the
beforeSession handler would take care of establishing the proxy, and
the afterSession would handle tearing down the proxy. I just need to
make sure the lifecycle there works the way I expect. But if it
doesn't, I'll just update Selenium to give us what we need :-)

--
Kevin

[Graham Abell]

Ah, I'm going to the conference but not the workshops - missed the boat on that one, I wasn't able to book until the speaker list was confirmed. I'll be looking for the outcome on that one! :)

That sounds like a good plan. At the minute I'm assuming the node uses the local system defined proxy? 

[Kevin Menard]

Hmm . . . Sorry, something may be confused here because of the double
meaning of proxy.

Right now, the node doesn't talk to browsermob proxy at all. The
browser is configured to talk to the browsermob proxy through the
client. The node proxy is a proxy for the Selenium Server so the
responses sent back to the hub are routed to the appropriate client.

What I'm suggesting is that since the node proxy has the opportunity
to handle the client request and it's bound to the node, we should be
able to extend the node proxy to create a proxy on browsermob proxy at
the start of the session and destroy it at the end of the session.

This is even more confusing because in RC mode, Selenium Server itself
can be a proxy. But that doesn't come into play at all here.

--
Kevin

[William Chang]

How I approached this was to start up standalone BMP processes on each remote machine. Then use selenium to configure the tests to run through the BMP proxy with each type of browser using a different port via DesiredCapabilities. Then using the REST api, I would do a HTTP PUT to start up a new HAR file and then a HTTP GET to get the HAR file from each node, targeting the specific remoteip:port for each node. 

[Graham Abell]

Hi Kevin, 

By client do you mean the machine where the code is being run from or the machine where the node is running? 

I assume it's the node machine since that would allow the BMP instance to be local to the node - that being the case my comment was in relation to the fact we have a corporate proxy which webdriver seems to pick up automatically when running on the node machines. If we specify a BMP then we just need to include that upstream proxy when we instantiate it. Does that make sense?

graham

[Kevin Menard]

By client I mean the machine running the code that's directing
WebDriver what to do. I think we're roughly on the same page with the
rest of it. It might be clearer if I can put some actual code out
there.

--
Kevin

[Graham Abell]

this is whiteboard territory. :)

[Matt Smith]

Hi All,

How did you all get on with this topic? Did you have much success?

I'm trying to do achieve this with a hub and two nodes. Ideally I'd like to have browsermob-proxy on the hub (is this possible?) or is it better to run browsermob-proxy on each node?  I've had it working locally (writing with ruby) using https://github.com/jarib/browsermob-proxy-rb, but so far have not seen any way of using this tool for a remote node.

Thanks

Matt

[Krishnan Mahadevan]

Matt,

Integrating browsermob-proxy with Grid is no big deal (Atleast in my humble opinion).

You can choose to have your proxy spawned anywhere (be it on the grid machine or the node machine or any other machine), just make sure that there is no firewall blockages and the machines can talk to each other via the network.

Your tests would be instructed to use the BMP proxy using 

DesiredCapabilities.setCapability(CapabilitiesType.PROXY, proxy)

That should pretty much do the entire thing.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

[AJ]

Krishnan,

Say if I want to setup the proxy in my nodes and I have 2 nodes - A and B. How will I specify different proxies for each node? Setting the capability can be done only once for the entire grid, isnt it?

Regards

AJ

[Krishnan Mahadevan]

Not necessarily.

All you would need to do is build the DesiredCapabilities object (wherein you would be plugging in your Proxy object) for every test.

So that way, everytime a RemoteWebDriver gets instantiated it would have its own proxy value.

Instead of binding your proxy settings into the environment, you should consider embedding this into your tests. That way it is more manageable 

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

[AJ]

Ok.. I do create a different RemoteWD object for each test. Even then, I wouldn't know to which node will the request be forwarded (since that is something which will be controlled by hub). Unless I know that, how will I be able to set the proxy value specific to that machine?

If you have a central machine which hosts proxy, then it should be easy enough to point all tests to that proxy. But maintaining the proxy at the node level doesn't seem to be that easy for me. May be I am missing something..

Regards

AJ

[Krishnan Mahadevan]

AJ,

Probably I am not being elaborate here.

DesiredCapabilities dc = DesiredCapabilities.firefox();

Proxy proxy = new Proxy();

proxy.setHttpProxy("proxyValueGoesHere");

dc.setCapability(CapabilityType.PROXY, proxy);

RemoteWebDriver rwd = new RemoteWebDriver(new URL("http://" + gridIP + ":4444/wd/hub"), dc);

With the above code,what you are essentially doing is, packing the proxy information as well into the capabilities.

This would mean that this proxy information gets related to the actual node (which the grid decides to route your test to) and the proxy gets set on that machine's firefox browser which would be spawned.

In the above scenario, you would still be running just one BMP instance [just as you would spin off a grid, you would run the bmp proxy on a machine]. The only catch here is, that the node machines should be able to talk to the machine on which the BMP is being spawned [network connectivity should be there]

This way, you have abstracted the proxy configuration from the environment and making it more manageable by embedding it in your tests.

Does that answer your question ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

[AJ]

Krishnan,

It doesn't answer my question. However, I think both of us are saying the same :). Here is what I said ealier:

It's possible and relatively easy to do this in the way mentioned below. Easy because you have a different machine in which proxy is setup. Now, I believe the assumption you are making here is, there will be only one proxy machine or a set of proxy machines and those will be static. 

My scenario is - I have n number of nodes in my grid. And I want to start the proxy for each machine in itself. As like, each machine will have its own proxy running within it. What do I achieve by doing it this way? - I don't have to rely on the network bandwidth for accessing the proxy. To do this, I would need to replace the proxy value with the value of proxy running in that machine. Its difficult (not impossible) to know which node will be used to run the test. 

Regards

AJ

[Krishnan Mahadevan]

On Wednesday, August 8, 2012, AJ wrote:

Krishnan,

It doesn't answer my question. However, I think both of us are saying the same :). Here is what I said ealier:

It's possible and relatively easy to do this in the way mentioned below. Easy because you have a different machine in which proxy is setup. Now, I believe the assumption you are making here is, there will be only one proxy machine or a set of proxy machines and those will be static. 

My scenario is - I have n number of nodes in my grid. And I want to start the proxy for each machine in itself. As like, each machine will have its own proxy running within it. What do I achieve by doing it this way? - I don't have to rely on the network bandwidth for accessing the proxy. To do this, I would need to replace the proxy value with the value of proxy running in that machine. Its difficult (not impossible) to know which node will be used to run the test. 

In that case why not refer to the proxy machine as "localhost" ? That should take care of your requirement. Through desired capabilities you would set the proxy host as "localhost".  

--

[AJ]

Yes. Localhost would work. But I was under the assumption that it would refer to the proxy in hub or the machine where code is residing. Since you are saying that proxy value would get relayed to the node, I will try that. Thanks!

[Hinling Yeung]

Hi AJ.

I know this is a pretty old thread but I was wondering whether you have any success in terms of getting browsermob proxy server running on each node instead of the grid server?  I am trying to solve the same problem right now and looking for some advice.

-Hinling

[Roy de Kleijn]

Huh, why each node?

Met vriendelijke groet / With kind regards, 

Roy de Kleijn

Verstuurd vanaf mijn iPhone

--
 
---
You received this message because you are subscribed to the Google Groups "BrowserMob Proxy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browsermob-pro...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.