Grid: Intercept and add capability in custom proxy or ?

153 views
Skip to first unread message

msar...@gmail.com

unread,
Jun 13, 2017, 1:39:58 AM6/13/17
to Selenium Users
Hello,
I'm trying to add browsermob proxy as a proxy to the browser by implementing a custom grid remote proxy and attempting to add the necessary capability (depending on browser type) in the getNewSession() method. For example, the Chrome implementation:

@Override
public TestSession getNewSession(Map<String, Object> requestedCapability) {
  options = new HashMap<>();
  options.put("args", Arrays.asList("--proxy-server=localhost:8080"));
  requestedCapability.put(ChromeOptions.CAPABILITY, options);
  return super.getNewSession(requestedCapability);

}

However, in the logs, I'm not seeing the proxy added as a capability:

Jun 13, 2017 12:50:54 AM org.openqa.grid.web.servlet.handler.RequestHandler process
INFO: Got a request to create a new session: Capabilities [{browserName=chrome, version=, platform=ANY}]
Jun 13, 2017 12:50:54 AM org.openqa.selenium.remote.server.SessionCleaner <init>
INFO: SessionCleaner initialized with insideBrowserTimeout 0 and clientGoneTimeout 1800000 polling every 180000
Jun 13, 2017 12:50:55 AM org.openqa.grid.internal.TestSlot getNewSession
INFO: Trying to create a new session on test slot {browserName=chrome, maxInstances=1, version=, platform=ANY}
Jun 13, 2017 12:50:55 AM org.openqa.selenium.remote.server.rest.ResultConfig handle
INFO: Executing: [new session: Capabilities [{browserName=chrome, version=, platform=ANY}]])
Jun 13, 2017 12:50:55 AM org.openqa.selenium.remote.server.DefaultDriverProvider newInstance
INFO: Creating a new session for Capabilities [{browserName=chrome, version=, platform=ANY}]
Starting ChromeDriver 2.30.477700 (0057494ad8732195794a7b32078424f92a5fce41) on port 31592
Only local connections are allowed.
Jun 13, 2017 12:50:57 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: OSS
Jun 13, 2017 12:50:57 AM org.openqa.selenium.remote.server.rest.ResultConfig handle
INFO: Done: [new session: Capabilities [{browserName=chrome, version=, platform=ANY}]]
Jun 13, 2017 12:50:57 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: OSS

Now, if I include the proxy capability in the RemoteWebDriver request in the client:

ChromeOptions options = new ChromeOptions();
options.addArguments("--proxy-server=localhost:8080");
DesiredCapabilities caps = DesiredCapabilities.chrome();

caps.setCapability(ChromeOptions.CAPABILITY, options);

driver = new RemoteWebDriver(hub.getWebDriverHubRequestURL(), caps);

Everything works as expected:

Jun 13, 2017 1:16:04 AM org.openqa.grid.web.servlet.handler.RequestHandler process
INFO: Got a request to create a new session: Capabilities [{browserName=chrome, chromeOptions={args=[--proxy-server=localhost:8080], extensions=[]}, version=, platform=ANY}]
Jun 13, 2017 1:16:04 AM org.openqa.selenium.remote.server.SessionCleaner <init>
INFO: SessionCleaner initialized with insideBrowserTimeout 0 and clientGoneTimeout 1800000 polling every 180000
Jun 13, 2017 1:16:05 AM org.openqa.grid.internal.TestSlot getNewSession
INFO: Trying to create a new session on test slot {browserName=chrome, maxInstances=1, version=, platform=ANY}
Jun 13, 2017 1:16:05 AM org.openqa.selenium.remote.server.rest.ResultConfig handle
INFO: Executing: [new session: Capabilities [{browserName=chrome, chromeOptions={args=[--proxy-server=localhost:8080], extensions=[]}, version=, platform=ANY}]])
Jun 13, 2017 1:16:05 AM org.openqa.selenium.remote.server.DefaultDriverProvider newInstance
INFO: Creating a new session for Capabilities [{browserName=chrome, chromeOptions={args=[--proxy-server=localhost:8080], extensions=[]}, version=, platform=ANY}]
Starting ChromeDriver 2.30.477700 (0057494ad8732195794a7b32078424f92a5fce41) on port 47865
Only local connections are allowed.
Jun 13, 2017 1:16:07 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: OSS
Jun 13, 2017 1:16:07 AM org.openqa.selenium.remote.server.rest.ResultConfig handle
INFO: Done: [new session: Capabilities [{browserName=chrome, chromeOptions={args=[--proxy-server=localhost:8080], extensions=[]}, version=, platform=ANY}]]
Jun 13, 2017 1:16:07 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: OSS

Can I not add new capabilities in my custom proxy in the getNewSession() method? Should I be extending some other class rather than creating a custom proxy to accomplish this task?

Thanks for your help.

Krishnan Mahadevan

unread,
Jun 13, 2017, 1:55:25 AM6/13/17
to seleniu...@googlegroups.com

You are not looking at this properly. The line

 

Jun 13, 2017 12:50:54 AM org.openqa.grid.web.servlet.handler.RequestHandler process

INFO: Got a request to create a new session: Capabilities [{browserName=chrome, version=, platform=ANY}]

 

Is shown when the Grid receives a new session request but before the session is created.

 

So this line is being printed even before your call to org.openqa.grid.internal.BaseRemoteProxy#getNewSession which is where your logic of adding/altering the capabilities is being invoked.

 

The way to identify this properly is by setting a breakpoint in your test code such that the code execution stops after the RemoteWebDriver instance is created [ which would mean that a browser was spun off at this point in the remote host at the node side], and then manually inspecting the browser’s settings (The one that was opened by your test code in your node) to see if the proxy settings were applied properly.

 

On a side note, I would be curious to know as to why would you want to inject proxy settings at the node level ? This would essentially cause your node to end up injecting the BMP proxy server for every new session (which means its done for every test) whereas if you do it when you are instantiating a new RemoteWebDriver instance, you have the choice of doing it based on your need.

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/51acf7a5-bac3-4c74-aa94-a58144f3840c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

msar...@gmail.com

unread,
Jun 13, 2017, 9:36:46 PM6/13/17
to Selenium Users
I inspected the proxy settings of the resulting browser in each of the two scenarios. The proxy gets applied correctly in the latter scenario, but does not get applied at all in the former scenario by the getNewSession() method. Any thoughts why the proxy capability added in getNewSession() is not being applied on the browser?

I want to inject these proxy settings at this level in order to control the lifecycle of the proxy based on when the session begins/ends and so that the client doesn't have to worry about pointing to the correct proxy, etc. I want every session to be proxied anyways, so it this approach makes the most sense to me, if it is possible.

Krishnan Mahadevan

unread,
Jun 14, 2017, 11:42:05 AM6/14/17
to seleniu...@googlegroups.com

Ok.. I spent some more time looking at the code. I don’t think what you are trying is going to work.

The capabilities map that gets exposed to you via the Proxy is an extracted copy of the original InputStream that’s embedded within the javax.servlet.http.HttpServletRequest (Remember your RemoteWebDriver instance eventually creates a HttpServletRequest

 

So in the grand scheme of things, what you are modifying, is a copy of the actual capability. The actual capability is still part of the original HttpServletRequest that your RemoteWebDriver instance forwarded to the Hub and which gets sent as is to the elected proxy node.

So you would need to change your logic and do one of the following :

 

  • Set the proxy via the capabilities when you instantiate a RemoteWebDriver instance (or)
  • You can setup the Proxy server when you spin off the selenium node via the JVM arguments (See here for more details)

 

I hope that explains the rationale behind the behavior you are currently seeing.

msar...@gmail.com

unread,
Jun 14, 2017, 10:51:10 PM6/14/17
to Selenium Users
So what is happening in org.openqa.grid.selenium.proxy.DefaultRemoteProxy#beforeSession then? If that method is also just operating on a copy of the capabilities, what purpose does that code have?

⇜Krishnan Mahadevan⇝

unread,
Jun 14, 2017, 10:55:23 PM6/14/17
to Selenium Users

Not sure what is the question. The beforeSession() is meant to act as a listener in the grid wherein it let's you do something before the session begins.

--

msar...@gmail.com

unread,
Jun 14, 2017, 11:09:30 PM6/14/17
to Selenium Users
Sorry, I did not explain. I have also tried attempting to add the proxy via the beforeSession() method in my extension of DefaultRemoteProxy, without any success. 

As I understand your previous statement, the capabilities map exposed in getNewSession() is a copy and therefore, additions to the capabilities copy would have no effect. I therefore assume that the capabilities map exposed via the TestSession in beforeSession() is also a copy and any attempts to add an additional capability there would not work either.

With that said, my latest question is: why does the implementation of org.openqa.grid.selenium.proxy.DefaultRemoteProxy#beforeSession set capabilities if it is merely operating on a copy of the original capabilities? I don't understand what the purpose would be.

Thanks.

Krishnan Mahadevan

unread,
Jun 15, 2017, 12:22:10 AM6/15/17
to seleniu...@googlegroups.com

Excellent question! Unfortunately I don’t have the answer.  My best guess is that the capability object is being exposed in the createSession() only to facilitate an easy inspection of what is being desired by the end user, so that some tweaks if necessary can be taken up. For e.g., if one were to be building an on-demand Grid wherein there are no nodes, but everything is backed by a docker end-point, one would need to inspect the incoming capability to decide what sort of a docker container needs to be spun off before forwarding the new session to it.  Or maybe one would want to switch on a VM and have it ready before the new session request can be forwarded to it.

 

Whatever I have told you so far, is all based on my experiences with the grid and by looking at the codebase.

 

I think currently the updates being made to the capabilities object via the beforeSession() method might actually be dead code and may not even be working (I am yet to vet out this theory, but that’s my observation).

 

Here’s the chain of events based on which I am saying this :

 

  1. When a new HttpServletRequest arrives to the org.openqa.grid.web.servlet.DriverServlet via the end-point http://GridIp:GridPort/wd/hub [ Remember this is the endpoint to which you refer to when you instantiate a new RemoteWebDriver ], a variant of SeleniumBasedRequest is created via the factory method org.openqa.grid.web.servlet.handler.SeleniumBasedRequest#createFromRequest. Since we are dealing with WebDriver requests only and not going to be considering the legacy RC code, that instance would be of type org.openqa.grid.web.servlet.handler.WebDriverRequest
  2. The org.openqa.grid.web.servlet.handler.WebDriverRequest as part of its instantiation takes a copy of the InputStream associated with a javax.servlet.http.HttpServletRequest.
  3. The InputStream thus copied in (2) is what is used by org.openqa.grid.internal.TestSession#prepareProxyRequest to prepare a HttpRequest.
  4. The HttpRequest thus prepared is what is forwarded to the actual node via org.openqa.grid.internal.TestSession#forward

 

So in all these events, the desiredCapability which is extracted out of an incoming HttpServletRequest (from step 1) inside a variant of SeleniumBasedRequest is NOT referred anywhere when it comes to forwarding it to the node.

 

That is why I say that the code within  org.openqa.grid.selenium.proxy.DefaultRemoteProxy#beforeSession which is updating the TestSession’s map is perhaps dead code.

 

You can quickly cross check this theory by nullifying this method in your customized proxy and run a test which sets one of the following via the capabilities object when instantiating a RemoteWebDriver:

  • org.openqa.selenium.firefox.FirefoxDriver#BINARY [ This would need you to add the same key into your node configuration JSON file in the node side when starting the node for it to take affect ]
  • org.openqa.selenium.chrome.ChromeOptions#CAPABILITY [ This would need you to add the key chrome_binary to your node configuration JSON file in the node side when starting the only for it to take affect ]

 

If you have turned on debug level logs at the node side, you would perhaps notice that the incoming capability object would still have these keys in it.

 

For your use-case, I would recommend that you resort to adding the proxy at the client side where the RemoteWebDriver is being instantiated. That’s easier to manage. If you have a framework, you can inject this information at the part wherein the RemoteWebDriver is being instantiated.

 

If you still feel that this perhaps makes no sense and what is being updated in a TestSession as its capability is what should be forwarded to the node by the DefaultRemoteProxy, then you can raise a bug against the selenium codebase.

Diego Fernando Molina

unread,
Jun 16, 2017, 5:34:20 AM6/16/17
to Selenium Users
Hi,

I think the main issue could be the way you are adding the new capability in the node, because it is done in a different way as it is done in the client, this results with an object with a different structure. 

Here is what I try to say:

When you add the option while creating the RemoteWebDriver you do it like this:
ChromeOptions options = new ChromeOptions();
options.addArguments("--proxy-server=localhost:8080");
DesiredCapabilities caps = DesiredCapabilities.chrome();

caps.setCapability(ChromeOptions.CAPABILITY, options);

And this creates an object with this structure: Capabilities [{browserName=chrome, chromeOptions={args=[--proxy-server=localhost:8080], extensions=[]}, version=, platform=ANY}]
Note the chromeOptions structure.

But if you create it like this:
  options = new HashMap<>();
  options.put("args", Arrays.asList("--proxy-server=localhost:8080"));
  requestedCapability.put(ChromeOptions.CAPABILITY, options);
And this just creates an object with a different structure.

So I suggest you try to create the whole ChromeOptions object in the Proxy as you do it in the client, for example:

@Override
public TestSession getNewSession(Map<String, Object> requestedCapability) {
ChromeOptions options = new ChromeOptions();
options.addArguments("--proxy-server=localhost:8080");
requestedCapability.put(ChromeOptions.CAPABILITY, options);
  return super.getNewSession(requestedCapability);
}

Let me know if it works,

Cheers,

Diego

⇜Krishnan Mahadevan⇝

unread,
Jun 16, 2017, 5:47:18 AM6/16/17
to Selenium Users
Diego probably has a point there. I went back to looking at the code once again. It looks like the capabilities that are part of the TestSession are what is being referred to. I missed it the last time I went through the codebase.

So I stand corrected. The part of my mail which talks about some dead code residing within DefaultRemoteProxy#beforeSession SHOULD BE completely ignored since its not true.

My apologies for the confusion.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-users+unsubscribe@googlegroups.com.
To post to this group, send email to selenium-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/05890dce-b1f4-47a9-a1f0-8523d3e1dad5%40googlegroups.com.

msar...@gmail.com

unread,
Jun 16, 2017, 9:54:22 PM6/16/17
to Selenium Users
Diego, Krishnan,
Thanks for your comments. I've tried your ideas and done some more digging on my own. The method of creating the ChromeOptions (whether via the ChromeOptions object or via a HashMap) doesn't make a difference. Neither are passed in the session request to the Node. I've confirmed this by examining the request passed from the Hub to Node with Wireshark.

What appears to happen is that the TestSession instance passed to the DefaultRemoteProxy#beforeSession() method has the requestedCapabilites attribute which is a reference to the desiredCapabilities attribute of the WebDriverRequest instance in the RequestHandler. Modifying the requestedCapabilities of the TestSession instance is all fine and well and ends up back in the WebDriverRequest instance, but the problem happens when the RequestHandler goes to forward the request to the Node.

While the WebDriverRequest at this point contains the modified desiredCapabilities map (containing my ChromeOptions that I added in the beforeSession method), the WebDriverRequest does not use this attribute in the request forwarded to the Node. Instead, the WebDriverRequest has a body attribute containing the original desiredCapabilities from the original remoteWebDriver request from the client. The body attribute is what gets sent in the request to the Node (confirmed via Wireshark).

The RequestHandler is doing nothing to sync the body and desiredCapabilities attributes of the WebDriverRequest between when beforeSession() is called and forwarding the request to the Node. Therefore, the beforeSession() method has no ability to modify the request sent to the Node.

Having said all that, it appears that the current implementation of DefaultRemoteProxy#beforeSession is dead code after all.

Does this appear to be a bug? I haven't been able to find any recent changes that would have made this code obsolete due to an architectural change.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.

Krishnan Mahadevan

unread,
Jun 19, 2017, 1:02:00 AM6/19/17
to seleniu...@googlegroups.com

This was a bug and I went about fixing this issue and have sent out a pull request.

 

PR : https://github.com/SeleniumHQ/selenium/pull/4196

 

Fingers crossed! Let’s hope it gets vetted out and merged.

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 


Reply-To: <seleniu...@googlegroups.com>
Date: Saturday, June 17, 2017 at 7:24 AM
To: Selenium Users <seleniu...@googlegroups.com>

msar...@gmail.com

unread,
Jun 19, 2017, 5:35:01 PM6/19/17
to Selenium Users
Thanks, Krishnan.

Best Regards,
Matt
Reply all
Reply to author
Forward
0 new messages