Can't create remote webdriver with python binding

[Kumar Gaurav]

Dear All, I am trying to create remote webdriver with python binding but getting WebDriverException with no message.

selenium grid: 3.12.0, Linux
python binding version: 3.12.0, running on OSX

python version: 2.7.15

There is also a Github ticket: https://github.com/SeleniumHQ/selenium/issues/6027. Does anyone have any idea of why this could be happening? Thanks a lot.

Python 2.7.15 (default, Jun  9 2018, 00:09:15) 
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from selenium import webdriver
>>> from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
>>> driver = webdriver.Remote(command_executor='http://myhub-url/wd/hub', desired_capabilities=DesiredCapabilities.CHROME)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 156, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 245, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 314, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 208, in check_response
    raise exception_class(value)
selenium.common.exceptions.WebDriverException: Message: 

>>>

[Kumar Gaurav]

Also tried with python 3.6.5: same error

My selenium grid is running on port 80. Even specifying port as 80, got same error.

Tried with different selenium grid: same error.

Both selenium grids are working fine (confirmed with other projects using java binding).

Any idea please? I will try to trace http request/response today to see what kind of request client is making to grid and what response it receives.

[Kumar Gaurav]

[Kumar Gaurav]

Update: works with grid running locally (127.0.0.1). But does not work with remote grid!

[Kumar Gaurav]

Tracing the request, python binding is sending following in request body 

{
 "capabilities": {
 "alwaysMatch": {
 "goog:chromeOptions": {
 "args": [],
 "extensions": []
 },
 "browserName": "chrome",
 "platformName": "any"
 },
 "firstMatch": [
 {}
 ]
 },
 "desiredCapabilities": {
 "goog:chromeOptions": {
 "args": [],
 "extensions": []
 },
 "platform": "ANY",
 "browserName": "chrome",
 "version": ""
 }
}

and gets response as:

{
    "status": 13,
    "value": {
        "class": "java.lang.NullPointerException",
        "stackTrace": []
    }
}

I also reproduced it using postman.

Now for java clients that work, following is the request:

{
 "desiredCapabilities": {
 "acceptSslCerts": true,
 "browserName": "chrome",
 "goog:chromeOptions": {
 "args": [],
 "extensions": []
 },
 "javascriptEnabled": true,
 "platform": "ANY",
 "takesScreenshot": true,
 "version": ""
 },
 "capabilities": {
 "desiredCapabilities": {
 "acceptSslCerts": true,
 "browserName": "chrome",
 "goog:chromeOptions": {
 "args": [],
 "extensions": []
 },
 "javascriptEnabled": true,
 "platform": "ANY",
 "takesScreenshot": true,
 "version": ""
 },
 "firstMatch": [
 {
 "browserName": "chrome",
 "goog:chromeOptions": {
 "args": [],
 "extensions": []
 }
 }
 ]
 }
}

And we get session id in the response:

{
    "state": null,
    "sessionId": "73edbb43-3318-44a2-b5b5-25cab1cda61e",
    "hCode": 1663785931,
    "value": {
        "mobileEmulationEnabled": false,
        "hasTouchScreen": true,
        "platform": "LINUX",
        "acceptSslCerts": false,
        "acceptInsecureCerts": false,
        "webStorageEnabled": true,
        "browserName": "chrome",
        "takesScreenshot": true,
        "javascriptEnabled": true,
        "setWindowRect": true,
        "unexpectedAlertBehaviour": "",
        "applicationCacheEnabled": false,
        "rotatable": false,
        "networkConnectionEnabled": true,
        "chrome": {
            "chromedriverVersion": "2.35.528139 (47ead77cb35ad2a9a83248b292151462a66cd881)",
            "userDataDir": "/tmp/.org.chromium.Chromium.Bewy5y"
        },
        "takesHeapSnapshot": true,
        "pageLoadStrategy": "normal",
        "unhandledPromptBehavior": "",
        "databaseEnabled": false,
        "handlesAlerts": true,
        "version": "64.0.3282.186",
        "browserConnectionEnabled": false,
        "nativeEvents": true,
        "webdriver.remote.sessionid": "73edbb43-3318-44a2-b5b5-25cab1cda61e",
        "locationContextEnabled": true,
        "cssSelectorsEnabled": true
    },
    "class": "org.openqa.selenium.remote.Response",
    "status": 0
}

So basically I need to provide more options & desired capabilities in my request, though I am not sure yet which ones are significant ones. Trying to mimic request created by java binding now.

[Kumar Gaurav]

Request that does not work:

{
 "capabilities": {
 
 "alwaysMatch": {
 "acceptInsecureCerts": true,

 "browserName": "chrome",
 "platformName": "any"
 },
 "firstMatch": [
 {}
 ]
 },
 "desiredCapabilities": {

 "takesScreenshot": true,
 "acceptSslCerts": true,

 "javascriptEnabled": true,
 "platform": "ANY",

 "browserName": "chrome",
 "version": ""
 }
}

Request that Works:

{
 "capabilities": {
 "desiredCapabilities": {
 "takesScreenshot": true,
 "acceptSslCerts": true,

 "javascriptEnabled": true,
 "platform": "ANY",

 "browserName": "chrome",
 "version": ""
 },
 "alwaysMatch": {
 "acceptInsecureCerts": true,

 "browserName": "chrome",
 "platformName": "any"
 },
 "firstMatch": [
 {}
 ]
 },
 "desiredCapabilities": {

 "takesScreenshot": true,
 "acceptSslCerts": true,

 "javascriptEnabled": true,
 "platform": "ANY",

 "browserName": "chrome",
 "version": ""
 }
}

Notice that in working version, there is an additional Json object in capabilities: desiredCapabilities. 

Without this, the grid throws NullPointer. 

This is with one grid. I think the issue is that the grid is old (still >3 version as far as I know) and is expecting different request format. I guess python binding is not compatible with old grid.

With another grid, the issue was the resolution of hostname to IP. In our scenario, selenium hub is behind load balancer so IP resolves in correctly. As resolve_ip is set to True by default and there is no way to change that while creating remote driver, it was resolving to wrong IP and was giving error.

Now solution for me is:

1. Find a version of python binding that is compatible with my old grid.

OR

2. Use new version of python binding with new grid: Override webdriver class and set resolve_ip to False while creating RemoteConnection object.