Using https with Chrome in headless mode fails to load page

[Geist Eng]

Folks,

I have just starting testing with Chrome in headless mode.  Works fine except in one case when I switch in the test run from using http to accessing the DUT with https:

• previous suite ends
• next suite setup runs
• close browser that was in headless mode
• set login to https
• create chrome web driver options object with settings including headless mode
• create web driver
• go to https url
• wait until visible: login button 

GUI never loads, even after 1 minute. On failure screen shot shows this.  Very repeatable.

If I disable headless mode code works fine.

Test rig versions:

• Linux 4.4.0-93-generic #116-Ubuntu SMP x86_64
• Python 2.7.12
  
• Robot Framework 3.0 (Python 2.7.12 on linux2)
  
• Selenium 2.53.1
  
• Google Chrome 60.0.3112.113 
  
• ChromeDriver 2.32.498513 (2c63aa53b2c658de596ed550eb5267ec5967b351)
  

For reference here is the RF code to open Chrome:

Open Chrome Browser to Page

| | [Documentation]

| | ... | Opens Google Chrome to a given web page using Create Webdriver so

| | ... | arguments can be passed to webdriver in the form of capabilities.

| | ... | For more information on what capabilities that Google Chrome

| | ... | supports, see [http://chromedriver.chromium.org/capabilities] and

| | ... | [https://peter.sh/experiments/chromium-command-line-switches/]

| | [Arguments] | ${url} |

| | ${ws}= | Set Variable | window-size=1050,800 |

| | ${chrome_options}= | Evaluate | sys.modules['selenium.webdriver'].ChromeOptions() | sys |

| | Call Method | ${chrome_options} | add_argument | disable-infobars          |

| | Call Method | ${chrome_options} | add_argument | test-type                 |

| | Call Method | ${chrome_options} | add_argument | disable-extensions        |

| | Call Method | ${chrome_options} | add_argument | ignore-certificate-errors |

| | Call Method | ${chrome_options} | add_argument | ${ws} |

| | Run Keyword If | ${USE CHROME HEADLESS} |

| | ... | Call Method | ${chrome_options} | add_argument | headless |

| | Create Webdriver | Chrome | chrome_options=${chrome_options} |

| | Go To | ${url} |

| | Wait Until Element Is Visible | id=header-loginout | timeout=${TIMEOUT} |

The failure comes on the Wait Until Element Is Visible since the page never loads. TIMEOUT for this DUT is 60 secs.

The DUT is an embedded system that takes a few secs to calculate the keys needed for https access. So when successful the Go To kw takes several secs to complete.  On failure the kw returns success in under 0.5 secs everytime:

Pass log:

KEYWORD Selenium2Library . Go To ${url}

Documentation:	Navigates the active browser instance to the provided URL.
Start / End / Elapsed:	20170909 22:45:41.357 / 20170909 22:45:48.112 / 00:00:06.755

22:45:41.358

INFO

Opening url 'https://10.20.35.50/'

22:45:41.358

DEBUG

POST http://127.0.0.1:36899/session/5211a8c017eed044108e62411ef3e6d4/url {"url": "https://10.20.35.50/", "sessionId": "5211a8c017eed044108e62411ef3e6d4"}

22:45:48.112 Fail log: Documentation: Navigates the active browser instance to the provided URL. Start / End / Elapsed: 20170909 07:44:17.393 / 20170909 07:44:17.776 / 00:00:00.383 07:44:17.394 INFO Opening url 'https://10.20.35.50/' 07:44:17.394 DEBUG POST http://127.0.0.1:52602/session/ad8f0e0b1bf2a491c9cb00194d8ac85c/url {"url": "https://10.20.35.50/", "sessionId": "ad8f0e0b1bf2a491c9cb00194d8ac85c"} 07:44:17.776 DEBUG Finished Request

DEBUG

Finished Request

[Geist Eng]

Well it looks like a Chrome web driver issue (or a GCE on my part) as I tested this directly in Selenium with python bindings and get the same behavior:  http works fine but https returns right away and then the wait fails.

Output:

using Chrome headless mode

open browser to https://192.168.123.105/

find login button

Traceback (most recent call last):

  File "./test_chrome_with_options.py", line 67, in <module>

    open_web_ui()

  File "./test_chrome_with_options.py", line 49, in open_web_ui

    loginElement = driver.find_element_by_id('header-loginout')

  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 266, in find_element_by_id

    return self.find_element(by=By.ID, value=id_)

  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 744, in find_element

    {'using': by, 'value': value})['value']

  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 233, in execute

    self.error_handler.check_response(response)

  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response

    raise exception_class(message, screen, stacktrace)

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"header-loginout"}

  (Session info: headless chrome=60.0.3112.113)

  (Driver info: chromedriver=2.32.498513 (2c63aa53b2c658de596ed550eb5267ec5967b351),platform=Linux 4.4.0-93-generic x86_64)

Code:

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

from selenium.common.exceptions import TimeoutException

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys

# DUT info

g_dut = 'https://192.168.123.105/'

g_user = 'admin'

g_pw = 'admin'

# global selenium objects

wait = None

driver = None

# flow control

wait_timeout = 20

use_headless = True

def open_web_ui():

    global driver

    global wait

    # Create a new instance of the web driver

    chrome_options = Options()

    chrome_options.add_argument("disable-extensions")

    chrome_options.add_argument("disable-infobars")

    chrome_options.add_argument("test-type")

    chrome_options.add_argument("ignore-certificate-errors")

    chrome_options.add_argument("window-size=1050,800")

    if use_headless:

        print 'using Chrome headless mode'

        chrome_options.add_argument("headless")

    driver = webdriver.Chrome(chrome_options=chrome_options)

    # Open DUT web page

    print 'open browser to {}'.format(g_dut)

    driver.get(g_dut)

    # Wait until element is visable

    wait = WebDriverWait(driver, wait_timeout)

    print 'find login button'

    loginElement = driver.find_element_by_id('header-loginout')

    loginElement = wait.until(EC.element_to_be_clickable((By.ID, 'header-loginout')))

def close_web_ui():

    print 'close browser'

    driver.quit()

open_web_ui()

print 'pause a few secs before closing'

time.sleep(5)

close_web_ui()

[Tatu Aalto]

Ugh

It would be great to raise an issue to the Selenium issue tracker[1], if there is not an issue about this already.

-Tatu
Send from my mobile

[1] https://github.com/SeleniumHQ/selenium/issues

On Sep 10, 2017 10:00, "Geist Eng" <geis...@gmail.com> wrote:

Well it looks like a Chrome web driver issue (or a GCE on my part) as I tested this directly in Selenium with python bindings and get the same behavior:  http works fine bug https returns right away and then the wait fails.

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.
To post to this group, send email to robotframework-users@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Hélio Guilherme]

Please note that he is using Selenium 2.53.1, I don't think they still support it.

Why not using Selenium 3.5.1?

My Favorite Open Source Projects
awsome-lists gretl meld robotframework wxPython
(sponsored/patrocinado) Recomendo servidores e alojamento Web em:
http://www.proalojamento.com/clientes/aff.php?aff=258

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsubscrib...@googlegroups.com.

To post to this group, send email to robotframework-users@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Geist Eng]

I have started a thread on the Chrome webdriver users group to get feedback.  I have not checked the issue in the bug tracking system.  I will open a ticket if one does not exist.

I noticed the other day I was way behind on versions for selenium and wanted to upgrade when I could.  So this issue prompted me to setup a vir env and tested again.

Not surprising, since I suspected the issue was in the driver, which I am using the latest, and not the python bindings of selenium, the issue is seen in the latest versions:

• Linux 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64
  
• Python 2.7.12
  
• Robot Framework 3.0.2 (Python 2.7.12 on linux2)
  
• Selenium 3.4.3
  
• Requests 2.18.4
  
• EasySNMP Version: 0.2.5
  
• Poster Version: 0.8.1
  
• Google Chrome 61.0.3163.79 
  
• ChromeDriver 2.32.498513 (2c63aa53b2c658de596ed550eb5267ec5967b351)
  

But it was worth the try.

(Note: pip installs Selenium 3.4.3 during install of robotframework-selenium2library.)

[Hélio Guilherme]

Thanks for clearing that out. One can never tell if old versions cause problems.

In this case, Chrome Headless is a new feature (I think it is still experimental).

My Favorite Open Source Projects
awsome-lists gretl meld robotframework wxPython
(sponsored/patrocinado) Recomendo servidores e alojamento Web em:
http://www.proalojamento.com/clientes/aff.php?aff=258