Python webdriver get() hangs. (Firefox 26.0, Selenium 2.39)

5,592 views
Skip to first unread message

Mike A

unread,
Jan 20, 2014, 2:51:36 PM1/20/14
to seleniu...@googlegroups.com
Any ideas why driver.get(url) hangs?
The script below opens Firefox, visits a website, finds all the links, and visits some links randomly.

After it visiting about random 10 urls it hangs.  In other words, after it does a few "driver.get(url)", Firefox stops responding.
At that point, I have to kill firefox manually, and the console output of this is shown at very bottom.

I'm using Firefox 26.0, Selenium 2.39.0.



--------------------
import random
import time

from selenium import webdriver

def browse_website(webpage, max_random_pages=20):
    '''Go to website and visit a few random links'''
    driver = webdriver.Firefox()
    driver.set_page_load_timeout(30)
    driver.implicitly_wait(10)
    # Visit website
    print webpage
    driver.get(webpage)
    # Find link urls
    link_urls = []
    for x in driver.find_elements_by_tag_name("a"):
        href = x.get_attribute('href')
        # Make sure href is not None before doing "startswith"
        if href and href.startswith("http"):
            link_urls.append(href)
    if len(link_urls) > max_random_pages:
        link_urls = random.sample(link_urls, max_random_pages)
    # Visit urls
    for url in link_urls:
        print url
        driver.get(url)
        time.sleep(5)
    driver.close()

if __name__ == '__main__':
    browse_website('http://www.yahoo.com')
--------------------


Here is the console output after running, firefox hanging, and killing it manually.
--------------------
55YWhvby5jb20-/RO=2/RU=aHR0cDovL3Nwb3J0cy55YWhvby5jb20vbmhsL3RlYW1zL2ZsYQ--/RS=%
5EADA8LzH5X5UGN2OKoV_E0Z1BdbZi5I-
EYnBvcwMxBGNjb2RlA29sZF9nYQRjcG9zAzE2BGN0AzEEZwMyMzE4Njg2OQRpbnRsA3VzBHBrZ3QDNAR
wb3MDMQRzZWMDdGQtc3RybQRzbGsDc3AEdGVzdAM5MDE-/RV=1/RE=1391456393/RH=aHNyZC55YWhv
by5jb20-/RO=2/RU=aHR0cDovL2hlbHAueWFob28uY29tL2tiL2luZGV4P3BhZ2U9Y29udGVudCZ5PVB
ST0RfRlJPTlQmbG9jYWxlPWVuX1VTJmlkPVNMTjE0NTUz/RS=%5EADAVbyNupooQJnHav7U3yZh.7tc8
uw-
55YWhvby5jb20-/RO=2/RU=aHR0cDovL3dlYXRoZXIueWFob28uY29tL3VuaXRlZC1zdGF0ZXMvVGV4Y
XMvUm91bmRSb2NrLTI0ODUxNzcv/RS=%5EADAKaTFwSt8sNQyynb8HuTJmcdNvDM-
55YWhvby5jb20-/RO=2/RU=aHR0cDovL21vdmllcy55YWhvby5jb20v/RS=%5EADAKYS007MGfAuQjvH
D9bQAPJVpRXg-
55YWhvby5jb20-/RO=2/RU=aHR0cDovL25ld3MueWFob28uY29tL2xpZ2h0Ym94L2RpbGJlcnQtc2xpZ
GVzaG93LzIwMTQwMTEzLWR0MTQwMTEzLWdpZi1waG90by0wNTAxNDE0MTQuaHRtbA--/RS=%5EADAS5P
sLR_UYMZEYD0QeTAneznf9BU-
55YWhvby5jb20-/RO=2/RU=aHR0cDovL2Fuc3dlcnMueWFob28uY29tLw--/RS=%5EADAeuvwfOZQnS1
9LOkwF12gaZpUEx0-
55YWhvby5jb20-/RO=2/RU=aHR0cDovL25ld3MueWFob28uY29tL2xpZ2h0Ym94L2RpbGJlcnQtc2xpZ
GVzaG93LzIwMTQwMTExLWR0MTQwMTExLWdpZi1waG90by0wNTAxMDIxOTkuaHRtbA--/RS=%5EADAAto
Q0mG.MRPEB6hMAryvoDM5uPk-
55YWhvby5jb20-/RO=2/RU=aHR0cDovL3Nwb3J0cy55YWhvby5jb20vbmJhL3RlYW1zL2RhbA--/RS=%
5EADAES.MVQ_1OM.iUSAj5rnFg818NUo-
55YWhvby5jb20-/RO=2/RU=aHR0cDovL3d3dy5mbGlja3IuY29tL3Bob3Rvcy95YWhvb2VkaXRvcnNwa
WNrcy9nYWxsZXJpZXMvNzIxNTc2Mzk4NzI1NDkxOTMvd2l0aC84NzIzNzQyOTY2L2xpZ2h0Ym94Lz95Y
z13d3cueWFob28uY29t/RS=%5EADAIEUuCahKtKrQZTMdOEKYOXBZrW0-
Traceback (most recent call last):
  File "test.py", line 31, in <module>
    browse_website('http://www.yahoo.com')
  File "test.py", line 26, in browse_website
    driver.get(url)
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 176, in get
    self.execute(Command.GET, {'url': url})
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 162, in execute
    response = self.command_executor.execute(driver_command, params)
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\remote_connectio
n.py", line 349, in execute
    return self._request(url, method=command_info[0], data=data)
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\remote_connectio
n.py", line 377, in _request
    resp = self._conn.getresponse()
  File "C:\Python27\lib\httplib.py", line 1045, in getresponse
    response.begin()
  File "C:\Python27\lib\httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "C:\Python27\lib\httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "C:\Python27\lib\socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
socket.error: [Errno 10054] An existing connection was forcibly closed by the re
mote host

Mike A

unread,
Jan 21, 2014, 10:56:58 AM1/21/14
to seleniu...@googlegroups.com
Changing one line in the script results in no hangs:

#driver = webdriver.Firefox()
driver = webdriver.Chrome()


So the Firefox driver seems to be the problem.


--
You received this message because you are subscribed to a topic in the Google Groups "Selenium Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/selenium-users/cOpsFtJfThQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/d3822039-da23-49d7-9c08-faabbfe84d47%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

David Lai

unread,
Jan 21, 2014, 1:05:39 PM1/21/14
to seleniu...@googlegroups.com
Is it the same site each time or different url ?  I'm thinking maybe there might be a certificate issue with one of the URLs which causes it to stop dead in it's tracks because of some warning prompt.

Mike A

unread,
Jan 21, 2014, 1:17:59 PM1/21/14
to seleniu...@googlegroups.com
> Is it the same site each time or different url ?  I'm thinking maybe there might be
> a certificate issue with one of the URLs which causes it to stop dead in it's tracks because of some warning prompt.

Firefox definitely hung on other websites frequently.
I only used "www.yahoo.com" as an example in my original post because I could consistently get hangs with Firefox after it did several "driver.get()" calls to random link urls on that page.

I'm now using Chrome driver and going to the top ~100 US websites and I have not experienced the browser hanging.  So I can at least move forward with what I needed to do.  Although I imagine it would be nice to track down this Firefox driver problem.

Mike


PS: I've found other kinds of exceptions in doing this, but those are easy to deal with.
For example, get_attribute() would occasionally return an exception, but I could safely get handle that:

    for x in driver.find_elements_by_tag_name("a"):
        try:

            href = x.get_attribute('href')
        except:
            continue



To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/fefe6e9c-1b88-462a-909e-55bce26a38ec%40googlegroups.com.

Martin Olsson

unread,
Feb 5, 2014, 2:01:39 AM2/5/14
to seleniu...@googlegroups.com
FWIW, I have the same problem when using selenium 2.39.0 install from pip along with Firefox 26.0+build2-0ubuntu0.13.04.2

My Python selenium script browsers various pages all on a single server (https+self sign cert+http auth) and it hangs after first successfully loading around 300 pages. If I switch to webdriver.Chrome() it runs for days straight without freezing. Just a few months back I was able to run similar multi day test runs using webdriver.Firefox() as well and I've not made any change to my selenium python script itself in this time. The stacktrace when it freezes is identical to what Mike reported above.



Regards,
Martin

Olivier Mireault

unread,
Feb 19, 2014, 11:32:04 AM2/19/14
to seleniu...@googlegroups.com
I believe we are also suffering from the same problems. 

Our firefox tests will just hang and the location seems consistent with Mike's. 

Olivier

Mikael Morvan

unread,
Feb 21, 2014, 4:10:50 AM2/21/14
to seleniu...@googlegroups.com
Facing the same issue. Tried with FF25 WebDriver 2.38.0, FF25 WD 2.38.4, FF27, WD 2.38.4, FF27.0.1 WD 27.0.1
Still the same problem that is Firefox doesn't respond. Must stop it manually. When I perform the stop, I have this trace log in my console :

  File "C:\SVN_ValidationScripts\Products\Ngpr\Manager\Gui\Tests\RecipientLists.py", line 112, in runTest
    page = WMPMetadataEditRecipientList(self.manager.web, rest_recipientlist.id).get()
  File "C:\SVN_ValidationScripts\Products\Ngpr\Manager\Gui\ProductLibs\WMP.py", line 36, in get
    self.manager.driver.get(self.url)
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 176, in get
    self.execute(Command.GET, {'url': url})
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 162, in execute
    response = self.command_executor.execute(driver_command, params)
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 349, in execute
    return self._request(command_info[0], url, body=data)
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 380, in _request
    resp = self._conn.getresponse()
  File "C:\Python27\lib\httplib.py", line 1045, in getresponse
    response.begin()
  File "C:\Python27\lib\httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "C:\Python27\lib\httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "C:\Python27\lib\socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
</PRE>

Glyn Davies

unread,
Feb 22, 2014, 4:28:31 AM2/22/14
to seleniu...@googlegroups.com
Hey Guys,

I have installed selenium for the first time today, and am having similar issues to those mentioned already. Here are the detail of my setup and the issues:

  mac os x 10.9
  python 2.7.5 ( in a virtualenv )
  selenium 2.40 ( installed via pip )
  phantomjs 1.9 ( installed via homebrew )
  chromedriver 2.9.248307

I have written a couple of small tests to try out the various webdrivers but of the following, Firefox, Chrome and Phantomjs not one of them can make a successful call to create a webdriver.

Here is the code sample I am running:

from selenium import webdriver

driver = webdriver.PhantomJS()
# driver = webdriver.Firefox()
# driver = webdriver.Chrome(executable_path='./chromedriver')
driver.get("http://duckduckgo.com/")
driver.find_element_by_id('search_form_input_homepage').send_keys("realpython")
driver.find_element_by_id("search_button_homepage").click()
print driver.current_url
driver.quit

If I run it with PhantomJS() I get the following
raise WebDriverException("Can not connect to GhostDriver")
selenium.common.exceptions.WebDriverException: Message: 'Can not connect to GhostDriver'

If I run it with Chrome() I get the following:
raise WebDriverException("Can not connect to the ChromeDriver")
selenium.common.exceptions.WebDriverException: Message: 'Can not connect to the ChromeDriver'

If I run it with Firefox() I get the following, the browser actually pops but then:
selenium.common.exceptions.WebDriverException: Message: "Can't load the profile. Profile Dir: /var/folders/fn/qhj2pg4j59dcdhwcmn2kn4gw0000gn/T/tmpae4jKO Firefox output: *** LOG addons.xpi: startup\n*** LOG addons.xpi: Skipping unavailable install location app-system-share\n*** LOG addons.xpi: checkForChanges\n*** LOG addons.xpi: No changes found\nOpenGL version detected: 210\nOpenGL version detected: 210\nJavaScript error: chrome://browser/content/urlbarBindings.xml, line 654: aUrl is undefined\n"

If anyone can suggest where I'm going wrong that would be greatly appreciated.

Cheers,
Glyn

Michael Shilov

unread,
Mar 11, 2014, 1:40:06 PM3/11/14
to seleniu...@googlegroups.com
+1, I also faced this problem

Simeon Shpiz

unread,
Mar 17, 2014, 4:49:52 AM3/17/14
to seleniu...@googlegroups.com
+1, 
Also facing similar problems. Using code that was written (and worked) a few months ago and now it hangs. Using chrome driver for the time being

Mario Niebes

unread,
Mar 18, 2014, 4:32:36 PM3/18/14
to seleniu...@googlegroups.com
that not only affects Python. With Java/Selenium/FF27.0.1 I have similar Problems. Sometimes it starts, but closing the browser only "closes" the browser without killing the process (I guess thats the same problem, otherwise I'll create a new thread).

Sergey Likhman

unread,
Mar 30, 2014, 4:01:28 AM3/30/14
to seleniu...@googlegroups.com
Same issue for me too. But what i have seen that if you stop the python process (by killing it) , the driver continue with finishing it last request.
Did you see such behavior?

понедельник, 20 января 2014 г., 21:51:36 UTC+2 пользователь Mike A написал:
Reply all
Reply to author
Forward
0 new messages