Extremely slow page loads

[bru...@corollarium.com]

I'm facing extremely slow load times for webpages when using selenium and firefox. A page like yahoo.com or google.com takes several seconds (usually around 10s; if I load it twice, second time is around 5s). I can't find out why, any ideas? I'll detail below what I've done to isolate the problem.

First, versions: it's a headless server using Xvfb with Centos 64 bits 6.6, Selenium 2.44, Firefox 31.3. I tried Java 1.6 and 1.7, no changes. The server has a DNS cache server, which means that it's not a DNS problem. Curl fetches these pages instantly. It makes no difference if pages are in remote hosts or localhost, even when domain resolves to 127.0.0.1. I tried both PHP (with codeception and selenium JAR) and Python bindings (pip, also v2.44), both are slow (for these tests I'm using a simple Python program with nothing except a browser.get() call). Server has plenty of RAM and low load.

Now, some positive results: downloading a single image instead of a webpage is fast (< 0.5s). A blank page is also very fast. Adding CSS and JS slows down the page, and *seems* to be the culprit. I also noticed a high sys% time when running the tests (around 10%, versus near zero when test is not running). Since it's headless on a remote server it's difficult to see what is happening in the browser.

If anybody has *any* idea of what might be causing this issue or suggestions to debug, fix... anything is appreciated. Really, no matter how crazy or simple, please post.

[Alasdair Macmillan]

> I also noticed a high sys% time when running the tests (around 10%, versus near zero when test is not running).

Could be dodgy hardware/drivers/kernel? You could try checking the syslog/kernel log. Also, is this a virtual machine?

Where is your test running? If remotely, what's the ping time to the box? Are you using a Grid (as in, you've got a Hub)? If so, where is you Hub?

> Since it's headless on a remote server it's difficult to see what is happening in the browser.

You could give this project a go: https://github.com/aimmac23/selenium-video-node. A video could provide some more information, perhaps?

I hope you've got this problem solved by now though :/

Alasdair Macmillan

[bru...@corollarium.com]

> I also noticed a high sys% time when running the tests (around 10%, versus near zero when test is not running).

Could be dodgy hardware/drivers/kernel? You could try checking the syslog/kernel log. Also, is this a virtual machine?

I don't think so, because this only happens when running a browser. Normal operation (the server has apache/mysql/etc) has near 0 sys time. I can't see any drivers that could create this problem.

This is a real dedicated server but running under virtualization (Xen). Would it influence?
 

Where is your test running? If remotely, what's the ping time to the box? Are you using a Grid (as in, you've got a Hub)? If so, where is you Hub?

Everything runs in localhost: Selenium, xvfb, apache and mysql are on the same server. I wondered if it was some sort of network delay, but page and its contents only are loaded from localhost. I do not have a Grid/Hub, it's just a single Selenium instance.
 

> Since it's headless on a remote server it's difficult to see what is happening in the browser.

You could give this project a go: https://github.com/aimmac23/selenium-video-node. A video could provide some more information, perhaps?

This seems nice! It seems that I need to run a Hub, but I can certainly test it.
 

I hope you've got this problem solved by now though :/

Unfortunately not! :( Still looking for ideas. Thanks!
 

[Alasdair Macmillan]

A few more things you can try is to get VNC running, and use that to run Firefox manually. Could be a bit harder to setup.

Also try to get Chrome running - its doing the same job, but the implementation is different. It could help narrow down the possibilities at least.

Alasdair