Testing site performance with Selenium

[Josh Fraser]

I'd like to use Selenium to test site performance. Ideally, I'd like
to measure the time it takes the browser to a) download the site, b)
reach the DOMContentReady event and c) fire the onload event. Is
there an easy way to do this with Selenium? I see a
waitForPageToLoad() function in the documentation, but don't see
anything equivalent for document ready.

It seems like the Selenium functions may not be getting called until
after the DOM is ready. Is that correct and if so, is there any way
to use Selenium for accurate performance timing or should I look for
something else?

[Adrian Yee]

If getting that data via a beacon works for you, Firebug with NetExport
will export a HAR file which contains the DOMContentReady and onload
timings (and more).

Adrian

[Patrick Lightbody]

Yes, there is some code (not 100% polished, but it works) in the browsermob-page-perf project that does exactly that. However, I'm not yet sure what options we've got for IE, Safari, Opera, Chrome, etc. That's really where the WTF idea came from: to borrow bits from webpagetest.org, BrowserMob, open source, etc to provide that type of functionality across the board.

--
We provide FREE website monitoring and load testing
http://browsermob.com

Patrick Lightbody
BrowserMob
(w) +1 (503) 828-9003 x 101
(m) +1 (415) 830-5488

[Patrick Lightbody]

Yes, there is some code (not 100% polished, but it works) in the browsermob-page-perf project that does exactly that. However, I'm not yet sure what options we've got for IE, Safari, Opera, Chrome, etc. That's really where the WTF idea came from: to borrow bits from webpagetest.org, BrowserMob, open source, etc to provide that type of functionality across the board.

--
We provide FREE website monitoring and load testing
http://browsermob.com

Patrick Lightbody
BrowserMob
(w) +1 (503) 828-9003 x 101
(m) +1 (415) 830-5488

On Oct 25, 2010, at 11:26 AM, Adrian Yee wrote:

[Patrick Lightbody]

Josh,

That is correct, and is one of the primary reasons we started this mailing list. Selenium is a fantastic browser automation tool, but it does a terrible job at capturing performance metrics. You are right: it only really knows when onload events fire or when certain elements appear in the DOM (visible, in the DOM, text present, etc).

Fortunately, other projects exist that can help. For IE, Pat Meenan and friends have built a fantastic tool called PageTest (hosted version available at http://webpagetest.org). It does an excellent job of capturing lots of really nice metrics inside of IE, including domReady, renderStart, onLoad, all the low-level object timings, and even a "filmstrip" showing how the page loaded. And best of all, it's all open source. 

Unfortunately, the biggest challenge right now is that PageTest and Selenium don't play very nice together at the moment. PageTest has its own basic browser automation toolkit, but Pat Meenan has mentioned that he feels it's a bit simplistic compared to what Selenium can do. This mailing list was meant to help get us (Selenium developers and PageTest developers) working together to achieve that. We're not there yet, but we'd like to. Perhaps Pat can chime in with more thoughts.

For Firefox, there is Firebug + the NetExport plugin. It can spit out all the data in the Net panel of Firebug, which is pretty detailed. For an example of how to integrate Selenium 2 (aka WebDriver) check out the browsermob-page-perf project here:

https://github.com/lightbody/browsermob-page-perf

The code is really simply - you don't have to use the project but can borrow the ideas. The key thing is to set up the FirefoxProfile with the proper Firebug and NetExport plugins pre-installed. Then when the browser closes, you need to make sure you grab that HAR file. In this project, we upload it to some basic "HAR server", but you can of course do whatever you want.

For all other browsers (or even Firefox and IE if you want consistency), the browsermob-proxy project might be of interest:

https://github.com/lightbody/browsermob-proxy

It's not 100% there yet (we're going to release some updates to it soon), but the idea is that it is a daemon process similar to Fiddler that is cross platform and can programmatically spin up additional ports (for concurrent browser sessions), capture detailed network traffic, etc. It will *not* be able to capture onload or DOM ready or anything else like that, and of course a proxy does inject some "observer effect", though we've found it's pretty darn accurate.

Hopefully that gives you some ideas to run with, and maybe it'll spawn some conversations on this list :)

Patrick

--

We provide FREE website monitoring and load testing

http://browsermob.com

Patrick Lightbody

BrowserMob

(w) +1 (503) 828-9003 x 101

(m) +1 (415) 830-5488

[Sergey Chernyshev]

Aaron Kulick also contributed YSlow / PageSpeed automation scripts that work with ShowSlow:

http://code.google.com/p/showslow/source/browse/#svn/trunk/automation

They do similar things and can work with NetExport as well.

As far as I know, dynaTrace team uses Selenium to drive their test nodes that send data to ShowSlow.com

I think there is a need for some simple framework around running test tools - independent from the receiving end and the tools themselves. After all most of them use real browsers and others emulate browsers so there is enough commonality among the use cases.

         Sergey


--
Sergey Chernyshev
http://www.sergeychernyshev.com/

http://www.meetup.com/Web-Performance-NY/

http://www.showslow.com/

[Patrick Lightbody]

Yes, I think between IE (WebPageTest & dynaTrace) and Firefox (Firebug
+ NetExport) we're pretty good. However, are there any solid options
for Chrome, Safari, and Opera today? What about mobile browsers?

Patrick

--
On-demand load testing and website monitoring
http://browsermob.com

Patrick Lightbody
BrowserMob
(w) +1 (503) 828-9003 x 101
(m) +1 (415) 830-5488