Webdriver API that captures http traffic

15,222 views
Skip to first unread message

Dreamer

unread,
Feb 7, 2012, 12:03:12 PM2/7/12
to webdriver
All,

I cant seem to capture http request and responses from a webdriver
session. I know you can do this in selenium 1

Selenium s = new DefaultSelenium(...);
s.start("captureNetworkTraffic=true");
s.open("http://www.google.com");
String traffic = s.captureNetworkTraffic("json");

What I'm looking for is a webdriver equivalent. I have had a look at
the proxy class but all that does is emulate/configure proxy, it does
not capture http traffic


Thanks

David

unread,
Feb 7, 2012, 3:25:59 PM2/7/12
to webdriver
Selenium RC had a captureNetworkTraffic option on startup of browser
session to enable such functionality, though the user of course had to
manually parse the desired data (or look for third party Selenium
tools that did that).

WebDriver unfortunately doesn't natively support this, but you can do
similar with third party tools combined with WebDriver.

Third party solutions to desired feature:
http://deanhume.com/Home/BlogPost/measuring-web-page-performance-with-selenium-2-and-the-web-timings-api/56
http://www.theautomatedtester.co.uk/blog/2010/selenium-webtimings-api.html
http://sevenseconds.wordpress.com/2011/06/28/velocity-conference-2011-workshop-on-automated-web-performance-testing/
http://www.supermind.org/blog/968/howto-collect-webdriver-http-request-and-response-headers

Frankly, it would be nice if WebDriver did incorporate said feature
same as Selenium RC.

Mark Collin

unread,
Feb 7, 2012, 4:03:57 PM2/7/12
to webd...@googlegroups.com
http://sqa.stackexchange.com/questions/1276/is-there-a-capturenetworktraffic
-implementation-in-selenium-2-via-webdriver

All,


Thanks

--
You received this message because you are subscribed to the Google Groups
"webdriver" group.
To post to this group, send email to webd...@googlegroups.com.
To unsubscribe from this group, send email to
webdriver+...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/webdriver?hl=en.


--
This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

If you have received this email in error please notify postm...@ardescosolutions.com

David

unread,
Feb 7, 2012, 8:41:15 PM2/7/12
to webdriver
This just got me thinking, maybe not officially incorporated with
WebDriver but could be done as third party work.

If you think of combining WebDriver Firefox extension (used to drive
FirefoxDriver behind the scenes) with some other Firefox extension
that offers HTTP traffic monitoring/analysis, and then expose an API,
you get the requested functionality.

I'm thinking combine WebDriver extension with something like parts of
Firebug (Network tab), liveHttpHeaders, HttpFox, etc.

I made similar suggestion for a related topic on debugger for
JSONWireProtocol (why not incorporate as part of WebDriver Firefox
extension).

http://code.google.com/p/selenium/issues/detail?id=861

On Feb 7, 1:03 pm, "Mark Collin" <m...@ardescosolutions.com> wrote:
> http://sqa.stackexchange.com/questions/1276/is-there-a-capturenetwork...
> -implementation-in-selenium-2-via-webdriver
>
>
>
>
>
>
>
> -----Original Message-----
> From: webd...@googlegroups.com [mailto:webd...@googlegroups.com] On
>
> Behalf Of Dreamer
> Sent: 07 February 2012 17:03
> To: webdriver
> Subject: [webdriver] Webdriver API that captures http traffic
>
> All,
>
> I cant seem to capture http request and responses from a webdriver session.
> I know you can do this in selenium 1
>
> Selenium s = new DefaultSelenium(...);
> s.start("captureNetworkTraffic=true");
> s.open("http://www.google.com");
> String traffic = s.captureNetworkTraffic("json");
>
> What I'm looking for is a webdriver equivalent. I have had a look at the
> proxy class but all that does is emulate/configure  proxy, it does not
> capture http traffic
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "webdriver" group.
> To post to this group, send email to webd...@googlegroups.com.
> To unsubscribe from this group, send email to
> webdriver+...@googlegroups.com.
> For more options, visit this group athttp://groups.google.com/group/webdriver?hl=en.
>
> --
> This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
>
> If you have received this email in error please notify postmas...@ardescosolutions.com

Mark Collin

unread,
Feb 8, 2012, 1:35:19 AM2/8/12
to webd...@googlegroups.com

I guess the reasons for not doing it are:

 

·         There is no need to do anything, you can already capture network traffic using a proxy.

·         It would not be cross browser and making it cross browser would be a ball ache.

·         Selenium is a browser automation framework, traffic monitoring is not part of its remit.

·         Most people don't want to do traffic monitoring and this won't be used by the vast majority.

For more options, visit this group at http://groups.google.com/group/webdriver?hl=en.

 


-- This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify postm...@ardescosolutions.com

Andre Gironda

unread,
Feb 8, 2012, 1:58:25 AM2/8/12
to webd...@googlegroups.com
On Tue, Feb 7, 2012 at 6:41 PM, David <mang...@gmail.com> wrote:
> This just got me thinking, maybe not officially incorporated with
> WebDriver but could be done as third party work.
>
> If you think of combining WebDriver Firefox extension (used to drive
> FirefoxDriver behind the scenes) with some other Firefox extension
> that offers HTTP traffic monitoring/analysis, and then expose an API,
> you get the requested functionality.

The Tamper Data Firefox extension allows a user to save as XML. There
is an example Python based TDParser for this XML here --
http://code.google.com/p/randomizedcode/

Cheers,
Andre

FireGuy

unread,
Feb 8, 2012, 9:29:34 AM2/8/12
to webdriver
I've used FiddlerCore for this, it slows things down quite a bit
(almost half the speed with multiple instances working against one
proxy) but for me, the ability to modify requests is worth it.

The only solution that will not be significantly slower, is monitoring
the connections from within the browser, but then, like people said,
you need an implementation for each one.

David

unread,
Feb 8, 2012, 4:44:48 PM2/8/12
to webdriver
On Mark's last comment, understandable why the feature isn't there in
WebDriver. But out of curiosity, was wondering why the traffic option
was added to Selenium RC. Why not just let people use proxies and
third party tools to do the job as they do now with WebDriver. Or did
Selenium RC function differently (than WebDriver in terms of proxies)
requiring that, or was there some other significant reason to add it
back then, or one of the developers just happen to feel like adding
it?

Would be nice to know this background. Otherwise, seems ironic that
feature is in Selenium RC but not WebDriver (unless it is left out due
to not being able to implement same interface/feature due to
architectural difference between the two).

On Feb 7, 10:35 pm, "Mark Collin" <m...@ardescosolutions.com> wrote:
> I guess the reasons for not doing it are:
>
> .         There is no need to do anything, you can already capture network
> traffic using a proxy.
>
> .         It would not be cross browser and making it cross browser would be
> a ball ache.
>
> .         Selenium is a browser automation framework, traffic monitoring is
> not part of its remit.
>
> .         Most people don't want to do traffic monitoring and this won't be
> used by the vast majority.
>
>
>
>
>
>
>
> -----Original Message-----
> From: webd...@googlegroups.com [mailto:webd...@googlegroups.com] On
>
> Behalf Of David
> Sent: 08 February 2012 01:41
> To: webdriver
> Subject: [webdriver] Re: Webdriver API that captures http traffic
>
> This just got me thinking, maybe not officially incorporated with WebDriver
> but could be done as third party work.
>
> If you think of combining WebDriver Firefox extension (used to drive
> FirefoxDriver behind the scenes) with some other Firefox extension that
> offers HTTP traffic monitoring/analysis, and then expose an API, you get the
> requested functionality.
>
> I'm thinking combine WebDriver extension with something like parts of
> Firebug (Network tab), liveHttpHeaders, HttpFox, etc.
>
> I made similar suggestion for a related topic on debugger for
> JSONWireProtocol (why not incorporate as part of WebDriver Firefox
> extension).
>
>  <http://code.google.com/p/selenium/issues/detail?id=861>http://code.google.com/p/selenium/issues/detail?id=861
>
> On Feb 7, 1:03 pm, "Mark Collin" < <mailto:m...@ardescosolutions.com>
> m...@ardescosolutions.com> wrote:
>
> >  <http://sqa.stackexchange.com/questions/1276/is-there-a-capturenetwork>http://sqa.stackexchange.com/questions/1276/is-there-a-capturenetwork...
>
> > -implementation-in-selenium-2-via-webdriver
>
> > -----Original Message-----
>
> > From:  <mailto:webd...@googlegroups.com> webd...@googlegroups.com
> <mailto:[mailto:webd...@googlegroups.com]>
> [mailto:webd...@googlegroups.com]
>
> > On
>
> > Behalf Of Dreamer
>
> > Sent: 07 February 2012 17:03
>
> > To: webdriver
>
> > Subject: [webdriver] Webdriver API that captures http traffic
>
> > All,
>
> > I cant seem to capture http request and responses from a webdriver
> session.
>
> > I know you can do this in selenium 1
>
> > Selenium s = new DefaultSelenium(...);
>
> > s.start("captureNetworkTraffic=true");
>
> > s.open(" <http://www.google.com>http://www.google.com");
>
> > String traffic = s.captureNetworkTraffic("json");
>
> > What I'm looking for is a webdriver equivalent. I have had a look at
>
> > the proxy class but all that does is emulate/configure  proxy, it does
>
> > not capture http traffic
>
> > Thanks
>
> > --
>
> > You received this message because you are subscribed to the Google
>
> > Groups "webdriver" group.
>
> > To post to this group, send email to  <mailto:webd...@googlegroups.com>
> webd...@googlegroups.com.
>
> > To unsubscribe from this group, send email to
>
> >  <mailto:webdriver+...@googlegroups.com>
> webdriver+...@googlegroups.com.
>
> > For more options, visit this group
> athttp://groups.google.com/group/webdriver?hl=en.
>
> > --
>
> > This message contains confidential information and is intended only for
> the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and delete
> this e-mail from your system. If you are not the intended recipient you are
> notified that disclosing, copying, distributing or taking any action in
> reliance on the contents of this information is strictly prohibited.
>
> > If you have received this email in error please notify
>
> >  <mailto:postmas...@ardescosolutions.com> postmas...@ardescosolutions.com
>
> --
>
> You received this message because you are subscribed to the Google Groups
> "webdriver" group.
>
> To post to this group, send email to  <mailto:webd...@googlegroups.com>
> webd...@googlegroups.com.
>
> To unsubscribe from this group, send email to
> <mailto:webdriver+...@googlegroups.com>
> webdriver+...@googlegroups.com.
>
> For more options, visit this group at
> <http://groups.google.com/group/webdriver?hl=en>http://groups.google.com/group/webdriver?hl=en.
>
> --
> This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
>
> If you have received this email in error please notify postmas...@ardescosolutions.com

David

unread,
Feb 8, 2012, 4:45:46 PM2/8/12
to webdriver
Andre, thanks for the info on Tamper Data.

On Feb 7, 10:58 pm, Andre Gironda <and...@gmail.com> wrote:

David

unread,
Feb 8, 2012, 4:55:02 PM2/8/12
to webdriver
Also, regarding Tamper Data, is that a GUI based manual use FF
extension, or is there some API/interface to use it with WebDriver? I
mean do you have to manually save/export the XML and then process it
or you can do that within code together with WebDriver?

Just asking because I made something similar for the "manual" case
using liveHttpHeaders and plain text parsing using Perl (and later
VBScript), quite some time back. It wasn't meant for Selenium/
WebDriver, but could also be used for it. There's also a matching
solution for IE via ieHttpHeaders plugin and my parser.

http://www.codeproject.com/Articles/15128/Web-HTTP-Automation-with-Perl

http://www.codeproject.com/Articles/19600/Web-HTTP-Request-and-Automation-Libraries

Andy Tinkham

unread,
Feb 8, 2012, 6:27:43 PM2/8/12
to webd...@googlegroups.com
As another point, you can do this already with Firebug like you mentioned - Firebug combined with the NetExport plugin allows our tests to automatically save off HTTP Archive files (which when viewed with a HAR viewer look just like the Net tab in Firebug). Basically, we create a Firefox Profile object before we instantiate the driver, add the two xpis to it, and configure a bunch of options for the plugins (turning on auto export and pointing it at a directory, mostly) and it mostly works well (it currently misses some of our AJAX calls - I think it triggers the export when the page loaded even is fired).

In Ruby, that looks like this:
#Add Firebug to Firefox for debugging purposes
profile = Selenium::WebDriver::Firefox::Profile.new
profile.add_extension(File.join(File.dirname(__FILE__), "firebug-1.9.0-fx.xpi"))
profile['extensions.firebug.currentVersion'] = "1.9.0"
profile['extensions.firebug.addonBarOpened'] = true
profile['extensions.firebug.console.enableSites'] = true
profile['extensions.firebug.allPagesActivation'] = "on"
profile['extensions.firebug.previousPlacement'] = 1
profile['extensions.firebug.onByDefault'] = true
profile['extensions.firebug.defaultPanelName'] = "console"
profile['extensions.firebug.script.enableSites'] = true
profile['extensions.firebug.net.enableSites'] = true
profile['extensions.firebug.console.logLimit'] = 50000
profile['extensions.firebug.openInWindow'] = true

# Add the NetExport Firebug plugin so that we get HAR files showing the performance
profile.add_extension(File.join(File.dirname(__FILE__), "netExport-0.8b20.xpi"))
profile['extensions.firebug.netexport.alwaysEnableAutoExport'] = true
profile['extensions.firebug.netexport.autoExportToFile'] = true
profile['extensions.firebug.netexport.autoExportToServer'] = false
profile['extensions.firebug.netexport.compress'] = true
Dir.chdir(output_dir) { Dir.mkdir "NetExportLogs" unless Dir.exists? "NetExportLogs" }
profile['extensions.firebug.netexport.defaultLogDir'] = File.join(output_dir, 'NetExportLogs').gsub("/", '\\')
Selenium::WebDriver.for @browser, :profile => profile

(If you try to do this, note the gsub in the next to last line. NetExport seems to be unhappy with forward slashes in the default log directory, but is fine with backslashes. Some of the Firebug preferences probably aren't needed, but I like having Firebug open as the test runs so I can see errors pop up on the console too if I'm watching.)

We'll probably eventually switch to the BrowserMob proxy in the hopes of getting stats for more browsers than just Firefox, but today is not that day.

Once we get the HTTP Archive files, I've built a script using Jari Bakken's har gem to parse everything out to a database that we can then query across runs. You could use a standard HAR viewer like that that comes with Jari's gem or the one Jan Odvarko has at softwareishard.com (http://www.softwareishard.com/blog/har-viewer/)

-----Original Message-----
From: webd...@googlegroups.com [mailto:webd...@googlegroups.com] On Behalf Of David
Sent: Wednesday, February 08, 2012 3:46 PM
To: webdriver

--

Mark Collin

unread,
Feb 9, 2012, 1:23:36 AM2/9/12
to webd...@googlegroups.com
It was never added properly to RC.

It was only ever available through FireFox and was at best incomplete, I
would regard the RC implementation as a spike that was never removed.

To post to this group, send email to webd...@googlegroups.com.


To unsubscribe from this group, send email to

webdriver+...@googlegroups.com.
For more options, visit this group at

http://groups.google.com/group/webdriver?hl=en.


--
This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

If you have received this email in error please notify postm...@ardescosolutions.com

Robson Peixoto

unread,
May 8, 2014, 8:10:19 PM5/8/14
to webd...@googlegroups.com
New about how to capture the Request using the Selenium WebDriver?

Thanks
Reply all
Reply to author
Forward
0 new messages