How to capture Browser Network traffic using Webdriver??

[Aditya Bhardwaj]

Like in Selenium RC there is a function "capture network traffic" which can capture all browser network traffic, Is there anything similar available in webdriver??

Thanks!!

[Krishnan Mahadevan]

No. There is none. You would need to use a ProxyServer such as BrowserMobProxy for doing this. 

- Krishnan M

iSent.iPad.

On 23-Mar-2015, at 23:38, Aditya Bhardwaj <eraditya...@gmail.com> wrote:

Like in Selenium RC there is a function "capture network traffic" which can capture all browser network traffic, Is there anything similar available in webdriver??

Thanks!!

--
You received this message because you are subscribed to the Google Groups "webdriver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to webdriver+...@googlegroups.com.
To post to this group, send email to webd...@googlegroups.com.
Visit this group at http://groups.google.com/group/webdriver.
For more options, visit https://groups.google.com/d/optout.

[Ravi Mundhe]

Hi,

The latest version of Selenium Webdriver does not really support traffic capture. You can however, use BrowserMob proxy to capture traffic. https://github.com/lightbody/browsermob-proxy . The README has examples on how to do that with Selenium.

Also you can try with Java .net package will provide some methods traffic related ....

Thanks,

-Ravi

--

Thanks & Regards

Ravi S. Mundhe

[Aditya Bhardwaj]

Thanks for the reference !!

[Aditya Bhardwaj]

Thanks for the Reference.

I am able to set up the Browsermob Proxy and able to generate the HAR file of the browser network tarffic.

But now i want  to get one specific call from the captured network calls for that i have to read or parse through the generated HAR file. I did some R&D for reading the HAR and fetching out the specific call but couldn't found the feasible solution, can you please provide some pointers for this as well??

Thank In Advance!!

P.S: I want to get the Web Analytics Omniture call from the network traffic.

[sunny sachdeva]

HI Aditya,

I am also using BMP for getting Omni Tags. I believe you are also looking for the same.

As far as reading HAR is concerned, I am using HARREADER. Its maven dependency is

<dependency>

<groupId>de.sstoehr</groupId>

<artifactId>har-reader</artifactId>

<version>1.0.5</version>

</dependency>

Or you can read about about it frm here

https://github.com/sdstoehr/har-reader

Thanks

Sunny

On Thursday, April 23, 2015 at 10:43:14 AM UTC+5:30, Aditya Bhardwaj wrote:

Thanks for the Reference.

I am able to set up the Browsermob Proxy and able to generate the HAR file of the browser network tarffic.

But now i want  to get one specific call from the captured network calls for that i have to read or parse througTo ah the generated HAR file. I did some R&D for reading the HAR and fetching out the specific call but couldn't found the feasible solution, can you please provide some pointers for this as well??

[Aditya Bhardwaj]

Hi Sunny,

Thanks for your response.

Yes, I am looking for Omni Tags. The solution provided by you does help. Also, I want to share one more possible solution i.e. using HarLib or Jackson Libraries that can read and parse the HAR files.

However, I am able to read the HAR files and fetch the calls from it but still facing one challenge i.e. Clearing out the data from HAR file, As i need to navigate through different web pages of the application so for every page load i would want to have a fresh HAR file or Override the existing file(Currently, HAR file is being appended with new network calls when new pages are loaded).This is required as the size of HAR file would increase with every page load which in turn impacts the performance while parsing the HAR file.

Thanks!!

[sunny sachdeva]

Thanks Aditya for sharing the other options for reading HAR.

To answer your query on overriding or creating new one, I guess it would depend on the application.

In my case , I need to get Tags for each and every button that exists o webpage. More over it was difficult to differentiate between the calls made to server, I mean ofcourse calls made to omni server would be save but property tag values would be different hence I start the proxy once but create different har files for each scenario.

Let me know if this helps out.

Thanks

Sunny Sachdeva

[Aditya Bhardwaj]

Yes Exactly, This serves the business i have also created different har files for different scenario. That meas one har file would generate per scenario.

Thanks!!

[sunny sachdeva]

Aditya,

from where you download HarLib binaries I cant get what to download from below link

https://sourceforge.net/projects/benchlab/files/HarLib/

[Suraj Ray]

Hey,

Need help on below item.

I tried setting up the BMP but failing when I execute Java main method with below exception

Caused by: java.lang.ClassNotFoundException: net.lightbody.bmp.BrowserMobProxy

at java.net.URLClassLoader$1.run(Unknown Source)

I am able to compile the java code properly and I can see the jars in the maven dependency folder but as soon as I try to run the program, encountering the above eexception.

I only have one entry to my POM

<dependency>

        <groupId>net.lightbody.bmp</groupId>

        <artifactId>browsermob-core-littleproxy</artifactId>

        <version>2.1.0-beta-1-SNAPSHOT</version>

        <scope>test</scope>

    </dependency>

And the code m trying to run is as below

public static void main(String[] args) throws Exception {

// TODO Auto-generated method stub

BrowserMobProxy server = new BrowserMobProxyServer();

  

        server.start(0);

   int port = server.getPort();

// get the Selenium proxy object

   Proxy proxy = ClientUtil.createSeleniumProxy(server);

   // configure it as a desired capability

   DesiredCapabilities capabilities = new DesiredCapabilities();

   capabilities.setCapability(CapabilityType.PROXY, proxy);

           

           // start the browser up

   WebDriver driver = new FirefoxDriver(capabilities);

   // create a new HAR with the label "yahoo.com"

   server.newHar("yahoo.com");

   // open yahoo.com

   driver.get("http://yahoo.com");

   // get the HAR data

   Har har = server.getHar();

        

}

[sunny sachdeva]

Suraj,

Dont use BMP beta version. You can try the below one. 

<dependency>

<groupId>net.lightbody.bmp</groupId>

<artifactId>browsermob-proxy</artifactId>

<version>2.0.0</version>

</dependency>

[Suraj Ray]

Hi Sunny,

Any reason for not using Beta version?? or just because it is not stable/final yet.

regards,

suraj

[Suraj Ray]

I can run now the ProxyServer to capture and get a HAR file but my HAR doesn't contain any data, any idea??

Regards,

Suraj

[sunny sachdeva]

Without seeing the code noone can help you out.

[Suraj Ray]

Code same as in my earlier mail....

ProxyServer server = new ProxyServer(8107);

        server.start();

        server.setCaptureHeaders(true);

        server.setCaptureContent(true);

        server.newHar("gooogle home page");

        DesiredCapabilities capabilities = new DesiredCapabilities();

capabilities.setCapability(CapabilityType.PROXY, proxy);

WebDriver driver = new FirefoxDriver(capabilities);

        driver.get("http://www.google.com");

        driver.findElement(By.id("lst-ib")).sendKeys("hello");

        driver.findElement(By.name("btnG")).click();

        Har har1 = server.getHar();

        System.out.println("HAR content is:   "+ har1);

        

        har1.writeTo(new File("har.json"));

        server.stop();

        driver.quit();

and the HAR I got is as below, is it correct??

log	Object
version	1.2
creator	Object
name	BrowserMob Proxy
version	2.0
comment
pages	Array [1]
0	Object
id	gooogle home page
startedDateTime	2015-04-30T16:46:32.402+05:30
title
pageTimings	Object
comment
comment
entries	Array [0]
comment

[Krishnan Mahadevan]

Suraj

I would suggest that you start posting BMP related queries on the browsermob proxy google forums than here.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

[Aditya Bhardwaj]

Hi Sunny,

You can download the HarLib from the below link.

https://code.google.com/p/zaproxy/source/browse/trunk/lib/harlib-jackson-1.1.1.jar?r=3946

Thanks,

Aditya

[suman kumar]

we are generating HAR  file with firefox + firebug+net export

               setup method

                -------------

                DesiredCapabilities capabilities = new DesiredCapabilities();

FirefoxProfile profile = new FirefoxProfile();

profile.addExtension(new File("D:\\suman\\firebug-2.0.8-fx.xpi"));

profile.addExtension(new File("D:\\suman\\netExport-0.9b6.xpi"));

profile.setEnableNativeEvents(true);

profile.setPreference("extensions.firebug.netexport.pageLoadedTimeout", 0);

profile.setPreference("extensions.firebug.currentVersion", "3.0.0");

profile.setPreference("extensions.firebug.defaultPanelName", "net");

profile.setPreference("extensions.firebug.net.enableSites", true);

profile.setPreference("extensions.firebug.allPagesActivation", "on");

profile.setPreference("extensions.firebug.showFirstRunPage", false);

profile.setPreference("extensions.firebug.netexport.defaultLogDir", "D:\\sumanHAR FIle");

profile.setPreference("extensions.firebug.netexport.saveFiles", true);

profile.setPreference("extensions.firebug.netexport.alwaysEnableAutoExport", true);

profile.setPreference("extensions.firebug.netexport.includeResponseBodies", false);

profile.setPreference("extensions.firebug.netexport.exportFromBFCache", true);

profile.setPreference("extensions.firebug.netexport.secretToken", "abcd");

profile.setPreference("extensions.firebug.netexport.showPreview", false);

profile.setPreference("extensions.firebug.net.defaultPersist", true);

profile.setPreference("extensions.firebug.console.defaultPersist", true);

capabilities.setCapability(FirefoxDriver.PROFILE, profile);

driver = new FirefoxDriver(capabilities);

driver.manage().window().maximize();

driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);

At end of test method

---------------------

JavascriptExecutor js= (JavascriptExecutor)driver;

js.executeScript("window.NetExport.triggerExport('abcd')");

[Ravi Teja Mygapula]

Hi,

I set up the Browsermob Proxy in windows 7. While trying to run sample script am getting an error as below. Googled and tried all possible ways but didn't get any proper solution

D:\E_Scripts\selenium>ruby bandwidth_limit.rb

C:/Ruby23-x64/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in

 `require': cannot load such file -- browsermob/proxy (LoadError)

        from C:/Ruby23-x64/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_req

uire.rb:55:in `require'

        from bandwidth_limit.rb:3:in `<main>'