BrowserMob Proxy - Selenium Firefox Driver - Windows 7 - Firefox not loading the page

[vasanth velayudham]

Hello Everyone,

I am currently using BrowserMob Proxy and Selenium to capture HTTP
requests using JAVA api for one of my project. But I am having issues
in establishing the connectivity between proxy and firefox in my code.
Could you please help me in resolving the issue? I have detailed about
my issue below for your reference and any help that you provide would
be greatly appreciated.

System details: Windows 7, Firefox 8.0, Selenium Java 2.20, BrowserMob
Proxy 2.0 beta 6, Java 1.6.0_32

Issue:

Whenever, I execute below mentioned code - system launches the firefox
browser but does not load the url in it. Firefox simply waits for the
browser to load but the page does not get loaded at all. I tried
several combinations to solve this issue, but none of them worked.
It would be great if you share your knowledge on this issue.

Code:
ProxyServer proxyServer = new ProxyServer(4545);
proxyServer.start();
Proxy proxy = proxyServer.seleniumProxy();

FirefoxProfile profile = new FirefoxProfile();
profile.setProxyPreferences(proxy);

DesiredCapabilities caps = DesiredCapabilities.firefox();
caps.setCapability(FirefoxDriver.PROFILE, profile);

WebDriver driver = new FirefoxDriver(caps);

driver.get("http://www.google.com");

[robin paul]

Please try this

ProxyServer server = new ProxyServer(4398);
server.start();
// get the Selenium proxy object
Proxy proxy = null;
proxy = server.seleniumProxy();
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(capabilities);
driver.get("http://www.google.com");


-robinpaul

[vasanth velayudham]

Thanks Robin.

But when I use your code, my firefox displays below mentioned message:

"The connection was reset

The connection to the server was reset while the page was loading.

The site could be temporarily unavailable or too busy. Try again in
a few
moments.
If you are unable to load any pages, check your computer's network
connection.
If your computer or network is protected by a firewall or proxy,
make sure
that Firefox is permitted to access the Web."

When I attempt to run the code, my windows firewall displayed a
warning and I chose 'Allow Access' option in the warning message.

I am wondering whether do I need to do any configurations in firewall.
Could you share your thoughts pls?

Thanks,
V.Vasanth

[robin paul]

what happens when you choose 'Allow Access'

[Ryan Schaffer]

That is correct. You will need to set the proxy settings on your
version of firefox to go through the proxy. There are a lot of
tutorials out there so just make sure you grab the one for your
version of firefox. If you choose to go this route then you will need
to make sure the firefox profile you are using is the one you save to
go through the proxy.

Alternatively you can just build the profile in the code to have all
the settings you want.

[vasanth velayudham]

@robin - 'Allow Access' popup is being displayed while starting the
proxy (using Java). When I choose 'Allow Access' option, I see below 4
lines of code in eclipse:

INFO 06/07 17:01:57 o.b.p.j.h.HttpServer - Version Jetty/5.1.x
INFO 06/07 17:01:57 o.b.p.j.u.Container - Started HttpContext[/,/]
INFO 06/07 17:01:57 o.b.p.j.h.SocketLis~ - Started SocketListener on
0.0.0.0:8100
INFO 06/07 17:01:57 o.b.p.j.u.Container - Started
org.browsermob.proxy.jetty.jetty.Server@10382a9

@Ryan - Thanks for your inputs. I even tried creating profile in
firefox using Java with below mentioned code - but still the same
issue occurs:

ProxyServer server = new ProxyServer(8100);
server.start();
server.setCaptureHeaders(true);
server.newHar("https://www.gmail.com");
Proxy proxy = server.seleniumProxy();

FirefoxProfile profile = new FirefoxProfile();

profile.setAcceptUntrustedCertificates(true);
profile.setAssumeUntrustedCertificateIssuer(true);
profile.setProxyPreferences(proxy);
capabilities.setCapability(FirefoxDriver.PROFILE,profile);

capabilities.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(capabilities);

driver.get("http://google.com");

Could you share any code snippet that you use to create firefox
profile and configure firefox-browsermobproxy??

Thanks..

[Ryan Schaffer]

Sure thing it just like you missed a few of the preferences. I code
in python but I'm pretty sure java is just replacing the
set_preference with setPreference.

here are a few that are handy for running through the proxy.. most
likely you will just need http and http_port

profile.set_preference("network.proxy.http", "localhost")
profile.set_preference("network.proxy.http_port", 8080)
profile.set_preference("network.proxy.ssl", "localhost")
profile.set_preference("network.proxy.ssl_port", 8080)
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.no_proxies_on", "")

hope that helps

[vasanth velayudham]

Hello All,

Apologies for not following up with this question...I am still facing the issue and I feel that I am missing something..

I discovered the reason for one of my issue is that I connect to internet using usb dongle, and I just found that usb dongle connects using dialup connection and the normal proxy settings would not apply to it.

So, now I switched over to wifi connection but still facing the same issue.

I have listed down the steps that I am following to retrieve the HTTPRequests generated using Selenium:

1. I am running below mentioned code directly in eclipse:

      ProxyServer server = new ProxyServer(8100); 

           server.start(); 

           server.setCaptureHeaders(true); 

           server.newHar("https://www.gmail.com"); 

           DesiredCapabilities capabilities = new DesiredCapabilities();

           Proxy proxy = server.seleniumProxy(); 

           FirefoxProfile profile = new FirefoxProfile(); 

           profile.setAcceptUntrustedCertificates(true); 

           profile.setAssumeUntrustedCertificateIssuer(true);

           profile.setPreference("network.proxy.http", "localhost");

           profile.setPreference("network.proxy.http_port", 8080); 

           profile.setPreference("network.proxy.ssl", "localhost"); 

           profile.setPreference("network.proxy.ssl_port", 8080); 

           profile.setPreference("network.proxy.type", 1); 

           profile.setPreference("network.proxy.no_proxies_on", ""); 

           profile.setProxyPreferences(proxy); 

           capabilities.setCapability(FirefoxDriver.PROFILE,profile); 

           capabilities.setCapability(CapabilityType.PROXY, proxy); 

           WebDriver driver = new FirefoxDriver(capabilities); 

           driver.get("http://www.google.com"); 

Please note that before running the code that I am NOT manually starting the browsermob-proxy.bat file. I hope this is not required as code itself takes care about it. I am using Firefox 8 browser and Selenium 2.20.

Request you to help me in solving this issue...

Thanks,

V.Vasanth

[Ryan Schaffer]

Almost there!

When you do:

ProxyServer server = new ProxyServer(8100); 

you are creating a proxy server on port 8100

but you are setting your firefox profile to comunicate with the proxy on 8080

match those 2 numbers up and you should have it.

[Krishnan Mahadevan]

Ryan,

Wouldnt he still be required to get a proxy server from the server?

I remember having to do a post via CURL command which would give me port number which was what I was to be using and not the port number on which the BMP server was started when I was working with it on standalone mode. i think the BMP also has a method which returns the Proxy instance which we would be embedding into the desiredcapabilities although I forgot the actual method name. 

--
Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

[VVasanth Kumar]

Hi Ryan,

I tried with your update as well, but the same result...

I believe, I am missing something here...in manual process, we start the server Browsermob-proxy.bat manually and set the proxy in the required port using cURL command...

Whereas, in the automated process with Selenium - we are just starting the server using one of the port, but are we using any code equivalent of cURL command...

Could you please help me in overcoming this issue? If you have any videos available that would help me a lot...

Thanks,

V.Vasanth

[VVasanth Kumar]

Hello everyone,

Atlast, I found the causes of my issue...First one is, I tried proxy using the internet from USB dongle...as it would connect using dialup connection, my proxy did not work..

Second issue is that I did not use the browsermob proxy dependencies properly...

Once I fixed both these issues, proxy worked for me...Thanks for your help!!!

[sahil]

Hello everyone,

Can you please explain what setting you change.Because am also getting the same blank page and error after that.

WebDriver driver = new FirefoxDriver(capabilities);

//

Exception in thread "main" java.lang.NoSuchMethodError: org.openqa.selenium.logging.LogEntry.<init>(Ljava/util/logging/Level;JLjava/lang/String;)V

at org.openqa.selenium.logging.profiler.ProfilerLogEntry.<init>(ProfilerLogEntry.java:25)

at org.openqa.selenium.logging.profiler.HttpProfilerLogEntry.<init>(HttpProfilerLogEntry.java:28)

at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:296)

at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.execute(NewProfileExtensionConnection.java:136)

at org.openqa.selenium.firefox.FirefoxDriver$LazyCommandExecutor.execute(FirefoxDriver.java:357)

at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:472)

at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:182)

at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:110)

at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:185)

at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:178)

at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:101)

at Testclass.main

[Michael]

I am getting the same error as Sahil:

java.lang.NoSuchMethodError: org.openqa.selenium.logging.LogEntry.(Ljava/util/logging/Level;JLjava/lang/String;)V
    at org.openqa.selenium.logging.profiler.ProfilerLogEntry.(ProfilerLogEntry.java:25)
    at org.openqa.selenium.logging.profiler.HttpProfilerLogEntry.(HttpProfilerLogEntry.java:28)

    at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:296)
    at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.execute(NewProfileExtensionConnection.java:136)
    at org.openqa.selenium.firefox.FirefoxDriver$LazyCommandExecutor.execute(FirefoxDriver.java:357)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:472)
    at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:182)

    at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.java:110)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:185)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:178)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:101)

All I did was remove my IVY dependency for Selenium 2.20:

<dependency org="org.thoughtworks" name="selenium-server-standalone" rev="2.20.0" conf="runtime->default" />

and replaced it with one for Selenium 2.25:

<dependency org="org.seleniumhq.selenium" name="selenium-server-standalone" rev="2.25.0" conf="runtime->default">
<artifact name="selenium-server-standalone" ext="jar"/>
</dependency>

And now FireFox comes up blank and I get that stacktrace.

In eclipse all works fine but when I try to run a test on a test machine it fails.

[Michael]

I just found what was messing up the works for me. The following jar was being added to my projects:

org.seleniumhq.selenium.selenium-api-2.20.0.jar

Once I got rid of this jar my test ran as expected with the "selenium-server-standalone-2.25.0.jar".

I haven't figured out why this "2.20 api jar" gets included (I don't have a <dependency/> entry requesting it) but at least now I know what I need to fix.