problem using proxy with selenium but not standalone

[Graham]

all,

I'm having a funny issue.

I've taken the latest code from github and compiled it locally.

I try to create and access a proxy via selenium like the example
provided:
ProxyServer server = new ProxyServer(4444);
server.start();

// get the Selenium proxy object
Proxy proxy = server.seleniumProxy();

// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);

When I then navigate to a url with my WebDriver it takes forever and
all the images etc are missing from the page.

In and effort to debug I then ran the binary version, created a new
proxy via the REST api and then set this in my browser. This worked
perfectly, page loaded reasonably quickly and all the assets were
there. I did see some errors in relation to google analytics and
facebook but this will be due to the fact we have a corporate firewall
and I'd need to set an upstream proxy and I wasn't sure how to do that
via REST.

any ideas?

[Patrick Lightbody]

Does the URL that you're hitting contain any SSL? Wondering if maybe the way you're launching it in you IDE might not be pulling in the resources from src/main/resources, which contains the sslSupport directory and other files necessary to operate. What IDE are you using?

--

Patrick Lightbody

+1 (415) 830-5488

[Graham Abell]

We're using intellij. I've compiled the source locally and then added the jar into our nexus repo which is where my project is pulling it from. 

There more than likely is some ssl on the page, is there a way I can tell whether the resources are loading?

[Patrick Lightbody]

Graham,

I'm not quite clear what your setup is. You're saying that you built the project from source and put it in to your own nexus repo. Did you include the pom as well, so that when you pull it back in all the dependencies are also pulled in? Also, we publish directly to the central maven repo, so any reasons you can't just that instead?

Patrick

--

Patrick Lightbody

+1 (415) 830-5488

[Graham]

Hi Patrick,

I didn't see the latest beta-6 in the public repo when I originally looked, I can see it there now so I've reverted to that version in the public repo. 

I've rerun my test and I'm still seeing very slow load times [~3 minutes] though images are being displayed now. The page loads in about 3 seconds if I go direct without using BrowserMob.

Any thoughts?

thanks,

Graham

[Patrick Lightbody]

Graham,

Is that still when only running it in your IDE? Does the page load fast when going through BMP running in the standard binary distribution?

Patrick

--

Patrick Lightbody

+1 (415) 830-5488

[Graham Abell]

Hi Patrick,

Yes, slow via the IDE while using BMP programmatically. It runs fine via the binary implementation.

graham

[Patrick Lightbody]

OK, I need the URL you're trying to hit and also if you can share a stripped down version of the IDE project (sans all your proprietary stuff, hopefully you can still recreate the issue).

--

Patrick Lightbody

+1 (415) 830-5488

[Graham Abell]

The IDE project shouldn't be a problem, the URL is internal but I'll verify the same thing happens with our production site in the  morning which is www.paddypower.com

[Patrick Lightbody]

Thanks let me know.

--

Patrick Lightbody

+1 (415) 830-5488

[Graham Abell]

Hi Patrick, 

I see the same issue with our production site. 

The demo test is:

import com.paddypower.reporter.ReporterInterface;

import com.paddypower.reporter.SeleniumReporterFactory;

import org.browsermob.core.har.Har;

import org.browsermob.proxy.ProxyServer;

import org.openqa.selenium.remote.CapabilityType;

import org.openqa.selenium.remote.DesiredCapabilities;

import org.openqa.selenium.remote.RemoteWebDriver;

import org.openqa.selenium.server.RemoteControlConfiguration;

import org.openqa.selenium.server.SeleniumServer;

import org.testng.annotations.Test;

import java.io.File;

import java.io.IOException;

import java.net.URL;

import java.util.ArrayList;

import java.util.HashMap;

import java.util.Map;

/**

 * User: gabell

 * Date: 27/03/12

 * Time: 10:18

 */

public class demo {

    @Test(priority = 1, groups = {"demo"},

            description = "demo test")

    public void test1() throws IOException {

        //System.setProperty("webdriver.chrome.driver", "library\\Chromedriver\\chromedriver.exe");

        DesiredCapabilities capabilities = DesiredCapabilities.chrome();

        ArrayList<String> switches = new ArrayList<>();

        switches.add("--start-maximized");

        switches.add("--disable-application-cache");

        switches.add("--disk-cache-size=1");

        switches.add("--media-cache-size=1");

        switches.add("--disk-cache-dir=c:\\cache");

        switches.add("enable-file-cookies");

        switches.add("--disable-popup-blocking");

        capabilities.setCapability("chrome.switches", switches);

        ProxyServer proxyServer = new ProxyServer(4443);

        try {

            proxyServer.start();

            Map options = new HashMap<String, String>();

            options.put("httpProxy", "ppwebaccess1:8080");

            proxyServer.setOptions(options);

        } catch (Exception e) {

            e.printStackTrace();

        }

        capabilities.setCapability(CapabilityType.PROXY, proxyServer.seleniumProxy());

        RemoteWebDriver driver = new RemoteWebDriver(new URL(startStandaloneServer()), capabilities);

        proxyServer.newHar("homepage");

        driver.get("http://www.paddypower.com/bet");

        Har har = proxyServer.getHar();

        har.writeTo(new File("c:\\test1.har"));

        ReporterInterface reporter = SeleniumReporterFactory.getSeleniumReporter();

        reporter.log("");

    }

    private static String startStandaloneServer() {

        //String hubURL = "http://localhost:8080/wd/hub";  //Android

        String hubURL = "http://127.0.0.1:4444/wd/hub";

        //Start the server

        RemoteControlConfiguration rc = new RemoteControlConfiguration();

        rc.setPort(4444);

        rc.setTimeoutInSeconds(20);

        try {

            SeleniumServer seleniumServer = new SeleniumServer(true, rc);

            seleniumServer.boot();

            seleniumServer.start();

            seleniumServer.getPort();

        } catch (Exception e) {

            e.printStackTrace();

        }

        return hubURL;

    }

}

This is the output I see in the console:

[TestNG] Running:

  C:\Documents and Settings\gabell\.IntelliJIdea11\system\temp-testng-customsuite.xml

log4j:WARN No appenders could be found for logger (org.browsermob.proxy.jetty.util.Container).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

INFO 03/27 09:27:43 o.o.s.r.s.DriverSer~ - Executing: [new session: {platform=ANY, browserName=chrome, proxy={httpProxy=LP-IT-HC2-1210.inh..., version=, chrome.switches=[--start-maximized, --disable...}] at URL: /session)

Started ChromeDriver

port=33095

version=18.0.1022.0

log=C:\Java\workspace\Automation\chromedriver.log

INFO 03/27 09:27:44 o.o.s.r.s.DriverSer~ - Done: /session

INFO 03/27 09:27:44 o.o.s.r.s.DriverSer~ - Executing: org.openqa.selenium.remote.server.handler.GetSessionCapabilities@19c98d3 at URL: /session/1332840462378)

INFO 03/27 09:27:44 o.o.s.r.s.DriverSer~ - Done: /session/1332840462378

INFO 03/27 09:27:44 o.o.s.r.s.DriverSer~ - Executing: [get: http://www.paddypower.com/bet] at URL: /session/1332840462378/url)

INFO 03/27 09:31:17 o.o.s.r.s.DriverSer~ - Done: /session/1332840462378/url

216069 [main] INFO  com.paddypower.reporter.Reporter  - 

===============================================

Custom suite

Total tests run: 1, Failures: 0, Skips: 0

===============================================

Process finished with exit code 0

I've attached the resulting Har though I'm not sure you'll be able to access it. 

[Graham]

Hi Patrick,

Did you have a chance to investigate this?

thanks,

Graham

[Patrick Lightbody]

Sorry, been swamped. I have about 20 emails to reply to for BMP - hoping to get to it tonight!

--

Patrick Lightbody

+1 (415) 830-5488

[Graham Abell]

that's great - thanks Patrick

[Patrick Lightbody]

Graham,

I haven't made any further progress on this - the IDE-specific issue is really tripping me up. Maybe you can send the entire project file? Not really sure where to go from here...

Patrick

--

Patrick Lightbody

+1 (415) 830-5488

<test1.har>

[Graham]

Hi Patrick, 

I've attached a project including the code which runs for me here - hopefully it will do the same for you.

thanks,

Graham

[Patrick Lightbody]

Graham,

I was able to get your project code to run fine, with a few tweaks:

1) I opened the project in IntelliJ IDEA 11 by importing the pom.xml file

2) demo.java was not in a standard Maven directory, so I moved it to src/main/java

3) I changed the test1() function to a simple "public static void main" so that I could run the code directly

Everything else just worked - I got a nice har file as the output. 

How are you running this code, and what problems are you having? I don't even understand how this project worked for you, since demo.java was in the wrong place for  Maven build.

--

Patrick Lightbody

+1 (415) 830-5488

<demo.zip>

[Graham]

Hi Patrick, 

Thanks for trying it. 

I ran it as a TestNg test, so I didn't need for it to be in a main method and I was using Maven purely to manage the dependencies. 

I've rerun this and it works perfectly now, I've no idea what has changed unless we're using a more up to date version of one of the dependencies or something has changed on our network in the office in here but either way it's working like a treat now. 

thanks for your time!

Graham

[webtester]

I have the same issue: But I think it's not IDE specific.

I have a Selenium Grid set up.

HUB: Ubuntu machine with Selenium server 2.24.1 and browsermob proxy
Nodes: windows machines with Selenium 2.24.1

The test execution is really fast if I don't use a proxy.
With proxy it takes almost 2 minutes.
Without proxy it takes 25 seconds.

I use the following code:
String PROXY = SELENIUM_HUB_IP + ":" + proxyPort;
Proxy proxy = new Proxy();
proxy.setHttpProxy(PROXY);

DesiredCapabilities cap = DesiredCapabilities.firefox();
cap.setCapability("platform", Platform.WINDOWS);
cap.setCapability("version", "9");
cap.setCapability("proxy", proxy);

driver = new RemoteWebDriver(new URL("http:// + SELENIUM_HUB_IP + ":
4444/wd/hub"), cap);


Thanks,

> >> On 26 March 2012 19:21, Patrick Lightbody <patr...@lightbody.net> wrote:
>
> >>> Thanks let me know.
>
> >>>   --
> >>> Patrick Lightbody
> >>> +1 (415) 830-5488
>
> >>> On Mar 26, 2012, at 11:20 AM, Graham Abell wrote:
>
> >>> The IDE project shouldn't be a problem, the URL is internal but I'll
> >>> verify the same thing happens with our production site in the  morning
> >>> which iswww.paddypower.com
>

> >>> On 26 March 2012 19:18, Patrick Lightbody <patr...@lightbody.net> wrote:
>
> >>>> OK, I need the URL you're trying to hit and also if you can share a
> >>>> stripped down version of the IDE project (sans all your proprietary stuff,
> >>>> hopefully you can still recreate the issue).
>
> >>>>   --
> >>>> Patrick Lightbody
> >>>> +1 (415) 830-5488
>
> >>>> On Mar 26, 2012, at 11:09 AM, Graham Abell wrote:
>
> >>>> Hi Patrick,
>
> >>>> Yes, slow via the IDE while using BMP programmatically. It runs fine
> >>>> via the binary implementation.
>
> >>>> graham
>

> ...
>
> meer lezen »

[Graham]

I've been continuing to see this intermittently. I actually think it's something to do with my network since the code doesn't change from day to day yet the results do. Anyway, what I've found resolves it is for me to disable the network interface in control panel and re-enable it. When I've done this I start getting normal run speeds again. very weird.

[webtester]

Well, we have an internal Grid and a 50 Mbit internet connection.

I don't think it is network related. Maybe there are other ideas.


Grtz

> ...
>
> meer lezen »

[Graham Abell]

so do we and we have a bigger pipe than that, but that's what I found fixed the issue for me.

[webtester]

Ok, I restarted the systems a few times.. same results :(

On 27 jun, 17:53, Graham Abell <nicegra...@gmail.com> wrote:
> so do we and we have a bigger pipe than that, but that's what I found fixed
> the issue for me.

> ...
>
> meer lezen »