BrowserMob Proxy as an embedded proxy for Web Mobile Appium HTTPS

[Salem Artin]

Hello,

The requirement:

- Use Chrome Web browser on Android (real device or an emulator)

- Proxy HTTP and HTTPS traffic to add request headers and read response headers

I am looking for an embedded proxy so that I programatically read and add headers. 

Note that I am aware that Appium does not support proxy https://github.com/appium/appium/issues/4968

I have been looking into BrowserMob Proxy (browsermob-core-littleproxy 2.1.0-beta-3) and while I was able to proxy HTTP traffic, I am unable to proxy HTTPS traffic!

When I try to proxy an HTTPS site, the Chrome browser tries to load the page but it fails after 30 seconds or so

Is there a certificate that I need to to install on the Android device? in order for this to work? Does anyone know what I exactly need to do so that I can proxy HTTPS traffic using BrowserMob Proxy?

In the code below,  if I set FULL_URL to an HTTP site like "http://www.urbandictionary.com/", BrowserMob Proxy will do an excellent job

But if I try do the same while setting FULL_URL to an HTTPS site, like "http://www.google.com/", BrowerMob Proxy does not work as expected

public class LittleProxyExampleMobile {
    private final static Logger LOGGER = LoggerFactory.getLogger(LittleProxyExampleMobile.class);
        private BrowserMobProxy proxy;
    private RemoteWebDriver driver;

    private final static String FULL_URL = "http://www.urbandictionary.com/";
//    private final static String FULL_URL = "http://www.google.com/";

    @BeforeSuite
    public void setup() {
        proxy = new BrowserMobProxyServer(9090);

        // configure it as a desired capability
        DesiredCapabilities caps = DesiredCapabilities.android();
        caps.setCapability(MobileCapabilityType.DEVICE_NAME, "Android Emulator");
        caps.setCapability(MobileCapabilityType.BROWSER_NAME, "Chrome");
        caps.setCapability(MobileCapabilityType.PLATFORM_NAME, "Android");
        caps.setCapability(MobileCapabilityType.PLATFORM_VERSION, "4.4.2");
        caps.setCapability(MobileCapabilityType.NEW_COMMAND_TIMEOUT, 0);
        caps.setCapability(MobileCapabilityType.ACCEPT_SSL_CERTS, true);

        ChromeOptions chromeOptions = new ChromeOptions();
        chromeOptions.addArguments("ignore-certificate-errors");
        caps.setCapability(ChromeOptions.CAPABILITY, chromeOptions);

        try {
            driver = new RemoteWebDriver(new URL("http://127.0.0.1:4723/wd/hub"), caps);
        } catch (MalformedURLException e) {
            e.printStackTrace();
            throw new RuntimeException("Failed to init RemoveWebDriver");
        }

        addProxyListenners();
    }

    @Test
    public void test() {
        // create a new HAR
        proxy.newHar(FULL_URL);
        driver.get(FULL_URL);
        // get the HAR data
        Har har = proxy.getHar();

    }

    @AfterSuite
    public void tearDown() {
        proxy.stop();
        driver.close();
        driver.quit();
    }

    private void addProxyListenners() {
        proxy.addRequestFilter(new RequestFilter() {
            @Override
            public HttpResponse filterRequest(HttpRequest request, HttpMessageContents contents, HttpMessageInfo messageInfo) {
                LOGGER.info("HttpRequest received with messageInfo " + messageInfo.getOriginalUrl());
                if (messageInfo.getOriginalUrl().endsWith(FULL_URL)) {
                    request.headers().add("key", "value");
                    LOGGER.info("Header added for URL: " + messageInfo.getOriginalUrl());
                }
                return null;
            }
        });

        proxy.addResponseFilter(new ResponseFilter() {
            @Override
            public void filterResponse(HttpResponse response, HttpMessageContents contents, HttpMessageInfo messageInfo) {
                LOGGER.info("HttpResponse received with messageInfo " + messageInfo.getOriginalUrl());
                if (messageInfo.getOriginalUrl().endsWith(FULL_URL)) {
                    HttpHeaders httpHeaders = response.headers();
                    for (Map.Entry<String, String> entry : httpHeaders.entries()) {
                        LOGGER.info(entry.getKey() + ": " + entry.getValue());
                    }
                }

            }
        });
    }
}

Thanks

[Krishnan Mahadevan]

If appium doesn't support proxies, how do you envision BMP would be able to help you out here ?

Here's a whackier way of doing this. 

Spin off a BMP server.

Use the RESTFul APIs to Create a Proxy instance and adding headers.

Now setup the proxy server information as shown in this SO Thread : http://stackoverflow.com/a/22451531

Start your appium server and try your tests against it.

PS : I am stating this as a theoretical solution and your actual mileage may vary.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--

---
You received this message because you are subscribed to the Google Groups "BrowserMob Proxy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browsermob-pro...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Salem Artin]

If appium doesn't support proxies, how do you envision BMP would be able to help you out here ?

As an embedded proxy that I can start, stop and manage from my TestNG code. Please try to execute the code I posted to see what I mean

Anyway, It seems Google Android emulators are having problems when proxying HTTPS traffic through Browsermob Proxy. I solvee my problem by using an alternative emulator solution; Genymotion!

[Krishnan Mahadevan]

I understand what you mean. All I was trying to say was that instead of you trying to start your BMP server via code and then create proxy instances out of it, you can try having a BMP server already running (you can use the same @BeforeSuite to start a BMP server using the batch file rather than via the code), and then have your code inject the proxy instance created via a RESTFul call, into your caps.

Glad to know you solved your problem. 

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

[je...@outlook.com]

You may also have better luck with beta-4, which was just released and contains better MITM support.

Please do post more info if you continue to see issues with BMP + Android -- it's not a use case that is well-tested, unfortunately.

[Fion Wang]

Hi, may I ask how you get the traffic like http from mobile using appium + BrowserMob proxy, do you need to configure the network from mobile with the same wireless with your desktop

...

[⇜Krishnan Mahadevan⇝]

Please see if this helps : https://github.com/appium/appium/issues/4968

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--

[Developer]

Thanks, not sure i'm understanding right about that,is that impossible to implement with appium with BrowserMob proxy in mobile?how to set up your own proxy wherever appium is running? 

[⇜Krishnan Mahadevan⇝]

This may sound silly.. But you may try setting the BMP proxy at the machine level.. [ So yeah, that kind of means that all the traffic goes via the proxy server..]..

PS : You might want to try posting this on the Appium forums to see if there is any progress on this front.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

[Nick S]

Hi Salem, 

I am facing the same issue with Google Android emulators when proxying HTTPS traffic through Browsermob Proxy. HTTP works just fine. Could you please share the network settings of Genymotion emulator and steps to make it work. 

Thank you!

N.