Modifying body content

[jan.magleb...@gmail.com]

Hi,

Is it possible to use LittleProxy to modify the content sent to and received from a web server? I'm not really interested in modifying the headers; just the body. Also, it should work over http and https. If so are there some examples of this.

I'v been playing around overwriting HttpFiltersAdapter but this seems not to work on the body content level.

Thanks

[Shyam Shankar]

Hi,

I'm also looking to solve the same problem. I need to inject a piece of javascript in the response which will collect all the javascript errors in the webpage. So by using selenium webdriver, I can display all the javascript errors in the page.

It would be great to get an example for modifying the response in the response filter. I was also reading from these forums and came across a suggestion by using cache mechanism. Nut I'm not sure how to go about it.


Thanks,
Syam

[Ray]

Here is my sample code which could capture request and response body.

HttpProxyServer server =

              DefaultHttpProxyServer.bootstrap().withAllowLocalOnly(false)

                      .withPort(8888)

                      .withManInTheMiddle(new SelfSignedMitmManager())

                      .withFiltersSource(new HttpFiltersSourceAdapter() {

                         public HttpFilters filterRequest(HttpRequest originalRequest, ChannelHandlerContext ctx) {

                            return new HttpFiltersAdapter(originalRequest) {

                               @Override

                               public HttpResponse requestPre(HttpObject httpObject) {

                                  System.out.println("requestPre: " + originalRequest.getMethod() + " " + originalRequest.getUri());

                                  if(httpObject instanceof HttpContent)

                                  {

                                     System.out.println((((HttpContent) httpObject)

                                             .content().toString(

                                                     Charset.forName("UTF-8"))));

                                  }

                                  return null;

                               }

                               @Override

                               public HttpResponse requestPost(HttpObject httpObject) {

                                  return null;

                               }

                               @Override

                               public HttpObject responsePre(HttpObject httpObject) {

                                  if (httpObject instanceof HttpResponse) {

                                  } else if (httpObject instanceof HttpContent) {

                                     System.out.println((((HttpContent) httpObject)

                                             .content().toString(

                                                     Charset.forName("UTF-8"))));

                                  }

                                  return httpObject;

                               }

                               @Override

                               public HttpObject responsePost(HttpObject httpObject) {

                                  // TODO: implement your filtering here

                                  return httpObject;

                               }

                            };

                         }

                      })

                      .start();

[Ox Cart]

Thanks Ray!  Couple of notes:

1. Some HttpObjects (like FullHttpResponse) actually implement both the HttpRequest interface and the HttpContent interface, so the if/else in your responsePre is a little buggy.

2. The HttpContent that you get may just be a portion of the body, as Netty processes the response in streaming fashion.  If your response is going to be of a limited size, you can implement the getMaximumResponseBufferSizeInBytes() method on HttpFiltersSource, which will cause LittleProxy to buffer the response.  The HttpContent will then be a FullHttpResponse, which gives you the fully body, which is easier to manipulate as you need.

[Ray]

Very appreciate for pointing out this. It's very useful for me.

[Shyam Shankar]

Thanks Ray and Ox Cart for your helpful answers.
Actually I was able to solve the problem of modifying the response body the next day. I had found this answer by Ox Cart, https://github.com/adamfisk/LittleProxy/issues/113 and had followed it successfully.
But I'm facing an another problem, when I try to inject a piece of javascript into the response, the original response body gets cut off. The response size seems to get fixed, so if you inject 50 characters to the response body, the last 50 characters of the original response body gets cut. So what parameter (related to response size) can I modify to reflect the new response size.
Here is the code I'm using : http://paste.ubuntu.com/7705044/

Thanks in advance.
Syam

[Ox Cart]

resp.content() refers to a fixed size buffer.  You need to construct a new large buffer, fill that with the modified body, and set it as resp.content().  If it's easier, you can also construct a new DefaultFullHttpResponse.

Make sure to release the original buffer before returning from your filter, otherwise you'll end up with a memory leak.

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

--
You received this message because you are subscribed to a topic in the Google Groups "LittleProxy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/littleproxy/iPY59IIiDgE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to littleproxy...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shyam Shankar]

Thank you Ox Cart for such a quick response. You are truly awesome :)
As you had suggested, I created a new DefaultFullHttpResponse and it is working fine. I'm just concerned that it takes a long time to finish loading the page. Is there something we can do to speed it up?

Thanks again for making this super proxy.

-Syam

[Ox Cart]

Unfortunately buffering the whole page before manipulating the content can slow things down considerably.  The alternative is to implement your logic on top of the data stream.  If you're just looking for the head section, you can buffer only the HEAD section inside of your filter class instead of having LittleProxy do the buffering.

1. As chunks come in, buffer them manually

2. Inspect each chunk looking for a closing </head> tag

3. Once you've found the closing </head> tag, insert your javascript into the buffered content, return that content to the browser, and then stop buffering

Cheers,

Ox

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Thanks again for your quick response. Your suggestion sounds brilliant, but I'm having trouble implementing it. This is my pre-Response Filter.

                    public HttpObject responsePre(HttpObject httpObject) {
   
                        if (httpObject instanceof HttpResponse) {

                         if (httpObject instanceof DefaultFullHttpResponse) {
                        
                         DefaultFullHttpResponse resp = (DefaultFullHttpResponse) httpObject;
                         System.out.println("hhtp reesp1");
                         //modifying the responsxe: adding a script to the response for collecting javacsript errors
                         String response = resp.content().toString(CharsetUtil.UTF_8);
                         String errorScript = "\n<script>" + "window.webdriver_javascript_errors = [];\n" + 
                         "window.onerror = function(errorMsg, url, lineNumber) {\n" + 
                         "  window.webdriver_javascript_errors.push(\n" + 
                         "    errorMsg +' (found at ' + url + ', line ' + lineNumber + ')');\n" + 
                         "};" + "</script>\n";
                        
                         Pattern pattern = Pattern.compile("<head>");
                             Matcher matcher = pattern.matcher(response);
                             int index = -1;
                             if (matcher.find()) {
                               index = matcher.start() + "<head>".length();
                             }
                             StringBuilder sb = new StringBuilder(response);
                             sb.insert(index,errorScript);
                             
                         ByteBuf contentNew = buffer(1024);
                         contentNew.writeBytes(sb.toString().getBytes(CharsetUtil.UTF_8));                             
                             DefaultFullHttpResponse resp2 = new DefaultFullHttpResponse(resp.getProtocolVersion(),resp.getStatus(),contentNew);

                         return resp2;
                         }
                        }
                        return null;
                    };

I would like to know how to break down the response into chunks, because the response I seem to get is FullHttpResponse. 

Also I've been successful in using this solution to capture Javascript errors with Selenium WebDriver for the three major browsers: Chrome, Firefox and Internet Explorer. But while using Little Proxy with Safari driver, the response filters are not getting activated. The strange part is the same proxy declaration works for Chrome and Firefox. Here is the code I'm using:

String urlString = "http://the-internet.herokuapp.com/javascript_error";

Proxy proxy = new Proxy();

proxy.setProxyType(Proxy.ProxyType.MANUAL);

String proxyStr = String.format("localhost:%d", PROXY_PORT);

proxy.setHttpProxy(proxyStr);

proxy.setSslProxy(proxyStr);

LoggingPreferences logs = new LoggingPreferences();

logs.enable(LogType.BROWSER, Level.ALL);

//Safari

// DesiredCapabilities capability = DesiredCapabilities.safari();

// capability.setCapability(CapabilityType.PROXY, proxy);

// capability.setCapability(CapabilityType.LOGGING_PREFS, logs);

// WebDriver driver = new SafariDriver(capability);

//Firefox

DesiredCapabilities capability = DesiredCapabilities.firefox();

capability.setCapability(CapabilityType.PROXY, proxy);

capability.setCapability(CapabilityType.LOGGING_PREFS, logs);

WebDriver driver = new FirefoxDriver(capability);

driver.get(urlString);

I'm sorry for dumping two questions in one email. Please let me know If I have to post this as a new question in the mailing group.

Thanks a lot.

-Syam

[Ox Cart]

Is your HttpFiltersSource returning a non-zero value from getMaximumResponseBufferSizeInBytes()?  That would cause LittleProxy to buffer the response and use a DefaultFullHttpResponse.

Concerning the Safari problem, you can turn up the log level in log4j.xml to DEBUG and see if requests are even making it to LittleProxy.  It they're not, then it sounds like an issue with Selenium/Safari, which I can't help with.

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Yes it was returning a non-zero value. When I tried by returning a zero value, it does not return a DefaultFullHttpResponse as you have mentioned. But I'm not sure how to access the chunk from HttpResponse as it doesn't have any content related variable like in the case of DefaultFullHttpResponse.

For the safari problem, I can see that Little Proxy is getting started for I can see this in the logs:

Jun 30, 2014 12:18:11 AM org.littleshoot.proxy.impl.DefaultHttpProxyServer start

INFO: Starting proxy at address: /127.0.0.1:8938

Jun 30, 2014 12:18:11 AM org.littleshoot.proxy.impl.DefaultHttpProxyServer doStart

INFO: Proxy listening with TCP transport

But now I am facing an even bigger problem. Actually the target webpage whose response I need to modify is using an HTTPS connection. So I tried to use MITM manager as given in the MITM test example , i.e       But the webpage I'm trying to access is not getting loaded and I'm getting this error.

The connection was reset

The connection to the server was reset while the page was loading.  

Also I want to know is it possible to get the content of the webpage in text format using this approach. If not, what can be the best approach to solve this issue?

Thanks,

Syam

[Ox Cart]

1. When not buffering responses, your filter method will be called multiple times per response.  You need to look for response instanceof HttpContent.  Casting to HttpContent will give you something from which you can extract the content buffer.

2. For your Safari problem, the question is not whether or not LittleProxy is started, but whether it's seeing any traffic.  Run with Chrome or Firefox, and observe what happens in the logs when LittleProxy gets traffic.  Now repeat with Safari and see if the same thing happens in the logs.

3. If you MITM the connection, the browser is going to get certificate errors.   If you try opening the page from a browser (not Selenium), what do you see?  If you see a certificate error, you will need to configure your browser and/or Selenium to ignore such errors.

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Thanks again for lightning fast response.

1. I'll try with HttpContent and let you know the results.

2. At present, I dont see any difference from the logs of Chrome/Firefox and Safari after receiving traffic. Could you tell me the location of  log4j.xml to change the logging level. I'm not able find it in src/test/resources.

3. I'm not sure I understood your suggestion correctly. When I try to open the https url from a regular browser I dont see any errors. I try to open the link in a new tab from a browser configured with MITM proxy opened with Selenium, then I see the error: The proxy server is refusing connections.

Please let me know if I answered your question correctly.

[Ox Cart]

1. Great

2. https://github.com/adamfisk/LittleProxy/blob/master/src/test/resources/log4j.xml

3. When you open the url from a regular browser, the page displays correctly without any errors?  Did you configure your browser to use your proxy?

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

2. I'm sorry to sound so noob, but I didn't get it. So should I clone the repo, edit the change in log4j file, then make a new jar from it and add the external jar to my project? That's the only way I can see :)

3. When I open the page from a regular browser without configuring the proxy, it loads without an error. When I try to load the page through the proxy, I get:

[Ox Cart]

2. You just need log4j.xml in your classpath somewhere, so you can just copy it over to your own project.  If you already have a log4j.xml (or log4j.properties), you can just update that.  The main thing is that you want the log level for org.lantern to be DEBUG.  That way, you'll see a bunch of messages every time LittleProxy handles a request.

3. Once you get DEBUG log output, please post your logs for the MITM issue - they should be of help

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

This is the log file generated:

https://dl.dropboxusercontent.com/u/18050220/log%20file.txt

This is the testcase:

https://dl.dropboxusercontent.com/u/18050220/HttpFilterTest.java

I'd want to confirm so after resolving this issue, we can actually get the content like in Http response, and edit it right?

[Ox Cart]

Thanks for sending that over.

I noticed that you are configuring your proxy with a ChainedProxyManager using the .withChainProxyManager() method. I don't think you need this, so as a first step I would try taking that out.

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Yes, I noticed a comment in the Mitm testcase, that it helps verify that Mitm manager is selected over Chained proxy manager as only either of them can be selected at a time. Very well, I'll remove the same and try again.

[Syam Sankar]

Here is the log file after having remove the chainproxyManager():

https://dl.dropboxusercontent.com/u/18050220/new%20log%20file.txt

[Ox Cart]

So, I tried running a proxy configured with your code (including filters), and then tried to access https://www.google.com from Chrome using that proxy.

Here's what I get (which is what I would expect):

The code I used is below.  If you do this test, what do you see?

import static io.netty.buffer.Unpooled.*;

import io.netty.buffer.ByteBuf;

import io.netty.handler.codec.http.DefaultFullHttpResponse;

import io.netty.handler.codec.http.HttpObject;

import io.netty.handler.codec.http.HttpRequest;

import io.netty.handler.codec.http.HttpResponse;

import io.netty.util.CharsetUtil;

import java.util.Queue;

import java.util.regex.Matcher;

import java.util.regex.Pattern;

import org.littleshoot.proxy.ChainedProxyManager;

import org.littleshoot.proxy.HttpFilters;

import org.littleshoot.proxy.HttpFiltersAdapter;

import org.littleshoot.proxy.HttpFiltersSource;

import org.littleshoot.proxy.HttpFiltersSourceAdapter;

import org.littleshoot.proxy.HttpProxyServer;

import org.littleshoot.proxy.extras.SelfSignedMitmManager;

import org.littleshoot.proxy.impl.DefaultHttpProxyServer;

public class SyamProblem {

    private static final int PROXY_PORT = 8938;

    public static void main(String[] args) {

        final HttpFiltersSource filtersSource = new HttpFiltersSourceAdapter() {

            public HttpFilters filterRequest(HttpRequest originalRequest) {

                return new HttpFiltersAdapter(originalRequest) {

                    public HttpObject responsePre(HttpObject httpObject) {

                        if (httpObject instanceof HttpResponse) {

                            if (httpObject instanceof DefaultFullHttpResponse) {

                                DefaultFullHttpResponse resp = (DefaultFullHttpResponse) httpObject;

                                System.out

                                        .println("http full resp created-------------------------------------------------------------------------");

                                // modifying the responsxe: adding a script to

                                // the

                                // response for collecting javacsript errors

                                String response = resp.content().toString(

                                        CharsetUtil.UTF_8);

                                // System.out.println("response");

                                // System.out.println(response);

                                String errorScript = "\n<script>"

                                        + "window.webdriver_javascript_errors = [];\n"

                                        +

                                        "window.onerror = function(errorMsg, url, lineNumber) {\n"

                                        +

                                        "  window.webdriver_javascript_errors.push(\n"

                                        +

                                        "    errorMsg +' (found at ' + url + ', line ' + lineNumber + ')');\n"

                                        +

                                        "};" + "</script>\n";

                                Pattern pattern = Pattern.compile("<head>");

                                Matcher matcher = pattern.matcher(response);

                                int index = -1;

                                StringBuilder sb = null;

                                if (matcher.find()) {

                                    index = matcher.start() + "<head>".length();

                                    sb = new StringBuilder(response);

                                    sb.insert(index, errorScript);

                                    ByteBuf contentNew = buffer(1024);

                                    contentNew.writeBytes(sb.toString()

                                            .getBytes(

                                                    CharsetUtil.UTF_8));

                                    DefaultFullHttpResponse resp2 = new DefaultFullHttpResponse(

                                            resp.getProtocolVersion(),

                                            resp.getStatus(), contentNew);

                                    // System.out.println("lalala---------------------------------------------------");

                                    // System.out.println(sb.toString());

                                    return resp2;

                                }

                            }

                        }

                        return null;

                    };

                };

            };

            public int getMaximumRequestBufferSizeInBytes() {

                return 102400 * 10240;

            };

            public int getMaximumResponseBufferSizeInBytes() {

                return 102400 * 10240;

            };

        };

        final HttpProxyServer proxyServer = DefaultHttpProxyServer.bootstrap()

                .withPort(PROXY_PORT)

                .withChainProxyManager(new ChainedProxyManager() {

                    public void lookupChainedProxies(

                            HttpRequest httpRequest,

                            Queue<org.littleshoot.proxy.ChainedProxy> chainedProxies) {

                    }

                })

                .withManInTheMiddle(new SelfSignedMitmManager())

                .withFiltersSource(filtersSource)

                .start();

    }

}

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

I'm getting the same response like you as shown in the screenshot. Let me try finding out how to ignore this warning as you had said in the previous post, using selenium and from the stand alone browser. But why am I not able to view the response (content in plain text format) recieved from the HTTPS server in my console while trying to access the site? Even though the browser gives the error, I should be able to view the content of the response right?

[Ox Cart]

The public keys for certain sites (like www.google.com) are pinned by the browser.  To override this, you need to install the certificate used by the MITM proxy as a trusted root certificate into your system's trust store (or in the case of Firefox, into Firefox's trust store).

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

But when I try to use Selenium webdriver for testing the same websites, I'm getting this error. 

I'm not able to understand what could be the issue. 

Here is the the DEBUG level log file: https://dl.dropboxusercontent.com/u/18050220/chrome%20log%20file.txt

Here is the webdriver code:

        String urlString = "https://www.paypal.com/home";

Proxy proxy = new Proxy();

proxy.setProxyType(Proxy.ProxyType.MANUAL);

String proxyStr = String.format("localhost:%d", PROXY_PORT);

proxy.setHttpProxy(proxyStr);

proxy.setSslProxy(proxyStr);

LoggingPreferences logs = new LoggingPreferences();

  logs.enable(LogType.BROWSER, Level.ALL);

 

String chromeDriverPath = "/Users/syasankar/software/chromedriver";

System.setProperty("webdriver.chrome.driver", chromeDriverPath);

DesiredCapabilities capability = DesiredCapabilities.chrome();

capability.setCapability(CapabilityType.LOGGING_PREFS, logs);

      capability.setCapability(CapabilityType.PROXY, proxy);

      capability.setCapability("chrome.switches", Arrays.asList("--ignore-certificate-errors"));

        WebDriver driver = new ChromeDriver(capability);

Any pointers to what it is causing this would be awesome.

[Ox Cart]

Does this work if you bypass the proxy (i.e. don't set capability CapabilityType.PROXY) ?

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Definitely.

[Ox Cart]

And Firefox gets a similar error when driven by Selenium?

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Ox Cart]

Okay, I just tried this, and got no data back:

curl -k -x localhost:8938 https://www.google.com/humans.txt

Taking a look now.

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Ox Cart]

Okay, if I run without the filters, but with MITM, it works.  So, something about the filter maybe ...

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Ox Cart]

Found your problem.  Your filter has a return null, which has the effect of swallowing the response.  You need to always at least return httpObject.

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Ox Cart]

BTW, it looks like when MITM is enabled, responses don't get buffered, so you'll have to implement a streaming version of your filter anyway.

For https://www.google.com/humans.txt, the response ends up being returned in 3 pieces:

DefaultHttpResponse

DefaultHttpContent (this has the beginning of the body)

LastHttpContent (this has the end of the body)

You can see this by just logging httpObject from within your filter.

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Thanks a lot. Let me look into it right away and let you know the results.

[Syam Sankar]

I'm not able to view any response body text from DefaultHttpContent ,LastHttpContent and HttpContent.

My HttpObject is not of the above 3 types, so I'm not able to view the content from those response. I'm only able to get a DefaultHttpResponse which is pretty useless as it doesnt have any content methods. Here is my response interceptor, do let me if I'm doing it all wrong.

                    public HttpObject responsePre(HttpObject httpObject) {

   

                        if (httpObject instanceof HttpResponse) {

                        

                        

                        if (httpObject instanceof DefaultHttpContent) {

                        DefaultHttpContent resp3 = (DefaultHttpContent) httpObject;

                        System.out.println("response default content ---------");

                        System.out.println(resp3.content().toString(CharsetUtil.UTF_8));

                        

                        }

                        

                        if (httpObject instanceof LastHttpContent) {

                        LastHttpContent resp3 = (LastHttpContent) httpObject;

                        System.out.println("response last content---------");

                        System.out.println(resp3.content().toString(CharsetUtil.UTF_8));

                        

                        } 

                        

                        if (httpObject instanceof DefaultHttpResponse) {

                        DefaultHttpResponse resp3 = (DefaultHttpResponse) httpObject;

                        System.out.println("default response---------");

                        System.out.println(resp3);

                        

                        } 

                        

                        if (httpObject instanceof HttpContent) {

                        HttpContent resp3 = (HttpContent) httpObject;

                        System.out.println("default content---------");

                        System.out.println(resp3.content().toString(CharsetUtil.UTF_8));

                        

                        } 

                        

[Ox Cart]

All you need to do is this:

public HttpObject responsePre(HttpObject httpObject) {

                        System.out.println("Response is: " + httpObject);

                        if (httpObject instanceof HttpContent) {

                            System.out.println(((HttpContent) httpObject)

                                    .content().toString(CharsetUtil.UTF_8));

                        }

                        return httpObject;

                    };

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

I tried that with Chrome and Firefox, but I'm getting a lot of gibberish (encrypted content) instead of plain text content. 

[Ox Cart]

Are you man in the middling the connection?  Does it work with curl?  What is your full code at this point?

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Yes I'm using MITM manager. Here is my code:

https://dl.dropboxusercontent.com/u/18050220/HttpFilterTest.java

 

[Ox Cart]

Does it work with curl?

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Yes it works with curl and gives me back a text response.

curl -k -x localhost:8919 https://www.google.com/humans.txt

Google is built by a large team of engineers, designers, researchers, robots, and others in many different sites across the globe. It is updated continuously, and built with more tools and technologies than we can shake a stick at. If you'd like to help us out, see google.com/careers.

[Ox Cart]

Okay, I see the issue.  It looks like Chrome accepts Gzip encoding, so the content is Gzip encoded.  You'll need to either decode it, or modify the request to remove the Accept-Encoding header.

At this point, I'm convinced that LittleProxy's MITM and filtering are working as intended.  I think your remaining issues will just be a matter of working with the HTTP protocol and understanding what the browser, server and Selenium are doing.  For this you'll be on your own.  My best advice to you is to add a lot of debug logging on both the request and response side of things, and to use curl with the -I and -v flags to test.

Best of luck to you!

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Let me try removing the Accept-encoding header from the request as you suggested and see how it goes.

Anyway thanks a lot mate for your awesomely detailed and insanely quick responses. Seriously man God bless you!

If you ever get around to visiting Bangalore in India, let me know and I'd love to buy you dinner.

-Syam

[Ox Cart]

You're very welcome!  If I'm ever in Bangalore, I might take you up on that!

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

Looking forward to it :)

...

[Gulshan Saini]

Hi Shayam,

I just started with Littleproxy API and my requirements are also similar. Can you please post updated code(with all changes you have done as per suggested by Ox Cart) as i am getting error at below line

"ByteBuf contentNew = buffer(1024);"

ERROR: The method buffer(int) is undefined for the type new HttpFiltersAdapter(){}

Also, I am not able to understand how to implement following changes

Unfortunately buffering the whole page before manipulating the content can slow things down considerably.  The alternative is to implement your logic on top of the data stream.  If you're just looking for the head section, you can buffer only the HEAD section inside of your filter class instead of having LittleProxy do the buffering.

1. As chunks come in, buffer them manually

2. Inspect each chunk looking for a closing </head> tag

3. Once you've found the closing </head> tag, insert your javascript into the buffered content, return that content to the browser, and then stop buffering

Thanks

Gulshan

[Syam Sankar]

So this is the final code I've got after implementing all of Ox Cart's suggestions as follows:

1. Removed gzip encoding header

2. Used MITM manager for https url support

3. Buffered the chunk wise response data and inserted a piece of Javascript code between the <head> tags.

I'm not sure how much of this will be useful for you, but do let me know if you are having trouble with it. Here is the code: https://dl.dropboxusercontent.com/u/18050220/littleproxy.java

--

You received this message because you are subscribed to a topic in the Google Groups "LittleProxy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/littleproxy/iPY59IIiDgE/unsubscribe.

[Ox Cart]

Syam,

Thanks for helping out!

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Syam Sankar]

My pleasure :)

-Syam

[Benny Fuhry]

Hi,

I want to solve a similar problem as Jan and Shyam and I read this whole thread. I also got Shyam's code running and it worked like a charm.

But my problem is a little bit harder as the content I want to add is not static. In fact I want to go through the body and change html elements dynamically. The problem with this is the content header in the HTTP request. Shyam could just rewrite it to his static length which he knew before. I have to edit the content first and the write the content header afterwards.

To my knowledge I can prevent Littleproxy to chunk the response messages with getMaximumResponseBufferSizeInBytes but the header and content is still separated and as the DefaultHttpResponse will come first it is not possible to change the body and set the content length header afterwards.

Does anybody know an approach to solve this problem? 

Thanks and regards,

Benny

[OS]

Hi Benny,

I'd like to implement the same - did you already solve a solution?

Thanks and regards

Olaf

Am Dienstag, 9. September 2014 11:15:08 UTC+2 schrieb Benny Fuhry:

... I want to go through the body and change html elements dynamically... 

...DefaultHttpResponse will come first it is not possible to change the body and set the content length header afterwards.

[Bruno Cunha]

The same here... I need to do some replaces of different lengths on body, and can't know how much it will change from response... How you solved it?

And another question... How to prevent the filter from receiving small chunks? I would prefer to receive the entire body and handle it with Jsoup...

Regards,
Bruno

[Ox Cart]

The filter is stateful and 1 per request.  Just store the chunks then do your thing when the last one arrives.

[Bruno Cunha]

Can you provide me some example, please?

I tried to do what you suggested (buffering responses in a ByteArrayOutputStream, and returning 0-length mid-responses), but I'm getting a lot of always-opened connections.

http://pastebin.com/617L7Su4

[Ox Cart]

Oh, sorry, there's a better way.

See https://github.com/adamfisk/LittleProxy/blob/master/src/main/java/org/littleshoot/proxy/HttpFiltersSource.java#L35

Just size that large enough for your responses, and your filter will get a FullHttpResponse.

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[OS]

FullResponse even with MTM received?
I've never seen it on ssl-connections - probably my fault!?
Thx

[Bruno Cunha]

Got this fully working: http://pastebin.com/gGbzACiZ

It works with dynamic sizes too if it filters a DefaultFullHttpResponse (probably way - I changed the buffer sizes to Integer.MAX_VALUE) - now I set the Content-Length response header to the length of changed content.

fullResponse.headers().set("Content-Length", proxyVO.getText().length());

Thank you!!

Bruno

[Ox Cart]

You're welcome!

Please do be careful setting the buffer size that high.  I don't know what kind of traffic you're filtering, but if it includes large file downloads, your program will end up trying to buffer way too much data.

Cheers,

Ox

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Sandeep Modgil]

Hi Bruna,

I have tried several ways to get a FullHttpResponse object but am unable to do so,i want to see the actual html content of the web pages using proxy.Let me know if it is possible.

I am using your code given in this link: http://pastebin.com/gGbzACiZ but it is not giving me any html content of the web page.Could you please suggest what more i need to do to make it work.

[Wissam Mallouli]

I tried your code and I have this problem:


[ERROR] /home/mallouli/workspace/LittleProxy/src/main/java/org/littleshoot/proxy/Launcher.java:[129,19] method does not override or implement a method from a supertype
[ERROR] /home/mallouli/workspace/LittleProxy/src/main/java/org/littleshoot/proxy/Launcher.java:[141,19] method does not override or implement a method from a supertype
[ERROR] /home/mallouli/workspace/LittleProxy/src/main/java/org/littleshoot/proxy/Launcher.java:[146,19] method does not override or implement a method from a supertype
[ERROR] /home/mallouli/workspace/LittleProxy/src/main/java/org/littleshoot/proxy/Launcher.java:[158,19] method does not override or implement a method from a supertype
[ERROR] -> [Help 1]

The "override" in your code correspond to requestPre, requestPost, responsePre, responsePost methods.