Trying new Portswigger "Web Security Academy" through ZAP, getting "Content Encoding Error"
449 views
Skip to first unread message
Bill Dietrich
Apr 7, 2019, 5:28:18 PM4/7/19
to OWASP ZAP User Group
[Sorry, I'm a newbie, hope I'm doing this right:]
Using Firefox 66.0.2 (64-bit) on Linux Mint 19.1, I've been working through the new Portswigger "Web Security Academy" (https://portswigger.net/ but you need to create an account). When you do an actual lab, their site redirects you to an URL such as https://acf92090389098d68063d3a2.web-security-academy.net/ which I assume is a just-spun-up VM.
Everything works fine if I just use Firefox. If I run ZAP D-2019-04-01 and have Firefox use the ZAP proxy, when the main site redirects to the VM, Firefox gives "Content Encoding Error".
It looks like the response from the GET of the VM URL has a header containing "Content-Encoding: gzip" but the response body just contains plain HTML (starts with "<!DOCTYPE html> <html> <head> ...").
In the zap.log I see "ERROR ProxyThread - Unable to uncompress gzip content: Not in GZIP format java.util.zip.ZipException: Not in GZIP format"
Why am I getting this error when using ZAP proxy ? Is the proxy being stricter than Firefox ? But the error message is coming in Firefox, a Mozilla-constructed web page that says "Content Encoding Error". Or maybe I'm completely wrong, and something else is going on ? Thanks for any help.
hauschu...@gmail.com
Apr 8, 2019, 2:25:53 AM4/8/19
to OWASP ZAP User Group
Hello!
Interesting!
It sounds like you've found what's going on, but exactly why is not too clear yet...could you include a screenshot of the error page and the accompanying proxied response headers/body?
One thing you could try is to the use the Replacer add-on in ZAP to regex for the 'Content-Encoding: gzip' header and replace it with empty string, and then see if that works. The replacer add-on has some default examples like 'remove HSTS' which could be used as an easy template to drop it or modify response headers to see under what exact circumstances the error occurs.
If you don't have that add-on yet, just click the little red/blue/green lego icon in the top toolbar, and then install it from the store!
hauschu...@gmail.com
Apr 8, 2019, 2:28:16 AM4/8/19
to OWASP ZAP User Group
Also, could you confirm that when you use Firefox without proxy, that the response header it receives is the exact same? How about the initial request headers?
I'm not sure why they would differ exactly, but it would help to pin down where things are going wrong...
Bill Dietrich
Apr 8, 2019, 3:38:35 AM4/8/19
to OWASP ZAP User Group
Okay, in ZAP, using Replacer to remove the "Content-Encoding: gzip" string from response headers DID fix the problem.
Without removing that string, attached files show what the request and response and error page look like. Trying to do a view-source on the error page also fails, not sure why.
hauschu...@gmail.com
Apr 8, 2019, 4:24:19 AM4/8/19
to OWASP ZAP User Group
Well, I'm glad the replacer works!
Let's try using the Replacer the other way....for some reason it looks like the Burp request has an 'accept-encoding: gzip' header, whereas ZAP does not (where it ends up in the response instead...??)
So instead of having the replacer check for and remove the Encoding header in the response, let's have it check for that encoding header in the initial request, and if it's not present, add it. (at least for that GET request)
Could we compare browser network requests through Burp and ZAP also, and ensure they have the same headers for both?
thc...@gmail.com
Apr 8, 2019, 4:37:15 AM4/8/19
to zaprox...@googlegroups.com
ZAP removes unsupported encodings by default:
https://github.com/zaproxy/zap-core-help/wiki/HelpUiDialogsOptionsLocalproxy#remove-unsupported-encodings
It's interesting that the response has the header despite that, I'd
check if there's any script (Proxy, HTTP Sender) changing the response.
Best regards.
https://github.com/zaproxy/zap-core-help/wiki/HelpUiDialogsOptionsLocalproxy#remove-unsupported-encodings
It's interesting that the response has the header despite that, I'd
check if there's any script (Proxy, HTTP Sender) changing the response.
Best regards.
Bill Dietrich
Apr 8, 2019, 5:13:49 AM4/8/19
to OWASP ZAP User Group
I can't get Replacer to add an 'Accept-Encoding: gzip, deflate' header to the request properly. In Tools / Replacer Options ... dialog, it requires a value in the "Match" field, won't allow it to be empty. If I put 'Accept-Encoding" in there, it doesn't match because the original request had no 'Accept-Encoding' directive at all. If I put "DNT: 1" in there (original request did have a "DNT: 1" in it), I end up with a bad request, duplicate DNT properties, because somehow Replacer left the original "DNT: 1" in there and added a "DNT: 1: Accept-Encoding: gzip, deflate". Is this a bug in Replacer ? Also, if I use Modify to try to modify a rule in Replacer, I can't do it, when I click Save it says "a rule with this value already exists", I have to delete the rule and add it again. Is this another bug ? Or maybe I'm just doing it wrong.
BUT: next time I tried it, Replacer DID replace "DNT: 1" with "Accept-Encoding: gzip, deflate". Maybe I was using "Request Header" instead of "Request Header String" the first time ? Not sure. Anyway, with "Accept-Encoding: gzip, deflate" added to the request, the response came back without the "Content-Encoding: gzip", and everything worked fine.
So, the situation seems to be:
- In all cases, I'm using same Firefox browser, and page content is always coming back as plain HTML (no encoding).
- With Burp proxy, request gets sent with "Accept-Encoding: gzip, deflate", response comes back without "Content-Encoding: gzip", all is well.
- I assume same happens with no proxy; I have no way of seeing headers in that case. All is well.
- With ZAP proxy, request gets
sent without "Accept-Encoding: gzip, deflate", response comes back with"Content-Encoding: gzip", get Firefox error.
Bill Dietrich
Apr 8, 2019, 5:17:54 AM4/8/19
to OWASP ZAP User Group
I'm running a pretty clean profile of Firefox when doing the testing, it has add-ons FoxyProxy, BuiltWith, WhatCMS in it. The last two only operate if I click on their icons, they're not reading or changing things on the fly.
Bill Dietrich
Apr 8, 2019, 5:23:59 AM4/8/19
to OWASP ZAP User Group
Okay, in ZAP I went to Tools / Options / Local Proxy and turned off "Remove Unsupported Encodings". And that fixed the problem. Now there's "Accept-Encoding: gzip, deflate, br" in the request header, and no "Content-Encoding: gzip" in the response header, all is well.
On Monday, April 8, 2019 at 10:37:15 AM UTC+2, thc202 wrote:
thc...@gmail.com
Apr 8, 2019, 5:28:26 AM4/8/19
to zaprox...@googlegroups.com
I meant scripts in ZAP:
https://github.com/zaproxy/zap-core-help/wiki/HelpAddonsScriptsScripts
Best regards.
https://github.com/zaproxy/zap-core-help/wiki/HelpAddonsScriptsScripts
Best regards.
hauschu...@gmail.com
Apr 8, 2019, 5:32:09 AM4/8/19
to OWASP ZAP User Group
Ah, nice!
I didn't know there was a simple checkbox option for that!
You can always check what network requests look like without a proxy by using the browser dev tools, but looks like we don't need to worry about that now!
Still, that only explains why ZAP drops the original accept-encoding request header....I wonder where the bad response header comes from. Ie, is ZAP adding it, or is the web-app adding it?
Because I think that's a bug if ZAP is adding it....
We could check by:
browser ---> ZAP ----> BURP ----> web-app
And then if we see the bad response header in BURP, we know it came from the web-app, but if it's in ZAP but not BURP, then we know ZAP did a weird thing!
Bill Dietrich
Apr 8, 2019, 5:43:02 AM4/8/19
to OWASP ZAP User Group
Re: scripts in ZAP: it's a fresh install of ZAP and I've never done any scripts. As far as I can tell from the tree view, the only scripts present are for HUD, and I'm not using HUD.
Bill Dietrich
Apr 8, 2019, 5:53:31 AM4/8/19
to OWASP ZAP User Group
The bad response is coming from the web-app. I ran Burp and its proxy, manually edited the request to remove "Accept-Encoding: gzip, deflate, br" from the request, and got a response with "Content-Encoding: gzip" in it, making Firefox show "Content Encoding Error".
hauschu...@gmail.com
Apr 8, 2019, 6:18:39 AM4/8/19
to OWASP ZAP User Group
Ah, okay then!
Thanks for looking into that!
Bill Dietrich
Apr 8, 2019, 6:24:09 AM4/8/19
to OWASP ZAP User Group
I can't get any tool to tell me what tech that server is running, so no idea how to generalize it to some broader case.
Simon Bennetts
Apr 8, 2019, 6:27:09 AM4/8/19
to OWASP ZAP User Group
You could always report the issue to Portswigger and see what they have to say about it?
Simon Bennetts
Apr 8, 2019, 6:27:27 AM4/8/19
to OWASP ZAP User Group
The apparent issue with their server, that is :)
Bill Dietrich
Apr 8, 2019, 6:42:22 AM4/8/19
to OWASP ZAP User Group
Okay, just sent email to Portswigger reporting the encoding issue.
Simon Bennetts
Apr 8, 2019, 6:47:54 AM4/8/19
to OWASP ZAP User Group
Thanks! Be interesting to hear what they say.
Reply all
Reply to author
Forward
0 new messages