Excluding URL's when proxying

[Ismail Lambat]

Hi,

I have set my Selenium Browser to proxy through the same host and port zap is configured with. When I run my selenium tests, I can see the URL's being logged within ZAP.

The issue I have is my tests are going to external websites, not linked to the one I'm testing. So I want to exclude all URL's besides any that start with my website URL whilst ZAP is proxying.

The reason I want to do this is so no external links are displayed on my report if they have any security vulnerabilities as it's out of our scope.

I've looked at the API and can see excludeFromScope etc, but that's further down the line during ascan or spider. I need the URL's excluding even before any of those are executed.

Can this be done? and if so, any pointers on how?

Thanks,

Ismail.

[Ismail Lambat]

On the back of this, I've figured out I can use the API by calling core.excludeFromProxy.

However, I want to exclude all URL's but my website. So instead of manually adding the external URL's (which could constantly change), I've tried to put regex in to exclude everything but my website.

E.g - my website - www.mywebsite.co.uk

_zapClient.core.excludeFromProxy(_zapApiKey, "[^www.mywebsite.co.uk]");

Zap doesn't seem to like that and when im running the tests, it's excluding my website URL and capturing the rest.

What regex do I need to supply to ZAP to exclude everything but my URL.

Thanks.

[kingthorin+owaspzap]

There are two problems with what you've done:

1. You've told ZAP to exclude your site... not sure why that would work.
2. You've anchored the URL to www which can't work, since all URLs have a scheme...

You probably really want a Regex with negative look ahead, something like:

^(?!http(s)?:\/\/www\.mywebsite\.co\.uk)

https://regex101.com/r/u0vgcK/1

[Ismail Lambat]

Hi,

I've tried this Regex statement with my website URL, and within the Regex101 website, it is behaving correctly. So all URL's that do not match my website are being selected.

However, when I put the same Regex into ZAP, it is not working and it is still displaying / proxying all the URL's. I've tried adding it to the Session Properties configuration with Zap under all the sections (Exclude from Proxy / Scanner / Spider) but it still doesn't exclude other URL's.

Tried it through the API with:

            _zapClient.core.excludeFromProxy(_zapApiKey, "^(?!http(s)?://my\\.testwebsite\\.co\\.uk)");  C# so escaping the characters.

Any idea on why it's working on Regex websites but not within Zap?

Could it be a bug in Zap?

Thanks.

[kingthorin+owaspzap]

I haven't had a chance to test it. I put the regex together yesterday purely based on regex101.

I did notice afterward that the brackets on the s in the protocol (scheme) aren't needed. In c# do you need to escape the slashes separating scheme from host?

The other option which might be easier and more intuitive (vs. a negative exclusion) would be to setup a context for your app (inclusively). Then you can:

• Set the passive scanner to "Scan only messages in scope: "
• Run spider/active scan against the context.
  

[thc...@gmail.com]

The regex just needs a tweak to match the URLs in ZAP:
^(?!https?://my\\.testwebsite\\.co\\.uk).*

Best regards.

On 17/10/17 13:19, kingthorin+owaspzap wrote:
> I haven't had a chance to test it. I put the regex together yesterday
> purely based on regex101.
>
> I did notice afterward that the brackets on the s in the protocol (scheme)
> aren't needed. In c# do you need to escape the slashes separating scheme
> from host?
>
> The other option which might be easier and more intuitive (vs. a negative
> exclusion) would be to setup a context for your app (inclusively). Then you
> can:
>

> - Set the passive scanner to "Scan only messages in scope: "
> - Run spider/active scan against the context.
>
>

[Ismail Lambat]

Just updating this one for anyone that is stuck.

The below Regex worked for me and what I needed it for.

^(?:(?!https://my-testwebsite.co.uk).)+$

Excludes my website url above and select all other URL's. Then I used this regex to put into excludeFromProxy

On Monday, 16 October 2017 13:25:46 UTC+1, Ismail Lambat wrote:

[Vadim Z]

Hi,

I'm trying to exclude every URL from proxy that is not *.test.com or *.test2.net

I need to see only requests to *.test.com, test.com, *.test2.net, test2.net

I'm doing this because everything else is out of scope.

The regex that works is ^(?:(?!https://test.com).)+$, mentioned in the thread, but it doesn't handle wildcards.

In Global Exclude URL (beta) I found an example: 

^https?://([^/])*\.?(getfoxyproxy\.org|getfirebug\.com|noscript\.net) (which doesn't work when enabled)

I modified it to:

^https?://([^/])*\.?(test\.com|test2\.net)

And added it to Proxy Exclude. I still got unwanted requests listed in Proxy.

What am I doing wrong?

Thanks.

[thc...@gmail.com]

Hi.

The following regex includes just the mentioned sites:
^(?!https://.*(?:test\.com|test2\.net)).*


Best regards.

[Vadim Z]

Thanks for your reply!

I've tried your regex for both "Exclude from proxy" and "Global exclude URL (Beta)" features, but I still can see requests to OOS APIs and ads services.

Regards.

[thc...@gmail.com]

Which ZAP version are you using?

Do you see the other requests while proxying? or using other tools (e.g.
AJAX Spider)?

Could you provide one of the URLs that were not excluded?


Best regards.

[Linesh Raghawan]

Hei

I am able to run my selenium tests in docker using ZAP proxy, but I want to exclude some of URL. 

How did you achieve that, the solution has mentioned above is this for GUI mode only?

-Linesh

[thc...@gmail.com]

Hi.

No, it can also be set through the API [1] using core [2] endpoint
excludeFromProxy.


[1] https://github.com/zaproxy/zaproxy/wiki/ApiDetails
[2] https://github.com/zaproxy/zaproxy/wiki/ApiGen_core

Best regards.

[seniorqaau...@gmail.com]

What is the best regex for including in proxy multiple endpoints? I use this one, but it works only for one endpoint          ^(?:(?!endpoint.*).$

понедельник, 16 октября 2017 г., 8:25:46 UTC-4 пользователь Ismail Lambat написал:

[hauschu...@gmail.com]

play around here, and pay special attention to using the |   (pipe) as an 'or' statement to combine multiple filters into a single regex

https://regex101.com/