Number of Requests for XSS scanner varies for the same Journey

[Eswar]

Hello Team

I have a query related to Zap XSS scanner. A journey in our application has Cross Site Scripting vulnerability. We are testing it with Zap to check if Zap can identify this vulnerability. We are getting mixed results depending upon how we run the active scan from Zap. 

So I was wondering if you could help us by pointing out what could possibly be happening. 

The Application:

The application collects user details over 6-7 pages and in the end displays a summary of the collected information before user can submit. 

For this test we are not submitting the application. We enter all the relevant details and stop from submitting the application so that we could run the Active scan on the same user transaction. There is a XSS vulnerability in the 3rd or 4th page, but the vulnerability itself will be exposed only when the user reaches the summary page. So we are expecting Zap to identify this Persistent XSS. 

The Test:

The journey is same for each of the below test and the user data is also exactly the same. Zap is quit and restarted at the end of every test.
Zap version 2.7.0.
Active Scanner: 32.0.0

Test 1:

- Launch the browser from Zap

- Fill the fields in various pages and view the summary page (no submission done) 

- Without adding the node to any context, perform an Active scan by right clicking on the node

Zap finds the vulnerability and this is what the console log shows. 

019-03-15 17:05:38,897 [Thread-201] INFO  HostProcess - start host http://localhost:9877 | TestPersistentXSSPrime strength MEDIUM threshold MEDIUM
2019-03-15 17:05:43,113 [Thread-201] INFO  HostProcess - completed host/plugin http://localhost:9877 | TestPersistentXSSPrime in 4.216s with 84 message(s) sent and 0 alert(s) raised.
2019-03-15 17:05:43,113 [Thread-201] INFO  HostProcess - start host http://localhost:9877 | TestPersistentXSSSpider strength MEDIUM threshold MEDIUM
2019-03-15 17:05:46,930 [Thread-201] INFO  HostProcess - completed host/plugin http://localhost:9877 | TestPersistentXSSSpider in 3.817s with 51 message(s) sent and 0 alert(s) raised.
2019-03-15 17:05:46,930 [Thread-201] INFO  HostProcess - start host http://localhost:9877 | TestPersistentXSSAttack strength MEDIUM threshold MEDIUM
2019-03-15 17:05:49,491 [Thread-201] INFO  HostProcess - completed host/plugin http://localhost:9877 | TestPersistentXSSAttack in 2.561s with 19 message(s) sent and 1 alert(s) raised.
2019-03-15 17:05:49,491 [Thread-201] INFO  HostProcess - start host http://localhost:9877 | TestSQLInjection strength MEDIUM threshold MEDIUM

Test 2: 

- Launch the browser from Zap
- Fill the fields in various pages and view the summary page (no submission done) 

- Add the node to the Default Context and perform Active Scan

Zap find extra vulnerabilities and this is what the console logs shows. You can see the number of requests for Persistent XSS attack is now different. 

2019-03-15 17:11:00,839 [Thread-176] INFO  HostProcess - start host http://localhost:9877 | TestPersistentXSSPrime strength MEDIUM threshold MEDIUM

2019-03-15 17:11:05,241 [Thread-176] INFO  HostProcess - completed host/plugin http://localhost:9877 | TestPersistentXSSPrime in 4.402s with 84 message(s) sent and 0 alert(s) raised.

2019-03-15 17:11:05,242 [Thread-176] INFO  HostProcess - start host http://localhost:9877 | TestPersistentXSSSpider strength MEDIUM threshold MEDIUM

2019-03-15 17:11:09,277 [Thread-176] INFO  HostProcess - completed host/plugin http://localhost:9877 | TestPersistentXSSSpider in 4.035s with 51 message(s) sent and 0 alert(s) raised.

2019-03-15 17:11:09,277 [Thread-176] INFO  HostProcess - start host http://localhost:9877 | TestPersistentXSSAttack strength MEDIUM threshold MEDIUM

2019-03-15 17:11:12,239 [Thread-176] INFO  HostProcess - completed host/plugin http://localhost:9877 | TestPersistentXSSAttack in 2.962s with 22 message(s) sent and 3 alert(s) raised.

2019-03-15 17:11:12,239 [Thread-176] INFO  HostProcess - start host http://localhost:9877 | TestSQLInjection strength MEDIUM threshold MEDIUM

Test 3:

 - Launch the browser from Zap

 - Fill the fields in various pages and view the summary page (no submission done) 

 - Add the node to the Default Context 

 - Set Authentication to Script Authentication and load a Zest script recorded previously to authenticate this application.

 - Configure the user and set the regex pattern identifier in the logged out response message (As we are performing the active scan immediately we don't expect Zap to have to login again as the session will be still active)
 - Run Active scan
Zap does not find the vulnerability and there are no requests sent for persistentXSSAttack:

2019-03-15 17:27:12,609 [Thread-176] INFO  HostProcess - completed host/plugin http://localhost:9877 | TestPersistentXSSPrime in 4.231s with 84 message(s) sent and 0 alert(s) raised.

2019-03-15 17:27:12,610 [Thread-176] INFO  HostProcess - start host http://localhost:9877 | TestPersistentXSSSpider strength MEDIUM threshold MEDIUM

2019-03-15 17:27:16,417 [Thread-176] INFO  HostProcess - completed host/plugin http://localhost:9877 | TestPersistentXSSSpider in 3.808s with 51 message(s) sent and 0 alert(s) raised.

2019-03-15 17:27:16,417 [Thread-176] INFO  HostProcess - start host http://localhost:9877 | TestPersistentXSSAttack strength MEDIUM threshold MEDIUM

2019-03-15 17:27:19,175 [Thread-176] INFO  HostProcess - completed host/plugin http://localhost:9877 | TestPersistentXSSAttack in 2.758s with 0 message(s) sent and 0 alert(s) raised.

2019-03-15 17:27:19,175 [Thread-176] INFO  HostProcess - start host http://localhost:9877 | TestSQLInjection strength MEDIUM threshold MEDIUM

This happens consistently when we include the Zest Authentication Script. 

So the question we would like to understand is:
 - Why Zap sends different number requests of requests for Test1 and Test 2 and does not send anything at all for Test 3
 - Does adding authentication script interferes XSS scanner. 



Any help on this would be greatly appreciated. 

Thanks

Eswar

[hauschu...@gmail.com]

So this is two different issues:

1. difference in messages sent (and results) based on context

2. difference in messages sent based on auth script

For #1, I would want to see the actual payloads sent in both cases. Ie, what were those 3 extra messages? Were the payloads all delivered to the same input vectors, or were they different? Then compare that to exactly how the context is set and see how that may be informing or restricting the message locations. 

For #2, I would do something similar, except this time also examine all of the network traffic to ensure it is logged in or is otherwise maintaining state for all of those scan requests. It is easy to think we have it set correctly or to otherwise assume it will behave one way, when in reality it doesn't see itself as logged in due to some configuration issue and will behave unexpectedly.

That's where I would start anyway!

[Eswar]

Thank you HauschulzPeter.

Apart from looking at the UI, console logs, is there a way to identify which request is being sent by which scanner? or is there a better way to map the requests sent by a particular scanner (in this case XSS) so that I can see the difference in requests for each run?

Thanks

Eswar

[hauschu...@gmail.com]

Not that I know of immediately....

I suppose you could bring up the application traffic logs and determine which is which (first 84 messages are scanner 1, next 51 are scanner 2, etc etc)

I don't remember if this applies to requests sent, but sometimes putting the ZAP log4j.properties file to DEBUG is helpful too (and then looking at zap.log)

[Eswar]

Thanks. I will give that a try. 

Regards

Eswar

[Eswar]

Hi

We did some more investigation on this issue. We think we may know why the XSS alerts are not consistent for our application. We would like to run it through you and hear your thoughts on this. 

As mentioned earlier, our application gathers user input across multiple pages and shows the summary of all the collected user information at the end. One of these pages has a field that has XSS vulnerability. But this will be exposed only when the user reaches the summary page. What we have noticed is Zap reports a vulnerability only if XSS script is injected in the last request (for an endpoint/page). If Zap injects a XSS script in request 2, and subsequently injects something else in request 3 in the same page, the request 3 will be overridden by request 2 when Zap eventually views the summary page. Which means any vulnerability that could have been exposed via request 2 may not be identified. I will try to explain this with an example below:

Endpoint POST /destination takes few params. In this request, the phone param has XSS vulnerability. The response from this endpoint is another page that takes more parameters. The values entered in /Destination page will show up eventually in the summary page later. During Active Scan I can see Zap injecting values into each of these fields one request at a time.

Request ID: 2786 has the below params:

address.line1=Building&address.line2=street&address.town=Town&address.county=County&address.postCode=SE1+0NB&phone=%22%3E%3Cscript%3Ealert%281%29%3B%3C%2Fscript%3E

Request ID: 2788 with the params:

address.line1=Building&address.line2=street&address.town=Town&address.county=County&address.postCode=SE1+0NB&phone=%22+onMouseOver%3D%22alert%281%29%3B

In the above run, request ID 2788 is the last request made for POST /destination endpoint. So eventually when ZAP views the summary page, the page will have the value 'onMouseOver' value in the summary page instead of 'script Alert' injected in request 2786. Request 2788 does not trigger a XSS alert in our application. So in the above sequence of requests, Zap reports no XSS alert found. 

Request 2786 exposes XSS vulnerability in our application. So, in a different run, when this was the last POST /destination request made by Zap, then subsequently when Zap views the summary page, the alert is found and zap reports the vulnerability. 

It appears that the summary page is accessed by ZAP only after all the iteration of injection in previous pages have been completed. Which mean ZAP potentially checks persisted XSS only for the last injected request for a particular endpoint instead of every request. 

Could you please advice if our understanding of the Zap behaviour is correct? If it is, then could you let us know if there is a way we can force Zap to overcome this?

Thanks
Eswar

[kingthorin+owaspzap]

I think there are two things you should consider here. (Well at least two....might be others I haven't thought of yet ;)    )

1) Check out the Sequence scanning addon available in the marketplace. (https://github.com/zaproxy/zap-extensions/wiki/HelpAddonsSequenceSequence)

2) If you're using the persistent XSS scanner you might be encountering: https://github.com/zaproxy/zaproxy/issues/4656

[hauschu...@gmail.com]

Interesting!

From my limited understanding/observations of the XSS and other scan rules, that sounds consistent. That is, ZAP will insert a payload in a request and immediately check the accompanying response....as far as I know it is not feasible for it to check every response it ever receives to see if any payload it has ever sent is showing up in a response. AKA, ZAP won't check message 15 for malicious input from messages 1-13...only 14. Otherwise, I imagine it would have to re-check every single response for every single input made prior, which sounds like a recipe for exponential problems.

ie 

send #1, receive and check #2 for #1

send #3, receive and check #4 for #1 and #3

send #5, receive and check #6 for #1 and #3 and #5

etc etc etc

I will think on ways around that...i haven't encountered any like that where it overwrites a field multiple times or the reflection is delayed! Obviously Kingthorin has good ideas too! 

[Eswar]

Thank you both. 

I gave the Sequence AddOn a quick try and it looks promising. It did find the XSS vulnerability. And I can see it executing in the sequence. 

I will play with it again tomorrow and come back to you for any queries. 

Thanks a lot again for you support. 

Regards

Eswar

[Eswar]

Sorry guys. I am back again. I might have too eagerly said that the sequencer found the vulnerability. I have played around it today and I can confirm that the sequencer does run the tests in Sequence. But what I see it not doing is, inject any XSS scripts at all. I did multiple runs and went through the POST requests to see if Zap is injecting XSS code, but I couldn't see anything at all when I run via sequencer. I was able to see it injecting XSS code when I run without sequencer. In the progress window, I can see requests for Prime and Spider, but not the XSS scanner itself. 

This is what I have done:

1.  I created a new sequence script
2.  Recorded the journey
3.  Stopped recording and saved the Script
4.  Ran the script from the right side pane to ensure the script works. This also created the tree under sites tab. 
5.  Right click on the zest script, under the script tab, and triggered 'Active Scan Sequence'

 I also tried triggering Active scan from the sites recorded in step 4 , by including the relevant Sequence script in the Advanced section of Active scan. Both of them does not appear to be injecting the code in the form fields to expose XSS vulnerability. 

Any thoughts why would this happen? Is there any additional set up required for the sequencer? Thanks again for your time on this.

Regards
Eswar

[Eswar]

So we did some more investigation on this and wanted to share our findings with you to hear your thoughts. 

We can now see Zap injecting the XSS script in the relevant field. But the script is not encoded. 

For example:

name='"<script>alert(1);</script> is sent in the name field. 

Instead of:

name=%27%22%3Cscript%3Ealert%281%29%3B%3C/script%3E

When it is not encoded, the script is not injected correctly in the HTML. The HTML looks like below. 

 <div class="cya-answer">       
'"<script>alert(1)

</div>

When we manually resend the request with the encoded script, we could see the HTML properly formed and an Alert pops up. 

Have you come across something similar before? Would you know why the script is not encoded? I have seen it being encoded in the past. Not sure if I am missing a plugin or if including sequencer is causing this. 

Any thoughts please? 

Regards
Eswar

[Eswar]

Found this issue in Zest Forum and wondering if this what preventing URL encoding when we use Sequence Plugin. 

https://groups.google.com/forum/#!topic/mozilla-zest/Eqsbwj6xN4w

Regards

Eswar

[hauschu...@gmail.com]

Good find! To my eyes anyway, that looks like that could be it!