Spidering and Scanning from imported context fails in replacing ZAP username and password

[Thierry Laurion]

Hi,

I'm using Zaproxy 2.5 and trying to automate spidering and active scan through context importation.
The spidering is done with "ZAP" username and password, even though imported context specifies right login and password, forced user, includes and excludes, login and logoff context and right login form replacements and method.

It was written elsewhere that this behavior would be normal if ZAP wouldn't know how to replace defined username and password, but that context was exported from a working session.
So I suppose I'm not respecting the right steps for it to use the context, username and passwords and login with the right credentials, even though I followed the guide

I tried a bunch of stuff, including forcing the user manually, using zap.spider.scan_as_user, etc, which are commented out in the provided code, which was based on this post.

Any hints of what i'm not doing right?

Thanks a bunch!

Scanner.py:

#!/usr/bin/env python
import time
from pprint import pprint
from zapv2 import ZAPv2
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select

target = 'https://172.20.120.25:1443/'
zapHostIp = '172.20.20.235'

zap = ZAPv2(proxies={'http': 'http://' + zapHostIp + ':8090', 'https': 'http://' + zapHostIp + ':8090'})
apikey = 'apikey!'


#new session
zap.core.new_session(apikey = apikey)

#print 'Accessing target %s' % target
#zap.urlopen(target)

# Give the sites tree a chance to get updated
#time.sleep(2)

# ConnectID.importContext
print "Importing Context"
contextId = zap.context.import_context('/root/.ZAP/contexts/PF2.context', apikey = apikey)
print "contextID = " + contextId
#zap.context.set_context_in_scope(contextId, True, apikey = apikey)
zap.context.set_context_in_scope('PF2', True, apikey = apikey)
#zap.context.context('PF2')

time.sleep(5)
userId = zap.forcedUser.get_forced_user(contextId)
print "Forced userId = " + userId
print "isForcedUserModeEnabled = " + zap.forcedUser.is_forced_user_mode_enabled
zap.forcedUser.set_forced_user_mode_enabled(True, apikey= apikey)
print "isForcedUserModeEnabled = " + zap.forcedUser.is_forced_user_mode_enabled
time.sleep(5)

#TODO: Update scripts before going further
print 'Accessing target %s' % target
# try have a unique enough session...
zap.urlopen(target)

# Give the sites tree a chance to get updated
time.sleep(2)

print 'Spidering target %s' % target
zap.spider.set_option_max_depth(10, apikey=apikey)
#zap.spider.scan_as_user (contextId, userId, target, recurse = True, apikey = apikey)
zap.spider.scan(url=target, contextname='PF2', apikey=apikey)
#zap.spider.scan_as_user (contextId, userId, target, apikey = apikey)
# Give the Spider a chance to start
time.sleep(5)
while (int(zap.spider.status()) < 100):
    print 'Spider progress %: ' + zap.spider.status()
    time.sleep(2)

print 'Spider completed'
# Give the passive scanner a chance to finish
time.sleep(10)

print 'Scanning target %s' % target
#zap.ascan.scan_as_user(target, 2, 2, recurse=True)
#zap.ascan.scan(target, recurse=True, inscopeonly=True)
zap.ascan.scan(target, recurse=True, apikey=apikey)

while (int(zap.ascan.status()) < 100):
    print 'Scan progress %: ' + zap.ascan.status()
    time.sleep(5)

print 'Scan completed'

# Report the results

print 'Hosts: ' + ', '.join(zap.core.hosts)
print 'Alerts: '
pprint (zap.core.alerts())

PF2.context:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
<context>
<name>PF2</name>
<desc/>
<inscope>true</inscope>
<incregexes>https://172.20.120.25:1443.*</incregexes>
<incregexes>.*172.20.120.25:1443.*</incregexes>
<excregexes>https://172.20.120.25:1443/admin/logout</excregexes>
<tech>
<include>Db.MySQL</include>
<include>Db.Oracle</include>
<include>Db.SQLite</include>
<include>Language.C</include>
<include>Language.XML</include>
<include>OS.Linux</include>
<include>SCM.Git</include>
<include>WS.Apache</include>
<exclude>Db</exclude>
<exclude>Db.Firebird</exclude>
<exclude>Db.HypersonicSQL</exclude>
<exclude>Db.IBM DB2</exclude>
<exclude>Db.Microsoft Access</exclude>
<exclude>Db.Microsoft SQL Server</exclude>
<exclude>Db.PostgreSQL</exclude>
<exclude>Db.SAP MaxDB</exclude>
<exclude>Db.Sybase</exclude>
<exclude>Language</exclude>
<exclude>Language.ASP</exclude>
<exclude>Language.PHP</exclude>
<exclude>OS</exclude>
<exclude>OS.MacOS</exclude>
<exclude>OS.Windows</exclude>
<exclude>SCM</exclude>
<exclude>SCM.SVN</exclude>
<exclude>WS</exclude>
<exclude>WS.IIS</exclude>
<exclude>WS.Tomcat</exclude>
</tech>
<urlparser>
<class>org.zaproxy.zap.model.StandardParameterParser</class>
<config>{"kvps":"&amp;","kvs":"=","struct":[]}</config>
</urlparser>
<postparser>
<class>org.zaproxy.zap.model.StandardParameterParser</class>
<config>{"kvps":"&amp;","kvs":"=","struct":[]}</config>
</postparser>
<authentication>
<type>2</type>
<loggedin>\Q Log out\E</loggedin>
<loggedout>\QYour session has expired.\E</loggedout>
<form>
<loginurl>https://172.20.120.25:1443/admin/login</loginurl>
<loginbody>redirect_url=https%3A%2F%2F172.20.120.25%3A1443%2Fadmin%2Fconfiguration&amp;username={%username%}&amp;password={%password%}</loginbody>
</form>
</authentication>
<users>
<user>1;true;YWRtaW4=&#13;
;2;YWRtaW4=&#13;
~dGVzdA==&#13;
~</user>
</users>
<forceduser>1</forceduser>
<session>
<type>0</type>
</session>
<authorization>
<type>0</type>
<basic>
<header/>
<body/>
<logic>AND</logic>
<code>401</code>
</basic>
</authorization>
</context>
</configuration>

[Wendell Fry]

Thierry, you're way ahead of me, and I'm trying to do something very similar (I think), but I'm not deeply steeped in programming experience, so this looks great before needed tweaks.  Where are you invoking this script?  Are you placing it in the scripts folder and then calling it from a command line, or are you calling it from the API, or within a running GUI of ZAP?  Sorry for so many questions, but you look like you're right in line with doing things I also need to do!  Thanks!

Wendell Fry (Wende...@dsainc.com)

[Simon Bennetts]

I always start by getting it working via the UI.
If you cant get it working that way then you have no chance with the API ;)
In particular see the troubleshooting guide here: https://github.com/zaproxy/zaproxy/wiki/FAQformauth#diagnosing-problems

Once I've got it working via the UI I start scripting it, but I run the scripts against ZAP in UI mode instead of daemon mode - they work just the same but you can see whats going on much easier.
You can mix and match them, eg use your script to load the context and set it up and then invoke the spider from the UI. That way you can easily look at the spider requests and see what ZAP is sending. You can also check that everything is set up as per the troubleshooting guide above.

Does that help at all?

Simon

[Thierry Laurion]

Hi,

Answers inline! :)

Le mercredi 15 février 2017 04:34:57 UTC-5, Simon Bennetts a écrit :

I always start by getting it working via the UI.

If you cant get it working that way then you have no chance with the API ;)

PF2.context was exported after having a working spidering and active scan session. So we know that the login and logoff conditions, include url and exclude url, username and password replacement etc are working from what was given previously.

In particular see the troubleshooting guide here: https://github.com/zaproxy/zaproxy/wiki/FAQformauth#diagnosing-problems

I'll redo this afternoon, just to be sure I haven't skipped something, but after having quickly read, that talks about forcing user from the GUI (which is done in the previous script provided.) Without success.

Once I've got it working via the UI I start scripting it, but I run the scripts against ZAP in UI mode instead of daemon mode - they work just the same but you can see whats going on much easier.
You can mix and match them, eg use your script to load the context and set it up and then invoke the spider from the UI. That way you can easily look at the spider requests and see what ZAP is sending. You can also check that everything is set up as per the troubleshooting guide above.

Does that help at all?

Thanks for your help. Does anyone use the context import calls from the API? Am i supposed to do other steps then the ones I do that are done through the gui that should be replicated through the API? A quick review of the steps done in my ocde would be greatly appreciated.
Thierry

[Thierry Laurion]

Le mardi 14 février 2017 16:41:22 UTC-5, Wendell Fry a écrit :

Thierry, you're way ahead of me, and I'm trying to do something very similar (I think), but I'm not deeply steeped in programming experience, so this looks great before needed tweaks.  Where are you invoking this script?

PF2.context should be placed under '/root/.ZAP/contexts/PF2.context'
chmod +x Scanner.py
./Scanner.py

 Are you placing it in the scripts folder and then calling it from a command line, or are you calling it from the API, or within a running GUI of ZAP?  Sorry for so many questions, but you look like you're right in line with doing things I also need to do!  Thanks!

Look at what Scanner.py does then. Play around and we might help each other, but playing with it is required!
The context file defines everything that should be needed from the scanner and spider to work. That should be adapted to your need, but the script itself just imports a working context. As Simon says, you are better to make it work from the GUI first, to make sure all your login/logout detection works, your exlusion are set (you do not want to crawl and test your logout url during a scan!) and authentication settings are set correctly before cgoing furter with the api. The API is there to automatise something the should work manually first.

So you should follow these steps religiously: https://github.com/zaproxy/zaproxy/wiki/FAQformauth, both for the gui and api paths of doing things.

Wendell Fry (Wende...@dsainc.com)

Have fun!
Thierry

[Thierry Laurion]

Corrections inline... Typos were involved, (url=target -> target) and realized after the fact that exclusions from the spider and scanner needs to be added from the API since they are not included in the profile.
In fact, anything involving urls should be called from the API, since it can be scripted out.

It would be nice if the API documentation would remind the developer that each call to the spider required passing the apikey, else those calls are neglicted.
Adapt to your needs! Maybe now is a good moment to update the example code being provided here?

Thanks!

Thierry

:

zap.spider.scan(target, contextname='PF2', apikey=apikey)

PF2.context:

[Wendell Fry]

Simon, 

That helps.  Right now, I use HTTP/NTLM Authentication, and use the Session Management setting of HTTP Authentication in the context, and then set up my user.  Everything works well in the GUI that way, and I'm learning to do it in the API in a limited fashion.  Your suggestion using the script to set up my context will be perfect, but I need to be pointed to a script I can modify that will do that.  Since I'm in a closed network on the side I use ZAP with, I have to manually type copy the scripts, and then modify them.  I've figured out I have to put the script in the proper scripts folder for it to show up in the scripts tab in the UI which I will work on getting to run before I can use them in the API.  I've gone through PYTHON training on Pluralsight's training website, which is great, so I can modify scripts fine.  I realize I've copied scripts I didn't need to, because I don't have wavsep installed.  Could you point me to a script or scripts I can copy and modify that will load my context, and that you recommend to help me spider and attack specific nodes in our code repository?  

I have to prove to my boss that ZAP will work in a timely fashion to give a before and after of a specified node in our repository (for now - later we will have developers do it as they go) .  We don't have to do the whole repository each time.  I hope what I"m asking is clear.  You've been a great help.  I wish I could hire you (don't have the authorization for it, and not even sure you could anyway) for a couple of hours to talk through just a few guiding points like I mentioned above.  I know jumping into a post is hard because you have to read so many and respond.  Your help is great!

Thanks!

Wendell