Can't start automated ZAP scan with Python script

[lxmi...@gmail.com]

Hi,  I'm relatively new to ZAP in general and totally new to python.  I have ZAP 2.1.0 running on Ubuntu 12.04.  I am using the zap.sh file from an older version to start the ZAP GUI as suggested by Simon because the new zap.sh that does the java version check gives the error that java 7 is required even though I have java 7 installed and running.  I have successfully run quite a few scans from the GUI and now need to get it automated.  I found a test python script "ZAP_Py_Test.py" in another thread on this forum and modified it slightly for my environment.  When I try to run the script I get the message: "ImportError: No module named zapv2".  I tried some variations of 2.1 but kept getting syntax errors.

I made sure I  had python installed.  I installed the Python API using "pip install python-owasp-zap". So I think things are ready??

My questions would be:

Where do I run the python script from?  Do I have to put the script somewhere in the ZAP directory structure?

Is there something I need to modify in the line 'from zapv2 import ZAPv2' since I am using 2.1.0?

Is there any configuration I need to do within ZAP to make sure the Python API is there and being used?

Thanks so much for any assistance you can give.  I am pretty sure this is a matter of missing something obvious but I won't know how obvious until I know what I am missing.  If there is an further information that needs to be included in this post to help please let me know and I will get it.

[thc202]

Hi.

The package "python-owasp-zap" is for ZAP version 1.4.

For ZAP versions 2.x you've to install the package "python-owasp-zap-v2". Once installed it should find the module "zapv2".

Best regards.

[lxmi...@gmail.com]

Thank you so much for the help.  That did indeed get me started but I quickly ran into another error.

The line:

while (int(zap.spider.status)<100): 

was raising the error: TypeError  int() arg must be a string or number not 'NoneType'

It seems this is a variable that ZAP should be providing.  Perhaps I don't have the plugin installed correctly?  Or would I need to purge the 1.4 version of python-owasp-zap?

Another question that came up as well.  I could not get the script to run if I used a specific local URL for the target.  It seems it would only run if the target was 127.0.0.1.  How can I get it to spider and scan the local site I want?

Thanks again for any help you can provide.

[lxmi...@gmail.com]

Just checking in.  I'm still beating my head against this one so if anyone can shed some light on it I would be very grateful.  Thanks.

[thc202]

Hi.

Sorry for taking so long to answer.

That's odd, that means that the API response object doesn't not contain the required property (in this case "status").
Are you sure that the script is connecting to ZAP? If it's connecting to ZAP it seems that ZAP is not answering correctly.

Would you mind check, using a browser, what is the output of the request "http://zap/JSON/spider/view/status/" while proxying through ZAP?
And check what's received in the script using the statement:
print zap.urlopen('http://zap/JSON/spider/view/status/')

Both should return something like:
{"status":"0"}

Let us know what's returning.


Regarding the target URL, are you getting a Python error or ZAP is not doing anything? Are you accessing the target before spidering?
It should be working with any target URL (note that there's an issue [1] that prevents the use of target URLs with an empty path component).


Regarding the first error that you mentioned in the first message (Java version is not correctly detected), it would be really useful, to fix the issue, if you could provide the output of the following command:
java -version


[1] https://code.google.com/p/zaproxy/issues/detail?id=665

Best regards.

[lxmi...@gmail.com]

Hey, no need for an appology.  I know you are most likely pretty busy and just answer questions here as time allows.  I am just very very grateful that you do!  

For your tests with the URL "http://zap/JSON/spider/view/status/" I did get {"status":"0"} from both the browser and the script.

I agree it would be great to get to the bottom of the java issue.  

When I do the java -version I get:

java version "1.7.0_21"

Java(TM) SE Runtime Environment (build 1.7.0_21-b11)

Java HotSpot(TM) SErver VM (build 23.21-b01, mixed mode)

I did some monkeying around with the whole localhost/127.0.0.1/actual target url (since I have multiple local sites that I am testing) and I was able to get the scan to work and complete.  I am using a script based largely on the one shown by Usman Waheed  (thanks to you also Usman!)

The scan completed very quickly in comparison to when I scan through the GUI.  When I do the scan with the GUI I do a manual walk through the related pages on the browser first

and then use the spider.  It does not appear that the spider finds all the pages that a manual walk through does.  Perhaps I could try to handle that by running multiple scans against 

specific urls?  Does ZAP do the same attacks when run via the script or do I have to tell ZAP what attacks to do via the script?

Also is there a way to save the resultant alerts to a file or output them as a report from the script?  That would be very helpful.  Or maybe there is a way to save the session via the script that way

I could at least reopen the session and export the report that way but it would be nicer to have the report fully automated as well.  

Again thank you for all your assistance!

Lee

[thc202]

Hi.

That's strange, it shouldn't be given that Python error if the output is {"status":"0"}.


Is the empty line also part of the output of "java -version"? That would cause the problem.


The option "Maximum depth to crawl" (Options > Spider) might help the spider to find all the pages (or more pages) in the first run. The more entry points (seeds) the spider has the better. Try increase the option and access more "key" pages before running the spider, that should help the spider to find all the pages in the first run. If after the changes the spider doesn't find all the pages you will have to run the spider a second time.

ZAP runs the same active scanner plug-ins that were selected in the GUI.


It's not possible to create a report using the API, you'll have to do it manually (at least for now).
Example script to save the session:

#!/usr/bin/env python

import time
import os
import subprocess
from zapv2 import ZAPv2

# Start ZAP
subprocess.Popen(['/path/to/zap.sh', '-daemon'], stdout=open(os.devnull, 'w'))
time.sleep(10)

zap = ZAPv2()

# Spider and scan...

print 'Session will be saved to: %s' % zap.core.home_directory

zap.core.save_session('Session Name')

# or if you want to save to other directory than "home directory"
#zap.core.save_session('/path/to/dir/Session Name')

# Shutdown ZAP
zap.core.shutdown

Best regards.

[lxmi...@gmail.com]

There are no blank lines in the java version.  I just used a blank line before and after to separate it from the other text and make it easier to read.  Ubuntu has since installed an Oracle Java update and now those lines read:

java version "1.7.0_25"

Java(TM) SE Runtime Environment (build 1.7.0_25-b15)

Java HotSpot(TM) Server VM (build 23.25-b01, mixed mode)

However when I try to run the development version ZAP_D-2013-04-29 I still get the java error.

Perhaps there is something I am missing in my understanding of how the spider works.  I increased the spider depth to 10 but it is still not finding any of the various pages of the site.  (It seems to be finding more since now I get a bunch of listings that are "OUT OF SCOPE".)  For example it finds the main URL which is my local version of our site (lee.beiservices.com) but it does not find (lee.beiservices.com/cgi01, lee.beiservices.com/cgi02 and so on).  Do I need to set up specific target URLs for each script?  (ex. lee.beiservices.com/cgi01?param01&param02&param03).  Perhaps it is because all of the links to these other scripts are in a javascript menu on the main page?  I thought ZAP could handle javascript menus.

When I use the GUI in conjunction with Firefox and manually navigate the site, all of the various cgi scripts (pages) show up under the lee.beiservices.com site but they do not show up for a quick start or a regular spider.  I think I am missing something very crucial and probably very basic.  I will continue to search the user group and documentation for more clues but any additional ideas you may have would be very appreciated.  Thanks.

Lee

[lxmi...@gmail.com]

This is something I just thought of and maybe it's a factor.  When I am manually navigating the local version of the web site I have firefox set to automatically log me in.  My assumption was that when I try to run the automated scan the automatic login would still be in place.  Is that the case?  Could that be causing the issues with the spider?

Lee

[Cosmin Stefan-Dobrin]

Hi,

Unfortunately that is not the case exactly. Unless set otherwise, the Spider is sending requests independent of what is set in Firefox. So, right now, if you want to spider a website from the point of view of an user (authenticated), you should:

1. first navigate to the website and log in (e.g. via automatic log in in Firefox)
   
2. go in ZAP in the HttpSession Tab and identify the Http Session on which you are logged in (should be the only one)
   
3. Mark the Session as "Active" via the right-click context menu
4. Start the Spider.
   

After setting a Session as active, all the subsequent requests sent by/through ZAP are modified to have their headers modified so that they match that particular session. So far, only Cookie based authentication is supported for this process. And all the before mentioned steps are available via API as well so they can be easily scripted.

We are currently working on a new set of features that will allow ZAP users to define Users and Roles in a more consistent and easy way so actions such as "Spidering" (and many others) can be run from the point of view of an User. But this is just in its inception stage, so in the meanwhile you should follow the steps above.

Cosmin

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[thc202]

Hi.

OK. Could you try the attached zap.sh and check what is the version reported?
(the file zap.sh was changed to report the version when it's "not" Java 7 [1]. It's already included in the latest weekly release)


The "normal" spider doesn't execute JavaScript code, so that might be the problem.
You can workaround that "issue" by accessing the pages before starting the spider. The spider will then use the accessed pages.
Something like:

target = 'http://lee.beiservices.com/'

zap.urlopen(target)
# Access other pages
zap.urlopen(target + 'cgi01')
zap.urlopen(target + 'cgi02')
zap.urlopen(target + 'cgi03')

# Give the sites tree a chance to get updated
time.sleep(2)

print 'Spidering target %s' % target
zap.spider.scan(target)

You can use the "Ajax Spider" add-on to spider sites that use (a lot of) JavaScript (not accessible through the ZAP API, though).

When I use the GUI in conjunction with Firefox and manually navigate the site, all of the various cgi scripts (pages) show up under the lee.beiservices.com site but they do not show up for a quick start or a regular spider.

Do not show up in the "Sites" combo box of the spider tab? or it's in other place?


[1] https://code.google.com/p/zaproxy/issues/detail?id=692

Best regards.

[lxmi...@gmail.com]

Ok, seems like I have a few different fronts going here.  Please bear with me.

The new zap.sh did not work with ZAP_D-2013-04-29.  The error message I got was:

zap.sh: 6: zap.sh: [[: not found
Exiting: ZAP requires a minimum of Java 7 to run. Found 1.7.0_25

Adding the zap.openurl(target + 'cgi01') seemed to do something.  The spider seemed to take slightly longer and the scan itself definitely ran a longer time.  However, when I tried to load the saved session back into ZAP I got this error

[lxmi...@gmail.com]

Third time's a charm I guess.  I was finally able to get the script to open ZAP run the spider and scan, save the session and close then successfully opened the resultant session.  

In the saved session noticed that there were not as many warnings as the manual scan so I am going to make sure I add some additional urls, per thc202's suggestion, to help the spider out.  One other interesting thing I did notice was there were no POST requests in the automated scan just GET.  Perhaps this will change when I add those urls.

I want to thank thc202 and Cosmin for all the assistance and other users of this forum for their inspiration and insight.  Now I just need to compare the sessions I do using the python automation against the manual scans and see how the results stack up.  I am thinking that the issue I was having before was not letting ZAP fully write out the session before trying to open it.  I noticed that it takes a bit of time when doing the manual scan to complete the task of writing out the session.  When using the Python script to do these tasks it's even harder to tell when it's done.

The java version error with the updated zap.sh file is still a problem.  See the previous post for the error message.  If anything new develops on that I would be interested to know.  Also if there is some additional information I can supply in that regard please let me know.  I would be glad to help if it brings us closer to a resolution.  Thanks.

Keep up the great work!

Lee

[lxmi...@gmail.com]

Another question:  Is there a status for saving a session similar to the status used for the spider and scan process?  Perhaps that would help me to know when the session is successfully saved if it exists.  Thanks.

[lxmi...@gmail.com]

Another question.  Is there a syntax I can use to add post data to a url and add that to my openurl statements?

[thc202]

Hi.

How are you executing the zap.sh script? It seems that it's not using bash.


Regarding the error while opening the session, can you check the log file to see if there's any error (file zap.log in the ZAP default directory [1])?

One other interesting thing I did notice was there were no POST requests in the automated scan just GET.  Perhaps this will change when I add those urls.

You should ensure that the spider is submitting POST forms (under "Options" > "Spider"). If the forms are not submitted they will not be scanned.
Let us know if the option is already enabled as it might be other problem.

Is there a status for saving a session similar to the status used for the spider and scan process?  Perhaps that would help me to know when the session is successfully saved if it exists.  Thanks

The save of the session is synchronous so it should return only once the session has been saved (or if an error occurs).

Is there a syntax I can use to add post data to a url and add that to my openurl statements?

Yes, you can do POST requests.
Example:

post_data = {'parameter1': 'value1', 'parameter2': 'value2'}

zap.urlopen("http://localhost/", urllib.urlencode(post_data))

(Note that you need to import urllib)


[1] https://code.google.com/p/zaproxy/wiki/FAQconfig

Best regards.

[lxmi...@gmail.com]

How are you executing the zap.sh script? It seems that it's not using bash.

I am not altogether sure what 'bash' refers to.  I am executing zap by opening a terminal, navigating to the /opt/zap/ZAP_D-2013-04-29 (into which I have placed the new zap.sh file you attached) then at the command line as a regular user I type sh zap.sh.  When I use ZAP_2.1.0 with the old zap.sh this opens the GUI normally but when I follow the same steps using ZAP_D-2013-04-29 and the updated zap.sh you sent, this is where I am getting the java error.

 

Regarding the error while opening the session, can you check the log file to see if there's any error (file zap.log in the ZAP default directory [1])?

The java version error only appears in the terminal, there is nothing about it in the log file.

Under Spider in Options, POST forms is already checked so it should be submitting them.  And indeed when I do a scan via the GUI the POST requests appear right alongside the GET requests.  So far it's just the session generated by the Automated Python script that do not have POST requests. I specifically added POST requests per your instructions and still none appear in the automatic scan.

When starting ZAP via the python script using:

   subprocess.Popen(['/opt/zap/ZAP_2.1.0/zap.sh','-daemon'],stdout=open(os.devnull,'w'))

the GUI is not opened (I assume that is because of -daemon).  Are the current settings of ZAP still available in daemon mode or are there additional settings I need to set via the script?  Does the GUI need to be fully opened for the script generated scan to fully work?

What I tried now is opening the GUI then opening Firefox and did a manual walk through of one of the sections of our site on my local machine.  Then I ran an active scan on that.  Then I went through the Sites pane and made a list of all of the urls in this pane and copy them into my python script then ran the script to do this automatically.  The automatic scan generated by the python script did take much longer than any other automatic scan to this point, though not quite as long as the active scan done by via the GUI.  And I saved the session via the script.  When I opened the automatic session and compared it to the GUI generated session.  The GUI generated session showed considerably more Alerts including 7 High Warnings that do not show up on the Python Script generated scan.  I am getting closer but I fear I don't see where the difference between the two scans (using the same urls) comes from.

gratefully yours

[thc202]

Hi.

I am executing zap by opening a terminal, navigating to the /opt/zap/ZAP_D-2013-04-29 (into which I have placed the new zap.sh file you attached) then at the command line as a regular user I type sh zap.sh.  When I use ZAP_2.1.0 with the old zap.sh this opens the GUI normally but when I follow the same steps using ZAP_D-2013-04-29 and the updated zap.sh you sent, this is where I am getting the java error.

OK, that's the problem. The default shell in Ubuntu is dash [1] but the (new) ZAP script is using bash [2] specific features (to compare the Java version).
Try run the script with:
./zap.sh

Note that you might need to set the file executable:
chmod +x zap.sh

Or, alternatively, you can run with bash (instead of default shell):
bash zap.sh

Under Spider in Options, POST forms is already checked so it should be submitting them.  And indeed when I do a scan via the GUI the POST requests appear right alongside the GET requests.  So far it's just the session generated by the Automated Python script that do not have POST requests. I specifically added POST requests per your instructions and still none appear in the automatic scan.

OK, that might be a bug.

When starting ZAP via the python script using:
   subprocess.Popen(['/opt/zap/ZAP_2.1.0/zap.sh','-daemon'],stdout=open(os.devnull,'w'))
the GUI is not opened (I assume that is because of -daemon).

Yes, that's because of the daemon flag (you can remove it to run in GUI mode).

Are the current settings of ZAP still available in daemon mode or are there additional settings I need to set via the script?

Both modes (daemon and GUI) use the same configurations.

Does the GUI need to be fully opened for the script generated scan to fully work?

No, it shouldn't.

What I tried now is opening the GUI then opening Firefox and did a manual walk through of one of the sections of our site on my local machine.  Then I ran an active scan on that.  Then I went through the Sites pane and made a list of all of the urls in this pane and copy them into my python script then ran the script to do this automatically.  The automatic scan generated by the python script did take much longer than any other automatic scan to this point, though not quite as long as the active scan done by via the GUI.  And I saved the session via the script.  When I opened the automatic session and compared it to the GUI generated session.  The GUI generated session showed considerably more Alerts including 7 High Warnings that do not show up on the Python Script generated scan.  I am getting closer but I fear I don't see where the difference between the two scans (using the same urls) comes from.

Could you try running the script with ZAP in GUI mode, this way it will allow to confirm the accessed URLs (especially the POST requests) and the alerts generated without the need to reopen the session.

Let us know if the POST requests are being done by the script or not.


[1] https://en.wikipedia.org/wiki/Debian_Almquist_shell
[2] https://en.wikipedia.org/wiki/Bash_%28Unix_shell%29

Best regards.

[lxmi...@gmail.com]

Thanks again thc202!  Spot on with bash vs dash.  Using bash I was able to open and run the GUI with the new version of zap.sh.  One issue down!

At this point I believe the initial reason I opened this thread has been accomplished.  I want to thank you and the rest for all the help you have given.  Keep up the great work and thanks for the super product!

Lee