ZAP Integration with Jenkins

[Osama Masood]

Hey everyone,

I hope you're all doing well and in the best of health! 

Brief Intro about me: I'm a young (read noob) and inexperienced (probably, when compared to most of you) cybersecurity enthusiast, but I make up for that with my passion and enthusiasm for this subject. I'm almost done with my Comp Sci/Security Degree, and I've already got an amazing job in my field and I discovered ZAP about a week and a half ago, and I'm aiming to integrate it to my company's nightly build to impress some important people :)

The important details FYI:

• My Linux server details: 
  • CentOS release 6.8 (Final)
  • LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
  • CentOS release 6.8 (Final)
  • CentOS release 6.8 (Final)
• Java version : 1.8.0_92
  
• Jenkins version : Jenkins ver. 2.32.2
  
• ZAP version : I have the latest stable version (2.5.0) and the latest weekly release (ZAP_D-2017-03-21)
  
• ZAP Jenkins Plugin version: Latest version, 1.0.8
  
  
• Firefox version (if running AJAX Spider or a Selenium build) : Not using selenium yet, so it's N/A
  
• Selenium (if applicable) : N/A
  
  
• Upload copies of the zap.log files and a copy of the console output of the Jenkins log to pastebin = my ZAP log folder seems to be empty, I think it's because no logs can be created since my server can't find the path to ZAP. Pastebin URL: https://pastebin.mozilla.org/8983536 (I highlighted most of the important stuff to save time, I think the issue is with the -installdir command or the $ZAPROXY_HOME variable)  
• Jenkins is always running on a master, is ZAP running on the master as well or on a separate slave machine? : Not sure, I installed ZAP on my company's main VM which is used for the nightly testing, so I guess it's the Master. 
• Relevant Jenkins Job Configurations sanitized screenshots: Attached in the post

My Objective: Is to integrate ZAP into jenkins, and be able to produce at least one generated report of a successful vulnerability scan or a spider scan.

My problems:

1. I lack experience and knowledge, but thanks to google I'm able to keep moving on
   
2. I'm not sure which ZAP to use, the official release or the latest weekly release (of 21st March 2017)
   
3. Since my server is headless, I am not sure if I need to setup selenium to do the automated vulnerability testing of my company's software. 
4. The ZAP path seems to be causing issues for me, and I'm not sure if my build will work even after I fix it

What I have done so far:

• Installed jenkins, installed ZAP, played around with jenkins and followed the official guide: https://wiki.jenkins-ci.org/display/JENKINS/zap+plugin
• I looked up ZAPR, ZAP REST API and ZAPROXY, and I don't know which of these tools are relevant to me or what they do exactly.
• I'm still learning linux, so I spent some time learning commands and revising other stuff,etc

Thank you very much for reading my post, I hope to hear your valuable input :)

Best Regards,

Osama Masood.

[JordanGS]

Hi, it's a pleasure to meet you. My name is Goran :)

I'll try to answer your questions one by one.

Documentation an be found on the wiki page, here.

The Jenkins Plugin does not support the stable release 2.5.0, only supports weekly release 2016-09-05 or later to run. See here. It will support 2.6.0 Stable release in the coming days as the standard.

Firefox version: See supported versions, link provided on the wiki as well as a link to the firefox download page.

Your environment variable ZAPROXY_HOME refers to the installation location of ZAP

The Jenkins variable ZAP SETTINGS DIRECTORY refers to the ZAP Home directory, not the installation directory as you provided. Not sure what the default would be on a linux box, on windows it's C:\Users\<username>\OWASP ZAP

See here for installation support. See configuration for ZAP Settings here (see heading: ZAP Settings, there is a link to FAQconfig).

I believe that should answer 1, 2 and 4.

--

In regards to 3, if ZAP is installed on a slave, it runs in daemon mode (headless) so that should not cause any issues.

-- As mentioned in the official guide you linked above, you want to use the links in the guide to find the right version. Currently that being the latest weekly release.

--

I hope that answers most of your questions, if you have any other questions, feel free to ask :)

Cheers, Goran.

[Lakshmi narayana]

Nice to meet you Osama Masood.

I liked the way you ask question here.

May be I and everyone learn from you - how to ask question in these professional forums.

Regards,

LnT

[Osama Masood]

Hey Goran, nice to meet you too!

Yes, that's the documentation i've been following, and that's the only reason I've progressed this far lol.

I installed both the latest weekly release, and the stable 2.5 version of ZAP, and I tried running both of them on jenkins, but I've been using the weekly release more frequently though

Yes, I did setup the Environment Variable on linux, when i input the command env on terminal, i get this: ZAPROXY_HOME=/opt/ZAP/ZAP_D-2017-03-21 , which is the location of the place where I installed the latest weekly release.

Ok, so I have to research look into the home directory of linux and see what's wrong over there and how to fix this issue. So is the ZAP PATH NOT FOUND error during the build is triggered by this? Because the ZAP SETTINGS PATH variable on jenkins is pointing to the place where I installed the latest weekly release of ZAP.

I can easily run ZAP in headless mode, using the command "./zap.sh -daemon -port 8045" and this is the log i receive: https://pastebin.com/y7UP2QLB

The next steps for me is to run ZAP on jenkins and generate a report, but since I've been following the wiki guide (for windows) i'm not sure what I may have missed or what I have done wrong.

[Osama Masood]

Nice to meet you too LnT,

Thank you very much! I'm very interested in communicating effectively and efficiently, and I know that I despise to be greeted with a wall of text on my screen :)

[JordanGS]

Yes, you can from the the machine where zap is installed because it assumes directory structure. Jenkins doesn't assume anything about the slave machine you're running zap on. It doesn't know where ZAP is installed or which directory it uses for it's home dir.

So for your Installation directory: ZAPROXY_HOME=/opt/ZAP/ZAP_D-2017-03-21 would be correct.

The ZAP wiki says the following: What is the default directory that ZAP uses? Linux is ~/.ZAP

So what i would do is terminal into the linux box, and run the following

$ cd ~/.ZAP
$ pwd

copy that absolute path that you get from pwd and use that as the path for ZAP Settings Directory. Remember that for weekly releases "_D" is appended to the above directory.

So it might be

$ cd ~/.ZAP_D
$ pwd

Remember that 2.5.0 will NOT work with Jenkins Plugin. It's NOT supported.

-- To summarize, when you run ZAP locally from the machine it's installed on, it assumes folders and installation paths. But when you want to run it from a master (jenkins) which has no knowledge of the type of slave it is or the directory structure. We have to specify those.

If you have not already done so, follow the changes outline here as well. You have to modify the .bat (windows) and the .sh (linux) so that they include the environment variable ZAPROXY_HOME as part of the path to the .jar file.

P.S. Very well structure question :D Made it very simple to answer +1

[JordanGS]

Any updates on this issue?

[Osama Masood]

Hi Goran,

Apologies for the late reply, I'll be working from Mondays to Wednesdays for April, will be back full time in May :)

Alright so here are the updates:

• After hours of struggling with the source of my problems, I managed to get the ZAP plugin for jenkins to build successfully! The "PATH IS MISSING" error was due to the fact that the Environment variable I set using SSH was only for that session! So to have the Environment variable permanently, I added it to the etc/environment file, and now it should load for every user at every boot up :D
  
• But after adding the real URL's to be tested in the "Session Properties" in Jenkins, my build failed because of this error:
  

• [ZAP Jenkins Plugin] CONFIGURE RUN COMMANDS for [ /opt/ZAP/ZAP_D-2017-03-21/zap.sh ]
  [ZAP Jenkins Plugin] EXECUTE LAUNCH COMMAND
  [ZAP_D-2017-03-21] $ /opt/ZAP/ZAP_D-2017-03-21/zap.sh -daemon -host 127.0.0.1 -port 8055 -config api.key=ZAPROXY-PLUGIN -dir /opt/ZAP/ZAP_D-2017-03-21 -installdir $ZAPROXY_HOME
  
  [ZAP Jenkins Plugin] INITIALIZATION [ START ]
  Found Java version 1.8.0_92
  Available memory: 3832 MB
  Setting jvm heap size: -Xmx958m
  Unable to initialize home directory! /opt/ZAP/ZAP_D-2017-03-21/log4j.properties (Permission denied)
  java.io.FileNotFoundException: /opt/ZAP/ZAP_D-2017-03-21/log4j.properties (Permission denied)
  	at java.io.FileOutputStream.open0(Native Method)
  	at java.io.FileOutputStream.open(FileOutputStream.java:270)
  	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
  	at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
  	at org.parosproxy.paros.model.FileCopier.copyLegacy(FileCopier.java:52)
  	at org.parosproxy.paros.model.FileCopier.copy(FileCopier.java:44)
  	at org.parosproxy.paros.Constant.initializeFilesAndDirectories(Constant.java:418)
  	at org.parosproxy.paros.Constant.<init>(Constant.java:319)
  	at org.parosproxy.paros.Constant.createInstance(Constant.java:903)
  	at org.parosproxy.paros.Constant.getInstance(Constant.java:895)
  	at org.zaproxy.zap.ZapBootstrap.start(ZapBootstrap.java:76)
  	at org.zaproxy.zap.DaemonBootstrap.start(DaemonBootstrap.java:48)
  	at org.zaproxy.zap.ZAP.main(ZAP.java:98)
  ERROR: java.net.ConnectException: Connection refused
  	at java.net.PlainSocketImpl.socketConnect(Native Method)
  	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
  	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  	at java.net.Socket.connect(Socket.java:589)
  	at org.jenkinsci.plugins.zap.ZAPDriver.waitForSuccessfulConnectionToZap(ZAPDriver.java:728)
  	at org.jenkinsci.plugins.zap.ZAPDriver.access$100(ZAPDriver.java:104)
  	at org.jenkinsci.plugins.zap.ZAPDriver$WaitZAPDriverInitCallable.invoke(ZAPDriver.java:2537)
  	at org.jenkinsci.plugins.zap.ZAPDriver$WaitZAPDriverInitCallable.invoke(ZAPDriver.java:2523)
  	at hudson.FilePath.act(FilePath.java:996)
  	at hudson.FilePath.act(FilePath.java:974)
  	at org.jenkinsci.plugins.zap.ZAPDriver.startZAP(ZAPDriver.java:641)
  	at org.jenkinsci.plugins.zap.ZAPBuilder.perform(ZAPBuilder.java:281)
  	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
  	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
  	at hudson.model.Build$BuildExecution.build(Build.java:205)
  	at hudson.model.Build$BuildExecution.doRun(Build.java:162)
  	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
  	at hudson.model.Run.execute(Run.java:1728)
  	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
  	at hudson.model.ResourceController.execute(ResourceController.java:98)
  	at hudson.model.Executor.run(Executor.java:404)
  
  ERROR: Unable to connect to ZAP's proxy after 60 seconds.
  	at org.jenkinsci.plugins.zap.ZAPDriver.waitForSuccessfulConnectionToZap(ZAPDriver.java:749)
  	at org.jenkinsci.plugins.zap.ZAPDriver.access$100(ZAPDriver.java:104)
  	at org.jenkinsci.plugins.zap.ZAPDriver$WaitZAPDriverInitCallable.invoke(ZAPDriver.java:2537)
  	at org.jenkinsci.plugins.zap.ZAPDriver$WaitZAPDriverInitCallable.invoke(ZAPDriver.java:2523)
  	at hudson.FilePath.act(FilePath.java:996)
  	at hudson.FilePath.act(FilePath.java:974)
  	at org.jenkinsci.plugins.zap.ZAPDriver.startZAP(ZAPDriver.java:641)
  	at org.jenkinsci.plugins.zap.ZAPBuilder.perform(ZAPBuilder.java:281)
  	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
  	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
  	at hudson.model.Build$BuildExecution.build(Build.java:205)
  	at hudson.model.Build$BuildExecution.doRun(Build.java:162)
  	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
  	at hudson.model.Run.execute(Run.java:1728)
  	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
  	at hudson.model.ResourceController.execute(ResourceController.java:98)
  	at hudson.model.Executor.run(Executor.java:404)
  
  Build step 'Execute ZAP' marked build as failure
  Started calculate disk usage of build
  Finished Calculation of disk usage of build in 0 seconds
  Started calculate disk usage of workspace
  Finished Calculation of disk usage of workspace in 0 seconds
  Finished: FAILURE

• I don't know what caused this error, or how, so I'm still figuring out a way to fix it, and make it point to the right address and then attack my target and scan for vulnerabilities
• My final step is to export the generated report to the existing nightly email which our team receives every night, so basically I'll be merging the reports of this vulnerability scan, to the existing nightly scan of some other automation tests

Thank you very much for your continued support guys, I hope I'm documenting my progress well enough so that this may benefit someone else in the future, especially if they're trying to use ZAP on a linux server. 

[JordanGS]

Your ZAP Settings Directory is still wrong. You set your zap installation directory and your zap settings directory to be the same thing, they shouldn't be. Read my previous reply, and look at the links i added.

[LnT]

I'm Adding few more comments ... ..

usually when Jenkins installed in Unix environment , a profile will be created with in Unix - called jenkins

Apparently , jenkins user to be given full access  - likely Admin.

which facilitates all plugin operations does smoothly.

Some cases , we need to provide explicit Write access to jenkins user for folders ( incase installation occupies @ /opt/ZAP/ZAP_D-2017-03-21/ )

[Osama Masood]

@LnT :

Yes, you're right, I ran into some problems because of these access controls, it took some time to get them approved and fixed by my manager as he's the one who has control over it. 

@Goran:

I followed the links you sent me previously, and I also changed the ZAP Settings Path on jenkins and tried some builds but they all failed. I changed the ZAP settings Path to "/root/.ZAP_D" after following your instructions from you previous post, but it didn't work.I've attached a screenshot of the changes i made, and a link to the log I get. ( https://pastebin.mozilla.org/9017931 )

By the way, If I keep the ZAP Settings Path to "/opt/ZAP/ZAP_D-2017-03-21" (the same as the installation path) and the build worked! This is the log from my console output:(https://pastebin.mozilla.org/9017937)

Please advise me how to solve this problem, I'm not sure what's going on lol.

[JordanGS]

@Osama, this is what i want you to do.

1) Download 2.6.0 from https://github.com/zaproxy/zaproxy/wiki/Downloads
2) Install the linux version.
3) Start ZAP (UI), NOT Jenkins. Just ZAP UI.
4) Find the ZAP Home directory it created and upload the zap.log that was generated when you opened and closed ZAP UI.

When you open the zap.log file, that was generated by ZAP UI. That's your HOME directory.

Provide me with the PATH to both your ZAP installation directory for 2.6.0 and the 2.6.0 home directory.

[Osama Masood]

Hi Goran,

So I've installed the latest version of OWASP ZAP (2.6) on my headless server, but I won't be able to run the GUI mode of ZAP on it because the server does not have a GUI, it just has a console. And after meeting up with the management team, I'm not sure when or even if it's possible to install a GUI on the server, so for now I'll use -daemon mode to run ZAP without a GUI on the server. I've attached the log of my results and the zap.log over here: (https://pastebin.com/0JtNxe5t).

So now I have got 2 paths: 

1. /root/.ZAP
   
2. /opt/ZAP/ZAP_2.6.0
   

I think I'm getting a better understanding of the big picture here, about how different the Home Directory and the Installation directory are.

Looking forward to hearing your valuable feedback :)

Also, I was wondering how to increase the scope of my attacks using ZAP. for now i've pointed the starting address to the login page of my company's software, but it does not spider all the URL's, even after I add a * at the end. I'm going to look for more resources to understand how to set and define the policy and other settings from command line.

Thanks for your help guys!

[Goran Dev]

Well, to configure alert filters and policies for active scanners you need the GUI for that. You can do that on any not headless machine then just move the xml files over to the headless machine.

What I would suggest is get ZAP setup on your local host where you have a GUI interface. Once you have that working, duplicate the settings over.

This is really a question for the ZAP user group. How to configure ZAP from a headless environment since Jenkins doesn't have access to all the settings but rather uses the config.xml which is modified from the GUI. You may be able to make changes from the command line to it. I'm not sure, ask in ZAP Users Group?

Cheers, Goran.

--
You received this message because you are subscribed to the Google Groups "ZAP Jenkins Plugin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-jenki...@googlegroups.com.
To post to this group, send email to zaproxy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-jenkins/26e9f2f0-cbe5-4d2c-bf4f-219c277fd92e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[JordanGS]

To clarify, I'm by no means a ZAP expert, i can help you with the Jenkins plugin but when it comes to configuring your scan for your specific software. It's best to ask the ZAP developers themselves. The ZAP Users Group is: https://groups.google.com/forum/#!forum/zaproxy-users

From my understand thought

ZAPROXY_HOME environment variable should be: /opt/ZAP/ZAP_2.6.0

and the ZAP Settings Value in the Jenkins Job Config should be /root/.ZAP

Let me know if that's successfully starts ZAP Jenkins for you

[Osama Masood]

Alright so the previous older version of ZAP is producing some basic results, I'll let it run for a week or two while I work on the upgrade for 2.6, it'll keep my colleagues happy and give me enough time to work on the upgrade :)

I created a new item on jenkins and copied my ZAP project, and made the appropriate configurations, here's my log report when I build my ZAP 2.6 : (https://pastebin.com/0WZCjVbm)

I'll also attach some screenshots of my settings on jenkins, I followed all your instructions and also made changes to the Environment variables. I'm still using ZAPROXY_HOME for the older ZAP, i created a new Environment variable called ZAPROXY_HOME2 for testing ZAP 2.6

As for the attack scope and how to get ZAP files on a headless linux server, I will contact the ZAP group that you linked in your previous post, thank you very much for that!

[JordanGS]

Just make sure to track version history: https://wiki.jenkins-ci.org/display/JENKINS/zap-plugin+History

That's good to hear. Keep me in the loop :D The permission issue needs to be resolved with your server administrator and jenkins run with administrator privileges.

[Osama Masood]

That's a wonderful resource! I will read through it and absorb and implement as much as possible, thank you very much goran!

So I managed to change the access controls of my entire ZAP folder, I gave jenkins the ownership and all the right privileges, but I'm still getting the error :

Failed to create directory /root/.ZAP
Unable to initialize home directory! /root/.ZAP/log4j.properties (Permission denied)
java.io.FileNotFoundException: /root/.ZAP/log4j.properties (Permission denied)

I've also attached a small screenshot of what my folder looks like, I'm not sure where or to what do I have to give access to create the directory in root. 

[JordanGS]

Forget Jenkins for the moment, if you open your terminal. What happens if you do the following:

$ZAPROXY_HOME/zap.sh -daemon -host 127.0.0.1 -port 8055 -config api.key=ZAPROXY-PLUGIN -dir /root/.ZAP -installdir $ZAPROXY_HOME

[Osama Masood]

my ZAPROXY_HOME is configured to run the latest weekly release of ZAP previous to the 2.6 version, so there are no issues running that: here's the log anyways (https://pastebin.com/dyy6hSkx)

As for ZAP 2.6, I created a new Environment variable to test 2.6 before officially using it, and it's called ZAPROXY_HOME2. It's configured in an entirely different  (https://pastebin.com/uNnn2ZfD)

[JordanGS]

So they both work from the terminal but only 2.6 doesn't work from Jenkins or Both don't work from Jenkins?

[Osama Masood]

So only 2.6 does not work from jenkins, the older weekly release works perfectly fine. 

I did some digging around, there are two instances of the weekly release in my ZAP folder, and I don't know which one is the one used by jenkins since they have the same path and name, but different capacity lol. I won't touch it because I don't want to break it for now, but I think the problem has to do with rights and ownership. I've attached some screenshots for a clearer picture. 

[JordanGS]

The easiest way is to compare permissions and openership between the folder that works and the folder that doesn't.

[Osama Masood]

Hey Jordan,

Hope you're well!

So after a 2 month break, I've been finally assigned to work on ZAP again, and it's in the exact same shape as it was 2 months ago.

The current problems:

• ZAP_D-2017-03-21 seems to be functioning well and generates a report daily on the login page of my target website
• ZAP 2.6 is not able to run from jenkins, but it works perfectly fine on the VM where all the testing is done

As for the permissions and file ownership, I entirely set both of the zap directories under the ownership of jenkins, just to make sure that jenkins doesn't cause any silly issues and has full rights.

So yeah, that's my current situation at the moment. Looking forward to hearing from you :)

Thanks!

[JordanGS]

Seems like an environment issue, not really sure without seeing any of the logs. I created redhat server myself on a vm to try to replicate this issue and i couldn't. If zap is working fine and able to create logs and write sessions in the default directories natively but jenkins is not. Then jenkins does not have the permissions to do so. Jenkins can only start zap with the same permissions it has, and not escalate them. In effect, if you were on windows and you tried to start jenkins normally, you would get this issue. However, if you started jenkins as administrator it would be resolved. In terms of linux, i would create a new permission group which has full read/write/execute permissions and then i would assign jenkins and zap to it. I would run zap first to make sure it's running properly and then i would run jenkins.

[Osama Masood]

Hey Goran,

So I was able to fix the issue finally!! basically the folders of jenkins and ZAP were not in the same directory, and although Jenkins had access rights to both of the folders entirely, it couldn't traverse in between my jenkins folder and my ZAP folder because Jenkins does not have rights to anything in between the folders lol...

SO, ZAP 2.6 is finally working and integrated in my CI environment, but I'm still not able to increase the attack scope of the target, I only get a report for the login page and that's it. I've asked around at the OWASP ZAP user group, and I'm also browsing the OWASP ZAP Scripts group to understand more, but I'm not sure who'll be able to help me at this point, so any help will be highly appreciated! 

[Pradnya Pawar]

Hello All,

I am Security Analyst and working on
After reading all these questions and answers I am confident that now I can integrate Jenkins with OWASP ZAP.

Tasks which have been completed.
1. We install Jenkins with required dependencies, OWASP zap with required dependencies.
2. Using local proxy: 127.0.0.1 and ports: 8080 for Jenkins, 8090 for ZAP.
3. Created master and slave (Installed jenkins on 1 windows machine which acts as master and installed ZAP on another windows machine which acts as Slave): Is this required for this scenario to go with master-slave concept/ can we do this on only 1 machine?
4.

Let me introduce blockers while doing this integration:
1. How to establish connectivity from master to slave and vice versa.
2. Facing too much issues and getting confused while following instructions on how to configure a Job from start to finish(Configure the Job to Execute ZAP)
3. Can you please guide us on pipeline script if we can add in CI/CD pipeline for ZAP execution.


It,s been a 15 days that we are working on only master-slave, AP job configuration. Today I read this page and thought to share my blockers with you all. Please help me out if it is possible for you guys.


Thank You in advance.

Regards,
Pradnya

[JordanGS]

@Osama Masood every website is different. It took me a long time to work out a script to configure my own sites. There isn't one specific way to setup a scan and it's hard for me to help since i don't have the expertise in that area or any knowledge of your site and it's structure. I would suggest to keep asking in the User Group. They might be slow to reply because they are working on 2.7 release. The best best would be to try and work out a procedure that works for you in ZAP (GUI). Not in jenkins but in the GUI version itself. Once you have something working there, it will be easy to duplicate the settings in the Jenkins plugin.