Problems with Jenkins OWASP plugin

[Tony Sweeney]

Hi,

One of our security analysts is interested in running the OWASP depencency check on my employer's software builds.  We have version 1.4.5 of the plugin and all latest dependencies installed.  The initial configuration was 100% default, and with this we were able initially to run analysis against some builds and publish results using the Jenkins plugins.  However, this is hugely wasteful of space as each build gets a private copy of the vulnerability database.  Further, some of our builds wipe out the workspace each time, and so were attempting to download the database afresh with every build.  I noticed today that some of the builds were failing due to OWASP.  Looking in the build workspace, the downloaded DB was not the expected 300MB, but exactly 65536 bytes in size.  There is no proxy between the build machines and the Internet.

So, I thought, given tha above, I'll just configure a global location, populate that on each of the build slaves and the builds should automatically use that location instead.  But it doesn't work:

[DependencyCheck] OWASP Dependency-Check Plugin v1.4.5
[DependencyCheck] Executing Dependency-Check with the following options:
[DependencyCheck]  -name = ttp-service
[DependencyCheck]  -scanPath = /usr/local/jenkins/workspace/ttp-service
[DependencyCheck]  -outputDirectory = /usr/local/jenkins/workspace/ttp-service
[DependencyCheck]  -dataDirectory = /usr/local/jenkins/owasp-nvd
[DependencyCheck]  -dataMirroringType = none
[DependencyCheck]  -isQuickQueryTimestampEnabled = true
[DependencyCheck]  -useMavenArtifactsScanPath = false
[DependencyCheck]  -jarAnalyzerEnabled = true
[DependencyCheck]  -nodeJsAnalyzerEnabled = true
[DependencyCheck]  -composerLockAnalyzerEnabled = true
[DependencyCheck]  -pythonDistributionAnalyzerEnabled = true
[DependencyCheck]  -pythonPackageAnalyzerEnabled = true
[DependencyCheck]  -rubyBundlerAuditAnalyzerEnabled = true
[DependencyCheck]  -rubyGemAnalyzerEnabled = true
[DependencyCheck]  -cocoaPodsAnalyzerEnabled = true
[DependencyCheck]  -swiftPackageManagerAnalyzerEnabled = true
[DependencyCheck]  -archiveAnalyzerEnabled = true
[DependencyCheck]  -assemblyAnalyzerEnabled = true
[DependencyCheck]  -centralAnalyzerEnabled = true
[DependencyCheck]  -nuspecAnalyzerEnabled = true
[DependencyCheck]  -nexusAnalyzerEnabled = false
[DependencyCheck]  -autoconfAnalyzerEnabled = true
[DependencyCheck]  -cmakeAnalyzerEnabled = true
[DependencyCheck]  -opensslAnalyzerEnabled = true
[DependencyCheck]  -tempPath = /tmp
[DependencyCheck]  -showEvidence = true
[DependencyCheck]  -format = XML
[DependencyCheck]  -autoUpdate = false
[DependencyCheck]  -updateOnly = false
[DependencyCheck] Scanning: /usr/local/jenkins/workspace/ttp-service
[DependencyCheck] Analyzing Dependencies
[DependencyCheck] One or more exceptions were thrown while executing Dependency-Check
[DependencyCheck] Exception Caught: org.owasp.dependencycheck.exception.InitializationException
[DependencyCheck] Cause: No such file or directory
[DependencyCheck] Message: Unable to create a temporary file
[DependencyCheck] Exception Caught: org.owasp.dependencycheck.exception.InitializationException
[DependencyCheck] Cause: No such file or directory
[DependencyCheck] Message: Unable to create a temporary file
Build step 'Invoke OWASP Dependency-Check analysis' marked build as failure
[DependencyCheck] Skipping publisher since build result is FAILURE
Skipped archiving because build is not successful

This plugin has thousands of installations, so it surely must work.  What am I doing wrong?

[Tony Sweeney]

I also tried with verbose logging enabled, which nets me this additional output line:

[DependencyCheck]  -verboseLogFile = /usr/local/jenkins/workspace/ttp-service/dependency-check.log

However, no output file is written even though permissions allow for it.

[Steve Springett]

The logging can be accessed in the Jenkins system log by specifying org.owasp as the logger. This was corrected awhile ago, and awaiting inclusion in the 2.0.0 release.

Best practice is to setup a global data directory, and for each job, specify the job to use the global data directory with auto-update disabled. Then setup another job that blocks all other DC jobs that runs once a day to update the global data directory. Basically, you want only a single job to be able to write to the global data directory and that job should run once a day. When you don’t have auto-update disabled on individual jobs and multiple jobs run simultaneously, the global data directory will become corrupted.

This approach can be used in combination with hosting a mirror of the NVD itself which can reduce update times and helps when build machines are locked down without access to the Internet.

— Steve

--
You received this message because you are subscribed to the Google Groups "Dependency Check" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dependency-che...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tony Sweeney]

This is essentially how I have set this up.  To give a bit of context, our Jenkins cluster has a Linux master and 12 Linux agents that we hope to be able to run this analysis on.  I created a matrix job to retrieve the NVD database nightly to a common location on each agent and configured that as teh global location location in the master node.  The individual analysis tasks should have that available to them at run time via the global config on the master (though there is in fact a separate DB instance in the same location on each agent).  The output in the relevant slave.log for a failing run is as follows:

Jun 14, 2017 3:44:13 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Analysis Started
Jun 14, 2017 3:44:13 PM org.owasp.dependencycheck.Engine initializeAnalyzer
SEVERE: Exception occurred initializing Archive Analyzer.
Jun 14, 2017 3:44:13 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished File Name Analyzer (0 seconds)
Jun 14, 2017 3:44:13 PM org.owasp.dependencycheck.Engine initializeAnalyzer
SEVERE: Exception occurred initializing Jar Analyzer.
Jun 14, 2017 3:44:13 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished Central Analyzer (0 seconds)
Jun 14, 2017 3:44:13 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished Dependency Merging Analyzer (0 seconds)
Jun 14, 2017 3:44:13 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished Version Filter Analyzer (0 seconds)
Jun 14, 2017 3:44:13 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished Hint Analyzer (0 seconds)
Jun 14, 2017 3:44:14 PM org.owasp.dependencycheck.analyzer.CPEAnalyzer open
INFO: Created CPE Index (0 seconds)
Jun 14, 2017 3:44:14 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished CPE Analyzer (1 seconds)
Jun 14, 2017 3:44:14 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished False Positive Analyzer (0 seconds)
Jun 14, 2017 3:44:14 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished Cpe Suppression Analyzer (0 seconds)
Jun 14, 2017 3:44:15 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished NVD CVE Analyzer (0 seconds)
Jun 14, 2017 3:44:15 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished Vulnerability Suppression Analyzer (0 seconds)
Jun 14, 2017 3:44:15 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Finished Dependency Bundling Analyzer (0 seconds)
Jun 14, 2017 3:44:15 PM org.owasp.dependencycheck.Engine analyzeDependencies
INFO: Analysis Complete (2 seconds)

The slave Java version is jdk1.8.0_60.  There is considerably more detail in the console log visible in the UI, where I have the log level set to ALL, but none of it indicates what is causing the build failure.

[Steve Springett]

I would recommend looking into the temp directory. It appears the errors stem from there. Dependency-Check will use the temp dir when unpacking jars and doing other things. If the user running Jenkins doesn’t have write access to the temp dir, or the temp dir space is full, DC will fail.

If this is the case, there is a global option to change the location of the temp directory DC will use. For example, use /usr/local/jenkins/dctemp

— Steve

[Tony Sweeney]

java.io.tmpdir is set to /tmp, which has over 150GB free.  I also tried explicitly setting it in the global config to both /tmp and /var/tmp to see if it might make a difference.  On a hunch I created /tmp/dependencycheck and made it writeable to the jenkins user, then ran the build under strace.  It turns out that dependency check is ignoring the global settings completely, even if I also set them in the build as well and despite the fact that it logs them correcty to the console.  It tries to download the db to ${WORKSPACE}/dependency-check-data, retrieves a truncated 64K dc.h2.db file and then fails to analyze using this corrupted DB.  If I replace the DB with a complete copy from another build, the analysis runs fine.