Build Failed due to " No space left on device" error - jenkins

[gowtha...@gmail.com]

Hi,

I have configured Zap-Jenkins successfully on my CI environment and was scanning an application that scan status was almost 80% (scanning took nearly 3 days) and today the build got failed due to following error:


468203657 [ZAP-ProxyThread-9414] FATAL hsqldb.db..ENGINE - shadow backup failure pos 2636267520 16384 java.io.IOException: No space left on device

Not sure why the build failed even though i have space left out in my device. While checked the workspace the session file size was nearly 30GB !! is this normal case ?? 

Is there any configuration available to clear out workspace to avoid these type of errors. I have chosen option "persist session" since running the job for the first time. 

Please help.

Console Output:

[ZAP Jenkins Plugin] ACTIVE SCAN STATUS [ 78% ]
[ZAP Jenkins Plugin] ALERTS COUNT [ 13894 ]
468203657 [ZAP-ProxyThread-9414] FATAL hsqldb.db..ENGINE  - shadow backup failure pos 2636267520 16384
java.io.IOException: No space left on device
	at java.io.RandomAccessFile.writeBytes(Native Method)
	at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
	at org.hsqldb.persist.RAFileSimple.write(Unknown Source)
	at org.hsqldb.persist.RAShadowFile.copy(Unknown Source)
	at org.hsqldb.persist.RAShadowFile.copy(Unknown Source)
	at org.hsqldb.persist.DataFileCache.copyShadow(Unknown Source)
	at org.hsqldb.persist.DataFileCache.saveRows(Unknown Source)
	at org.hsqldb.persist.Cache.saveRows(Unknown Source)
	at org.hsqldb.persist.Cache.cleanUp(Unknown Source)
	at org.hsqldb.persist.Cache.preparePut(Unknown Source)
	at org.hsqldb.persist.Cache.put(Unknown Source)
	at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
	at org.hsqldb.persist.DataFileCache.get(Unknown Source)
	at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
	at org.hsqldb.index.NodeAVLDisk.findNode(Unknown Source)
	at org.hsqldb.index.NodeAVLDisk.getLeft(Unknown Source)
	at org.hsqldb.index.IndexAVL.next(Unknown Source)
	at org.hsqldb.index.IndexAVL.next(Unknown Source)
	at org.hsqldb.index.IndexAVL$IndexRowIterator.getNextRow(Unknown Source)
	at org.hsqldb.RangeVariable$RangeIteratorMain.findNext(Unknown Source)
	at org.hsqldb.RangeVariable$RangeIteratorMain.next(Unknown Source)
	at org.hsqldb.QuerySpecification.buildResult(Unknown Source)
	at org.hsqldb.QuerySpecification.getSingleResult(Unknown Source)
	at org.hsqldb.QuerySpecification.getResult(Unknown Source)
	at org.hsqldb.StatementQuery.getResult(Unknown Source)
	at org.hsqldb.StatementDMQL.execute(Unknown Source)
	at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
	at org.hsqldb.Session.execute(Unknown Source)
	at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
	at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source)
	at org.parosproxy.paros.db.paros.ParosTableHistory.read(ParosTableHistory.java:255)
	at org.zaproxy.zap.extension.api.CoreAPI.processHttpMessages(CoreAPI.java:1541)
	at org.zaproxy.zap.extension.api.CoreAPI.handleApiView(CoreAPI.java:1005)
	at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:467)
	at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:456)
	at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:317)
	at java.lang.Thread.run(Thread.java:748)
468203658 [ZAP-ProxyThread-9414] FATAL hsqldb.db..ENGINE  - DataFileCache.copyShadow 2636257024
java.io.IOException: No space left on device
	at java.io.RandomAccessFile.writeBytes(Native Method)
	at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
	at org.hsqldb.persist.RAFileSimple.write(Unknown Source)
	at org.hsqldb.persist.RAShadowFile.copy(Unknown Source)
	at org.hsqldb.persist.RAShadowFile.copy(Unknown Source)
	at org.hsqldb.persist.DataFileCache.copyShadow(Unknown Source)
	at org.hsqldb.persist.DataFileCache.saveRows(Unknown Source)
	at org.hsqldb.persist.Cache.saveRows(Unknown Source)
	at org.hsqldb.persist.Cache.cleanUp(Unknown Source)
	at org.hsqldb.persist.Cache.preparePut(Unknown Source)
	at org.hsqldb.persist.Cache.put(Unknown Source)
	at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
	at org.hsqldb.persist.DataFileCache.get(Unknown Source)
	at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
	at org.hsqldb.index.NodeAVLDisk.findNode(Unknown Source)
	at org.hsqldb.index.NodeAVLDisk.getLeft(Unknown Source)
	at org.hsqldb.index.IndexAVL.next(Unknown Source)
	at org.hsqldb.index.IndexAVL.next(Unknown Source)
	at org.hsqldb.index.IndexAVL$IndexRowIterator.getNextRow(Unknown Source)
	at org.hsqldb.RangeVariable$RangeIteratorMain.findNext(Unknown Source)
	at org.hsqldb.RangeVariable$RangeIteratorMain.next(Unknown Source)
	at org.hsqldb.QuerySpecification.buildResult(Unknown Source)
	at org.hsqldb.QuerySpecification.getSingleResult(Unknown Source)
	at org.hsqldb.QuerySpecification.getResult(Unknown Source)
	at org.hsqldb.StatementQuery.getResult(Unknown Source)
	at org.hsqldb.St

[ZAP Jenkins Plugin] SHUTDOWN [ SUCCESSFUL ]

[ZAP Jenkins Plugin] LOG SEARCH...
	ZAP HOME DIRECTORY [ /var/lib/jenkins/ZAP ]
	JENKINS WORKSPACE [ /var/lib/jenkins/workspace/security-test-ZAP ]
	CLEARING WORKSPACE/LOGS

	[ /var/lib/jenkins/ZAP/zap.log.1 ] LOG HAS BEEN FOUND
ERROR: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:313)
	at org.apache.commons.io.IOUtils.write(IOUtils.java:1527)
	at org.apache.commons.io.FileUtils.writeStringToFile(FileUtils.java:1929)
	at org.apache.commons.io.FileUtils.writeStringToFile(FileUtils.java:1962)
	at org.jenkinsci.plugins.zap.ZAPBuilder$CopyFileCallable.invoke(ZAPBuilder.java:514)
	at org.jenkinsci.plugins.zap.ZAPBuilder$CopyFileCallable.invoke(ZAPBuilder.java:495)
	at hudson.FilePath.act(FilePath.java:1009)
	at hudson.FilePath.act(FilePath.java:987)
	at org.jenkinsci.plugins.zap.ZAPBuilder.perform(ZAPBuilder.java:325)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
	at hudson.model.Run.execute(Run.java:1727)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)

Build step 'Execute ZAP' marked build as failure
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Started calculate disk usage of workspace
Finished Calculation of disk usage of workspace in  5 second
Finished: FAILURE

~Gowtham

[SRINANNAPA]

Hi ,  I'm also facing the similar issue. Did you find any solution to this ?

Thanks,

[hauschu...@gmail.com]

Hello!

I would recommend against using 'persist session'. 

That doesn't just save the site tree and settings, that persists every single alert (and accompanying request, response, etc), and is cumulative every time you use it. The first time you use it, it finds 10 alerts and so has 10 alerts, the second time it has 10 alerts and finds the same 10 (now has 20), then 30, etc etc. 

It's better to have a consistent baseline session with your settings, site tree, and no alerts and import it every time to a new session. (unless you have some defined use case for needing to persist session)

So try it with a clean session and see if that helps the speed and resource usage!