Running as a daemon (OutOfMemory)

[codemonkey]

Hi,

For nightly automation, Is it better to run zap as a daemon or to start a new process for each run?

What are the pros and cons of continuing to use the same session indefinitely versus starting a new one for each run?

Part of the reason I ask is because we have just hit some OutOfMemory issues which seem to have corrupted our session.data file.

We just got zap set up as a daemon for use by our nightly build process. Each night, we run our test automation through the zap session and then kick off the scans. Everything seemed to be running great for a week or so, but then the proxy started refusing connections and we found OutOfMemory exceptions in the ZAP log -

102705282 [pool-27-thread-1] INFO com.crawljax.core.CrawlController  - Time is up! Shutting down...
212019438 [ZAP-PassiveScanner] ERROR org.zaproxy.zap.ZAP$UncaughtExceptionLogger  - Exception in thread "ZAP-PassiveScanner"
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3236)
        at java.lang.StringCoding.safeTrim(StringCoding.java:79)
        at java.lang.StringCoding.access$300(StringCoding.java:50)
        at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:305)
        at java.lang.StringCoding.encode(StringCoding.java:344)
        at java.lang.String.getBytes(String.java:918)
        at org.zaproxy.zap.network.HttpResponseBody.toUTF8(Unknown Source)
        at org.zaproxy.zap.network.HttpResponseBody.createCachedStringWithMetaCharset(Unknown Source)
        at org.zaproxy.zap.network.HttpResponseBody.createCachedString(Unknown Source)
        at org.parosproxy.paros.network.HttpBody.toString(Unknown Source)
        at org.zaproxy.zap.extension.pscan.PassiveScanThread.run(Unknown Source)
215283567 [ZAP-ProxyThread-8876] ERROR org.zaproxy.zap.ZAP$UncaughtExceptionLogger  - Exception in thread "ZAP-ProxyThread-8876"
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.sun.crypto.provider.CipherCore.update(CipherCore.java:750)
        at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:400)
        at javax.crypto.Cipher.update(Cipher.java:1945)
        at sun.security.ssl.CipherBox.encrypt(CipherBox.java:323)
        at sun.security.ssl.OutputRecord.encrypt(OutputRecord.java:264)
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:859)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:847)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
        at org.parosproxy.paros.network.HttpOutputStream.write(Unknown Source)
        at org.parosproxy.paros.network.HttpOutputStream.write(Unknown Source)
        at org.parosproxy.paros.core.proxy.ProxyThread.writeHttpResponse(Unknown Source)
        at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(Unknown Source)
        at org.parosproxy.paros.core.proxy.ProxyThread.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:745)


I figured out how to increase the JVM heap size using the .ZAP_JVM.properties file, but then I found that zap could not restart because the session.data file had problems -

14265 [ZAP-daemon] FATAL hsqldb.db.HSQLDB379AF3DEBD.ENGINE  - /build/build/.ZAP/session/DailyBuild.session.data getFromFile 58142865
org.hsqldb.HsqlException: java.io.EOFException
        at org.hsqldb.error.Error.error(Unknown Source)
        at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
        at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
        at org.hsqldb.persist.DataFileCache.get(Unknown Source)
        at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
        at org.hsqldb.index.NodeAVLDisk.findNode(Unknown Source)
        at org.hsqldb.index.NodeAVLDisk.getRight(Unknown Source)
        at org.hsqldb.index.IndexAVL.next(Unknown Source)
        at org.hsqldb.index.IndexAVL.next(Unknown Source)
        at org.hsqldb.index.IndexAVL$IndexRowIterator.getNextRow(Unknown Source)
        at org.hsqldb.RangeVariable$RangeIteratorMain.findNext(Unknown Source)
        at org.hsqldb.RangeVariable$RangeIteratorMain.next(Unknown Source)
        at org.hsqldb.QuerySpecification.buildResult(Unknown Source)
        at org.hsqldb.QuerySpecification.getSingleResult(Unknown Source)
        at org.hsqldb.QuerySpecification.getResult(Unknown Source)
        at org.hsqldb.StatementQuery.getResult(Unknown Source)
        at org.hsqldb.StatementDMQL.execute(Unknown Source)
        at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
        at org.hsqldb.Session.execute(Unknown Source)
        at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
        at org.hsqldb.jdbc.JDBCPreparedStatement.executeQuery(Unknown Source)
        at org.parosproxy.paros.db.paros.ParosTableHistory.getHistoryIdsOfHistType(Unknown Source)
        at org.parosproxy.paros.model.Session.open(Unknown Source)
        at org.parosproxy.paros.model.Model.openSession(Unknown Source)
        at org.parosproxy.paros.control.Control.runCommandLineOpenSession(Unknown Source)
        at org.zaproxy.zap.HeadlessBootstrap.handleCmdLineSessionArgsSynchronously(Unknown Source)
        at org.zaproxy.zap.DaemonBootstrap$1.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:745)

When I inspected the session.data file, I found it had grown to 3.6 GB and the corresponding session.lobs file was 148 MB.

I also found a bunch of AJAXSpider errors in the log that are described on this page - https://github.com/zaproxy/zaproxy/issues/2149. I updated the Selenium add-on to version 4, but still see the same errors. So, I've disabled the AJAX spider for now. Is it possible that this issue is the cause of the out-of-memory condition?

We are running on Centos 7, with ZAP 2.4.3 and firefox 38.4.0. I am now upgrading firefox to 38.7.0.

Thanks!

Chris

[Simon Bennetts]

Right now I'd definitely recommend running a new ZAP process for each scan, although scheduling a regular (eg daily) restart is also an option.

Historically ZAP has been a desktop tool and builds a lot of data structures in memory so that the Swing UI reacts quickly.
The daemon option just removes the Swing UI, it doesnt reduce the memory usage very much.
We _do_ have a -lowmem option but thats still experimental so I wouldnt rely on it yet.

Note that you can run multiple ZAP instances on the same machine by making sure that they use different home directories (-dir option) and different ports (-port option)

Cheers,

Simon

[codemonkey]

Hello again,

Still having this problem after changing our setup to restart the zap daemon process at the start of each new (daily) run. Since making the change, we've restarted the daemon and run a scan 15 times, but then today it hit the OutOfMemory condition again.

The jvm heap size is set to 7 GB. Our session data file is currently 4 GB, and the session lobs file is 572 MB.

I have 2 questions:

1. Is this perhaps due to the nature of our application? Could it be that sometimes zap gets into some kind of loop exploring the interface and spins until the data grows very large? Any suggestions on how I might debug something like that?

2. I want to change our config to use the -newsession flag, to see if that helps. But, that means it won't be running with the right context and so won't have our custom authentication scripts, etc. How do I set the context and other session configuration values if we are starting a new session each time? Is this something I have to do via the API?

Well, I guess that's really 5 questions...

Thanks, Chris

[codemonkey]

BTW, our application is not that complex really. The most extensive successful scan we've run took just about 9 minutes. That included running the spider, the ajax spider and then the active scan. Since the 15th, we haven't been running the ajax spider at all.

[Simon Bennetts]

Very strange :/
Does your application have any very large files? eg very big images or videos?

Re the contexts, export your context using the UI - this will give you an XML file.
You can then load this via the API in a new session using the context/importContext action: https://github.com/zaproxy/zaproxy/wiki/ApiGen_context

Cheers,

Simon

[codemonkey]

No large files, as far as I know. It's a fairly simple SNMP monitoring application. The user inferface has pages for managing users, groups and the target devices. Also, pages for viewing logs and alerts.

I do see these messages appearing in the log, are they unusual? I see 47 of them in our last good log, before we hit the OutOfMemory problem. There are about 89 such messages in the logs for each of the previous runs.

2016-03-23 13:45:58,292 [ZAP-ProxyThread-2203] WARN  HttpMethodBase - Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.

My plan is to get the newsession change into place, then I will drill down into the details to see if I can figure out what's happening.

Is there any size data in the logs I can analyze, perhaps by enabling debugging or something?

Is there a utility I can use to view the contents of these large session data files?

Chris

[codemonkey]

I'm finally getting around to updating my automation to automatically populate the context so I can start a new session for each run. I went down the list of context settings in the interactive user interface and added code to handle each value I had configured.

I'm almost done now, but I can't find any api for setting the values in the Authorization section. In particular, I just want to set the 'HTTPS Status Code' value to 401 - how can I do that via the api?

Thanks!