Zap daemon blocked during a scan

[Albert]

Hi,

Would really appreciate if someone can give me some direction on the following as I been stuck for some time.

I been working on prototyping ZAP for continuos security testing using Jenkins. To achieve that I used the zap-client-api and integrate the ZAP scans within JUnit tests. 

So each JUnit test does the following:

1. Authenticates to the site.
   
2. Spiders the site
   
3. Scans the Site. 
   

This is done for every JUnit test. I guess a good refactoring would be to Authenticate and Spider in the Fixture and then only Scan en each test. But not sure if that is feasible. But this is another question. 

The situation I am facing is that my prototype start zap as a daemon within the Java code of the Junit tests:

        ProcessBuilder ZapProcess = new ProcessBuilder(ZAPConstants.ZAP_HOME_DIRECTORY + "zap.sh", "-daemon");

Then all the Junit tests are run (Authentication + Spidering + Scanning) xtest.

I see a consistent pattern that Zap hung out during the scanning process of some of the Unit tests. 

To debug I started ZAP in UI mode listening to the same port and loading the same config.xml file as the Jenkins process does. I can then see in the UI all the actions done by each JUnit test. Everything works fine. All tests are executed and I can see the spider bar and scan bar getting to 100% for each test. 

So the problem seems to be with the daemon mode. 

I got the following thread dump at the moment it got stuck. Can someone point me to what you think is really going on? I am not that familiar with the core zap code and this is the last issue I need to solve.

THREAD DUMP:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.77-b03 mixed mode):

"ZAP-ActiveScanner-0" #77 daemon prio=3 os_prio=0 tid=0x00007fc2a008b800 nid=0x55ee waiting for monitor entry [0x00007fc294bd1000]

  java.lang.Thread.State: BLOCKED (on object monitor)

at org.zaproxy.zap.users.User.processMessageToMatchUser(Unknown Source)

- waiting to lock <0x00000000d9f981d8> (a org.zaproxy.zap.users.User)

at org.parosproxy.paros.network.HttpSender.sendAuthenticated(Unknown Source)

at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractPlugin.sendAndReceive(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractPlugin.sendAndReceive(Unknown Source)

at org.zaproxy.zap.extension.ascanrulesBeta.SQLInjectionPostgresql.scan(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractAppParamPlugin.scan(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractAppParamPlugin.scanVariant(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractAppParamPlugin.scan(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractPlugin.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"ZAP-ActiveScanner-1" #73 daemon prio=3 os_prio=0 tid=0x00007fc2a0071800 nid=0x55ea runnable [0x00007fc287c7c000]

  java.lang.Thread.State: RUNNABLE

at java.io.FileOutputStream.writeBytes(Native Method)

at java.io.FileOutputStream.write(Unknown Source)

at java.io.BufferedOutputStream.write(Unknown Source)

- locked <0x00000000d838cc78> (a java.io.BufferedOutputStream)

at java.io.PrintStream.write(Unknown Source)

- locked <0x00000000d8318a30> (a java.io.PrintStream)

at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)

at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)

at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)

at sun.nio.cs.StreamEncoder.flush(Unknown Source)

- locked <0x00000000d85cab28> (a java.io.OutputStreamWriter)

at java.io.OutputStreamWriter.flush(Unknown Source)

at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)

at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)

at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)

at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)

- locked <0x00000000d85358a8> (a org.apache.log4j.ConsoleAppender)

at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)

at org.apache.log4j.Category.callAppenders(Category.java:206)

- locked <0x00000000d835cd68> (a org.apache.log4j.spi.RootLogger)

at org.apache.log4j.Category.forcedLog(Category.java:391)

at org.apache.log4j.Category.info(Category.java:666)

at org.zaproxy.zap.extension.httpsessions.HttpSessionsSite.processHttpResponseMessage(Unknown Source)

at org.zaproxy.zap.extension.httpsessions.ExtensionHttpSessions.onHttpResponseReceive(Unknown Source)

at org.parosproxy.paros.network.HttpSender.notifyResponseListeners(Unknown Source)

at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)

at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)

at org.zaproxy.zap.authentication.FormBasedAuthenticationMethodType$FormBasedAuthenticationMethod.authenticate(Unknown Source)

at org.zaproxy.zap.users.User.authenticate(Unknown Source)

at org.zaproxy.zap.users.User.processMessageToMatchUser(Unknown Source)

- locked <0x00000000d9f981d8> (a org.zaproxy.zap.users.User)

at org.parosproxy.paros.network.HttpSender.sendAuthenticated(Unknown Source)

at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractPlugin.sendAndReceive(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractPlugin.sendAndReceive(Unknown Source)

at org.zaproxy.zap.extension.ascanrulesBeta.SQLInjectionPostgresql.scan(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractAppParamPlugin.scan(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractAppParamPlugin.scanVariant(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractAppParamPlugin.scan(Unknown Source)

at org.parosproxy.paros.core.scanner.AbstractPlugin.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"Thread-11" #69 daemon prio=3 os_prio=0 tid=0x00007fc28c544000 nid=0x55e6 sleeping[0x00007fc287d7e000]

  java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.parosproxy.paros.core.scanner.Util.sleep(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.scanSingleNode(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.access$200(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess$1.apply(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.traverse(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.traverse(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.traverse(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.traverse(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.traverse(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.traverse(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.traverse(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.processPlugin(Unknown Source)

at org.parosproxy.paros.core.scanner.HostProcess.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"Thread-10" #66 daemon prio=3 os_prio=0 tid=0x00007fc280003000 nid=0x55e3 in Object.wait() [0x00007fc2955d3000]

  java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.Thread.join(Unknown Source)

- locked <0x00000000c5412770> (a java.lang.Thread)

at org.parosproxy.paros.common.ThreadPool.waitAllThreadComplete(Unknown Source)

at org.parosproxy.paros.core.scanner.Scanner.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"MultiThreadedHttpConnectionManager cleanup" #50 daemon prio=5 os_prio=0 tid=0x00007fc2880cc800 nid=0x559a in Object.wait() [0x00007fc2a8725000]

  java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.ref.ReferenceQueue.remove(Unknown Source)

- locked <0x00000000d9f846f8> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(Unknown Source)

at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

"Java2D Disposer" #29 daemon prio=10 os_prio=0 tid=0x00007fc2a004e800 nid=0x5584 in Object.wait() [0x00007fc294455000]

  java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.ref.ReferenceQueue.remove(Unknown Source)

- locked <0x00000000d9d3b650> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(Unknown Source)

at sun.java2d.Disposer.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"TimerQueue" #28 daemon prio=5 os_prio=0 tid=0x00007fc2a0036000 nid=0x5583 waiting on condition [0x00007fc2947cf000]

  java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for  <0x00000000d9d3b808> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)

at java.util.concurrent.DelayQueue.take(Unknown Source)

at javax.swing.TimerQueue.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"ZAP-PassiveScanner" #27 daemon prio=4 os_prio=0 tid=0x00007fc2880db800 nid=0x5582 sleeping[0x00007fc2948d0000]

  java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.zaproxy.zap.extension.pscan.PassiveScanThread.run(Unknown Source)

"HSQLDB Timer @6193932a" #26 daemon prio=4 os_prio=0 tid=0x00007fc2881ba000 nid=0x5581 in Object.wait() [0x00007fc2bc2b0000]

  java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at org.hsqldb.lib.HsqlTimer$TaskQueue.park(Unknown Source)

- locked <0x00000000d84f49d0> (a org.hsqldb.lib.HsqlTimer$TaskQueue)

at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)

- locked <0x00000000d84f49d0> (a org.hsqldb.lib.HsqlTimer$TaskQueue)

at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"ZAP-ProxyServer" #19 daemon prio=6 os_prio=0 tid=0x00007fc28c52a800 nid=0x554c runnable [0x00007fc294cd2000]

  java.lang.Thread.State: RUNNABLE

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.AbstractPlainSocketImpl.accept(Unknown Source)

at java.net.ServerSocket.implAccept(Unknown Source)

at java.net.ServerSocket.accept(Unknown Source)

at org.parosproxy.paros.core.proxy.ProxyServer.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"Timer-0" #16 daemon prio=5 os_prio=0 tid=0x00007fc28cf39800 nid=0x5540 in Object.wait() [0x00007fc2958d4000]

  java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.util.TimerThread.mainLoop(Unknown Source)

- locked <0x00000000d90d3868> (a java.util.TaskQueue)

at java.util.TimerThread.run(Unknown Source)

"ZAP-ExtensionFilter" #15 daemon prio=5 os_prio=0 tid=0x00007fc28cf25000 nid=0x553f sleeping[0x00007fc2959d5000]

  java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.parosproxy.paros.extension.filter.ExtensionFilter.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"ZAP-DownloadManager" #13 daemon prio=5 os_prio=0 tid=0x00007fc28c6de000 nid=0x553e sleeping[0x00007fc2a8523000]

  java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.zaproxy.zap.extension.autoupdate.DownloadManager.run(Unknown Source)

"DestroyJavaVM" #10 prio=5 os_prio=0 tid=0x00007fc2b8009000 nid=0x552b waiting on condition [0x0000000000000000]

  java.lang.Thread.State: RUNNABLE

"ZAP-daemon" #9 prio=5 os_prio=0 tid=0x00007fc2b87c7800 nid=0x553b sleeping[0x00007fc2bc1af000]

  java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.zaproxy.zap.DaemonBootstrap$1.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007fc2b80d2800 nid=0x5532 runnable [0x0000000000000000]

  java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fc2b80cb000 nid=0x5531 waiting on condition [0x0000000000000000]

  java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fc2b80c8800 nid=0x5530 runnable [0x0000000000000000]

  java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fc2b80c6800 nid=0x552f waiting on condition [0x0000000000000000]

  java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fc2b80a1000 nid=0x552e in Object.wait() [0x00007fc2bdfe5000]

  java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.ref.ReferenceQueue.remove(Unknown Source)

- locked <0x00000000d82d91a0> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(Unknown Source)

at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fc2b809c000 nid=0x552d in Object.wait() [0x00007fc2be0e6000]

  java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Unknown Source)

at java.lang.ref.Reference.tryHandlePending(Unknown Source)

- locked <0x00000000d82d9358> (a java.lang.ref.Reference$Lock)

at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

"VM Thread" os_prio=0 tid=0x00007fc2b8094800 nid=0x552c runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fc2b80d6000 nid=0x5533 waiting on condition

JNI global references: 1727

<writer thread='140474386896640'/>

Heap

def new generation   total 18496K, used 5021K [0x00000000c4200000, 0x00000000c5610000, 0x00000000d8150000)

 eden space 16448K,  24% used [0x00000000c4200000, 0x00000000c45e5540, 0x00000000c5210000)

 from space 2048K,  50% used [0x00000000c5410000, 0x00000000c5512208, 0x00000000c5610000)

 to   space 2048K,   0% used [0x00000000c5210000, 0x00000000c5210000, 0x00000000c5410000)

tenured generation   total 40960K, used 39661K [0x00000000d8150000, 0x00000000da950000, 0x0000000100000000)

  the space 40960K,  96% used [0x00000000d8150000, 0x00000000da80b5c8, 0x00000000da80b600, 0x00000000da950000)

Metaspace       used 49426K, capacity 51378K, committed 51712K, reserved 1091584K

 class space    used 8476K, capacity 9147K, committed 9216K, reserved 1048576K

[Stephen de Vries]

Hi Albert,

I can’t answer why you’re having that problem, but here’s how we’re starting/stopping ZAP from BDD-Security: https://github.com/continuumsecurity/bdd-security/blob/master/src/test/java/net/continuumsecurity/scanner/ZapManager.java

BTW, we’ve built the same sort of process for JUnit except that before doing ZAP spidering we do selenium navigation, here’s the project: https://github.com/continuumsecurity/zap-webdriver

Java friendly ZAP api might be of interest: https://github.com/continuumsecurity/zap-java-api

cheers,

Stephen

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/zaproxy-develop/b9751756-d4fe-44a3-809b-67ff5dfbaaef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Albert]

Hi Stephen, 

Thanks a lot for the links. That's a very interesting project and it aligns with what I am trying to achieve. 

I will check it out. I also use a ProcessBuilder to start ZAP as daemon but I get into infinite loops/hung outs while scanning eventually. 

Will see if I can get inspired by your work.