Active scan stop during run with error does_not_exist

[Asaf Sahar]

I will start describing how I run tests to scan vulnerabilities (Reflected XSS) after implementing Zap API.

I am testing different endpoints from the same site, running tests in parallel that written in Python and using the pytest testing framework.

For Example:

I am sending 12 tests to run in parallel using xdist -n 10 (https://pypi.org/project/pytest-xdist/). A first batch of 10 tests run, in most of the times 1-2 tests are failing with this error: 

2021-10-11 20:42:08,661 [ZAP-ProxyThread-965] WARN  API - Bad request to API endpoint [/JSON/ascan/view/status/] from [127.0.0.1]:

org.zaproxy.zap.extension.api.ApiException: does_not_exist

at org.zaproxy.zap.extension.ascan.ActiveScanAPI.getActiveScan(ActiveScanAPI.java:769) ~[zap-2.10.0.jar:2.10.0]

at org.zaproxy.zap.extension.ascan.ActiveScanAPI.handleApiView(ActiveScanAPI.java:963) ~[zap-2.10.0.jar:2.10.0]

at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:526) [zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:497) [zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:333) [zap-2.10.0.jar:2.10.0]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]

More info:

1. I start new session (overwrite=True) for each endpoint
2. For POST requests I am building openapi file and uploading it to zap
3. For GET requests I am using Spider scan before the active scan

I can't understand why the active scan stop in the middle, I suspected that it's because that new session is starting and overwrite existing scans but I created a waiter using zap.ascan.scans to check that all scans has status FINISHED before starting a new session

[Simon Bennetts]

Dont do that :)

Starting a new session wipes the existing db and stops any outstanding scans.

I'd recommend launching a new ZAP instance for each scan, making sure you use different ports and directories.

Cheers,

Simon

[Asaf Sahar]

If I am adding a waiter according to zap.ascan.scans and waiting until all scans has status FINISHED, it's not a good option?

[Asaf Sahar]

Wanted to consult on other approach.

I removed the creation of new session and just running the scans against the same currents session.

It turns to be more stable for around 70 scans and then I started to get this exception:

2021-10-14 09:31:05,343 [ZAP-ProxyThread-5207] ERROR API - Exception while handling API request:

java.lang.ArrayIndexOutOfBoundsException: 156 > 0

at java.util.Vector.insertElementAt(Vector.java:603) ~[?:1.8.0_275]

at javax.swing.tree.DefaultMutableTreeNode.insert(DefaultMutableTreeNode.java:191) ~[?:1.8.0_275]

at javax.swing.tree.VariableHeightLayoutCache.createNodeAt(VariableHeightLayoutCache.java:781) ~[?:1.8.0_275]

at javax.swing.tree.VariableHeightLayoutCache.treeNodesInserted(VariableHeightLayoutCache.java:491) ~[?:1.8.0_275]

at javax.swing.plaf.basic.BasicTreeUI$Handler.treeNodesInserted(BasicTreeUI.java:3878) ~[?:1.8.0_275]

at javax.swing.tree.DefaultTreeModel.fireTreeNodesInserted(DefaultTreeModel.java:517) ~[?:1.8.0_275]

at javax.swing.tree.DefaultTreeModel.nodesWereInserted(DefaultTreeModel.java:314) ~[?:1.8.0_275]

at javax.swing.tree.DefaultTreeModel.insertNodeInto(DefaultTreeModel.java:241) ~[?:1.8.0_275]

at org.parosproxy.paros.view.AbstractParamContainerPanel.addNewNode(AbstractParamContainerPanel.java:395) ~[zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.view.AbstractParamContainerPanel.addParamPanel(AbstractParamContainerPanel.java:425) ~[zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.view.AbstractParamDialog.addParamPanel(AbstractParamDialog.java:293) ~[zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.view.SessionDialog.addParamPanel(SessionDialog.java:83) ~[zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.view.AbstractParamDialog.addParamPanel(AbstractParamDialog.java:308) ~[zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.view.View.addContext(View.java:767) ~[zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.model.Session.addContext(Session.java:1397) ~[zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.model.Session.getNewContext(Session.java:1336) ~[zap-2.10.0.jar:2.10.0]

at org.zaproxy.zap.extension.api.ContextAPI.handleApiAction(ContextAPI.java:253) ~[zap-2.10.0.jar:2.10.0]

at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:507) [zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:497) [zap-2.10.0.jar:2.10.0]

at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:333) [zap-2.10.0.jar:2.10.0]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]

And wasn't able to make zap running active scans again.

Would like to hear you opinion.

Thank you very much for your support

[Simon Bennetts]

See the line in the above stack trace: Session.getNewContext(Session.java:1336) ?

That means you are still starting new ZAP sessions.

FYI ZAP is not designed as a long running service.

We did look into how much work would be involved changing ZAP to allow it to be a long running service but we decided that it would take too much time and effort for the relatively small number of people we have available.

I still recommend that new ZAP instances are launched for each scan in this sort of environment.

Cheers,

Simon

[Asaf Sahar]

I am not starting new session but I do create a new context for each endpoint, it's also not recommended?

[Simon Bennetts]

Arrg, sorry - I misread that - creating a new context for each endpoint is fine.

Are you running ZAP is desktop mode?

If so, why?

Cheers,

Simon

[Asaf Sahar]

I am running ZAP on desktop mode just to have visualization of the scans for testing purposes. My real tests are running vs docker.

I was trying to use the zap web but I wasn't able to run automated scans there. 
Other than starting zap on docker container for each URL I want to scan, what is your opinion on creating 1 zap instance on docker, without creating new session for each, and having context for each URL.

I was able to run around 150 scans until got the first failure on contextId: “contextId isn’t valid, contextId is internal_error” (I don't have the exception). Creating a new session (while there aren't active scans running) solved the problem 

[Simon Bennetts]

Creating a new context before scanning each URL makes sence, but make sure you put in suitable "sleeps"

I hit a problem once where I was using the same db and it was taking longer to shutdown that I expecte - I ended up getting strange exceptions so you might have hit something similar.

However I still dont recommend running one ZAP instance for a long period of time, it wasnt designed to be run like that and we do not test it in that sort of environment.

I'd be very happy to accept PRs which make ZAP work better for a longer period of time but its not something that I'm going to focus on for now.

Cheers,

Simon

[Asaf Sahar]

Thank you very much for your support.

The behavior you described about the strange exceptions is exactly what I experienced. Though since I stopped creating new session for each scan and having the same session make it more stable.

I am now checking how much scans I succeed before the exceptions start and in parallel investigating framework that will use docker container for each scan.

In case of instability in the scans I think that creating new session from /UI/core/action/newSession/ can be a workaround for now