How to spider,scan using ZAP API for the loaded session using (zap.core.load_session(<filepath>))

[lakshmi]

How to spider,scan using ZAP API for the loaded session using (zap.core.load_session(<filepath>))

We have to perform REST APIs security testing and following below approach:

1. Record POST, GET and other methods with params using Postmand and get all in ZAP UI

2. Save the session> copy these session files to linux server

3. User Python client API to load the session and spider by contextName, scan and get results.

Able to load the session, but when tried to spider by contextName facing an issue. Please advice.

Python script:

target = 'https://api-xxx/v2/yyy/user/auth/login'

apiKey = '9876'

zap = ZAPv2(apikey=apiKey, proxies={'http': 'http://127.0.0.1:8090', 'https': 'http://127.0.0.1:8090'})

core = zap.core

sessionName = 'NewSession1'

core.new_session(name=sessionName, overwrite=True, apikey=apiKey)

zap.core.load_session('/home/ubuntu/postman-login.session', apikey=apiKey)

time.sleep(20)

contextName = 'v2'

# zap.context.set_context_in_scope('ConnectID', True)

zap.context.set_context_in_scope(contextName, True)

zap.context.set_context_in_scope('Default Context', False)

zap.core.access_url(url=target, followredirects=True, apikey=apiKey)

print('Spidering context %s' % contextName)

scanId = zap.spider.scan(contextName)

print('printing scanId %s' % scanId)

# Give the Spider a chance to start

time.sleep(5)

while (int(zap.spider.status()) < 100):

    print('Spider progress %: ' + zap.spider.status())

    time.sleep(2)

print('Spider completed')

Error details:

Spidering context v2

printing scanId Provided parameter has illegal or unrecognized value

Traceback (most recent call last):

  File "load.py", line 60, in <module>

    while (int(zap.spider.status()) < 100):

ValueError: invalid literal for int() with base 10: 'Does Not Exist'

Please suggest.

[lakshmi]

to add more info about the loaded session using zap.core.load_session: session has context V2 with included, scope and other details defined hence defining the contextName as v2 in the script.

Please advice as we are blocked due to this.

[psiinon]

Have you checked to see if the load_session(...) call is working?

I dont think you need to include the ".session" part of the file name - that might be causing it to fail.

Cheers

Simon

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/e330775b-b683-4d43-ac0a-79c8933e2bd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lakshmi]

Thanks for the reply. Realized the session is not loading by looking at zap.log details. below is the exception it shows: Please guide.

few details extracted from zap.log:

2017-05-16 09:01:42,038 [ZAP-ProxyThread-1] DEBUG API - handleApiRequest http://zap/JSON/core/action/loadSession/?apikey=&name=%2Fhome%2Fubuntu%2F.ZAP%2Fsession%2Fpostman-login.session

2017-05-16 09:01:42,093 [ZAP-ProxyThread-1] DEBUG Control - runCommandLineOpenSession /home/ubuntu/.ZAP/session/postman-login.session

2017-05-16 09:01:42,093 [ZAP-ProxyThread-1] DEBUG ExtensionLoader - sessionAboutToChangeAllPlugin

2017-05-16 09:01:42,094 [ZAP-ProxyThread-1] DEBUG AttackModeScanner - Stopping

2017-05-16 09:01:42,104 [ZAP-ProxyThread-1] DEBUG ParosDatabase - close

2017-05-16 09:01:42,114 [ZAP-ProxyThread-1] INFO  ENGINE - dataFileCache commit start

2017-05-16 09:01:42,116 [ZAP-ProxyThread-1] INFO  ENGINE - dataFileCache commit end

2017-05-16 09:01:42,126 [ZAP-ProxyThread-1] INFO  ENGINE - Database closed

2017-05-16 09:01:42,227 [ZAP-ProxyThread-1] DEBUG ParosDatabase - open /home/ubuntu/.ZAP/session/postman-login.session

2017-05-16 09:01:42,361 [ZAP-ProxyThread-1] INFO  ENGINE - open start - state modified

2017-05-16 09:01:42,403 [ZAP-ProxyThread-1] INFO  ENGINE - dataFileCache open start

2017-05-16 09:01:42,406 [ZAP-ProxyThread-1] INFO  ENGINE - dataFileCache open end

2017-05-16 09:01:42,422 [ZAP-ProxyThread-1] WARN  ENGINE - NIO access failed

java.lang.IndexOutOfBoundsException

        at java.nio.Buffer.checkBounds(Buffer.java:559)

        at java.nio.ByteBuffer.get(ByteBuffer.java:668)

        at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:279)

        at org.hsqldb.persist.RAFileNIO.read(Unknown Source)

        at org.hsqldb.persist.RAFileHybrid.read(Unknown Source)

        at org.hsqldb.persist.DataFileCache.readObject(Unknown Source)

        at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)

        at org.hsqldb.persist.DataFileCache.get(Unknown Source)

        at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)

        at org.hsqldb.persist.RowStoreAVLDisk.setAccessor(Unknown Source)

        at org.hsqldb.Table.setIndexRoots(Unknown Source)

        at org.hsqldb.Table.setIndexRoots(Unknown Source)

        at org.hsqldb.StatementCommand.getResult(Unknown Source)

        at org.hsqldb.StatementCommand.execute(Unknown Source)

        at org.hsqldb.Session.executeCompiledStatement(Unknown Source)

        at org.hsqldb.scriptio.ScriptReaderText.readDDL(Unknown Source)

        at org.hsqldb.scriptio.ScriptReaderBase.readAll(Unknown Source)

        at org.hsqldb.persist.Log.processScript(Unknown Source)

        at org.hsqldb.persist.Log.open(Unknown Source)

        at org.hsqldb.persist.Logger.open(Unknown Source)

        at org.hsqldb.Database.reopen(Unknown Source)

        at org.hsqldb.Database.open(Unknown Source)

        at org.hsqldb.DatabaseManager.getDatabase(Unknown Source)

        at org.hsqldb.DatabaseManager.newSession(Unknown Source)

        at org.hsqldb.jdbc.JDBCConnection.<init>(Unknown Source)

        at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)

        at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)

        at java.sql.DriverManager.getConnection(DriverManager.java:571)

        at java.sql.DriverManager.getConnection(DriverManager.java:215)

        at org.parosproxy.paros.db.paros.ParosDatabaseServer.start(Unknown Source)

        at org.parosproxy.paros.db.paros.ParosDatabaseServer.<init>(Unknown Source)

        at org.parosproxy.paros.db.paros.ParosDatabase.open(Unknown Source)

        at org.parosproxy.paros.model.Session.open(Unknown Source)

        at org.parosproxy.paros.model.Model.openSession(Unknown Source)

        at org.parosproxy.paros.control.Control.runCommandLineOpenSession(Unknown Source)

        at org.zaproxy.zap.extension.api.CoreAPI.handleApiAction(Unknown Source)

        at org.zaproxy.zap.extension.api.API.handleApiRequest(Unknown Source)

        at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(Unknown Source)

        at org.parosproxy.paros.core.proxy.ProxyThread.run(Unknown Source)

        at java.lang.Thread.run(Thread.java:745)

2017-05-16 09:01:42,430 [ZAP-ProxyThread-1] FATAL ENGINE - DataFileCache.readObject 325687

java.io.IOException: java.lang.IndexOutOfBoundsException

        at org.hsqldb.lib.java.JavaSystem.toIOException(Unknown Source)

        at org.hsqldb.persist.RAFileNIO.read(Unknown Source)

        at org.hsqldb.persist.RAFileHybrid.read(Unknown Source)

        at org.hsqldb.persist.DataFileCache.readObject(Unknown Source)

Please guide.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-user...@googlegroups.com.

[lakshmi]

tried with not giving '.session' also. facing the same issue.

zap.core.load_session('/home/ubuntu/.ZAP/session/postman-login')

On Tuesday, May 16, 2017 at 2:03:19 PM UTC+5:30, Simon Bennetts wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-user...@googlegroups.com.

[thc...@gmail.com]

Was ZAP cleanly shutdown before copying the session files? Seems the
session is corrupt.

Best regards.

On 16/05/17 10:10, lakshmi wrote:
> tried with not giving '.session' also. facing the same issue.
> zap.core.load_session('/home/ubuntu/.ZAP/session/postman-login')
>
>
> On Tuesday, May 16, 2017 at 2:03:19 PM UTC+5:30, Simon Bennetts wrote:
>>
>> Have you checked to see if the load_session(...) call is working?
>> I dont think you need to include the ".session" part of the file name -
>> that might be causing it to fail.
>>
>> Cheers
>>
>> Simon
>>

>>> email to zaproxy-user...@googlegroups.com <javascript:>.

>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/zaproxy-users/e330775b-b683-4d43-ac0a-79c8933e2bd6%40googlegroups.com

>>> <https://groups.google.com/d/msgid/zaproxy-users/e330775b-b683-4d43-ac0a-79c8933e2bd6%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .

[lakshmi]

Thanks for help. .session was corrupted. now able to load the file successfully and same printed in zap.log file. However when tried to spider.scan getting errors. Please guide.

want to spider using the context so wanted to pass contextName=auth

scanId = zap.spider.scan(contextName=auth)

print('printing scanId %s' % scanId)

error:

Traceback (most recent call last):

  File "inprogress.py", line 52, in <module>

    scanId = zap.spider.scan(contextName=auth)

NameError: name 'auth' is not defined

different approaches:

**********************

scanId = zap.spider.scan(contextName='auth')

TypeError: scan() got an unexpected keyword argument 'contextName'

other approach:

scanId = zap.spider.scan('auth')

printing scanId Provided parameter has illegal or unrecognized value

Traceback (most recent call last):

  File "inprogress.py", line 57, in <module>

    while (int(zap.spider.status()) < 100):

ValueError: invalid literal for int() with base 10: 'Does Not Exist'

zap.log:

2017-05-16 10:14:11,567 [ZAP-ProxyThread-15] DEBUG API - handleApiRequest http://zap/JSON/spider/action/scan/?url=auth&apikey=

2017-05-16 10:14:11,567 [ZAP-ProxyThread-15] DEBUG SpiderAPI - Request for handleApiAction: scan (params: {"url":"auth","apikey":""})

2017-05-16 10:14:11,567 [ZAP-ProxyThread-15] DEBUG SpiderAPI - API Spider scanning url: auth

how to pass contextName in spider.scan(<contextName>)?

when looked at ZAP API: contextName=auth works fine

Please suggest.

[lakshmi]

the contextName in the loaded session is 'auth' and able to print using context.context(auth) also:

contextName='auth'

# zap.context.set_context_in_scope(name, True)

print("printing contextname")

pprint(zap.context.context(contextName))

but spider.scan(contextName=auth) doesn't work. please guide.

On Tuesday, May 16, 2017 at 2:42:34 PM UTC+5:30, thc202 wrote:

[psiinon]

It looks like you're using the default directory.

Have you tried:

zap.core.load_session('postman-login', apikey=apiKey)

Cheers

Simon

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/4deca196-82b3-4d69-bd2e-938f92ab3e85%40googlegroups.com.

[thc...@gmail.com]

The parameter name should be all lowercase:
spider.scan(contextname='auth')

or, since you already have a variable with the context name:
spider.scan(contextname=contextName)

Best regards.

[lakshmi]

Thanks for the help. If i load the session zap.core.load_session('postman-login', apikey=apiKey), works (loads the file) but spider.scan(contextName) is not working (in the ZAP API it shows as url=auth whereas expected contextName=auth). Please suggest. below is the python script:

zap = ZAPv2(proxies={'http': 'http://127.0.0.1:8090', 'https': 'http://127.0.0.1:8090'})

zap.core.load_session('postman-auth')

time.sleep(10)

contextName='auth'

# zap.context.set_context_in_scope(name, True)

print("printing contextname")

pprint(zap.context.context(contextName))

print('Spidering context %s' % contextName)

scanId = zap.spider.scan(contextName)

print('printing scanId %s' % scanId)

# Give the Spider a chance to start

time.sleep(5)

while (int(zap.spider.status()) < 100):

    print('Spider progress %: ' + zap.spider.status())

    time.sleep(2)

print('Spider completed')

error details:

 DEBUG API - handleApiRequest http://zap/JSON/spider/action/scan/?url=auth&apikey=

 DEBUG SpiderAPI - Request for handleApiAction: scan (params: {"url":"auth","apikey":""})

DEBUG SpiderAPI - API Spider scanning url: auth

[thc...@gmail.com]

As mentioned in a previous message you need to call:
zap.spider.scan(contextname=contextName)

Best regards.

>>>>> <https://groups.google.com/d/msgid/zaproxy-users/e330775b-b683-4d43-ac0a-79c8933e2bd6%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .

>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "OWASP ZAP User Group" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to zaproxy-user...@googlegroups.com <javascript:>.

>>> To view this discussion on the web visit

>>> https://groups.google.com/d/msgid/zaproxy-users/4deca196-82b3-4d69-bd2e-938f92ab3e85%40googlegroups.com
>>> <https://groups.google.com/d/msgid/zaproxy-users/4deca196-82b3-4d69-bd2e-938f92ab3e85%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .

[lakshmi]

Thanks again for the help. Now able to spider successfully.. :). Able to print spider urls list, it shows two urls. However it should be POST call as in my session. but it takes as GET call and giving 404 not found. Please suggest.

Also when tried to print allurls using pprint(zap.spider.all_urls()) getting error message as below: but able to see the list of urls in zap.log file

Traceback (most recent call last):

  File "inprogress.py", line 68, in <module>

    pprint (zap.spider.all_urls())

TypeError: 'list' object is not callable

[thc...@gmail.com]

It should be just:
pprint (zap.spider.all_urls)

(all_urls is a property.)

Re the GET/POST question, answered in the other thread.