ZAP API with my python Project

217 views
Skip to first unread message

wwed jiu

unread,
Dec 19, 2023, 5:10:08 AM12/19/23
to ZAP Developer Group
Hi Simon, I tying apply OWASP ZAP API in my final year project.

my Topic: real time security assessment of web testing framework.

my idea is:
1. create a browser extension capture real time user browsing any website.
2. passing to localhost python file by using flash in port 8000
3. ZAP API are trying to scanning this url. (for this demo, i was use spider scanning first)
4. after scanning return result & pass to php to store in my localhost database.

problem facing:
for my code, I trying capture each url when user browsing, but now the problem is when user visit 1st Website link, zap can scanning and return result, when user 2nd visit next website Step2 can capture URL, only ZAP API showing error link below here:


ERROR:__main__:Error connecting to ZAP: HTTPConnectionPool(host='127.0.0.1', port=8080): Max retries exceeded with url: http://zap/JSON/spider/view/results/?scanId=119 (Caused by ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))) INFO:werkzeug:127.0.0.1 - - [19/Dec/2023 17:41:39] "POST /GetURL3 HTTP/1.1" 200 -


for my browser extension passing URL to this python file:
from flask import Flask, request, jsonify
from spider_scanning1 import process_urls #get the function from scannning.py file
import logging
import requests  # Import the requests library

app = Flask(__name__)


# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

@app.route('/GetURL3', methods=['POST'])
def handle_data():
    data = request.get_json()

    # Access the URLs
    urls = data.get('urls', [])

    # Process each URL individually (e.g., store them in a database)
    for url in urls:
       
        # Skip chrome://new-tab-page/ OR about:blank OR https://www.google.com/search?q=
        #if url.startswith('chrome://new-tab-page/'):
        if url.startswith('chrome://new-tab-page/') or url.startswith('about:blank') or 'https://www.google.com/search?q=' in url:
            #print(f"Skipping new tab page URL: {url}")
            continue

        else:
             
            try:
                #process_urls(url)

                # Process the URL using the function from process_urls.py
                #spider_results = process_urls(url)

                # Send results to PHP script
                #send_results_to_php(spider_results)
                scanID, spider_results = process_urls(url)
                send_results_to_php(scanID, spider_results)



            except Exception as e:
                logger.error(f"Error connecting to ZAP: {e}")


            # Process the URLs using the function from process_urls.py
            ##process_urls(urls)
            #print(f"Processing URL: {url}")
        # Your processing logic here...

    # Send a response (optional)
    return jsonify({'status': 'success'})


def send_results_to_php(scan_id, spider_results):
    print("Sending data to PHP script:", scan_id, spider_results)
    response = requests.post(php_script_url, json={"scan_id": scan_id, "spider_results": spider_results})
    print("PHP script response:", response.text)




if __name__ == '__main__':
    app.run(port=8000)




my spider scanning.py
# scanning.py

#!/usr/bin/env python
import time
#import requests  # Import the requests library
from zapv2 import ZAPv2

def process_urls(url):
    # The URL of the application to be tested
    target = url
    # Change to match the API key set in ZAP, or use None if the API key is disabled
    apiKey = 'ddfqwgs2fk32jfr4b52fwe9n6fw'


    # Change this line in scanning.py
    zap = ZAPv2(apikey=apiKey, proxies={'http': 'http://127.0.0.1:8080', 'https': 'http://127.0.0.1:8080'})

    # Add the following line to set a timeout for requests made by ZAP
    #zap.requests_timeout = 10  # Use an appropriate value for the timeout

    print('Spidering target {}'.format(target))
    # The scan returns a scan id to support concurrent scanning
    scanID = zap.spider.scan(target)
    while int(zap.spider.status(scanID)) < 100:
        # Poll the status until it completes
        print('Spider progress %: {}'.format(zap.spider.status(scanID)))
        time.sleep(1)

    print('Spider has completed!')
    # Prints the URLs the spider has crawled
    print('\n'.join(map(str, zap.spider.results(scanID))))
    #spider_results = '\n'.join(map(str, zap.spider.results(scanID)))

    # Retrieve and return spider results
    spider_results = zap.spider.results(scanID)
    #return spider_results #pass bk the python main function
    return scanID, spider_results


    # TODO: Explore the Application more with Ajax Spider or Start scanning the application for vulnerabilities



here is error message showing:
error_message.jpg

Do you any advise or solutions ? 

Regrads,
Henley.
error_message.jpg

psiinon

unread,
Dec 19, 2023, 8:08:39 AM12/19/23
to ZAP Developer Group
Hiya Henley,

I'm a bit confused by the browser extension.
Whats is purpose?
Why not just proxy your browser directly through ZAP to collect the "real time data"?

Regarding the error you are getting - are you sure you are specifying the right API key?
Have a look in the zap.log file - that should give you more details:

It is worth noting that ZAP is not designed to be a long running multi-user service.

Cheers,

Simon

wwed jiu

unread,
Dec 19, 2023, 8:12:20 AM12/19/23
to zaproxy...@googlegroups.com
Hi Simon, for my browser extension. I just used for capture real time URL purpose.
The API Key are correct, it can work in 1st target URL, but 2nd URL become above errors message.


--
You received this message because you are subscribed to the Google Groups "ZAP Developer Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/zaproxy-develop/6080332b-79b3-43c3-b4d2-2db845239d44n%40googlegroups.com.

psiinon

unread,
Dec 19, 2023, 8:19:00 AM12/19/23
to ZAP Developer Group
Hi Simon, for my browser extension. I just used for capture real time URL purpose.

Thats what proxying is for :) Why not use that as its supported by ZAP out-of-the-box?
 
The API Key are correct, it can work in 1st target URL, but 2nd URL become above errors message.

Then look at the zap.log file as per the link I sent :)

Cheers,

Simon

wwed jiu

unread,
Dec 19, 2023, 8:25:57 AM12/19/23
to zaproxy...@googlegroups.com
Education purpose, to used extension capture it.

Sure I will check log first.

wwed jiu

unread,
Dec 19, 2023, 9:52:57 PM12/19/23
to ZAP Developer Group
Hi Simon,

When I trying again and review the log file it become like this:

zap.log

2023-12-20 10:50:31,258 [ZAP-SpiderThreadPool-5-thread-7] WARN  HttpSenderApache - An error occurred while sending the request:
java.lang.IllegalStateException: Connection pool shut down
at org.apache.hc.core5.util.Asserts.check(Asserts.java:38) ~[httpcore5-5.2.jar:5.2]
at org.apache.hc.core5.pool.LaxConnPool.lease(LaxConnPool.java:163) ~[httpcore5-5.2.jar:5.2]
at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(PoolingHttpClientConnectionManager.java:298) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:103) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:125) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ZapProtocolExec.execute(ZapProtocolExec.java:178) ~[network-beta-0.13.0.zap:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec.execute(ZapHttpRequestRetryExec.java:81) ~[network-beta-0.13.0.zap:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ZapInternalHttpClient.doExecute(ZapInternalHttpClient.java:173) ~[network-beta-0.13.0.zap:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188) ~[httpclient5-5.2.1.jar:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl0(HttpSenderApache.java:481) ~[network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:362) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:116) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendRateLimited(BaseHttpSender.java:413) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAuthenticated(BaseHttpSender.java:382) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendNoRedirections(BaseHttpSender.java:350) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.send(BaseHttpSender.java:306) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:277) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:233) [network-beta-0.13.0.zap:?]
at org.parosproxy.paros.network.HttpSender.sendImpl(HttpSender.java:529) [zap-2.14.0.jar:2.14.0]
at org.parosproxy.paros.network.HttpSender.sendAndReceive(HttpSender.java:349) [zap-2.14.0.jar:2.14.0]
at org.zaproxy.addon.spider.SpiderTask.fetchResource(SpiderTask.java:420) [spider-release-0.7.0.zap:?]
at org.zaproxy.addon.spider.SpiderTask.runImpl(SpiderTask.java:185) [spider-release-0.7.0.zap:?]
at org.zaproxy.addon.spider.SpiderTask.run(SpiderTask.java:157) [spider-release-0.7.0.zap:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
2023-12-20 10:50:31,258 [ZAP-SpiderThreadPool-5-thread-7] ERROR SpiderTask - An error occurred while fetching the resource [https://github.com/*setup_organization=]: java.lang.IllegalStateException: Connection pool shut down
java.io.IOException: java.lang.IllegalStateException: Connection pool shut down
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:374) ~[?:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:116) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendRateLimited(BaseHttpSender.java:413) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAuthenticated(BaseHttpSender.java:382) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendNoRedirections(BaseHttpSender.java:350) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.send(BaseHttpSender.java:306) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:277) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:233) ~[?:?]
at org.parosproxy.paros.network.HttpSender.sendImpl(HttpSender.java:529) ~[zap-2.14.0.jar:2.14.0]
at org.parosproxy.paros.network.HttpSender.sendAndReceive(HttpSender.java:349) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.addon.spider.SpiderTask.fetchResource(SpiderTask.java:420) [spider-release-0.7.0.zap:?]
at org.zaproxy.addon.spider.SpiderTask.runImpl(SpiderTask.java:185) [spider-release-0.7.0.zap:?]
at org.zaproxy.addon.spider.SpiderTask.run(SpiderTask.java:157) [spider-release-0.7.0.zap:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.hc.core5.util.Asserts.check(Asserts.java:38) ~[?:?]
at org.apache.hc.core5.pool.LaxConnPool.lease(LaxConnPool.java:163) ~[?:?]
at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(PoolingHttpClientConnectionManager.java:298) ~[?:?]
at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:103) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:125) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ZapProtocolExec.execute(ZapProtocolExec.java:178) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec.execute(ZapHttpRequestRetryExec.java:81) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ZapInternalHttpClient.doExecute(ZapInternalHttpClient.java:173) ~[?:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245) ~[?:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188) ~[?:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl0(HttpSenderApache.java:481) ~[?:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:362) ~[?:?]
... 15 more
2023-12-20 10:50:31,259 [ZAP-SpiderThreadPool-5-thread-7] INFO  Spider - Spidering process is complete. Shutting down...
2023-12-20 10:50:31,261 [ZAP-SpiderShutdownThread-5] INFO  SpiderThread - Spider scanning complete: true on https://github..main/version_2 at 2023-12-20T10:50:31.260+0800


May I how any solutions it? 

Regards,
Henley.

thc...@gmail.com

unread,
Dec 24, 2023, 8:28:18 AM12/24/23
to zaproxy...@googlegroups.com
Looks like ZAP is shutting down or memory exhausted. You would have to
show the log before those exceptions to know for sure.

Best regards.
>>> <https://groups.google.com/d/msgid/zaproxy-develop/6080332b-79b3-43c3-b4d2-2db845239d44n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "ZAP Developer Group" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to zaproxy-devel...@googlegroups.com.
>>>
>> To view this discussion on the web, visit
>>> https://groups.google.com/d/msgid/zaproxy-develop/1de2ddf7-9e15-4fa1-9e7b-9d1e186f2132n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/zaproxy-develop/1de2ddf7-9e15-4fa1-9e7b-9d1e186f2132n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>

Akhmad Syarifurrahman

unread,
Mar 14, 2024, 11:31:13 PMMar 14
to ZAP Developer Group
I've same issue show in zap.log, I ran scan with zap automation framework, but after Automation plan succeeded ! always hanging / freeze.
zap(1).log

thc...@gmail.com

unread,
Mar 15, 2024, 4:43:02 AMMar 15
to zaproxy...@googlegroups.com
Hi,

Provide a thread dump when that happens. What OS are you using?

Best regards.

Akhmad Syarifurrahman

unread,
Mar 17, 2024, 9:56:16 PMMar 17
to ZAP Developer Group
Hi,

I'm using Dagger to run the Zap automation framework. After investigating, I found out that the error occurs when I update the addon (zap.sh -cmd -addonupdate).

Regarding the thread dump, I'm currently looking for a way to obtain it inside the Dagger container. I will attach it once I get it.


Best Regards.

Reply all
Reply to author
Forward
0 new messages