ZAP API with my python Project

[wwed jiu]

Hi Simon, I tying apply OWASP ZAP API in my final year project.

my Topic: real time security assessment of web testing framework.

my idea is:

1. create a browser extension capture real time user browsing any website.

2. passing to localhost python file by using flash in port 8000

3. ZAP API are trying to scanning this url. (for this demo, i was use spider scanning first)

4. after scanning return result & pass to php to store in my localhost database.

problem facing:

for my code, I trying capture each url when user browsing, but now the problem is when user visit 1st Website link, zap can scanning and return result, when user 2nd visit next website Step2 can capture URL, only ZAP API showing error link below here:

ERROR:__main__:Error connecting to ZAP: HTTPConnectionPool(host='127.0.0.1', port=8080): Max retries exceeded with url: http://zap/JSON/spider/view/results/?scanId=119 (Caused by ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))) INFO:werkzeug:127.0.0.1 - - [19/Dec/2023 17:41:39] "POST /GetURL3 HTTP/1.1" 200 -

for my browser extension passing URL to this python file:

from flask import Flask, request, jsonify

from spider_scanning1 import process_urls #get the function from scannning.py file

import logging

import requests  # Import the requests library

app = Flask(__name__)

# Configure logging

logging.basicConfig(level=logging.INFO)

logger = logging.getLogger(__name__)

@app.route('/GetURL3', methods=['POST'])

def handle_data():

    data = request.get_json()

    # Access the URLs

    urls = data.get('urls', [])

    # Process each URL individually (e.g., store them in a database)

    for url in urls:

       

        # Skip chrome://new-tab-page/ OR about:blank OR https://www.google.com/search?q=

        #if url.startswith('chrome://new-tab-page/'):

        if url.startswith('chrome://new-tab-page/') or url.startswith('about:blank') or 'https://www.google.com/search?q=' in url:

            #print(f"Skipping new tab page URL: {url}")

            continue

        else:

             

            try:

                #process_urls(url)

                # Process the URL using the function from process_urls.py

                #spider_results = process_urls(url)

                # Send results to PHP script

                #send_results_to_php(spider_results)

                scanID, spider_results = process_urls(url)

                send_results_to_php(scanID, spider_results)

            except Exception as e:

                logger.error(f"Error connecting to ZAP: {e}")

            # Process the URLs using the function from process_urls.py

            ##process_urls(urls)

            #print(f"Processing URL: {url}")

        # Your processing logic here...

    # Send a response (optional)

    return jsonify({'status': 'success'})

def send_results_to_php(scan_id, spider_results):

    print("Sending data to PHP script:", scan_id, spider_results)

    php_script_url = "http://localhost/FYP/Web-OWASP/save_result.php"

    response = requests.post(php_script_url, json={"scan_id": scan_id, "spider_results": spider_results})

    print("PHP script response:", response.text)

if __name__ == '__main__':

    app.run(port=8000)

my spider scanning.py

# scanning.py

#!/usr/bin/env python

import time

#import requests  # Import the requests library

from zapv2 import ZAPv2

def process_urls(url):

    # The URL of the application to be tested

    target = url

    # Change to match the API key set in ZAP, or use None if the API key is disabled

    apiKey = 'ddfqwgs2fk32jfr4b52fwe9n6fw'

    # Change this line in scanning.py

    zap = ZAPv2(apikey=apiKey, proxies={'http': 'http://127.0.0.1:8080', 'https': 'http://127.0.0.1:8080'})

    # Add the following line to set a timeout for requests made by ZAP

    #zap.requests_timeout = 10  # Use an appropriate value for the timeout

    print('Spidering target {}'.format(target))

    # The scan returns a scan id to support concurrent scanning

    scanID = zap.spider.scan(target)

    while int(zap.spider.status(scanID)) < 100:

        # Poll the status until it completes

        print('Spider progress %: {}'.format(zap.spider.status(scanID)))

        time.sleep(1)

    print('Spider has completed!')

    # Prints the URLs the spider has crawled

    print('\n'.join(map(str, zap.spider.results(scanID))))

    #spider_results = '\n'.join(map(str, zap.spider.results(scanID)))

    # Retrieve and return spider results

    spider_results = zap.spider.results(scanID)

    #return spider_results #pass bk the python main function

    return scanID, spider_results

    # TODO: Explore the Application more with Ajax Spider or Start scanning the application for vulnerabilities

here is error message showing:

error_message.jpg

Do you any advise or solutions ? 

Regrads,

Henley.

[psiinon]

Hiya Henley,

I'm a bit confused by the browser extension.

Whats is purpose?

Why not just proxy your browser directly through ZAP to collect the "real time data"?

Regarding the error you are getting - are you sure you are specifying the right API key?

Have a look in the zap.log file - that should give you more details:

https://www.zaproxy.org/faq/somethings-not-working-what-should-i-do/#check-the-log-file

It is worth noting that ZAP is not designed to be a long running multi-user service.

Cheers,

Simon

[wwed jiu]

Hi Simon, for my browser extension. I just used for capture real time URL purpose.

The API Key are correct, it can work in 1st target URL, but 2nd URL become above errors message.

--
You received this message because you are subscribed to the Google Groups "ZAP Developer Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/zaproxy-develop/6080332b-79b3-43c3-b4d2-2db845239d44n%40googlegroups.com.

[psiinon]

Hi Simon, for my browser extension. I just used for capture real time URL purpose.

Thats what proxying is for :) Why not use that as its supported by ZAP out-of-the-box?

 

The API Key are correct, it can work in 1st target URL, but 2nd URL become above errors message.

Then look at the zap.log file as per the link I sent :)

Cheers,

Simon

[wwed jiu]

Education purpose, to used extension capture it.

Sure I will check log first.

To view this discussion on the web, visit https://groups.google.com/d/msgid/zaproxy-develop/1de2ddf7-9e15-4fa1-9e7b-9d1e186f2132n%40googlegroups.com.

[wwed jiu]

Hi Simon,

When I trying again and review the log file it become like this:

zap.log

2023-12-20 10:50:31,258 [ZAP-SpiderThreadPool-5-thread-7] WARN  HttpSenderApache - An error occurred while sending the request:
java.lang.IllegalStateException: Connection pool shut down
at org.apache.hc.core5.util.Asserts.check(Asserts.java:38) ~[httpcore5-5.2.jar:5.2]
at org.apache.hc.core5.pool.LaxConnPool.lease(LaxConnPool.java:163) ~[httpcore5-5.2.jar:5.2]
at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(PoolingHttpClientConnectionManager.java:298) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:103) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:125) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ZapProtocolExec.execute(ZapProtocolExec.java:178) ~[network-beta-0.13.0.zap:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec.execute(ZapHttpRequestRetryExec.java:81) ~[network-beta-0.13.0.zap:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.ZapInternalHttpClient.doExecute(ZapInternalHttpClient.java:173) ~[network-beta-0.13.0.zap:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245) ~[httpclient5-5.2.1.jar:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188) ~[httpclient5-5.2.1.jar:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl0(HttpSenderApache.java:481) ~[network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:362) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:116) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendRateLimited(BaseHttpSender.java:413) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAuthenticated(BaseHttpSender.java:382) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendNoRedirections(BaseHttpSender.java:350) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.send(BaseHttpSender.java:306) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:277) [network-beta-0.13.0.zap:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:233) [network-beta-0.13.0.zap:?]
at org.parosproxy.paros.network.HttpSender.sendImpl(HttpSender.java:529) [zap-2.14.0.jar:2.14.0]
at org.parosproxy.paros.network.HttpSender.sendAndReceive(HttpSender.java:349) [zap-2.14.0.jar:2.14.0]
at org.zaproxy.addon.spider.SpiderTask.fetchResource(SpiderTask.java:420) [spider-release-0.7.0.zap:?]
at org.zaproxy.addon.spider.SpiderTask.runImpl(SpiderTask.java:185) [spider-release-0.7.0.zap:?]
at org.zaproxy.addon.spider.SpiderTask.run(SpiderTask.java:157) [spider-release-0.7.0.zap:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
2023-12-20 10:50:31,258 [ZAP-SpiderThreadPool-5-thread-7] ERROR SpiderTask - An error occurred while fetching the resource [https://github.com/*setup_organization=]: java.lang.IllegalStateException: Connection pool shut down
java.io.IOException: java.lang.IllegalStateException: Connection pool shut down
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:374) ~[?:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:116) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendRateLimited(BaseHttpSender.java:413) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAuthenticated(BaseHttpSender.java:382) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendNoRedirections(BaseHttpSender.java:350) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.send(BaseHttpSender.java:306) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:277) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:233) ~[?:?]
at org.parosproxy.paros.network.HttpSender.sendImpl(HttpSender.java:529) ~[zap-2.14.0.jar:2.14.0]
at org.parosproxy.paros.network.HttpSender.sendAndReceive(HttpSender.java:349) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.addon.spider.SpiderTask.fetchResource(SpiderTask.java:420) [spider-release-0.7.0.zap:?]
at org.zaproxy.addon.spider.SpiderTask.runImpl(SpiderTask.java:185) [spider-release-0.7.0.zap:?]
at org.zaproxy.addon.spider.SpiderTask.run(SpiderTask.java:157) [spider-release-0.7.0.zap:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.hc.core5.util.Asserts.check(Asserts.java:38) ~[?:?]
at org.apache.hc.core5.pool.LaxConnPool.lease(LaxConnPool.java:163) ~[?:?]
at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(PoolingHttpClientConnectionManager.java:298) ~[?:?]
at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:103) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:125) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ZapProtocolExec.execute(ZapProtocolExec.java:178) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec.execute(ZapHttpRequestRetryExec.java:81) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ZapInternalHttpClient.doExecute(ZapInternalHttpClient.java:173) ~[?:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245) ~[?:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188) ~[?:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl0(HttpSenderApache.java:481) ~[?:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:362) ~[?:?]
... 15 more
2023-12-20 10:50:31,259 [ZAP-SpiderThreadPool-5-thread-7] INFO  Spider - Spidering process is complete. Shutting down...
2023-12-20 10:50:31,261 [ZAP-SpiderShutdownThread-5] INFO  SpiderThread - Spider scanning complete: true on https://github..main/version_2 at 2023-12-20T10:50:31.260+0800

May I how any solutions it? 

Regards,

Henley.

[thc...@gmail.com]

Looks like ZAP is shutting down or memory exhausted. You would have to
show the log before those exceptions to know for sure.

Best regards.

>>> <https://groups.google.com/d/msgid/zaproxy-develop/6080332b-79b3-43c3-b4d2-2db845239d44n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .

>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "ZAP Developer Group" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to zaproxy-devel...@googlegroups.com.
>>>
>> To view this discussion on the web, visit

>>> https://groups.google.com/d/msgid/zaproxy-develop/1de2ddf7-9e15-4fa1-9e7b-9d1e186f2132n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/zaproxy-develop/1de2ddf7-9e15-4fa1-9e7b-9d1e186f2132n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>

[Akhmad Syarifurrahman]

I've same issue show in zap.log, I ran scan with zap automation framework, but after Automation plan succeeded ! always hanging / freeze.

[thc...@gmail.com]

Hi,

Provide a thread dump when that happens. What OS are you using?

Best regards.

[Akhmad Syarifurrahman]

Hi,

I'm using Dagger to run the Zap automation framework. After investigating, I found out that the error occurs when I update the addon (zap.sh -cmd -addonupdate).

Regarding the thread dump, I'm currently looking for a way to obtain it inside the Dagger container. I will attach it once I get it.

Best Regards.