"Check for updates call failed" when Running ZAP fullscan inside docker container

[reda benfilali]

Dear,

We are running ZAP full scan in a jenkins pipeline from RHEL8 server, we have opened the conexion from the server to URL: https://news.zaproxy.org:443.

1)  We run zap in the pipeline as following:

sh "docker run -v \$(pwd):/zap/wrk/:rw -w /zap/wrk --userns=keep-id --user=\$(id -ur):\$(id -gr) --name ${OwaspContainerName} -p 8080:8080 --env http_proxy=HTTP_PROXY_URL:8080 --env https_proxy=HTTsP_PROXY_URL:8080 -t owasp/zap2docker-stable zap-full-scan.py -t $target -r evvet_vet_report.html -j -a"

==> We got the following error :

43955 [ZAP-cfu] INFO  org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec - Recoverable I/O exception (java.net.SocketException) caught when processing request to {s}->https://cfu.zaproxy.org:443

Thanks in advance,

Regards,

Reda

[reda benfilali]

FYI:

We can CURL the https://news.zaproxy.org:443 from the pipeline without docker and with docker, attaching print screen "CURL_OWASP_ORG". 

Regards,
Reda Benfilali.

[thc...@gmail.com]

Hi.

You also need to allow https://cfu.zaproxy.org/ (cfu vs news).

The screenshot was not attached.

Best regards.

[Simon Bennetts]

FYI all of the ZAP 'calls home' are documented on https://www.zaproxy.org/faq/what-calls-home-does-zap-make/

[reda benfilali]

Dear,

I see we should open cfu, news and tel:

We will try and give feedback.

Regards,
Reda BENFILALI

--
You received this message because you are subscribed to a topic in the Google Groups "ZAP User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-users/H1K2w-v2K6I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zaproxy-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/cf995422-c008-9bce-8bfb-fd597b52dbff%40gmail.com.

[reda benfilali]

Dear,

We are still getting the error :

org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec - Recoverable I/O exception (java.net.SocketException) caught when processing request to {s}->https://news.zaproxy.org:443
164000 [ZAP-cfu] ERROR org.zaproxy.addon.callhome.ExtensionCallHome - Network is unreachable (connect failed)

java.net.SocketException: Network is unreachable (connect failed)

 

However, the connexions are open, we can CURL all the following URLs: https://news.zaproxy.org:443, https://cfu.zaproxy.org:443, https://tel.zaproxy.org:443

Regards,
Reda BENFILALI

[thc...@gmail.com]

If you need the proxy to access external sites you'll have to configure
it in ZAP. (ZAP does not read those env vars.)

Best regards.

[reda benfilali]

Dear,

I'm not able to run it in the jenkins pipeline using the -z -config params:

+ docker run -v PWD_PATH:/zap/wrk/:rw -w /zap/wrk --userns=keep-id --user=1011:1011 --name ZAP_CONTAINER_NAME -p 8080:8080 --env NO_PROXY=google.com --env http_proxy=HTTP_PROXY --env https_proxy=HTTPS_PROXY -t owasp/zap2docker-stable zap-full-scan.py -z -config network.connection.timeoutInSecs=60 network.connection.httpProxy.enabled=true network.connection.httpProxy.host=HTTP_PROXY network.connection.httpProxy.port=8080 -t WEB_APP_TARGET_URL -r Report.html -j -a

Please see attached print screen "ZAP_CONFIG_PARAMS" .

Regards,
Reda

[thc...@gmail.com]

The value to -z should be quoted as that should be passed to ZAP, rather
than being processed by the scan script, e.g.:
-z "-config … -config …"

Best regards.

[reda benfilali]

Dear,

set the pipeline sh:

sh "docker run -v \$(pwd):/zap/wrk/:rw -w /zap/wrk --userns=keep-id --user=\$(id -ur):\$(id -gr) --name ${OwaspContainerName} -p 8080:8080 --env NO_PROXY=google.com --env http_proxy=http_proxy:8080 --env https_proxy=https_proxy:8080 -t owasp/zap2docker-stable $scan_type -z \"-config network.connection.timeoutInSecs=60 -config network.connection.httpProxy.enabled=true -config network.connection.httpProxy.host=http_proxy -config network.connection.httpProxy.port=8080\" -t $target -r report.html"

Got logs as follow, it does consider the proxy settings but got error :  

<< [ZAP-cfu] ERROR org.zaproxy.addon.callhome.ExtensionCallHome - http://proxy.eudra.org >>

bellow the logs:

+ docker run -v PWD_PATH:/zap/wrk/:rw -w /zap/wrk --userns=keep-id --user=1011:1011 --name owasp-zap-container-xxx -p 8080:8080 --env NO_PROXY=google.com --env http_proxy=http_proxy:8080 --env https_proxy=http_proxy:8080 -t owasp/zap2docker-stable zap-full-scan.py -z '-config network.connection.timeoutInSecs=60 -config network.connection.httpProxy.enabled=true -config network.connection.httpProxy.host=http_proxy -config network.connection.httpProxy.port=8080' -t HTTPS://WEB_APP_URL -r report.html 

Found Java version 11.0.20
Available memory: 7762 MB
Using JVM args: -Xmx1940m
724 [main] INFO  org.zaproxy.zap.DaemonBootstrap - OWASP ZAP 2.13.0 started 08/09/2023, 16:30:33 with home /zap/wrk/.ZAP/
761 [main] INFO  org.parosproxy.paros.common.AbstractParam - Setting config database.recoverylog = false was false
761 [main] INFO  org.parosproxy.paros.common.AbstractParam - Setting config api.disablekey = true was true
761 [main] INFO  org.parosproxy.paros.common.AbstractParam - Setting config api.addrs.addr.name = .* was .*
761 [main] INFO  org.parosproxy.paros.common.AbstractParam - Setting config api.addrs.addr.regex = true was true
762 [main] INFO  org.parosproxy.paros.common.AbstractParam - Setting config spider.maxDuration = 0 was 0
762 [main] INFO  org.parosproxy.paros.common.AbstractParam - Setting config network.connection.timeoutInSecs = 60 was null
762 [main] INFO  org.parosproxy.paros.common.AbstractParam - Setting config network.connection.httpProxy.enabled = true was null
763 [main] INFO  org.parosproxy.paros.common.AbstractParam - Setting config network.connection.httpProxy.host = http_proxy was null
763 [main] INFO  org.parosproxy.paros.common.AbstractParam - Setting config network.connection.httpProxy.port = 8080 was null

...

3811 [ZAP-daemon] INFO  org.flywaydb.core.internal.database.base.BaseDatabaseType - Database: jdbc:hsqldb:file:/zap/wrk/.ZAP/db/permanent (HSQL Database Engine 2.7)
3816 [ZAP-daemon] WARN  org.flywaydb.core.internal.database.base.Database - Flyway upgrade recommended: HSQLDB 2.7 is newer than this version of Flyway and support has not been tested. The latest supported version of HSQLDB is 2.6.
3836 [ZAP-daemon] INFO  org.flywaydb.core.internal.command.DbValidate - Successfully validated 1 migration (execution time 00:00.013s)
3842 [ZAP-daemon] INFO  org.flywaydb.core.internal.command.DbMigrate - Current version of schema "PUBLIC": 1
3845 [ZAP-daemon] INFO  org.flywaydb.core.internal.command.DbMigrate - Schema "PUBLIC" is up to date. No migration necessary.
3852 [ZAP-daemon] INFO  org.zaproxy.addon.oast.services.callback.CallbackService - Started callback service on 0.0.0.0:43967
3942 [ZAP-cfu] ERROR org.zaproxy.addon.callhome.ExtensionCallHome - http_proxy
org.zaproxy.addon.network.common.ZapUnknownHostException:  http_proxy
at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]
at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:883) ~[?:?]
at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1386) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1307) ~[?:?]
at org.apache.hc.client5.http.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:43) ~[?:?]
at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[?:?]
at org.apache.hc.client5.http.impl.io.ZapHttpClientConnectionOperator.connect(ZapHttpClientConnectionOperator.java:95) ~[?:?]
at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:447) ~[?:?]
at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162) ~[?:?]
at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:146) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ZapProtocolExec.execute(ZapProtocolExec.java:178) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec.execute(ZapHttpRequestRetryExec.java:81) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
at org.apache.hc.client5.http.impl.classic.ZapInternalHttpClient.doExecute(ZapInternalHttpClient.java:173) ~[?:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245) ~[?:?]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188) ~[?:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl0(HttpSenderApache.java:481) ~[?:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:362) ~[?:?]
at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:116) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendRateLimited(BaseHttpSender.java:412) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAuthenticated(BaseHttpSender.java:381) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendNoRedirections(BaseHttpSender.java:349) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.send(BaseHttpSender.java:305) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:276) ~[?:?]
at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:233) ~[?:?]
at org.parosproxy.paros.network.HttpSender.sendImpl(HttpSender.java:524) ~[zap-2.13.0.jar:2.13.0]
at org.parosproxy.paros.network.HttpSender.sendAndReceive(HttpSender.java:517) ~[zap-2.13.0.jar:2.13.0]
at org.zaproxy.addon.callhome.ExtensionCallHome.sendServiceRequest(ExtensionCallHome.java:197) ~[?:?]
at org.zaproxy.addon.callhome.ExtensionCallHome.getServiceData(ExtensionCallHome.java:216) ~[?:?]
at org.zaproxy.addon.callhome.ExtensionCallHome.getCheckForUpdatesData(ExtensionCallHome.java:225) ~[?:?]
at org.zaproxy.addon.callhome.ExtensionCallHome.get(ExtensionCallHome.java:235) ~[?:?]
at org.zaproxy.zap.extension.autoupdate.ExtensionAutoUpdate.getRemoteConfiguration(ExtensionAutoUpdate.java:1002) [zap-2.13.0.jar:2.13.0]
at org.zaproxy.zap.extension.autoupdate.ExtensionAutoUpdate$4.run(ExtensionAutoUpdate.java:1130) [zap-2.13.0.jar:2.13.0]
4899 [ZAP-daemon] ERROR org.parosproxy.paros.CommandLine - Check for updates call failed
4901 [ZAP-cfu] ERROR org.zaproxy.addon.callhome.ExtensionCallHome - http_proxy

Thanks in advance.
Regards.

[Simon Bennetts]

It looks like "http_proxy" cannot be resolved in your docker container.

Is that really the address you want to use?

What happens when you run:

• docker run -t owasp/zap2docker-stable curl http://http_proxy

Cheers,

Simon

[reda benfilali]

We are able to connect the application inside docker through the prox:

+ docker run --rm -p 8080:8080 --env NO_PROXY=google.com --env http_proxy=http_proxy:8080 --env https_proxy=https_proxy:8080 -t owasp/zap2docker-stable /usr/bin/curl -i --proxy proxy:8080 https://web_app_url
HTTP/1.1 200 Connection established

HTTP/2 401

We are not able to connect to the proxy itself, due to security reason, the proxy allow only connexion on port 8080:

+ docker run -t owasp/zap2docker-stable curl http://proxy.domain.org
curl: (7) Failed to connect to proxy.domain.org port 80: Connection refused



pipeline 79:
CURL the proxy on port 8080, the proxy is accessible but no connexion allowed to the proxy itself

+ docker run --rm -p 8080:8080 --env NO_PROXY=google.com --env http_proxy=http_proxy:8080 --env https_proxy=http_proxy:8080 -t owasp/zap2docker-stable /usr/bin/curl -i http_proxy:8080
HTTP/1.1 403 Forbidden

[Simon Bennetts]

The error message:

•  3942 [ZAP-cfu] ERROR org.zaproxy.addon.callhome.ExtensionCallHome - http_proxy
  org.zaproxy.addon.network.common.ZapUnknownHostException:  http_proxy

implies that Java cannot resolve the hostname http_proxy in the docker container.

Thats what I think you need to fix.

If java cannot do that then ZAP has no chance of being able to use it as a proxy.

Cheers,

Simon

[reda benfilali]

==> Updated the /etc/hosts inside docker adding the proxy IP address: 

+ docker exec -i zap-container cat /etc/hosts xxx.xxx.xxx.xxx proxy.domain.org

==> but still getting the same issue in zap.out file "ZapUnknownHostException": 

3937 [ZAP-cfu] ERROR org.zaproxy.addon.callhome.ExtensionCallHome - http://proxy.eudra.org
org.zaproxy.addon.network.common.ZapUnknownHostException: http://proxy.eudra.org

[Simon Bennetts]

Looks like you need an entry for http://proxy.eudra.org/ as well then :)

[reda benfilali]

Yes it is the entry I added for the proxy.

adding to that, we are able to curl the proxy inside docker:

root@d3e124396966:/zap/wrk# curl http_proxy_url

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" http://www.w3.org/TR/html4/strict.dtd>

<html><head>

Thanks.

Le mercredi 13 septembre 2023 à 17:57:15 UTC+2, psi...@gmail.com a écrit :

Looks like you need an entry for  as well then :)

On Wednesday, 13 September 2023 at 17:55:43 UTC+2 benfil...@gmail.com wrote:

==> Updated the /etc/hosts inside docker adding the proxy IP address: 

+ docker exec -i zap-container cat /etc/hosts xxx.xxx.xxx.xxx proxy.domain.org

==> but still getting the same issue in zap.out file "ZapUnknownHostException": 

3937 [ZAP-cfu] ERROR org.zaproxy.addon.callhome.ExtensionCallHome - http://proxy.domain.org
org.zaproxy.addon.network.common.ZapUnknownHostException: http://proxy.domain.org

[reda benfilali]

==> Question 2: I got a result saying all pass, but in the logs I got error failed to read the application ?

...

PASS: Server Side Template Injection [90035]

PASS: Server Side Template Injection (Blind) [90036]

FAIL-NEW: 0 FAIL-INPROG: 0 WARN-NEW: 0 WARN-INPROG: 0 INFO: 0 IGNORE: 0 PASS: 102

15768 [ZAP-Scanner-0] INFO  org.parosproxy.paros.core.scanner.HostProcess - start host https://app_url | PathTraversalScanRule strength MEDIUM threshold MEDIUM
15769 [ZAP-Scanner-0] WARN  org.parosproxy.paros.core.scanner.HostProcess - Failed to obtain the HTTP response for href [id=2, type=0, URL=https://app_url]: http://proxy.domain.org

Regards

[Simon Bennetts]

PASS in this case just means that the relevent rule did not raise any alerts.

The packaged scans should warn you if they fail to access the target:

docker run -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t https://www.this-does-not-exist-dgdfsg3464.com

Using the Automation Framework
Automation plan failures:
Job spider failed to access URL https://www.this-does-not-exist-dgdfsg3464.com check that it is valid : www.this-does-not-exist-dgdfsg3464.com: Name or service not known

Cheers,

Simon