Ajax Spider found 0 URL's

[Praveen PB]

Hello Team,

Good day!

Running ZAP scans via GitHub workflows, not allowed to use ZAP actions due to restriction on using external images. Hence, installing the ZAP package with the container along with Firefox. With the help of AF plan running the workflows. The AF plan runs as expected using ZAP GUI.

However, Please find the below issue:

1. Ajax Spider says 0 URL's found
2. Post the plan completes with Automation Plan succeeded, runs for 60 mins to complete the build.

Attached is the logs FYR. Kind request to fix the issue.

Many Thanks

[kingthorin+zap]

1) It seems like your authentication isn't setup properly.

2) It seems that however you're installing Firefox, it isn't found or can't be launched.

2024-04-18 08:21:41,705 [ZAP-SpiderThreadPool-0-thread-7] INFO  User - Authentication failed for user:
2024-04-18T09:22:42.5639193Z 2024-04-18 08:21:41,712 [ZAP-SpiderThreadPool-0-thread-7] INFO  Spider - Spidering process is complete. Shutting down...
2024-04-18T09:22:42.5640885Z 2024-04-18 08:21:41,713 [ZAP-SpiderShutdownThread-0] INFO  SpiderThread - Spider scanning complete: true on Context: *************** at 2024-04-18T08:21:41.713+0000
2024-04-18T09:22:42.5642218Z 2024-04-18 08:21:41,880 [main ] INFO  CommandLine - Job spider found 10 URLs
2024-04-18T09:22:42.5643442Z 2024-04-18 08:21:41,884 [main ] INFO  CommandLine - Job spider test of type stats failed: At least 100 URLs found [10 < 100]
2024-04-18T09:22:42.5644648Z 2024-04-18 08:21:41,885 [main ] INFO  CommandLine - Job spider finished, time taken: 00:00:01
2024-04-18T09:22:42.5645602Z 2024-04-18 08:21:41,890 [main ] INFO  CommandLine - Job spiderAjax started
2024-04-18T09:22:42.5646891Z 2024-04-18 08:21:41,895 [ZAP-AjaxSpiderAuto] INFO  SpiderThread - Running Crawljax (with firefox-headless): Auto - Context: ***************
2024-04-18T09:22:42.5648148Z 2024-04-18 08:21:41,897 [ZAP-AjaxSpiderAuto] INFO  SpiderThread - Starting proxy...
2024-04-18T09:22:42.5649307Z 2024-04-18 08:21:41,899 [ZAP-AjaxSpiderAuto] INFO  SpiderThread - Proxy started, listening at port [42863].
2024-04-18T09:22:42.5651000Z 2024-04-18 08:21:42,144 [ZAP-AjaxSpiderAuto] INFO  Plugins - Loaded org.zaproxy.zap.extension.spiderAjax.SpiderThread$DummyPlugin@59d7e089 as a OnBrowserCreatedPlugin
2024-04-18T09:22:42.5652643Z 2024-04-18 08:21:42,305 [ZAP-AjaxSpiderAuto] WARN  SpiderThread - Failed to start browser firefox-headless

[kingthorin+zap]

Also with regard to updating add-ons when using an automation plan : https://www.zaproxy.org/docs/desktop/addons/automation-framework/job-addons/ ➡ https://www.zaproxy.org/docs/automate/automation-framework/#updating-add-ons

[Praveen PB]

Hi There,

Many thanks for your feedback.

1) It seems like your authentication isn't setup properly.

>>Authentication has been set up and tested in the ZAP GUI, authentication was success. I had removed those details while sharing the logs. 

2) It seems that however you're installing Firefox, it isn't found or can't be launched.

>> Any other way or command to test or debug more on this?

Please let me know if any further information required.

Awaiting your valuable response, Thanks

[thc...@gmail.com]

Don't run as root, Firefox will not start.

Best regards.

>>> 1. Ajax Spider says 0 URL's found
>>> 2. Post the plan completes with *Automation Plan succeeded,* runs for

[Praveen PB]

Hi There,

Thanks for the update, let me check and revert.

[Praveen PB]

Hi Team,

Good Day!

In continuation to the above Issue, still Ajax Spider not working as expected. In-spite of running as zap user. Attached logs - https://paste.mozilla.org/63PMZRbZ.

Kind request to check and assist,

Thanks

[Simon Bennetts]

Please include the full zap.log file.

One of the things we will check is the ZAP and add-on versions. We cannot see then in the truncated log you have supplied.

Cheers,

Simon

[Praveen PB]

Hi Simon,

Please find the complete logs at - https://paste.mozilla.org/pe6StHVz

Thanks,

[Praveen PB]

Hi Team,

Any update on the issue. Awaiting your valuable response.

Thanks,

[Simon Bennetts]

That does not looks like the standard zap.log file - there are a lot of messages missing, including those which tell us exactly which add-ons ZAP is using.

Have you changed the log config?

Details of where to find the zap.log file are given here: https://www.zaproxy.org/faq/somethings-not-working-what-should-i-do/#check-the-log-file

However, from the limited log you have supplied:

• Add-on downloaded to: /home/zap/.ZAP/plugin/webdriverlinux-release-81.zap

This was released on April 26th.

The current version is v88.

Please update and try again. If it still doesnot work please supply the _full_ zap.log file with the default log settings.

Cheers,

Simon

[Praveen PB]

Hello Team,

Good day!

The issue seems to surfacing again, not sure what or where the problem is.No changes were made. Please find the complete logs in the below link.

Logs URL : https://paste.mozilla.org/Mk81ZK1s

Thanks in advance.

Please let me know if any other information needed.

[kingthorin+zap]

You should update add-ons separately form running the automation plan, if it's done in a single step you can't be sure things get loaded properly:

https://www.zaproxy.org/docs/automate/automation-framework/#updating-add-ons

Your report can't be written because you didn't map a writable location?

" Job report failed to generate report: Cannot create directory '/opt/zap/AM_Test'."

Your authentication seems to be failing because ZAP can't find Firefox:

"2024-07-01 12:47:42,827 [ZAP-ActiveScanner-9] INFO  User - Authentication failed for user: Jsmith
2024-07-01 12:47:42,829 [ZAP-ActiveScanner-8] INFO  User - Authenticating user: Jsmith
2024-07-01 12:47:42,839 [ZAP-ActiveScanner-8] ERROR User - An error occurred while authenticating:
org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Response code 500. Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line"

Similar to auth, if the ajaxSpider can't find Firefox then it's not going to spider anything and yes it's going to find zero URLs.