Browser Authentication

[Hector Luna]

I am running Browser-Based Authentication on my website. It is all working well and I am using `firefox-headless` as the browser, but for some reason it has decided that it wants to spend a lot of time "calling home".

Is there a way to prevent it from doing that?
Right now it is spitting all this into the console (I have a script that intercepts requests and tells me who is the request initiator, and what URL it is attempting to get to...)

I am also using the following file in /home/zap/.mozilla/firefox/distribution/policies.json

{

"policies": {

"AppAutoUpdate": false,

"BlockAboutConfig": true,

"DisableAppUpdate": true,

"DisableTelemetry": true

}

}

The logs look like this, but there are many more. The issue is that the browser authentication spends so  much time dealing with firefox junk that it isn't traversing my app like it should, or at the very least, that is what it looks like it is doing. Any help/suggestions would be greatly appreciated.

Thanks!

2024-04-30 02:54:14.072 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/monitor/collections/changes/changeset?_expected=0

2024-04-30 02:54:14.139 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/addons-bloomfilters/changeset?_expected=1714394162447&_since=%221712342161400%22

2024-04-30 02:54:14.211 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2024-06-09-11-51-09.chain

2024-04-30 02:54:14.287 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/gfx?_expected=1692730580117

2024-04-30 02:54:14.359 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/normandy-recipes-capabilities/changeset?_expected=1714435264307

2024-04-30 02:54:14.446 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/search-telemetry-v2/changeset?_expected=1713187389066&_since=%221711574102228%22

2024-04-30 02:54:14.534 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/sites-classification?_expected=1544035467383

2024-04-30 02:54:14.578 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/anti-tracking-url-decoration?_expected=1564511755134

2024-04-30 02:54:14.634 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/public-suffix-list/changeset?_expected=1575468539758

2024-04-30 02:54:14.692 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/hijack-blocklists?_expected=1605801189258

2024-04-30 02:54:14.744 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/pioneer-study-addons-v1/changeset?_expected=1607042143590

2024-04-30 02:54:14.811 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/top-sites?_expected=1647020600359

2024-04-30 02:54:14.871 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/doh-providers/changeset?_expected=1647549722107&_since=%221621943542621%22

2024-04-30 02:54:14.937 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/doh-config/changeset?_expected=1651753780606&_since=%221621943462970%22

2024-04-30 02:54:15.014 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/devtools-devices?_expected=1653469171354

2024-04-30 02:54:15.067 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/websites-with-shared-credential-backends?_expected=1659924446436

2024-04-30 02:54:15.134 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/language-dictionaries?_expected=1673270322227

2024-04-30 02:54:15.192 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/password-recipes?_expected=1674595048726

2024-04-30 02:54:15.284 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/password-rules?_expected=1679600032742

2024-04-30 02:54:15.331 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/fxmonitor-breaches/changeset?_expected=1683667257606

2024-04-30 02:54:15.559 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/addons-manager-settings/changeset?_expected=1688747728721

2024-04-30 02:54:15.616 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/url-classifier-skip-urls?_expected=1701090424142

2024-04-30 02:54:15.680 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/search-default-override-allowlist?_expected=1710168995103

2024-04-30 02:54:15.727 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/search-config?_expected=1710766850143

2024-04-30 02:54:15.801 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/devtools-compatibility-browsers/changeset?_expected=1713943930239&_since=%221711547229147%22

2024-04-30 02:54:15.935 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/cookie-banner-rules-list/changeset?_expected=1713281881442&_since=%221711550955136%22

2024-04-30 02:54:16.028 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/cert-revocations/changeset?_expected=1714427834885

2024-04-30 02:54:16.083 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://content-signature-2.cdn.mozilla.net/chains/onecrl.content-signature.mozilla.org-2024-06-09-11-51-06.chain

2024-04-30 02:54:16.421 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/intermediates/changeset?_expected=1714427823465&_since=%221712311023721%22

2024-04-30 02:54:16.887 -- 000076 -- INF -- AUTHENTICATION_HELPER -- GET https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/onecrl?_expected=1710189695302

[Hector Luna]

The weekly and stable docker image exhibit this behaviour as well.

[Hector Luna]

So maybe the right question to ask is:

How can I duplicate the same behaviour from the "Authentication Tester" when using the automation framework?

It works well from the UI, and it only inspects the URLs that are relevant. None of the firefox noise I see when I run it from the automation framework.
I am hoping to get some information on how to duplicate this behaviour using the automation framework because it is just not happening for me.

Running it from the GUI generates a number of alerts.
Running it from the automation framework generates a fraction of the alerts. No limits on the number of alerts, no rules off.

What am I doing wrong?
Is it even possible to get the same results using the automation framework?
Do proxy scripts even work when using browser authentication via automation framework?

Thank you!

[Simon Bennetts]

Hiya,

You should be able to reproduce the results from the auth tester with the automation framework (AF) - they use the same underlying code.
Lets leave the Firefox noise for now, unless you think this is causing some of your problems?

How are you running your AF plan?

If you are running it from the commandline then try running it from within the desktop.
Does it work in the desktop?
If not, can you share your plan?
Feel free to obfuscate sensitive info.

Cheers,

Simon

[Hector Luna]

Hi Simon,

I am running the AF from docker using the ZAP weekly image directly from the source (I have a different image that I build based on the one from you guys, but that is only to eliminate webswing which I don't use for our CI environments).
But the same result occurs when using the weekly image zaproxy/zap-weekly.

Here is the yaml file:

---

env:

contexts:

- name: "Application - Browser Authentication"

urls:

- "https://${targetEnvironment}"

includePaths:

- "https://${targetEnvironment}.*"

excludePaths: []

authentication:

method: "browser"

parameters:

loginPageUrl: "https://${targetEnvironment}"

loginPageWait: 30

browserId: "firefox-headless"

verification:

method: "autodetect"

sessionManagement:

method: "autodetect"

parameters: {}

technology:

exclude: []

users:

- name: "user-credentials"

credentials:

username: "${targetUsername}"

password: "${targetPassword}"

parameters:

failOnError: false

failOnWarning: false

progressToStdout: true

vars:

targetEnvironment: "OBFUSCATED_APPLICATION_DOMAIN"

targetUsername: "${ENV_VAR_USERNAME}"

targetPassword: "${ENV_VAR_PASSWORD}"

jobs:

- parameters:

maxAlertsPerRule: 0

scanOnlyInScope: false

maxBodySizeInBytesToScan: 0

enableTags: true

disableAllRules: false

rules: []

name: "passiveScan-config"

type: "passiveScan-config"

- parameters:

action: "add"

type: "standalone"

engine: "ECMAScript : Oracle Nashorn"

name: "Environment - Configuration.js"

file: "/zap/wrk/scripts/standalone/Environment - Configuration.js"

name: "script"

type: "script"

- parameters:

action: "run"

type: "standalone"

engine: ""

name: "Environment - Configuration.js"

name: "script"

type: "script"

- parameters:

action: "add"

type: "httpsender"

engine: "ECMAScript : Oracle Nashorn"

name: "HttpRequest - Browser Processor.js"

file: "/zap/wrk/scripts/httpsender/HttpRequest - Browser Processor.js"

name: "script"

type: "script"

- parameters:

user: "user-credentials"

requests:

- url: "http://${targetEnvironment}"

name: "initial-request"

method: "GET"

httpVersion: "HTTP/1.1"

headers: []

data: ""

responseCode: 301

name: "requestor"

type: "requestor"

- parameters:

maxDuration: 0

name: "passiveScan-wait"

type: "passiveScan-wait"

- parameters:

template: "modern"

theme: "corporate"

reportDir: "/zap/wrk/reports"

reportFile: "{{yyyy-MM-dd}}-Modern-Report-app-browser.${targetEnvironment}"

reportTitle: "Application - Browser Authentication Report"

reportDescription: ""

displayReport: true

risks:

- "info"

- "low"

- "medium"

- "high"

confidences:

- "low"

- "medium"

- "high"

- "confirmed"

sections:

- "instancecount"

- "alertdetails"

- "alertcount"

- "params"

- "chart"

- "statistics"

name: "report"

type: "report"

A few things to note:

• The Configuration script is used to store some values into ScriptVars. Things such as where I want the output generated by the httpsender script.
• The Browser Processor.js script is used to create a log file with information about the URLs I am hitting, and who initiated the request (as seen in the firefox noise at the start of the thread).
• Excluding these two scripts has no effect. I get the same results.
• The `http` in the requestor is fine. It triggers the whole process the same way as if I used `https`.
• I am letting ZAP detect the session management and verification.
• Setting "enableTags" to true or false has no effect on the results.

The Actual Issue:

• Running the authentication tester via ZAP GUI with the same details yields some 30+ alerts (No passive scan rules restrictions).
  
• Running it from the AF yields 4, sometimes 5 alerts.
• I do get to see all of our sites in the "Sites" section of the report, as well as details about the cookies, etc in the report, but the alerts are not showing, and external sources (ie goole things) do not show.
• The ZAP GUI reports on all the sites that the authentication tester visits, and these are included in the report when generated manually.

So that is about it. Please do advise if there is something that I am doing wrong (which is likely the case).
Thank you very much!!!

[Hector Luna]

So I have been doing further research into this and I think that the issue I am having has to do with one of several potential issues:

1. The passive scan wait job doesn't fully wait for the authentication helper/browser authentication to stop.
2. The report generation jobs do not wait for the authentication helper/browser authentication to stop.
3. There is some sort of disconnect between the authentication helper/browser authentication and the report generation.

The reason I am getting to this conclusion is because of the following observations:

1. When I run the exact same automation job via a docker container, or directly using zap.sh I get the same result: only a few alerts (count: X). A number that varies from one run to another and usually results in a lower alert number. Note that I can only see the number of alerts via the reports I am generating, so it is possible that all of the expected alerts are still being generated.
2. When I run the exact same automation job via the ZAP GUI (automation tab), I get most of the alerts (count: Y) I would expect to get but the reports only include a small portion of the alerts (count: X). These alerts show up on the alerts tab, and that number is greater than the number of alerts presented in the report generated via AF (Y > X). Naturally, they all show up if I generate the report via GUI once it is all said and done.
3. Running the authentication tester yields Z number of alerts. Generating an automation plan (via the instructions given here https://www.zaproxy.org/docs/authentication/) results in getting the same number of alerts as observed in step #2 (count: Y). Which leads me to believe that there is a difference in how the tester operates vs AF. 
4. Another thing to note is that the authentication tester does not produce any of the mozilla noise I see when running the AF via GUI, docker, or directly using zap.sh -cmd -autorun...

So I know it isn't my automation plan doing weird things, or how it is configured, but there is something happening when trying to capture all the things that went on during browser authentication that prevents all of the alerts to be included in any of the reports generated.

Any advice? 

Thanks!

[Simon Bennetts]

Hiya,

I think the main difference you are seeing is this:

When the Authentication Tester runs then its just the same as launching a browser yourself, they are "Proxy" requests which then get passively scanned.

When you use Browser Based Authentication then the browser uses a separate proxy - the requests will appear in the History table as "Authentication" requests, these are not passively scanned.

You are right in that the passive scan wait job doesn't fully wait for the authentication helper/browser authentication to stop.

It just waits for the passive scan queue to empty.

I would have expected this to only empty after the authentication and the associated request has finished.

However in this case I suspect the following is happening:

1. The requester initiates the request - this will be made in a new thread
   
2. Theres no session, so the authentication starts, all in another thread
3. The passive scan wait job starts. No requests have been made, so the passive scan queue is empty, the job exits
4. The authentication finishes
5. The requester can now make its request, this will get added to the passive scan queue, but its too late :)

The easiest solution to this would be to add a delay job before the passive scan wait job.

Does that help?

Cheers,

Simon

[Hector Luna]

Thank you Simon!
That does make a lot of sense. I will try that out and report back!

[Hector Luna]

I tried the delay and now I can see the entire process complete before the reports get generated.

I think the real issue here is my understanding of the process as you describe it here:

"When the Authentication Tester runs then its just the same as launching a browser yourself, they are "Proxy" requests which then get passively scanned.

When you use Browser Based Authentication then the browser uses a separate proxy - the requests will appear in the History table as "Authentication" requests, these are not passively scanned."

I was under the impression that all of the requests were passively scanned. Is there a way for me to get all of the requests from the browser based authentication passively scanned? My aim is to inspect the authentication process and all the things our app does during the authentication process and find any vulnerabilities without me having periodically record this process, and it seemed like browser based authentication was the answer.

Thank you for your help!!!

[Simon Bennetts]

Right now authentication requests will not be passively scanned.

You can proxy the authentication requests through ZAP, e.g. via your own unit tests, or you could record a client side Zest script: https://www.zaproxy.org/blog/2023-09-11-browser-recorder/

Cheers,

Simon

[Hector Luna]

That is quite unfortunate.
I have been using the script recorder to inspect the authentication process, but we have so much dynamic content that loads based on many different factors that it can become quite difficult to maintain and also to capture, such as requests that occur from the server side after authentication.

[thc...@gmail.com]

So do you want the passive scanning to inspect the auth process?

Best regards.

On 21/05/2024 16:41, Hector Luna wrote:
> That is quite unfortunate.
> I have been using the script recorder to inspect the authentication
> process, but we have so much dynamic content that loads based on many
> different factors that it can become quite difficult to maintain and also
> to capture, such as requests that occur from the server side after
> authentication.
>
>
> On Monday, May 20, 2024 at 4:31:37 AM UTC-5 psi...@gmail.com wrote:
>
>> Right now authentication requests will not be passively scanned.
>> You can proxy the authentication requests through ZAP, e.g. via your own
>> unit tests, or you could record a client side Zest script:
>> https://www.zaproxy.org/blog/2023-09-11-browser-recorder/
>>
>> Cheers,
>>
>> Simon
>>
>> On Tuesday 14 May 2024 at 17:50:12 UTC+1 hmlu...@gmail.com wrote:
>>
>>> I tried the delay and now I can see the entire process complete before
>>> the reports get generated.
>>>
>>> I think the real issue here is my understanding of the process as you
>>> describe it here:
>>>

>>> "*When the Authentication Tester runs then its just the same as

>>> launching a browser yourself, they are "Proxy" requests which then get

>>> passively scanned.*
>>> *When you use Browser Based Authentication then the browser uses a

>>> separate proxy - the requests will appear in the History table as

>>> "Authentication" requests, these are not passively scanned.*"

>>>>> 1. The requester initiates the request - this will be made in a new
>>>>> thread
>>>>> 2. Theres no session, so the authentication starts, all in another
>>>>> thread
>>>>> 3. The passive scan wait job starts. No requests have been made, so

>>>>> the passive scan queue is empty, the job exits

>>>>> 4. The authentication finishes
>>>>> 5. The requester can now make its request, this will get added to

>>>>> the passive scan queue, but its too late :)
>>>>>
>>>>> The easiest solution to this would be to add a delay job before the
>>>>> passive scan wait job.
>>>>>
>>>>> Does that help?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Simon
>>>>>
>>>>> On Tuesday 14 May 2024 at 00:07:19 UTC+1 hmlu...@gmail.com wrote:
>>>>>
>>>>>> So I have been doing further research into this and I think that the
>>>>>> issue I am having has to do with one of several potential issues:
>>>>>>

>>>>>> 1. The passive scan wait job doesn't fully wait for the

>>>>>> authentication helper/browser authentication to stop.

>>>>>> 2. The report generation jobs do not wait for the authentication
>>>>>> helper/browser authentication to stop.
>>>>>> 3. There is some sort of disconnect between the authentication

>>>>>> helper/browser authentication and the report generation.
>>>>>>
>>>>>> The reason I am getting to this conclusion is because of the following
>>>>>> observations:
>>>>>>

>>>>>> 1. When I run the exact same automation job via a docker

>>>>>> container, or directly using zap.sh I get the same result: only a few
>>>>>> alerts (count: X). A number that varies from one run to another and usually
>>>>>> results in a lower alert number. Note that I can only see the number of
>>>>>> alerts via the reports I am generating, so it is possible that all of the
>>>>>> expected alerts are still being generated.

>>>>>> 2. When I run the exact same automation job via the ZAP GUI
>>>>>> (automation tab), *I get most of the alerts (count: Y) I would

>>>>>> expect to get but the reports only include a small portion of the alerts

>>>>>> (count: X)*. These alerts show up on the alerts tab, and that

>>>>>> number is greater than the number of alerts presented in the report
>>>>>> generated via AF (Y > X). Naturally, they all show up if I generate the
>>>>>> report via GUI once it is all said and done.

>>>>>> 3. Running the authentication tester yields Z number of alerts.

>>>>>> Generating an automation plan (via the instructions given here
>>>>>> https://www.zaproxy.org/docs/authentication/) results in getting
>>>>>> the same number of alerts as observed in step #2 (count: Y). Which leads me
>>>>>> to believe that there is a difference in how the tester operates vs AF.

>>>>>> 4. Another thing to note is that the authentication tester does

>>>>>> not produce any of the mozilla noise I see when running the AF via GUI,
>>>>>> docker, or directly using zap.sh -cmd -autorun...
>>>>>>
>>>>>> So I know it isn't my automation plan doing weird things, or how it is
>>>>>> configured, but there is something happening when trying to capture all the
>>>>>> things that went on during browser authentication that prevents all of the
>>>>>> alerts to be included in any of the reports generated.
>>>>>>
>>>>>> Any advice?
>>>>>> Thanks!
>>>>>>
>>>>>> On Friday, May 10, 2024 at 12:52:46 PM UTC-5 Hector Luna wrote:
>>>>>>
>>>>>>> Hi Simon,
>>>>>>>
>>>>>>> I am running the AF from docker using the ZAP weekly image directly
>>>>>>> from the source (I have a different image that I build based on the one
>>>>>>> from you guys, but that is only to eliminate webswing which I don't use for
>>>>>>> our CI environments).
>>>>>>> But the same result occurs when using the weekly image

>>>>>>> *zaproxy/zap-weekly.*

>>>>>>> - The Configuration script is used to store some values into

>>>>>>> ScriptVars. Things such as where I want the output generated by the
>>>>>>> httpsender script.

>>>>>>> - The Browser Processor.js script is used to create a log file

>>>>>>> with information about the URLs I am hitting, and who initiated the request
>>>>>>> (as seen in the firefox noise at the start of the thread).

>>>>>>> - Excluding these two scripts has no effect. I get the same
>>>>>>> results.
>>>>>>> - The `http` in the requestor is fine. It triggers the whole

>>>>>>> process the same way as if I used `https`.

>>>>>>> - I am letting ZAP detect the session management and verification.
>>>>>>> - Setting "enableTags" to true or false has no effect on the
>>>>>>> results.
>>>>>>>
>>>>>>> The Actual Issue:
>>>>>>>
>>>>>>> - Running the authentication tester via ZAP GUI with the same

>>>>>>> details yields some 30+ alerts (No passive scan rules restrictions).

>>>>>>> - Running it from the AF yields 4, sometimes 5 alerts.
>>>>>>> - I do get to see all of our sites in the "Sites" section of the

>>>>>>> report, as well as details about the cookies, etc in the report, but the
>>>>>>> alerts are not showing, and external sources (ie goole things) do not show.

>>>>>>> - The ZAP GUI reports on all the sites that the authentication

[Hector Luna]

That is correct. I would like to passively scan the authentication process.
Looks like I could possibly look at the history and perhaps replay it and get the scanning done that way. I am trying to see how to make that work right now :)

Unless there is another way to get the same work done :)
Thank you!

[Bibin Paul]

Hey

How to block the POST methods while using the jar file to execute

[thc...@gmail.com]

Please, don't ask different questions than the subject of the thread,
your question has been answered:
https://groups.google.com/g/zaproxy-users/c/RpxfKTK65wg/m/O6QA5U1vBQAJ

Best regards.

On 22/05/2024 12:07, 'Bibin Paul' via ZAP User Group wrote:
> Hey
> How to block the POST methods while using the jar file to execute
>
> On Monday, May 20, 2024 at 3:01:37 PM UTC+5:30 psi...@gmail.com wrote:
>
>> Right now authentication requests will not be passively scanned.
>> You can proxy the authentication requests through ZAP, e.g. via your own
>> unit tests, or you could record a client side Zest script:
>> https://www.zaproxy.org/blog/2023-09-11-browser-recorder/
>>
>> Cheers,
>>
>> Simon
>>
>> On Tuesday 14 May 2024 at 17:50:12 UTC+1 hmlu...@gmail.com wrote:
>>
>>> I tried the delay and now I can see the entire process complete before
>>> the reports get generated.
>>>
>>> I think the real issue here is my understanding of the process as you
>>> describe it here:
>>>

>>> "*When the Authentication Tester runs then its just the same as

>>> launching a browser yourself, they are "Proxy" requests which then get

>>> passively scanned.*
>>> *When you use Browser Based Authentication then the browser uses a

>>> separate proxy - the requests will appear in the History table as

>>> "Authentication" requests, these are not passively scanned.*"

>>>>> 1. The requester initiates the request - this will be made in a new
>>>>> thread
>>>>> 2. Theres no session, so the authentication starts, all in another
>>>>> thread
>>>>> 3. The passive scan wait job starts. No requests have been made, so

>>>>> the passive scan queue is empty, the job exits

>>>>> 4. The authentication finishes
>>>>> 5. The requester can now make its request, this will get added to

>>>>> the passive scan queue, but its too late :)
>>>>>
>>>>> The easiest solution to this would be to add a delay job before the
>>>>> passive scan wait job.
>>>>>
>>>>> Does that help?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Simon
>>>>>
>>>>> On Tuesday 14 May 2024 at 00:07:19 UTC+1 hmlu...@gmail.com wrote:
>>>>>
>>>>>> So I have been doing further research into this and I think that the
>>>>>> issue I am having has to do with one of several potential issues:
>>>>>>

>>>>>> 1. The passive scan wait job doesn't fully wait for the

>>>>>> authentication helper/browser authentication to stop.

>>>>>> 2. The report generation jobs do not wait for the authentication
>>>>>> helper/browser authentication to stop.
>>>>>> 3. There is some sort of disconnect between the authentication

>>>>>> helper/browser authentication and the report generation.
>>>>>>
>>>>>> The reason I am getting to this conclusion is because of the following
>>>>>> observations:
>>>>>>

>>>>>> 1. When I run the exact same automation job via a docker

>>>>>> container, or directly using zap.sh I get the same result: only a few
>>>>>> alerts (count: X). A number that varies from one run to another and usually
>>>>>> results in a lower alert number. Note that I can only see the number of
>>>>>> alerts via the reports I am generating, so it is possible that all of the
>>>>>> expected alerts are still being generated.

>>>>>> 2. When I run the exact same automation job via the ZAP GUI
>>>>>> (automation tab), *I get most of the alerts (count: Y) I would

>>>>>> expect to get but the reports only include a small portion of the alerts

>>>>>> (count: X)*. These alerts show up on the alerts tab, and that

>>>>>> number is greater than the number of alerts presented in the report
>>>>>> generated via AF (Y > X). Naturally, they all show up if I generate the
>>>>>> report via GUI once it is all said and done.

>>>>>> 3. Running the authentication tester yields Z number of alerts.

>>>>>> Generating an automation plan (via the instructions given here
>>>>>> https://www.zaproxy.org/docs/authentication/) results in getting
>>>>>> the same number of alerts as observed in step #2 (count: Y). Which leads me
>>>>>> to believe that there is a difference in how the tester operates vs AF.

>>>>>> 4. Another thing to note is that the authentication tester does

>>>>>> not produce any of the mozilla noise I see when running the AF via GUI,
>>>>>> docker, or directly using zap.sh -cmd -autorun...
>>>>>>
>>>>>> So I know it isn't my automation plan doing weird things, or how it is
>>>>>> configured, but there is something happening when trying to capture all the
>>>>>> things that went on during browser authentication that prevents all of the
>>>>>> alerts to be included in any of the reports generated.
>>>>>>
>>>>>> Any advice?
>>>>>> Thanks!
>>>>>>
>>>>>> On Friday, May 10, 2024 at 12:52:46 PM UTC-5 Hector Luna wrote:
>>>>>>
>>>>>>> Hi Simon,
>>>>>>>
>>>>>>> I am running the AF from docker using the ZAP weekly image directly
>>>>>>> from the source (I have a different image that I build based on the one
>>>>>>> from you guys, but that is only to eliminate webswing which I don't use for
>>>>>>> our CI environments).
>>>>>>> But the same result occurs when using the weekly image

>>>>>>> *zaproxy/zap-weekly.*

>>>>>>> - The Configuration script is used to store some values into

>>>>>>> ScriptVars. Things such as where I want the output generated by the
>>>>>>> httpsender script.

>>>>>>> - The Browser Processor.js script is used to create a log file

>>>>>>> with information about the URLs I am hitting, and who initiated the request
>>>>>>> (as seen in the firefox noise at the start of the thread).

>>>>>>> - Excluding these two scripts has no effect. I get the same
>>>>>>> results.
>>>>>>> - The `http` in the requestor is fine. It triggers the whole

>>>>>>> process the same way as if I used `https`.

>>>>>>> - I am letting ZAP detect the session management and verification.
>>>>>>> - Setting "enableTags" to true or false has no effect on the
>>>>>>> results.
>>>>>>>
>>>>>>> The Actual Issue:
>>>>>>>
>>>>>>> - Running the authentication tester via ZAP GUI with the same

>>>>>>> details yields some 30+ alerts (No passive scan rules restrictions).

>>>>>>> - Running it from the AF yields 4, sometimes 5 alerts.
>>>>>>> - I do get to see all of our sites in the "Sites" section of the

>>>>>>> report, as well as details about the cookies, etc in the report, but the
>>>>>>> alerts are not showing, and external sources (ie goole things) do not show.

>>>>>>> - The ZAP GUI reports on all the sites that the authentication

[Hector Luna]

So getting back to this...

So I execute browser authentication and no passive scanning occurs. I understand that.
I have created a script that does the following:

1. Executes browser authentication.
2. Runs a new script that keeps track of the history of all inspected urls with cookies, etc.
3. Changes the parameters so that the context is no longer "browser authentication"
4. Iterates through the items in the history in order to attempt to trigger passive scanning that way.

My question is whether or not there is a better way to do this?
Would saving the session and loading it up later with a different automation plan be the better option?

If so, how do I do this?
Thanks!

[Simon Bennetts]

Right now yes, although you can simplify this by just looking for requests from the "Authentication" initiator.

You also will not have to change the context - just dont make authenticated requests.

FYI we are looking at whether to make the initiators which are passively scanned completely configurable .. but that will require a core change.

Cheers,

Simon