Automation Framework with Auth

[Jason Ryan]

I configured the automation framework in the GUI, which works great but after manually exploring the site. Without the manual exploration, it seems the spider cannot find the other parts of my site and thus doesn't know where to go post-login.

I was hoping to run the baseline scan from command line from CI/CD but I feel I'm missing something regarding how to nudge the spider tool in the right direction of where all my pages are. Can I import URL's or something?

Thanks,
Jason

[Jason Ryan]

I'm almost instantly realizing I could probably just add a ton of includeUrl's in the URL specification.

[psiinon]

Have you tried using the ajax spider?

Have you checked the spiders are authenticating correctly while crawling?

You can check that via the "auth" stats: https://www.zaproxy.org/docs/desktop/addons/automation-framework/authentication/

Manually including URLs is _ok_ but it would be better if we can work out a way to explore the site properly.

Cheers,

Simon

[Jason Ryan]

The spider now appears to be finding everything after adding every site URL in the yaml under contexts/urls. I'm having trouble verifying if the authentication is working. It seems like all the warnings / alerts are things that authentication wouldn't be needed for like header flags. Is there a way to view a log of the http responses (I'm using the docker image and passing in the yaml file)?

What has me concerned is this:

I am getting the warning: "Automation plan warnings: Unrecognised parameter for job users : credentials"

My yaml file was generated by default using a working context. The structure for users is as follows:

env:

    contexts:

    - name: "Default Context"

       urls:

           - "https://site.example.org"

           - [... every page on my site]

      ...

     - authentication:

            method: "form"

            parameters:

                   ...

            verification:

                   ...

            sessionManagement:

                    ...

             users:

              - name: "Username"

              credentials:

                    password: "password123"

                    username: "Username"

[psiinon]

This worries me: "Automation plan warnings: Unrecognised parameter for job users : credentials"

Have you made sure that ZAP is up to date before running it in CI/CD?

The way to do that is to start ZAP twice, once to update the add-ons and the second time to run the job as per https://www.zaproxy.org/docs/automate/automation-framework/

• ./zap.sh -cmd -addonupdate
  
• ./zap.sh -cmd -autorun zap.yaml <any other ZAP options>

Dont mess around adding all of the URLs to the context- that is not a good approach.

Lets fix this credentials issue, and then I can explain how to check if authentication is really working.

Cheers,

Simon

[Jason Ryan]

OK, removed all the URLs. I pulled the latest weekly image and no longer getting the credentials error, good catch there. The ajaxSpider is in action but appears to be erroring out, I've attached the error output. 

Spider found 10 urls, ajax still found 41 despite the error. Ajax finds all the same as spider but nothing more that I actually want (just some out of scope JS dependency urls are found)

I checked the api stats, and here they are:

- stats.auth.10.state.loggedout: 3

- stats.auth.10.state.nothtml: 15

- stats.auth.spider.state.loggedout: 2

- stats.auth.spider.state.nothtml: 5

- stats.auth.spider.state.notsuccess: 3

I'll continue my investigations. I really appreciate your time/help so far. Thank you.

[Jason Ryan]

Also for completeness I'm attaching my yaml file that was generated from a context established with working form-based authentication.

[Jason Ryan]

I got it working!!! 

a. I didn't realize you could double-click on the jobs in the GUI to add authentication information for both spiders.

b. Instead of spidering the root URL, I set the spider to start at our homepage that our login page redirects to after successful auth.

I can now add this to our CI/CD :-)

Thanks for your help and all the hard work you poured into this tool. 

[psiinon]

Great - thanks for letting us know.

And do say if you think any of our docs could be improved ...

Cheers,

Simon

[Jason Ryan]

I'd say to improve it perhaps just adding some mention of those two items above. For automation, I would have found it helpful to have seen mention of the fact that you can configure each of the jobs in the GUI by double-clicking. Also (and please correct me if this is wrong) for successful authenticated spidering, set your spider to start at the home page directed to after login (if it's not the same as the login page). 

Having said all that, perhaps these things were already mentioned and I just missed them ha.

[psiinon]

The double clicking is mentions in the "Plan sub-tab" section of https://www.zaproxy.org/docs/desktop/addons/automation-framework/gui/

Re the spidering - that is probably specific to your app, I've not needed to do that before.

Cheers,

Simon