Automation Framework :: Working example or a Demo

[Vicky]

Hello Zap Team,

Thanks for bringing up the new stuff - Automation Framework.

I have gone through the documents. However, do you have any example so that the users can refer it, study it and then make changes to it as per their needs?

From the questions which the end users are asking on this google group, it seems that they are still struggling just to get the Automation Framework working. And many other users in future will also be struggling and will be asking the similar question on the errors which they encounter while setting the basic Automation Framework with example.com. Also, it must be monotonous for the Zap team to answer the similar types of questions again and again to different users.

In order to save the time of the users and the Zap team, would it not be easy just to create a 10-15 mins video which will explain step by step how to set up the Automation framework from scratch. So, the users can go through the video and perform the steps, just to get on track and then change it as per their needs?

Thanks and Regards,

Vicky

[Simon Bennetts]

Hi Vicky,

We already have one video which introduced the Automation Framework (AF) at ZAPCon :) https://www.youtube.com/watch?v=aZmS9NiQlJA

However this is out of date, and at the rate the AF is changing any other videos we record in the near future will end up out of date pretty quickly as well.

We will record one, but possibly when it is a bit more stable.

TBH right now we are focusing on the core functionality rather than usability, but we will be looking to add a UI fairly soon which should make it easier to use.

In the meantime a quick example using the latest weekly ZAP release is:

1. Generate a minimal config file using
   •  ./zap.sh -cmd -autogenmin `pwd`/zap.yaml
2. Edit the zap.yaml file with your favourite text editor and add a suitable target URL eg
   • urls
   • - https://www.example.com
     
3. Run the yaml file using:
   • ./zap.sh -cmd -autorun `pwd`/zap.yaml

I've used `pwd` as relative paths can be more problematic. You can also specify the full path instead.

You'll probably find that the plan fails:

Job graphql started
Endpoint URL must be specified.
Automation plan failures:
    Endpoint URL must be specified.

This is a bug that should be fixed in this weeks weekly release.

To work around it remove the graphql job (or spacify a valid graphql URL).

Does that help?

Cheers,

Simon

[Vicky]

Hi Simon,

Thanks for your reply. I have tried your solution. I am getting below error

I removed the graphql related job , also added contexts urls and spider parameters url and re-ran. But, still getting the same error. How do I get it working?

You mentioned that it is being fixed in the weekly release. When would the fix be integrated in ZAP desktop?

Also, is this Automation Framework is fully ready for use or till then users are required to use the package scans?

Thanks and Regards,

Vikrant

[Simon Bennetts]

Hi Vikrant,

I think you have a different error: "Context URLs should be a list"

The URL should appear after the "urls" line and start with a dash, as I posted above, eg

    urls:

    - https://www.example.com

We released a new version of the Automation Framework (AF) yesterday and that includes all of the latest fixes and enhancements.

Just 'check-for-updates' and update all of the add-ons.

The AF is still alpha quality and so while we would like people to try it out and give us feedback we cant say that its completely "ready for use".

However if it works for you then great. I'm hoping it can get to beta status fairly soon.

Cheers,

Simon

[Vicky]

Hi Simon,

Thanks for your prompt reply.

I have updated all the add-ons. It says Zap is up-to-date (2.10.0) and the Update All button is also disabled. So, I assume update is successful.

Also, I corrected the url as a list. This time I ran the Automation Framework for Bodgeit store configured on my local system.

I have few queries and I will be grateful to you if you can answer them. Actually I wanted to ask them tomorrow in the ZapCon but since I have already started working on Automation Framework, I write here.

Q1. I believe Automation Framework has used Firefox browser installed in the system. Is it the same Firefox browser which is available in the Quick Start tab ?

Q2. Should we bother about the errors and warnings highlighted in red color? I mean do we need to fix them if it is impacting the result?

Q3. As highlighted in green, it says Job spider found 28 URLs. Where do we see those URLs? I could not find them in zap.log either

Q3. As highlighted in green, it says Job spiderAjax found 1,170 URLs. Where do we see those URLs? I could not find them in zap.log either

Q4. Does it not generate any report? How to configure it?

Q5. Where do I see the progress of scanning? It helps to get an idea what is going on and how much is left!

Q6. What type of Authentication is supported by Automation Framework? If it does not support the desired authentication then does it help if first I run my functional happy path automated test cases proxying through Zap so that it learns my application and its authentication and then start the Automation Framework? I hope it helps Zap to learn the pages hidden behind the authentication and then access, scan and attack them while passive and active scanning. If it is so, do I even need the authentication at all?

Q7. In package scan, we have option -c config_file  to provide the config file and option  -p progress_file to provide the progress file. Can we do the same in case with Automation Framework?

Q8. How do I save the session and then open it in ZAP GUI?

Q9. Can I use my functional tests proxying through Zap and then start the Automation Framework?

Q10. If I want to use the config.xml and My_Context.xml and do not want to mention these parameters in yaml file, is it possible to do so? I mean can I instruct zap to use config.xml and My_Context.xml  and ignore these parameters from yaml file? I am not sure if this will beat the whole purpose of Automation Framework and the yaml configuration.

As per zap.log

Q11. Why is it scanning http://localhost:8081? Do I need to include in context only http://bodgeit:8080/bodgeit.* so that it will not scan http://localhost:8081? Or is it because the DirectoryBrowsingScan Rule?

Q12. An error is thrown in the logs below. Any idea what does it indicate to?

Sorry for so many questions but answers to them will certainly help to understand Automation Framework more better.

Thanks and Regards,

Vicky

[Vicky]

Hello Simon,

I would be very grateful if you can help me in getting answers of the questions.

Thanks and Regards,

Vicky

[Simon Bennetts]

Answers inline...

On Tuesday, 29 June 2021 at 22:17:53 UTC+2 Vicky wrote:

Hi Simon,

Thanks for your prompt reply.

I have updated all the add-ons. It says Zap is up-to-date (2.10.0) and the Update All button is also disabled. So, I assume update is successful.

Also, I corrected the url as a list. This time I ran the Automation Framework for Bodgeit store configured on my local system.

I noticed that you are running ZAP via "java -jar" - this is not recommended, we recommend you use the zap.sh script instead.

 

I have few queries and I will be grateful to you if you can answer them. Actually I wanted to ask them tomorrow in the ZapCon but since I have already started working on Automation Framework, I write here.

Q1. I believe Automation Framework has used Firefox browser installed in the system. Is it the same Firefox browser which is available in the Quick Start tab ?

We include Firefox in the ZAP Docker images, we do not include it in the ZAP installation. If Firefox is available on your system then ZAP should be able to use it. It will use the same version in all cases.

 

Q2. Should we bother about the errors and warnings highlighted in red color? I mean do we need to fix them if it is impacting the result?

That all depends on how serious they are for you. ZAP cant tell that - we try to give you some indication of there relative importance but you need to look at them and understand how they could impact your apps.

 

Q3. As highlighted in green, it says Job spider found 28 URLs. Where do we see those URLs? I could not find them in zap.log either

They are stored in the ZAP session.

If you're using the ZAP Desktop then they are shown in the Spider tab and the Sites tree.

You can also access them via the API.

 

Q3. As highlighted in green, it says Job spiderAjax found 1,170 URLs. Where do we see those URLs? I could not find them in zap.log either

As above but in the Ajax Spider tab instead of the Spider tab.

 

Q4. Does it not generate any report? How to configure it?

You need to define a report job: https://www.zaproxy.org/docs/desktop/addons/report-generation/automation/

 

Q5. Where do I see the progress of scanning? It helps to get an idea what is going on and how much is left!

Right now you cant. That may change in the future.

However you can use ZAP in desktop mode the first time and then it will show you the progress.

 

Q6. What type of Authentication is supported by Automation Framework? If it does not support the desired authentication then does it help if first I run my functional happy path automated test cases proxying through Zap so that it learns my application and its authentication and then start the Automation Framework? I hope it helps Zap to learn the pages hidden behind the authentication and then access, scan and attack them while passive and active scanning. If it is so, do I even need the authentication at all?

The Automation Framework does not currently support authentication, however we plan to add  that soon.

If your app supports authentication then yes, you will really need this support.

 

Q7. In package scan, we have option -c config_file  to provide the config file and option  -p progress_file to provide the progress file. Can we do the same in case with Automation Framework?

Neither of these are currently supported but we are working on that right now.

We will post to this group as updates are made but you can also see whats supported in the relevant scripts: https://github.com/zaproxy/zaproxy/blob/main/docker/zap-baseline.py#L108-L143

 

Q8. How do I save the session and then open it in ZAP GUI?

Thats not currently supported explicitly by the Automation Framework but I'm fairly sure the exsisting '-newsession' command line option will work https://www.zaproxy.org/docs/desktop/cmdline/ - why not try it out?

 

Q9. Can I use my functional tests proxying through Zap and then start the Automation Framework?

Not yet no, we do not currently have a way to pause the Automation Framework. That is planned.

 

Q10. If I want to use the config.xml and My_Context.xml and do not want to mention these parameters in yaml file, is it possible to do so? I mean can I instruct zap to use config.xml and My_Context.xml  and ignore these parameters from yaml file? I am not sure if this will beat the whole purpose of Automation Framework and the yaml configuration.

No, this is not supported, and tbh I dont think we will want to support it, unless you can convince us otherwise.

We will provide an option to convert context files to the AF yaml file.

 

As per zap.log

Q11. Why is it scanning http://localhost:8081? Do I need to include in context only http://bodgeit:8080/bodgeit.* so that it will not scan http://localhost:8081? Or is it because the DirectoryBrowsingScan Rule?

I domnt know - whats your configuration?

 

Q12. An error is thrown in the logs below. Any idea what does it indicate to?

They indicate that controlling browsers is hard :)

I think they can be safely ignored.