Saved Session but Context is Missing

[Lia]

Hi there,

In my ZAP Session, I have created a context called "TEST1" and configured all the login configurations there. I am planning to use this ZAP session to run the scan with my automation framework.

But when I reopen the "TEST1" session via ZAP Desktop, the context which I have created has "disappeared". Anyone knows what is the cause?

When I first open ZAP Desktop I chose the option "Yes, I want to persist this session but I want to specify the name and location"

Thanks!

[Simon Bennetts]

Hiya,

That should not happen.

But before we look into that - why do you plan to use the ZAP session with the Automation Framework?

There are valid reasons, but I just want to check that you are doing it for one of the right reasons :)

Cheers,

Simon

[Lia]

Hi Simon,

It is for authentication purpose. 

In another thread, I have asked you before on using "cmd", "autorun" and "session" in the same command line.

Here is the link to our previous discussion if you're interested: https://groups.google.com/g/zaproxy-users/c/B9sns2EweR0/m/bv-6FMMhAwAJ

Thanks!

[Simon Bennetts]

Cool.

Can you describe the exact set of steps that you are following the create, configure, save and then load the context?

cheers,

Simon

[Lia]

Hi Simon,

Sure. These are the steps:

1.  Choose Yes, I want to persist this session but I want to specify the name and location and press Start

2. Enter my preferred location to save the session file and put in File Name as "TEST1"

3. In ZAP Desktop, click on Manual Explore

4. Enter the my site's URL (https://example.com) and click on Explore. HUD is disabled and browser is Chrome

5. Enter my site's username and password in the pop up Chrome browser

6. Wait for site to log then logout from the site.

7. In ZAP Desktop, I right-clicked my site (example.com) under Sites and choose Include in Context > New Context

8. In pop-up window, I changed the context name to "TEST1" and press OK button

9. I click on the TEST1 context and click on the red circle icon to only show my Site's URL.

10. Under my site's URL, I search for POST:login and right click and select Flag as Context > TEST1: Form-based Auth Login Request

11. In the pop up window, I check that the Authentication is Form-based Authentication

12. I configured the Username parameter to my site's parameters for username and password.

13. I entered the Regex pattern for both Logged in and Logged out

14. For Users tab, I clicked on Add button and enter my site's credentials.

15. I ensure that the Enabled checkbox is ticked and click on Add button, then I press OK button

16. I search for my site's logout URL and right clicked and select Exclude from Context > TEST1

17. Then I clicked on OK button.

18. Under the Automation tab, I clicked on icon with green plus sign to add a new plan.

19. When the New Plan window pop-up, choose TEST1 for Contexts.

20. I selected import, active scan and report jobs and press Save button

21. I double clicked on import job and choose my URL.txt files which contains a list of URLs to be scanned. Then clicked Save button

22. I double clicked on the active scan job and configured the Default Strength to High and checked Add Query Parameter, Handle Anti CSRF Tokens, Scan All Headers under Advanced tab.

23. Click Save button

24. I double clicked on report job and configured the Report Name, Report location and Template to PDF Report.

25. I click on Save button.

26. I clicked on Save Plan icon and name the file as TEST1.yaml and put at my preferred location.

27. Then I exit ZAP Desktop by clicking on the upper right "x" 

 I run the scan using ".\zap.bat -cmd -addonupdate -config network.connection.timeoutInSecs=180 -config rules.domxss.browserid=chrome-headless -autorun TEST1.yaml -session "C:\\Users\\Local User\\TEST1.session"

After the scan has completed, I checked my site's audit log and noticed that it is not showing all modules which should be scanned

That is when I reopen back the session file and noticed that the TEST1 context is missing.

Appreciate your help on this.

[Simon Bennetts]

Have you tried running the plan you have created in the ZAP desktop with a new session?

If not, try that and see what happens - esp checking to see if it creates the context.

I'm not really sure why you are loading the original context as the plan should include everything you need, but that shouldnt affect whether the context is added or not.

Cheers,

Simon

[Lia]

Hi Simon,

Thank you for your advice.

I have created a folder called "TEST3". I opened ZAP Desktop and saved a new empty session (TEST3.session) in this location.

I then run the scan using this command line: 

 .\zap.bat -cmd -config database.response.bodysize=524288000;.\zap.bat -cmd -addonupdate -config network.connection.timeoutInSecs=90 -config rules.domxss.browserid=chrome-headless -autorun TEST2.yaml -session "C:\\Users\\Local User\\Documents\\TEST3\\TEST3.session"

But then the scanning time was shorter than usual. It took around 4 minutes to complete the scan when the usual time is usually 20-30 mins.

The audit log does not show all modules were scanned and also the context TEST2 is missing when I open TEST3.session via ZAP Desktop.

Appreciate if you can provide advice on this please.

Many thanks.

[Simon Bennetts]

Thats not actually what I suggested ;)

Try running:

 .\zap.bat -config network.connection.timeoutInSecs=90 -config rules.domxss.browserid=chrome-headless -autorun TEST2.yaml

This will start the GUI and run the AF plan.

You will then be able to see what it does :)

Does it create the context defined in the plan?

Cheers,

Simon

[Lia]

Hi Simon,

I have tried and yes it does create the context defined in my plan.

I also tried running the plan in the ZAP Desktop (new session) and it also creates the defined context.

But if I run the plan via commandline, after the scan, the defined context is missing when I open the session via ZAP GUI.

I am expecting the context to still be in the ZAP session.

The zap.log file does not show anything is wrong and it seems like the scan is working as normal.

The problem is that the target website is nor showing the modules has been scanned in the audit log listing.

[Simon Bennetts]

Hiya,

Pro tip - focus on one problem at a time ;)

I started with the context not being created by the plan, but that seems to be ok now.

Looks like you have 2 more problems - the context not being present when you open the session and the target website looking like its not being fully attacked.

Which would you like to look at first?

Cheers,

Simon

[Xeno 23]

Hi there Simon,

Got it. Thanks for the fast reply.

I would like to start with the target website not being fully attacked please.

--
You received this message because you are subscribed to a topic in the Google Groups "ZAP User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-users/H8Yp-_ogqpc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zaproxy-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/d3610812-b5ec-4625-9225-19895af032f6n%40googlegroups.com.

[Simon Bennetts]

OK, so how are you exploring your app, and why do you think its not being fully explored?

[Xeno 23]

Hi Simon,

I'm exploring via the Automation Framework plan which I have configured using the same steps provided earlier in this same thread.

About why is it not working, I'm really not sure. Previously i have create and configured sessions and AF plan many times and only one session file worked.

By worked = audit log is showing all modules scanned.

Now i want to create a new session with a new context and AF plan to scan a new target website. This new target website is not up yet, so right now I'm just scanning against the same target website as the previous one.

But even with the same steps, I can't get the new created session and AF plan to work.

Not sure why it isn't working. Would like your thoughts/advice on this.

To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/a2605924-8e3c-48c9-bd4e-9ade22c56fa2n%40googlegroups.com.

[Simon Bennetts]

I know nothing about your app or have any access to it.

I have absolutely no idea why its not working or even what you mean by "its not working".

You will need to tell me a _lot_ more.

Lets start with: what do you mean by "not working"?

[Xeno 23]

Hi Simon,

Sorry, let me explain more.

The target website has a login page where user will have to enter username and password. These credentials I have configured in ZAP session via ZAP desktop and AF plan.

By not working, I mean when I scan using this created session and AF plan, after scan, the target website does not show all modules being scanned in the audit log listing.

Hope this helps to clarify.

To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/cb0097c9-e97c-4dd7-b449-085556202583n%40googlegroups.com.

[Simon Bennetts]

How are you exploring the target?

Can you see all of the URLs your would expect ZAP to find in the Sites tree?

Cheers,

Simon

[Xeno 23]

Hi there,

If I run my AF plan in ZAP desktop, then yes, I can see the URLs in the site tree. 

To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/9acc6edb-9a69-46b1-9aa7-a5838fd59482n%40googlegroups.com.

[psiinon]

If you run the plan in the ZAP desktop then can you see the expected coverage in your app's logs?

Does the scan take roughly the same amount of time?

You received this message because you are subscribed to the Google Groups "ZAP User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/CAAUWTg0W5GsP2CO-7AyP6KK8QpDi_QE6%3DdgN7px0rx_ey%2B-LyQ%40mail.gmail.com.

--

ZAP Project leader

[Xeno 23]

Hi there,

I ran it once today but the target website's audit log was only showing login modules and not the scanned modules.

I also did not take note of the time taken for the scan to complete

I am planning to run the scan one more time tomorrow via ZAP desktop. Once I have done that, I will let you know of the results.

Besides this, do you have anything else which (hopefully) I can clarify currently?

To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/CAORxfg5u3jykfoL35h01SL0GENXJk5Lg8gEkn66ktZaxJGdbeQ%40mail.gmail.com.

[Lia]

Hi there Simon,

I have run the AF plan in ZAP desktop. The active scan job is taking very less time: around 2 minutes compared to the usual 20-30 minutes.

I have checked the site tree and all the imported URLs are present there. 

When I checked the "History" tab, at the POST login, it is showing 500 Internal Server error.

I guess thats why it is ZAP is not login in and doing the scanning? Due to to many requests?

[Simon Bennetts]

That will be a problem :)

Why that is happenning is impossible for us to say.

You will need to investigate in your app.

Cheers,

Simon

[Lia]

Hi there Simon,

I think it is authentication problem. I have been seeing a bunch of this kind of messages in the target server where the target website is located on in Event Viewer.

Exception information:      Exception type: HttpRequestValidationException      Exception message: A potentially dangerous Request.Form value was detected from the client (LoginID1="...0A[%]0d[%]0a//</stYle/</titLe/</te...").    at System.Web.HttpRequest.ValidateString(String value, String collectionKey, RequestValidationSource requestCollection)    at System.Web.HttpRequest.ValidateHttpValueCollection(HttpValueCollection collection, RequestValidationSource requestCollection)    at System.Web.HttpRequest.get_Form()    at System.Web.HttpRequest.get_HasForm()    at System.Web.UI.Page.GetCollectionBasedOnMethod(Boolean dontReturnNull)    at System.Web.UI.Page.DeterminePostBackMode()    at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)    at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)    at System.Web.UI.Page.ProcessRequest()    at System.Web.UI.Page.ProcessRequestWithNoAssert(HttpContext context)    at System.Web.UI.Page.ProcessRequest(HttpContext context)    at ASP.login_aspx.ProcessRequest(HttpContext context)    at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()    at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)    at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Why is ZAP entering random value inputs for LoginID parameter? Based on the list of steps which I had provided to you earlier in this thread, I have specify the credentials for username (LoginID) and the password parameter with the correct values.

Similarly, in the audit log I can see many login attempts and a few failed ones. 

Appreciate your thoughts on this please.

[Simon Bennetts]

That does not look like a login attempt - that looks like an attack.

Run the plan in the desktop and look at the initial requests that the active scan makes. You can stop the active scanner once it has made the initial requests.

Are there any messages about authentication in the Output tab?

Can you see valid session tokens in the requests?

Cheers,

Simon

[Lia]

Hi Simon,

I see. 

Yes, under the "Output" tab, I can see "Authentication Successful" statement.

Under the "Active Scan" tab, I check the initial request and I can see the session tokens in the request like below:

Cookie: ASP.NET_SessionId=zevaunrugbbprbc0scanukwt; __AntiXsrfToken=b0ee8ce3b0584e45bfe911d5fb541d57; mySession=e7e09154-4863-492a-a4e8-697395296411

[Lia]

But still the audit log is not showing any scanned modules

[Simon Bennetts]

Do the responses to the requests indicate that the user is logged in?

Can you see any requests made by ZAP that do not appear in your audit logs?

The fact that the active scans are significantly shorter than before does imply that something is wrong.

[Lia]

Hi Simon,

I run the plan in ZAP desktop again and waited for the scan to finish.

The response to the request to the response does not seem to indicate that it is logged in.

Under the "History" tab, the "Request" tab for the POST login url is showing the configured username and password parameters with the correct credentials to the target website.

However, the "Response" tab is showing 500 Internal Server Error and message like "Validation of viewstate MAC failed".

Please see 500Error.png

I checked the "Active Scan" tab and I see a mixed of 200, 301, 403 and 500 codes on the URLs.

• For the login URL with the 200 code, in the "Response" tab, I can see the html of the login page. The same if I were to right-click and select "view page source" on the login page of the target website. Is this considered logged in?
• For 301 code, it just show the "Moved permanently" message
• For 403, I am seeing Forbidden:Access Denied kind of html message under the "Request" tab
• For 500 code, the "Request" tab is showing the same message as the attached png image.

In the audit log, it is not showing any scanned modules. And the scan this time is around 10 minutes.

Can you see any requests made by ZAP that do not appear in your audit logs?

To answer your above question, yes I can see ZAP sending request to the imported URLs and login page and is getting either one of the code above. Some of the URLs under "Active Scan" is also having random inputs in the query parameter which I guess is ZAP trying to attack.

But in the audit log, it is not showing any scanned modules. And the scan this time is around 10 minutes.

Appreciating your help and advice Simon!

[Simon Bennetts]

Its looking like it is an authentication and/or access control problem.

We can not really tell if ZAP is logged in or not - we dont have access to your app, and there are no standards in this area.

So thats something you will need to work out.

We can help with general advice and guidance but not with the specifics unless you give us enough information.

So, focus on authentication to start with. If that isnt working then nothing else will.

You can create an AF plan which just makes one authenticated request to a suitable URL - use the same AF environment, but then just add a "requester" job which specifies the user you have set up.

Run that in the desktop and examine the requests and responses.

This page may help a bit too: https://www.zaproxy.org/docs/authentication/finding-a-verification-url/

Cheers,

Simon

[Lia]

Hi Simon,

Got it. Can you tell me how to check using ZAP Desktop to see whether it is logged in or not?

Is it by looking at the "Response" tab and checking if can see the html content of the target website? Or is it something to look at as a big picture?

About the AF plan,  do you mean to configure the login and user as usual and only run the plan with a "requestor" job?

Is the login page a suitable URL or it must be a URL which can only be accessed after login?

Thank you for your help all this while Simon!

[Simon Bennetts]

Replies inline

On Thursday, 2 November 2023 at 10:27:35 UTC Lia wrote:

Hi Simon,

Got it. Can you tell me how to check using ZAP Desktop to see whether it is logged in or not?

Is it by looking at the "Response" tab and checking if can see the html content of the target website? Or is it something to look at as a big picture?

Pretty much yes.

But looking at the "right" requests and responses is key, and thats something we cant help you with.

But there are some suggestions on https://www.zaproxy.org/docs/authentication/finding-a-verification-url/ which is why I linked to it before.

 

About the AF plan,  do you mean to configure the login and user as usual and only run the plan with a "requestor" job?

Yes.

 

Is the login page a suitable URL or it must be a URL which can only be accessed after login?

The Logon page is the URL which displays the Login form which asks for your credentials.

The URL you request via the "requester" job should be one where you can tell from the response whether you are logged in or not.

So the verification URL discovered by ZAP (or yourself) would be fine.

Cheers,

Simon

[Lia]

Hi Simon,

I tried using the requestor job and request a few different URLs which can only be seen if the user is logged in.

After the requestor job run finish, I check the "Output" tab. It always show "Authentication successful" statement.

Then if I check the "History" tab, all the URLs requested via requestor job will be shown with 200 code and GET method.

But in the response tab, it is only showing the part of code where it will redirect back to the logout link.

Under the "History" tab, it is also showing the POST login URL, but still with 500 error code.

The response tab still show same error message which I showed you before:

Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that &lt;machineKey&gt; configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.<br><br>See http://go.microsoft.com/fwlink/?LinkID=314055 for more information

I noticed that only for the login URL which I have explored manually through ZAP Desktop, under the Response tab it is showing the string which indicates it is logged in. For this URL, it is having 200 code.

Then I compared the Request for both login URLs, I noticed that the password value for the password parameter which I have configured under Authentication are different.

• The login URL with 500 code is showing the password value as plaintext and the symbols replaced with HTML values
• The login URL with 200 code is showing the password value as encrypted.

However, when I change the password value for Users in the same session file and run the requestor job again, the POST Login URL under the "History" tab is still showing 500 code error with same error message in Response tab and the password value in Request tab is showing as  plaintext and the symbols replaced with HTML values.

Just wondering your input on this, Simon.

Thank you so much for replying!

[Lia]

Hey there Simon, 

I wonder if we could look into the other issue, context not being present after opening saved session file after a scan with the -session parameter.
I noticed sometimes, not always, the context will be missing after running a scan with -cmd, -session and -autorun parameters via zap.bat.

But if I open the session file before running any scan with it, the context that I created is always there.

DO you have any idea about this?

[Simon Bennetts]

Hi Lia,

Its not something I've seen before.

Are there any errors in the zap.log file?

https://www.zaproxy.org/faq/somethings-not-working-what-should-i-do/#check-the-log-file

Can you reproduce this with a very simple AF plan, for example one which just sets up a context for https://www.example.com ?

If not then we'll need more details of what you are doing - if we cant reproduce the problem then it will be much harder to fox.

Cheers,

Simon

[Lia]

Hi Simon,

Thanks for the reply.

The zap.log file looks normal and I don't see anything out of place.

Sure, but the site which I'm scanning has a login page. Hence one of the steps when I am creating the AF plan includes conf the authenticated user with the credentials.This https://www.example.com/ site does not seem to have a login page and no username and password to set up.

I'm not sure if having a login site affects/cause the "missing" context but I can try.

I'll let you know if I manage to reproduce the issue using this site.

[Lia]

Also, I noticed after moving to ZAP v2.14.0 from v2.13.0, the session files no longer have .session.lobs file

May I know why is that and how does it affects the scan (if applicable)?

[Simon Bennetts]

That is the HSQLDB Large Objects file and I think its created as needed.

So I dont thonk its relevant to this problem.

Cheers,

Simon

[Lia]

Hi Simon,

Bringing this issue back up - "context not being present after opening saved session file after a scan"

I have tried running a scan via cmd with the following command on https://www.example.com/

.\zap.bat -cmd -autorun Example.yaml -session "C:\Users\user-name\Documents\Example-Scan\Example.session"

Attached are the session files and AF plan after the scan is run. The "Example" context which I have created before running the scan is missing.

Below are the steps which I executed:

1. Choose Yes, I want to persist this session but I want to specify the name and location and press Start

1. Enter my preferred location to save the session file and put in File Name as "Example"

1. In ZAP Desktop, click on Manual Explore

1. Enter the site's URL ( https://www.example.com/  ) and click on Explore. HUD is disabled and browser is Chrome

1. In ZAP Desktop, I right-clicked my site (example.com) under Sites and choose Include in Context > New Context

1. In pop-up window, I changed the context name to "Example" and press OK button
2. I click on the Example context and click on the red circle icon to only show the Site's URL.

1. Under the Automation tab, I clicked on icon with green plus sign to add a new plan.

1. When the New Plan window pop-up, choose Example for Contexts.
2. I selected active scan jobs and press Save button

1. I double clicked on the active scan job and configured the Default Strength to High and checked Add Query Parameter, Handle Anti CSRF Tokens, Scan All Headers under Advanced tab.

1. Click Save button
2. I clicked on Save Plan icon and name the file as Example.yaml and put at my preferred location.

1. Then I exit ZAP Desktop by clicking on the upper right "x" 

1. Run the command line via Powershell
2. After scan is done, open back the Example.session via ZAP GUI and the context is missing

[Simon Bennetts]

FYI I am looking at this .. and I think I can reproduce it :)

I'll update this thread when I find out more..

[Simon Bennetts]

It was a bug - thanks for reporting it!

The fix is in https://github.com/zaproxy/zap-extensions/pull/5093

Cheers,

Simon

[Lia]

Hi Simon,

You're welcome.

Weird how nobody else notice it before.

Just to confirm, ZAP just needs to be updated to get the fix right?

[Lia]

By the way, I noticed that I am potentially missing Selenium bundle when checking the marketplace.

I thought Selenium is installed together with ZAP, do I have to install it separately?

[Simon Bennetts]

I guess most people dont use the AF with an existing ZAP session.

We will need to release a new version of the automation add-on to include this fix - thats in progress so should be available soon.

Yes, the Selenium add-on is included with ZAP - if you dont have it installed then you should add it.

Cheers,

Simon

[Simon Bennetts]

FYI the fixed automation add-on is available now :)

[Lia]

Hi Simon,

Thanks for the speedy update and fix!

Will let you know if I encounter any issue.

Please see my queries below:

1. I have updated my Selenium add-ons but I am still seeing the same message saying "Missing Requirements - Selenium" at the marketplace window even after restarting ZAP. Please see the attached image.
   
   I also checked " C:\Users\Local User\ZAP\selenium\extensions" path and the folder is empty.
   
   
2. Regarding the below, I also noticed that after I have run the scan no matter via GUI or CMD, after I have close the ZAP session, if I reopen it .session file, there is no scan result under "Active Scan" tab. 

1. 
   
   I guess most people dont use the AF with an existing ZAP session.
   
   

1. Is this intended? Because I did noticed that the file size has increased compared to before running a scan on the session file.