Error 401 Unauthorized when scan intranet application asp.net with windows authentication

[l043nz0]

I'm trying to scan a web application in my lan, developed in asp.net
I've used windows authentication in web app.

For testing, i've tried to insert my credentials in tools/option/Authentication but, when i tried to scan webapp, I've received error 401.

[Simon Bennetts]

Windows authentication in a web app - you sure you really want to do that??

OK, I'm biased ;)

I must admit I dont know much about Windows authentication. Is this using Kerberos, NTLM or ??
Can you access your application when proxying your browser through ZAP?
If so then you can authenticate that way and then use the ZAP spiders and scanners.
Thats no good if you want to automate your scanning of course.

Cheers,

Simon

[l043nz0]

Thank y.

Yes, I access to application with proxy through ZAP, but the scanner result error 401.
I'm using NTLM.

[thc202]

Hi.

What version of ZAP are you using?

Could you provide an excerpt of the log file (file zap.log located in ZAP's default directory [1]) when active scanning the site?

The file log4.properties (located in the same directory as zap.log) must be changed so all the useful information is logged (it has to be changed before starting ZAP).
The following line has to be added:

log4j.logger.httpclient.wire.header=DEBUG

and the following line changed:

log4j.logger.org.apache.commons.httpclient=ERROR

replace ERROR with DEBUG.

It should log something like:
[...]
2013-10-17 15:15:45,398 INFO  Scanner - scanner started
[...]
2013-10-17 15:15:45,433 DEBUG header - >> "GET http://example.com/834083552227703082 HTTP/1.1[\r][\n]"
2013-10-17 15:15:45,433 DEBUG HttpMethodBase - Adding Host request header
2013-10-17 15:15:45,438 DEBUG header - >> "Connection: keep-alive[\r][\n]"
2013-10-17 15:15:45,439 DEBUG header - >> "Pragma: no-cache[\r][\n]"
2013-10-17 15:15:45,439 DEBUG header - >> "Cache-Control: no-cache[\r][\n]"
2013-10-17 15:15:45,439 DEBUG header - >> "Host: example.com[\r][\n]"
2013-10-17 15:15:45,439 DEBUG header - >> "Proxy-Connection: Keep-Alive[\r][\n]"
2013-10-17 15:15:45,439 DEBUG header - >> "[\r][\n]"
2013-10-17 15:15:45,495 DEBUG header - << "HTTP/1.1 401 Unauthorized[\r][\n]"
2013-10-17 15:15:45,495 DEBUG header - << "HTTP/1.1 401 Unauthorized[\r][\n]"
2013-10-17 15:15:45,495 DEBUG header - << "WWW-Authenticate: NTLM[\r][\n]"
2013-10-17 15:15:45,495 DEBUG header - << "Content-Length: 0[\r][\n]"
2013-10-17 15:15:45,495 DEBUG header - << "Connection: keep-alive[\r][\n]"
2013-10-17 15:15:45,495 DEBUG header - << "[\r][\n]"
2013-10-17 15:15:45,496 DEBUG HttpMethodDirector - Authorization required
2013-10-17 15:15:45,497 DEBUG AuthChallengeProcessor - Supported authentication schemes in the order of preference: [ntlm, digest, basic, ntlm]
2013-10-17 15:15:45,497 INFO  AuthChallengeProcessor - ntlm authentication scheme selected
2013-10-17 15:15:45,510 DEBUG AuthChallengeProcessor - Using authentication scheme: ntlm
2013-10-17 15:15:45,510 DEBUG AuthChallengeProcessor - Authorization challenge processed
2013-10-17 15:15:45,510 DEBUG HttpMethodDirector - Authentication scope: NTLM <any realm>@example.com
2013-10-17 15:15:45,510 DEBUG HttpMethodDirector - Retry authentication
2013-10-17 15:15:45,511 DEBUG HttpMethodDirector - Authenticating with NTLM <any realm>@example.com
2013-10-17 15:15:45,536 DEBUG header - >> "GET http://example.com/834083552227703082 HTTP/1.1[\r][\n]"
[...]


[1] https://code.google.com/p/zaproxy/wiki/FAQconfig

Best regards.

[Mike Ward]

Sorry to bump an old thread, but I have this exact problem myself.  Here's an excerpt from my log file.  When I browse the site normally I get the 401 -> 401 -> 200 pattern of messages as it performs the authentication, but it looks like when doing an active scan it just sees the first 401 then stops and reports that as the result.  This is on ZAP 2.3.0.1.  Thanks!

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Path Traversal

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Remote File Inclusion

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Server side include

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Cross Site Scripting (Reflected)

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Cross Site Scripting (Persistent)

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin SQL Injection

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Server Side Code Injection Plugin

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Remote OS Command Injection Plugin

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Directory browsing

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Secure page browser cache

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin External redirect

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin CRLF injection

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Parameter tampering

2014-04-30 15:24:39,127 INFO  PluginFactory - loaded plugin Cross Site Scripting (Persistent) - Prime

2014-04-30 15:24:39,143 INFO  PluginFactory - loaded plugin Cross Site Scripting (Persistent) - Spider

2014-04-30 15:24:39,158 INFO  PluginFactory - loaded plugin Script active scan rules

2014-04-30 15:24:39,158 INFO  Scanner - scanner started

2014-04-30 15:24:39,689 INFO  HostProcess - start host http://<redacted_domain> | TestPathTraversal strength MEDIUM threshold MEDIUM

2014-04-30 15:25:40,779 ERROR HttpMethodDirector - Out of sequence NTLM response message

org.apache.commons.httpclient.auth.MalformedChallengeException: Out of sequence NTLM response message

at org.zaproxy.zap.network.ZapNTLMScheme.processChallenge(Unknown Source)

at org.apache.commons.httpclient.auth.AuthChallengeProcessor.processChallenge(AuthChallengeProcessor.java:162)

at org.apache.commons.httpclient.HttpMethodDirector.processWWWAuthChallenge(HttpMethodDirector.java:694)

at org.apache.commons.httpclient.HttpMethodDirector.processAuthenticationResponse(HttpMethodDirector.java:668)

at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:193)

at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)

at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)

at org.parosproxy.paros.network.HttpSender.executeMethod(Unknown Source)

at org.parosproxy.paros.network.HttpSender.runMethod(Unknown Source)

at org.parosproxy.paros.network.HttpSender.send(Unknown Source)

at org.parosproxy.paros.network.HttpSender.sendAuthenticated(Unknown Source)

at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)

at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)

at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(Unknown Source)

at org.parosproxy.paros.core.proxy.ProxyThread.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

[Simon Bennetts]

No need to apologize, I think its better to resurrect old threads as it makes it easier to see the previous discussions.

How many active scan threads do you have?
If its more than 1 then can you set it to one and retry?
I'm just wondering if the threads are causing the "Out of sequence NTLM response message".

Secondly, could you try setting an active session before scanning?
To do this:

1. Browse the site, authenticating as required
2. Find you session in the Http Sessions tab (if there isnt one then we have a different problem;)
3. Right click it and "Set as active"
4. Try scanning again

This should cause that session to be reused for your scan and therefore not require NTLM authentication again.

Cheers,

Simon

[Mike Ward]

Thanks for the quick reply Simon.

1) I'm just using one active scan thread, set to one host at a time too (although I'm only scanning the one host to begin with).  

2) I tried double checking that the session in ZAP is marked as active, and the same thing happens.  

So neither of the things to try there had an effect.

If I try to manually browse to a URL that the active scan hit, then I still get the 401->401->200 pattern, which tells me that even normal browsing is re-doing the NTLM negotiation on every get request, and I don't know enough about how NTLM is supposed to work to know if that's proper or not.  Maybe the issue is that ZAP expects to only do the negotiation once, and the app is forcing it more frequently?

Also of note is that the stack trace error does not happen for every get request that the active scanner makes, only for a relatively small number of them.

Thanks again for the help!

Mike

[thc...@gmail.com]

Hi.

How did you configure the user authentication credentials in ZAP?


Could you provide again an excerpt of the log but doing the following changes to log4.properties (located in the same directory as zap.log) before starting ZAP:

The following line has to be added:

log4j.logger.httpclient.wire.header=DEBUG

and the following line changed:

log4j.logger.org.apache.commons.httpclient=ERROR

replace ERROR with DEBUG.

This would show the content of the HTTP request/response headers and hopefully it would give the clues to why the NTLM messages are out of sequence.
Note that you might need to remove any sensitive information. Regarding the NTLM messages (sent in Authorization/WWW-Authenticate headers after "NTLM ") it would be great if you could leave the first 12 chars of each NTLM message (or check the NTLM message type yourself, you can base64 decode it in ZAP and check in an hex editor the 8th byte, it should contain 1, 2 or 3).

 Maybe the issue is that ZAP expects to only do the negotiation once, and the app is forcing it more frequently?

That's the normal sequence (although I think that ZAP could reuse the Authorization request header to avoid the need to re-authenticate each time).

Also of note is that the stack trace error does not happen for every get request that the active scanner makes, only for a relatively small number of them.

Maybe the active scanner is using/attacking one of the (intermediate) authentication messages...


Best regards.

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Ward]

Hi thc202,

Thanks for the response.  I followed the instructions you gave, and posted an excerpt from the log file at the bottom.  I didn't include any of the NTLM out of sequence messages, because I believe the source of the problem is in the excerpt below, where ZAP can't find the credentials it needs to complete the NTLM authentication request:

2014-05-02 16:17:13,759 DEBUG HttpMethodDirector - Authentication scope: NTLM <any realm>@[redacted]:80

2014-05-02 16:17:13,759 DEBUG HttpMethodDirector - Credentials required

2014-05-02 16:17:13,759 DEBUG HttpMethodDirector - Credentials provider not available

2014-05-02 16:17:13,759 INFO  HttpMethodDirector - No credentials available for NTLM <any realm>@[redacted]:80

Once it sees this, it closes the connection and goes on to the next get.  This leads me to believe either I'm configuring ZAP wrong, or this piece isn't working properly.  I configured NTLM for the session context using the sessions properties dialog, and added a user with the proper credentials, but it still doesn't appear to get picked up by the active scanner.  I read through the help documentation too, but couldn't see anything there that indicated something I could change configuration-wise to try to fix this.  Any ideas?  

Thanks again,

Mike

2014-05-02 16:17:13,619 DEBUG MultiThreadedHttpConnectionManager - HttpConnectionManager.getConnection:  config = HostConfiguration[host=http://[redacted]], timeout = 0

2014-05-02 16:17:13,619 DEBUG MultiThreadedHttpConnectionManager - Allocating new connection, hostConfig=HostConfiguration[host=http://[redacted]]

2014-05-02 16:17:13,619 DEBUG HttpConnection - Open connection to [redacted]:80

2014-05-02 16:17:13,619 DEBUG header - >> "GET /3144274607455470462 HTTP/1.0[\r][\n]"

2014-05-02 16:17:13,619 DEBUG HttpMethodBase - Adding Host request header

2014-05-02 16:17:13,619 DEBUG header - >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"

2014-05-02 16:17:13,619 DEBUG header - >> "Cookie: ASP.NET_SessionId=[redacted][\r][\n]"

2014-05-02 16:17:13,619 DEBUG header - >> "Host: [redacted][\r][\n]"

2014-05-02 16:17:13,619 DEBUG header - >> "[\r][\n]"

2014-05-02 16:17:13,634 DEBUG header - << "HTTP/1.1 404 Not Found[\r][\n]"

2014-05-02 16:17:13,634 DEBUG header - << "HTTP/1.1 404 Not Found[\r][\n]"

2014-05-02 16:17:13,634 DEBUG header - << "Content-Type: text/html[\r][\n]"

2014-05-02 16:17:13,634 DEBUG header - << "Server: Microsoft-IIS/7.5[\r][\n]"

2014-05-02 16:17:13,634 DEBUG header - << "X-Powered-By: ASP.NET[\r][\n]"

2014-05-02 16:17:13,634 DEBUG header - << "Date: Fri, 02 May 2014 23:17:13 GMT[\r][\n]"

2014-05-02 16:17:13,634 DEBUG header - << "Connection: close[\r][\n]"

2014-05-02 16:17:13,634 DEBUG header - << "Content-Length: 1245[\r][\n]"

2014-05-02 16:17:13,634 DEBUG header - << "[\r][\n]"

2014-05-02 16:17:13,634 DEBUG HttpMethodBase - Buffering response body

2014-05-02 16:17:13,634 DEBUG HttpMethodBase - Should close connection in response to directive: close

2014-05-02 16:17:13,634 DEBUG HttpConnection - Releasing connection back to connection manager.

2014-05-02 16:17:13,634 DEBUG MultiThreadedHttpConnectionManager - Freeing connection, hostConfig=HostConfiguration[host=http://[redacted]]

2014-05-02 16:17:13,634 DEBUG IdleConnectionHandler - Adding connection at: 1399072633634

2014-05-02 16:17:13,634 DEBUG MultiThreadedHttpConnectionManager - Notifying no-one, there are no waiting threads

2014-05-02 16:17:13,743 DEBUG DefaultHttpParams - Set parameter http.protocol.version = HTTP/1.0

2014-05-02 16:17:13,743 DEBUG MultiThreadedHttpConnectionManager - HttpConnectionManager.getConnection:  config = HostConfiguration[host=http://[redacted]], timeout = 0

2014-05-02 16:17:13,743 DEBUG MultiThreadedHttpConnectionManager - Getting free connection, hostConfig=HostConfiguration[host=http://[redacted]]

2014-05-02 16:17:13,743 DEBUG HttpConnection - Open connection to [redacted]:80

2014-05-02 16:17:13,743 DEBUG header - >> "GET /[redacted]/4824003649570109115 HTTP/1.0[\r][\n]"

2014-05-02 16:17:13,743 DEBUG HttpMethodBase - Adding Host request header

2014-05-02 16:17:13,743 DEBUG header - >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"

2014-05-02 16:17:13,743 DEBUG header - >> "Cookie: ASP.NET_SessionId=[redacted][\r][\n]"

2014-05-02 16:17:13,743 DEBUG header - >> "Host: [redacted][\r][\n]"

2014-05-02 16:17:13,743 DEBUG header - >> "[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "HTTP/1.1 401 Unauthorized[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "HTTP/1.1 401 Unauthorized[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "Content-Type: text/html[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "Server: Microsoft-IIS/7.5[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "WWW-Authenticate: NTLM[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "WWW-Authenticate: Negotiate[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "X-UA-Compatible: IE=EmulateIE8[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "Date: Fri, 02 May 2014 23:17:13 GMT[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "Connection: close[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "Content-Length: 1293[\r][\n]"

2014-05-02 16:17:13,759 DEBUG header - << "[\r][\n]"

2014-05-02 16:17:13,759 DEBUG HttpMethodDirector - Authorization required

2014-05-02 16:17:13,759 DEBUG AuthChallengeProcessor - Supported authentication schemes in the order of preference: [ntlm, digest, basic, ntlm]

2014-05-02 16:17:13,759 INFO  AuthChallengeProcessor - ntlm authentication scheme selected

2014-05-02 16:17:13,759 DEBUG AuthChallengeProcessor - Using authentication scheme: ntlm

2014-05-02 16:17:13,759 DEBUG AuthChallengeProcessor - Authorization challenge processed

2014-05-02 16:17:13,759 DEBUG HttpMethodDirector - Authentication scope: NTLM <any realm>@[redacted]:80

2014-05-02 16:17:13,759 DEBUG HttpMethodDirector - Credentials required

2014-05-02 16:17:13,759 DEBUG HttpMethodDirector - Credentials provider not available

2014-05-02 16:17:13,759 INFO  HttpMethodDirector - No credentials available for NTLM <any realm>@[redacted]:80

2014-05-02 16:17:13,759 DEBUG HttpMethodBase - Buffering response body

2014-05-02 16:17:13,759 DEBUG HttpMethodBase - Should close connection in response to directive: close

2014-05-02 16:17:13,759 DEBUG HttpConnection - Releasing connection back to connection manager.

2014-05-02 16:17:13,759 DEBUG MultiThreadedHttpConnectionManager - Freeing connection, hostConfig=HostConfiguration[host=http://[redacted]]

[Pontus Frendin]

I am having this exact problem. Was it ever solved? And do you by any chance have a suggestion to what the solution was? Considering the age of this thread I realize that it is a longshot to ask but since the problem still is reproducible in the version of ZAP that I use (2.5.0) others could be interested of the answer as well.

[Ilya Tsapin]

Hi, 

Got the same problem. Do you happened to find the solution? 

Thank you

[l043nz0]

I'm sorry but I don't remember.