ZAP - USER_AGENT SETTING when using it through the JAVA API

[Usman Waheed]

Hi,

I am facing a rather unusual issue when i use ZAP to spider a URL

If i start it using the JAVA API and try to spider a website the USER-AGENT is set to: Java/1.7.0_21

Noted below are the REQUEST_HEADERS i extracted out.

User-Agent: Java/1.7.0_21

Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

Proxy-Connection: keep-alive

Content-length: 0

If i start it through the UI and try to spider the website the USER-AGENT is set to: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;)

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;)

Pragma: no-cache

Cache-control: no-cache

Content-Type: application/x-www-form-urlencoded

Content-length: 0

Is there a way i can change the USER_AGENT setting in daemon mode to be: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;)

The problem is with the USER_AGENT set to: Java/1.7.0_21 , a redirect happens to a webpage which ZAP cannot comprehend and i am getting the following error message:

I wonder if i can change the USER_AGENT to be: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;) when using the JAVA API?

[java] 0 [main] INFO org.zaproxy.zap.ZAP  - OWASP ZAP 2.1.0 started.
     [java] 924 [main] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE  - dataFileCache open start
     [java] 951 [main] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE  - dataFileCache open end
     [java] 1401 [main] INFO org.parosproxy.paros.view.View  - Initialising View
     [java] 3284 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin Path Traversal
     [java] 3284 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin Remote File Inclusion
     [java] 3284 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin URL Redirector Abuse
     [java] 3285 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin Server side include
     [java] 3287 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin Cross Site Scripting (Reflected)
     [java] 3287 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin SQL Injection
     [java] 3288 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin Directory browsing
     [java] 3288 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin Session ID in URL rewrite
     [java] 3288 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin Secure page browser cache
     [java] 3288 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin External redirect
     [java] 3288 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin CRLF injection
     [java] 3288 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugin Parameter tampering
     [java] 4256 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Change user agent to other browsers. 
     [java] 4256 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Detect insecure or potentially malicious content in HTTP responses.
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Detect and alert 'Set-cookie' attempt in HTTP response for modification.
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Avoid browser cache (strip off IfModifiedSince)
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Log cookies sent by browser.
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Log unique GET queries into file:filter/get.xls
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Log unique POST queries into file:  filter/post.xls
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Log request and response into file: filter/message.txt
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Replace HTTP request body using defined pattern.
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Replace HTTP request header using defined pattern.
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Replace HTTP response body using defined pattern.
     [java] 4257 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Replace HTTP response header using defined pattern.
     [java] 4258 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded filter Send ZAP session request ID
     [java] 5743 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: Incomplete or no cache-control and pragma HTTPHeader set
     [java] 5744 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: Content-Type header missing
     [java] 5744 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: Cookie no http-only flag
     [java] 5744 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: Cookie without secure flag
     [java] 5744 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: Cross-domain JavaScript source file inclusion
     [java] 5745 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: IE8's XSS protection filter not disabled
     [java] 5745 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: Secure pages including mixed content
     [java] 5745 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: Password Autocomplete in browser
     [java] 5745 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: Private IP disclosure
     [java] 5746 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: X-Content-Type-Options header missing
     [java] 5746 [main] INFO org.zaproxy.zap.extension.pscan.ExtensionPassiveScan  - loaded passive scan rule: X-Frame-Options header not set
     [java] 8694 [main] INFO org.parosproxy.paros.control.Control  - New Session
     [java] 19358 [ZAP-ProxyThread] ERROR org.zaproxy.zap.ZAP$UncaughtExceptionLogger  - Exception in thread "ZAP-ProxyThread"
     [java] java.lang.NullPointerException
     [java] 	at org.zaproxy.zap.extension.api.API.handleApiRequest(Unknown Source)
     [java] 	at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(Unknown Source)
     [java] 	at org.parosproxy.paros.core.proxy.ProxyThread.run(Unknown Source)
     [java] 	at java.lang.Thread.run(Thread.java:722)
     [java] 19366 [main] ERROR org.zaproxy.zap.ZAP$UncaughtExceptionLogger  - Exception in thread "main"
     [java] java.net.SocketException: Unexpected end of file from server
     [java] 	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:718)
     [java] 	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579)
     [java] 	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:715)
     [java] 	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579)
     [java] 	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1322)
     [java] 	at SecurityRegressionTest.openUrlViaProxy(SecurityRegressionTest.java:24)
     [java] 	at SecurityRegressionTest.testDaemonWave(SecurityRegressionTest.java:58)
     [java] 	at SecurityRegressionTest.main(SecurityRegressionTest.java:125)
     [java] 19367 [ZAP-ProxyThread] ERROR org.zaproxy.zap.ZAP$UncaughtExceptionLogger  - Exception in thread "ZAP-ProxyThread"
     [java] java.lang.NullPointerException
     [java] 	at org.zaproxy.zap.extension.api.API.handleApiRequest(Unknown Source)
     [java] 	at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(Unknown Source)
     [java] 	at org.parosproxy.paros.core.proxy.ProxyThread.run(Unknown Source)
     [java] 	at java.lang.Thread.run(Thread.java:722)

[thc202]

Hi.

The NullPointerException seems to be Issue 665 [1], does the accessed URI contain a non-empty path component?
"http://example.com/" (note the "/" at the end of the URI) instead of "http://example.com".

I wonder if i can change the USER_AGENT to be: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;) when using the JAVA API?

You can try set the system property "http.agent" to the value you want but it will still append the Java version.
You have to start your Java client API with -Dhttp.agent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;)"


[1] https://code.google.com/p/zaproxy/issues/detail?id=665

Best regards.

[Usman Waheed]

Hi,

Yes there is a / at the end of the URL i am spidering on the start. 

I have http://www.example.com/ but because of the USER AGENT setting it redirects to http://www.example2.com without the / and as a result we get the NULLPOINTER EXCEPTION.

Your suggestion worked, Thank You :)

I set the system property "http.agent" using the code below and everything works now.

import java.util.Properties;

Properties props = System.getProperties();

props.setProperty("http.agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;)");

ZAP now starts using the USER AGENT setting i have set and not Java/1.7.0_21.

Best Regards,

Usman