api scan against openapi seems to only scan localhost

[Barry Kaplan]

I'm running the zap2docker-weekly using

command:
  zap-api-scan.py -d
   -z "-configfile zap.config"
   -S -f openapi -t api.json
   -g api-scan.conf
   -r report.html

The logs indicate it sees the target

2019-04-23 16:45:19,456 Target: api.json

But after that I can't tell what is happening

2019-04-23 16:45:19,459 Starting ZAP
2019-04-23 16:45:19,459 Params: ['zap-x.sh', '-daemon', '-port', '36334', '-host', '0.0.0.0', '-config', 'api.disablekey=true', '-config', 'api.addrs.addr.name=.*', '-config', 'api.addrs.addr.regex=true', '-addonupdate', '-addoninstall', 'pscanrulesBeta', '-configfile', 'zap.config']
2019-04-23 16:45:19,465 Starting new HTTP connection (1): localhost:36334
_XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created.
2019-04-23 16:45:20,468 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:21,471 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:22,473 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:23,476 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:24,479 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:25,481 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:26,485 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:27,489 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:28,495 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:29,498 Starting new HTTP connection (1): localhost:36334
2019-04-23 16:45:30,502 Starting new HTTP connection (1): localhost:36334

And it goes on like. No other logs. No report is generated. No errors it appears.  

What is supposed to running at localhost:36344?

[Barry Kaplan]

I have also used -t /zap/wrk/api.json, execed into the container and saw that that api.json is where it's expected. Same output.

[Simon Bennetts]

Whats in your api.json file?

If thats specifying localhost then it will be working as expected :)

[Barry Kaplan]

Ok, hacking the paths I seem to have gotten a bit further. from docker-compose.yml

  zap:
    image: owasp/zap2docker-weekly
    container_name: zap
    command:
      zap-api-scan.py
      -z "-configfile /zap/wrk/zap.config"
      -S -f openapi -t api.yml
      -r report.html
    volumes:
      - .:/zap/wrk

I am getting an error parsing the openapi spec

7956 [ZAP-Import-OpenAPI-1] ERROR io.swagger.parser.SwaggerCompatConverter  - failed to read resource listing
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'openapi': was expecting ('true', 'false' or 'null')
 at [Source: /tmp/openapi8739339247369534937.defn; line: 1, column: 9]

The api.yml is valid, but is 3.x. Does zap not support openapi 3?

[Barry Kaplan]

I change from api.yml to api.json, the errors are a bit different. Kinda seems like some code is missing from the container?

6900 [ZAP-daemon] INFO org.zaproxy.zap.DaemonBootstrap  - ZAP is now listening on 0.0.0.0:41724
7170 [ZAP-ProxyThread-2] WARN org.zaproxy.zap.extension.api.API  - Bad request to API endpoint [/JSON/script/action/load/] from [127.0.0.1]:
Does Not Exist (does_not_exist) : /home/zap/.ZAP_D/scripts/scripts/httpsender/Alert_on_HTTP_Response_Code_Errors.js
 at org.zaproxy.zap.extension.script.ScriptAPI.handleApiAction(ScriptAPI.java:238)
 at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:449)
 at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:463)
 at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:320)
 at java.lang.Thread.run(Thread.java:748)
7182 [ZAP-ProxyThread-3] WARN org.zaproxy.zap.extension.api.API  - Bad request to API endpoint [/JSON/script/action/enable/] from [127.0.0.1]:
Does Not Exist (does_not_exist) : scriptName
 at org.zaproxy.zap.extension.script.ScriptAPI.handleApiAction(ScriptAPI.java:206)
 at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:449)
 at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:463)
 at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:320)
 at java.lang.Thread.run(Thread.java:748)
7187 [ZAP-ProxyThread-4] WARN org.zaproxy.zap.extension.api.API  - Bad request to API endpoint [/JSON/script/action/load/] from [127.0.0.1]:
Does Not Exist (does_not_exist) : /home/zap/.ZAP_D/scripts/scripts/httpsender/Alert_on_Unexpected_Content_Types.js
 at org.zaproxy.zap.extension.script.ScriptAPI.handleApiAction(ScriptAPI.java:238)
 at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:449)
 at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:463)
 at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:320)
 at java.lang.Thread.run(Thread.java:748)
7192 [ZAP-ProxyThread-5] WARN org.zaproxy.zap.extension.api.API  - Bad request to API endpoint [/JSON/script/action/enable/] from [127.0.0.1]:
Does Not Exist (does_not_exist) : scriptName
 at org.zaproxy.zap.extension.script.ScriptAPI.handleApiAction(ScriptAPI.java:206)
 at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:449)
 at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:463)
 at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:320)
 at java.lang.Thread.run(Thread.java:748)
7593 [ZAP-Import-OpenAPI-1] WARN org.zaproxy.zap.extension.openapi.ExtensionOpenApi  - Failed to parse swagger defn {"openapi":"3.0.0","info":{"description":"VIMANA's API","version":"2.0","title":"VIMANA API"},"servers":[{"url":"https://api.staging.vimana.us/api","description":"Staging environment"}],"security":[{"BearerAuth":[]}],"paths":{"/v3/calendar/timezone":{"get":{"tags":["Calendar"],"summary":"Get timezone for a plant","parameters":[{"$ref":"#/components/parameters/xTenantHeader"}],"responses":{"200":{"description":"OK","content":{"application/json":{"schema":{"$ref":"#/components/schemas/timezone"}}}},
...

[Barry Kaplan]

Bit of above log that got truncated

7593 [ZAP-Import-OpenAPI-1] WARN org.zaproxy.zap.extension.openapi.ExtensionOpenApi  - Failed to parse swagger defn {"openapi":"3.0.0","info":{"description":"VIMANA's API","version":"2.0","title":"VIMANA API"},"servers":[{"url":"https://api.staging.vimana.us/api","description":"Staging environment"}],"security":[{"BearerAuth":[]}],"paths":{"/v3/calendar/timezone":{"get":{"tags":["Calendar"],"summary":"Get timezone for a plant","parameters":[{"$ref":"#/components/parameters/xTenantHeader"}],"responses":{"200":{"description":"OK","content":{"application/json":{"schema":{"$ref":"#/components/schemas/timezone"}}}},"401":{"$ref":"#/components/responses/Unauthorized"},"403":{"$ref":"#/components/responses/InsufficientPerms"},

[thc...@gmail.com]

> The api.yml is valid, but is 3.x. Does zap not support openapi 3?

No, the OpenAPI add-on does not support that version yet.

Best regards.

[Barry Kaplan]

Seems the severs/localhost was a decoy. The parsing of the api is what is failing.

[kingthorin+owaspzap]

OpenAPI v3 spec isn't supported yet:

https://github.com/zaproxy/zaproxy/issues/4549

[Barry Kaplan]

Ok, I converted to swagger2.

I still get a whole bunch messages like:

8771 [ZAP-Import-OpenAPI-1] WARN io.swagger.util.PropertyDeserializer  - no property from null, null, {ENUM=null, TITLE=null, DESCRIPTION=null, DEFAULT=null, PATTERN=null, DESCRIMINATOR=null, MIN_ITEMS=null, MAX_ITEMS=null, MIN_PROPERTIES=null, MAX_PROPERTIES=null, MIN_LENGTH=null, MAX_LENGTH=null, MINIMUM=null, MAXIMUM=null, EXCLUSIVE_MINIMUM=null, EXCLUSIVE_MAXIMUM=null, UNIQUE_ITEMS=null, EXAMPLE=null, TYPE=null, FORMAT=null, READ_ONLY=null, VENDOR_EXTENSIONS={}, MULTIPLE_OF=null}
8774 [ZAP-Import-OpenAPI-1] WARN io.swagger.util.PropertyDeserializer  - no property from null, null, {ENUM=null, TITLE=null, DESCRIPTION=null, DEFAULT=null, PATTERN=null, DESCRIMINATOR=null, MIN_ITEMS=null, MAX_ITEMS=null, MIN_PROPERTIES=null, MAX_PROPERTIES=null, MIN_LENGTH=null, MAX_LENGTH=null, MINIMUM=null, MAXIMUM=null, EXCLUSIVE_MINIMUM=null, EXCLUSIVE_MAXIMUM=null, UNIQUE_ITEMS=null, EXAMPLE=null, TYPE=null, FORMAT=null, READ_ONLY=null, VENDOR_EXTENSIONS={}, MULTIPLE_OF=null}
8775 [ZAP-Import-OpenAPI-1] WARN io.swagger.util.PropertyDeserializer  - no property from null, null, {ENUM=null, TITLE=null, DESCRIPTION=null, DEFAULT=null, PATTERN=null, DESCRIMINATOR=null, MIN_ITEMS=null, MAX_ITEMS=null, MIN_PROPERTIES=null, MAX_PROPERTIES=null, MIN_LENGTH=null, MAX_LENGTH=null, MINIMUM=null, MAXIMUM=null, EXCLUSIVE_MINIMUM=null, EXCLUSIVE_MAXIMUM=null, UNIQUE_ITEMS=null, EXAMPLE=null, TYPE=null, FORMAT=null, READ_ONLY=null, VENDOR_EXTENSIONS={}, MULTIPLE_OF=null}

I can't see how these provide any info as to what might be wrong. There no "null" values -- or even the string "null" -- in the swagger.json

I have `-d` for debug on. Is there any way to get zap to log a bit more of what its working on?

At the end of these message it emits an NPR, then starts zap again and emits the "Starting new HTTP connection" in a loop it seems. I'm not at clear on what zap is doing.

8923 [ZAP-Import-OpenAPI-1] WARN io.swagger.util.PropertyDeserializer  - no property from null, null, {ENUM=null, TITLE=null, DESCRIPTION=null, DEFAULT=null, PATTERN=null, DESCRIMINATOR=null, MIN_ITEMS=null, MAX_ITEMS=null, MIN_PROPERTIES=null, MAX_PROPERTIES=null, MIN_LENGTH=null, MAX_LENGTH=null, MINIMUM=null, MAXIMUM=null, EXCLUSIVE_MINIMUM=null, EXCLUSIVE_MAXIMUM=null, UNIQUE_ITEMS=null, EXAMPLE=null, TYPE=null, FORMAT=null, READ_ONLY=null, VENDOR_EXTENSIONS={}, MULTIPLE_OF=null}
8965 [ZAP-Import-OpenAPI-1] WARN org.zaproxy.zap.extension.openapi.ExtensionOpenApi  -
java.lang.NullPointerException
 at org.zaproxy.zap.extension.openapi.generators.ArrayGenerator.generate(ArrayGenerator.java:53)
 at org.zaproxy.zap.extension.openapi.generators.DataGenerator.generateBodyValue(DataGenerator.java:107)
 at org.zaproxy.zap.extension.openapi.generators.BodyGenerator.generate(BodyGenerator.java:90)
 at org.zaproxy.zap.extension.openapi.converter.swagger.RequestModelConverter.generateBody(RequestModelConverter.java:76)
 at org.zaproxy.zap.extension.openapi.converter.swagger.RequestModelConverter.convert(RequestModelConverter.java:49)
 at org.zaproxy.zap.extension.openapi.converter.swagger.SwaggerConverter.convertToRequest(SwaggerConverter.java:83)
 at org.zaproxy.zap.extension.openapi.converter.swagger.SwaggerConverter.getRequestModels(SwaggerConverter.java:77)
 at org.zaproxy.zap.extension.openapi.ExtensionOpenApi$3.run(ExtensionOpenApi.java:218)
2019-04-23 23:00:37,352 Trigger hook: pre_exit, args: 3
2019-04-24 01:23:43,238 Target: swagger2.json
2019-04-24 01:23:43,238 Could not find custom hooks file at /home/zap/.zap_hooks.py
2019-04-24 01:23:43,239 Trigger hook: cli_opts, args: 1
2019-04-24 01:23:43,243 Using port: 53696
2019-04-24 01:23:43,243 Trigger hook: start_zap, args: 2
2019-04-24 01:23:43,243 Starting ZAP
2019-04-24 01:23:43,243 Params: ['zap-x.sh', '-daemon', '-port', '53696', '-host', '0.0.0.0', '-config', 'api.disablekey=true', '-config', 'api.addrs.addr.name=.*', '-config', 'api.addrs.addr.regex=true', '-addonupdate', '-addoninstall', 'pscanrulesBeta', '-configfile', '/zap/wrk/zap.config']
2019-04-24 01:23:43,252 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:44,260 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:45,264 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:46,269 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:47,274 Starting new HTTP connection (1): localhost:53696
/zap/zap-x.sh: 10: kill: No such process


2019-04-24 01:23:48,279 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:49,289 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:50,296 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:51,302 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:52,310 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:53,313 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:54,320 Starting new HTTP connection (1): localhost:53696
2019-04-24 01:23:55,324 Starting new HTTP connection (1): localhost:53696
...

[Barry Kaplan]

I should add that there are other errors in the swagger.json that got left over from converting from openapi 3. (eg, "schema": {"oneOf": ...)

I will clean these up. But it would be nice to know if I am wasting my time. That is it would be nice to know exactly what zap is unhappy about.

[Barry Kaplan]

I convert fully to valid swagger2. Pretty got same NPRs and logs as above. At least from those messages I have no way of knowing what the problem is. Gonna table this for now.

[T M]

I am seeing similar errors. I am able to use ZAP to import the swagger petstore API: https://petstore.swagger.io/v2/swagger.json just fine. Any clue on what these error messages mean?