Using Swagger Spec to run automated tests against your API

[chrokh]

I'm new to Swagger but I believe I'm getting a clearer picture of the eco-system, and it looks fantastic.

However, I'm wondering whether there's a module that allows me to run automated tests against my actual API to see whether it conforms with my Swagger spec. Without server integration.

Assuming you have a Swagger spec-file and a running API it sounds reasonable to me that this should be possible? Basically, black-box testing. Is there something like this?

Thanks in advance!

[tony tam]

Hi, there is a tool to analyze your swagger spec files to see that they're valid.  A javascript version will be available shortly but in the mean time, you can use this:

https://github.com/wordnik/swagger-codegen/#validating-your-swagger-spec

If you are on the previous (1.1) version of the spec, you can use this tool to convert it to 1.2:

https://github.com/wordnik/swagger-codegen/#migrating-from-swagger-11-to-12-format

And once you're using 1.2, you can use this JSON-schema for validation of each api declaration:

https://github.com/wordnik/swagger-core/blob/master/schemas/api-declaration-schema.json

[chrokh]

Thanks for the answer!

Ok, so maybe I'm misunderstanding but I think that I'm not :)

What I mean is that it should be possible to first write a spec file manually, then write part of the server (api-provider) code manually, and then run some automated tests (that get generated from the spec file), and thus get results on how much of the spec has been implemented.

I'm basically talking about TDD using the spec as the "outermost" test. The test-runner would probably use curl to make sensible requests and analyze whether the responses match those defined in the spec file.

Or was that actually what you answered?

[W. Trevor King]

On Friday, 16 August 2013 08:31:13 UTC-7, chrokh wrote:

What I mean is that it should be possible to first write a spec file manually, then write part of the server (api-provider) code manually, and then run some automated tests (that get generated from the spec file), and thus get results on how much of the spec has been implemented.

I'm basically talking about TDD using the spec as the "outermost" test. The test-runner would probably use curl to make sensible requests and analyze whether the responses match those defined in the spec file.

Has anyone written up something like this yet?  I'm looking around and haven't found it.  I want to be able to do something like:

  $ swagger-test --parameters=test-parameters.json http://petstore.swagger.wordnik.com/api/api-docs
  GET /pets         [ok]
  POST /pets        [ok]
  GET /pets/{id}    [ok]
  DELETE /pets/{id} [ok]
  4/4 paths passed

Things like auth or parameters (e.g. id=0) would be stored in the file configured via --parameters.  If its not written yet, but other folks are interested in getting together to write it, let me know too ;).

Thanks,
Trevor

[Dave Bryand]

I'm also trying to find something like this. It would basically be what Dredd is to API Blueprint.

[Wolfgang Grieskamp]

There is a lot what could be done here, and a bunch of work exists usually attributed as "model-based testing". However, just based on the swagger spec, I don't think you would get very far, as it doesn't allow to describe valid sequences of API calls. For interesting test cases, you would need to make at least 2-3 valid calls to establish a certain state on the server. There needs to be a way how to specify that sequences, because a test generator cannot easily guess them.

--
You received this message because you are subscribed to the Google Groups "Swagger" group.
To unsubscribe from this group and stop receiving emails from it, send an email to swagger-swaggers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[W. Trevor King]

On Mon, Nov 24, 2014 at 04:36:07PM -0800, 'Wolfgang Grieskamp' via Swagger wrote:
> There is a lot what could be done here, and a bunch of work exists
> usually attributed as "model-based testing".

Ah, thanks for pointing out the right buzzword :).

> However, just based on the swagger spec, I don't think you would get
> very far, as it doesn't allow to describe valid sequences of API
> calls. For interesting test cases, you would need to make at least
> 2-3 valid calls to establish a certain state on the server. There
> needs to be a way how to specify that sequences, because a test
> generator cannot easily guess them.

That's part of what --parameters is for in my suggested:

> >> $ swagger-test --parameters=test-parameters.json http://
> >> petstore.swagger.wordnik.com/api/api-docs

I'd handle the rest of it by launching the test backend with a
predefined state. Obviously, this would still just test that the
responses matched the expected schema, not that they made sense
(e.g. that PUT calls succesfully updated the attribute in question),
so it's not really a great way to test your backend implementation.
It would just be an easy way to make sure your spec matches reality.

Cheers,
Trevor

--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[tony tam]

I do think the code generator can drastically speed up the process of writing automated tests.  It may not be end-to-end without a fair amount of custom template work for your use case, but doing the basic plumbing of writing the tests could be really sped up.

[Gallagher Polyn]

I had the same interest as Trevor, and I think based on his original post and this earlier group thread, groups.google.com/forum/#!searchin/swagger-swaggersocket/test/swagger-swaggersocket/OsUi_Rz_erM/pKXgIa_Uw8MJ, the answer is 'no, there is no project support for automated testing with swagger docs.'

G

[Andy Sharp]

Have a look at this project: https://www.npmjs.com/package/swagger-test

I'm also keen to include the swagger spec in our TDD approach. It seems wrong to have a system artefact that is not subject to tests

[mission liao]

I think the tool you need is something like this:

 - able to load Swagger spec(handwritten by you)

 - generate a corresponding static client (or a dynamic client, without any pre-processing)

 - then you can send request/receive response via the client to your server without composing raw requests in http.

There are tools for this purpose:

 - https://github.com/swagger-api/swagger-js in javascript

 - in python, there are many of them (keyword: dynamic client)

   - swagger-py https://github.com/digium/swagger-py (supports Swagger 1.2)

   - pyswagger https://github.com/mission-liao/pyswagger (maintained by me)

   - bravado https://github.com/Yelp/bravado (forked from swagger-py and maintained by Yelp, supports Swagger 2.0)

chrokh於 2013年8月16日星期五 UTC+8下午7時38分35秒寫道：

[R. Rajesh Jeba Anbiah]

Very interesting thread.

I'm also looking for automated testing option for Swagger. Any new tools available now? 

[Morris Brodersen]

You could try https://github.com/mobilcom-debitel/got-swag

[Cody Reichert]

I've been really interested in this problem for a while now, and your assumption is correct that it's definitely possible. We've been working on solving this at Assertible, and a couple of the harder parts we've hit are:

Route parameters and other variables. Required "setup" for PUT/DELETE/etc requests. Setting up auth. In a lot of cases there just has to be some additional configuration.

That said, we've built what I this is a very good starting point to automatically generate a test suite from a Swagger spec.

We put together a post on Testing an API using Swagger.

Basically, when you import a Swagger spec, a test is generated for each unique endpoint/method combination (like GET /users and POST /users). Route parameters are automatically filled in as "variables" and auth is stubbed out from the "securityDefinitions" found in the swagger spec. The actual tests/validations are done by creating "assertions" that are created for the expected status code response.

For example, importing the Twitter Swagger spec would give you something like this:

What are your (or anyone elses) thoughts on this approach? We've got a lot of plans/ideas for how we can dynamically generate even more tests and assertions from a Swagger spec. The idea is that, you simply have to import your spec once, and you're set up with automated tests, monitoring, failure alerts, etc, right out of the box.

Let me know what you think - I'm really interested in others people's thoughts about testing from a spec!

- Cody Reichert

codyre...@gmail.com

https://assertible.com

[Noah Dietz]

Hey y'all!

Great conversation! This is something I've been working in/on for a few years now, here and there. I started by working on github.com/apigee-127/swagger-test-templates , a unit test scaffolding generator (the concept of which I think was mentioned a bit in conversation here earlier?). It parses a swagger spec and generates unit tests for every path/operation/status code defined, while exposing configurations for supplying parameters and expected response bodies. The main goal is to accelerate API development by enabling developers to create test code for their APIs early and often, thus strengthening the contract between implementation and specification. It was meant to be integrated with swagger-node, but has gotten a decent amount of use as an individual module. 

I've since been working on an unofficial rewrite built on github.com/apigee-127/sway . The rewrite can be found at github.com/noahdietz/oatts and I'd love feedback/contributors as it is in the early stages.

@Cody, I love the problems you're solving and bringing up. One thing I have found is valuable is if you can't assert against actual values in the expected responses, to merely validate the schema of the response body (for JSON, at least) is definitely a win (unsure if you Assertible is actually doing that just after a quick glance). WRT populating auth values, that's a tough one. With the aforementioned swagger-test-templates, because it was designed to generate test code locally, we added support for loading values from a config file or the environment for such sensitive things, rather than hard code them. This is tougher when all done in the cloud for them. Maybe you can expose a hook that they configure to pull some creds at runtime, that are not stored and they can change whenever? Lets chat if you're interested in sharing ideas! I'm @no_d_here on Twitter if thats easier than this or email.

[Cody Reichert]

Noah - thanks for your thoughts here! oatts looks like a really interesting project - it's awesome to see more push towards OpenAPI tooling. I'm about to install it and try it out on one of my specs locally. I'll let you know how it goes and report any issues (if any). Would love to contribute, so I'll keep an eye on the repo. I really like the idea of contract testing the spec against the backend early in development, we're trying to push that it should absolutely be done after development as well.

What you said about the problem/solution space is definitely in-line with what I've been working on with Assertible.

if you can't assert against actual values in the expected responses, to merely validate the schema of the response body (for JSON, at least) is definitely a win (unsure if you Assertible is actually doing that just after a quick glance). WRT populating auth values, that's a tough one. ... Maybe you can expose a hook that they configure to pull some creds at runtime, that are not stored and they can change whenever?

The JSON Schema validation is a really good idea - we do already have a test assertion for validating an HTTP response against JSON Schema, enabling this for Swagger specs out of the box is doable, I'll take a look. On the auth problem again, that's right on the money! Assertible has a concept of setup steps, which are basically requests run before the actual test that can be used to populate variables. Auth and route parameters are the driving use-case there: make a request to fetch a token or similar, and use a {{variable}} in the header of the test. Route parameters work the same, but if a `default` value is specified for the parameter in the spec, we can actually fully populate this and not require any additional setup. Cases like this really make it easy to just import -> test.

I like that oatts and Assertible are at separate ends of the testing spectrum, too -- the more tools we can get around this the better.

Lets chat if you're interested in sharing ideas! I'm @no_d_here on Twitter if thats easier than this or email.

That sounds great -- I just watched oatts and followed you on Twitter. Looking forward to your updates. If you're able to try out Assertible, I'd love to get your feedback. It will probably be easier to chat over email (codyre...@gmail.com) so we don't pull this thread too far off topic.

- Cody Reichert