Writing scenarios for APIs with randomly generated IDs

[Matt Hauck]

I've run into this problem in the past and have run into it again so I thought I'd ask the group and see if anyone has a better solution for this issue than mine.

So, I'm (again) using cucumber to spec a publicly facing backend REST API. I quite like how I am able to specify the exact inputs and outputs of the API in full text so that consumers of the API can see exactly what needs to be input and what exactly they can expect as output. 

Anyhoo, part of this is that I want to be able to explicitly list out the URL inside the step; something like:

When I send a POST to the URL "/foo-resource/12"

The difficulty with this is that I do not know the value of "12" at the time of writing the steps because that gets generated randomly by the database, and it could obviously change every time the test is run, depending on what order the tests are run in. (To make matters worse perhaps, our code doesn't even use the id value, it uses a java.util.UUID.randomUUID()). The point is, it is a value different each time the test is run, but I still want it to show up inside the text of the scenario definition.

I have come up with the following two ways to solve this in times past doing something similar with cucumber, but neither are entirely satisfying:

1) Embed a substitution variable in the definition

Use something like "/foo-resource/:foo-id" in the scenario definition, and then do a string replacement in the step definition that swaps out ":foo-id" for the actual foo-id that was captured at some other step definition when the foo-resource was actually created. yuck. this is hard to follow and it just isn't a clean way of doing it.

2) Embed what the id should be in the creation step, and make the change in the database

In the step that creates the foo-resource, say something like: `When I create a foo-resource (with id "12")`, which follows the normal way of creating a foo-resource, and then sneaks into the database and updates the appropriate value to the expected id value. The problem with this is it requires backend manipulation doesn't follow the normal way of doing things and so is a bit funky.

This seems like a general enough problem that I'm hoping the community has a better answer than mine. Thanks for any help you may offer!

[Seb Rose]

Matt,

Can you explain why you "want it to show up inside the text of the scenario definition"?

What value does the actual UUID this give the readers of the scenario beyond saying "the UUID of foo-resource"?

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
ACCU - Professionalism in Programming - http://accu.org

http://www.claysnow.co.uk
http://twitter.com/#!/sebrose
http://uk.linkedin.com/in/sebrose

[Matt Hauck]

On Wednesday, January 29, 2014 12:41:11 AM UTC-8, Seb Rose wrote:

Matt,

Can you explain why you "want it to show up inside the text of the scenario definition"?

What value does the actual UUID this give the readers of the scenario beyond saying "the UUID of foo-resource"?

I suppose in the end it doesn't make a whole lot of difference, and it is just about the same to use something like "<fooResourceId>" in the url. What I really want to know is not so much how to get the string into the scenario -- but how to handle ids like these. Do you put them in the scenario (and modify the db to match that id); or do you put some generic text in the scenario (and keep the actual value in some hidden variable)? Or is there a better way? Is there a "best practices" for something like this?

[Timo Lehto]

On Wed, Jan 29, 2014 at 10:17 PM, Matt Hauck <matt...@gmail.com> wrote:

On Wednesday, January 29, 2014 12:41:11 AM UTC-8, Seb Rose wrote:

Matt,

Can you explain why you "want it to show up inside the text of the scenario definition"?

What value does the actual UUID this give the readers of the scenario beyond saying "the UUID of foo-resource"?

I suppose in the end it doesn't make a whole lot of difference, and it is just about the same to use something like "<fooResourceId>" in the url. What I really want to know is not so much how to get the string into the scenario -- but how to handle ids like these. Do you put them in the scenario (and modify the db to match that id); or do you put some generic text in the scenario (and keep the actual value in some hidden variable)? Or is there a better way? Is there a "best practices" for something like this?

Hi, I got no answers to you Matt, but wanted to chip in and share that I've been pondering very similar thoughts to those of Matts and used identical approaches to those of Matts with API testing and would like to hear any ideas for better ways from more eminent Cukers. Why I've decided to use these approaches is simply that since the audience is programmers the fact that the ids and such appear as valid ids rather than "the UUID of foo resource". I find this to be more informational:

When I send a GET request to /api with data { type: "something", id: 124562 }

When I send a GET request to /somethings/1234562

as oppose to these:

When I send a GET request to /api with data { type: "something", id: _the UUID of foo resource_ }

When I send a GET request to /somethings/the UUID of foo resource

When I send a GET request to /somethings/:uuid with the UUID of resource foo

I like them above because they are clean, they reveal what are the ids are like and what is their type (UUID, string, slug, numeric, etc..) without you actually having to read it. I've also used "Embed a substitution variable in the definition" (sorry, I may be slightly slipping out of topic here, but I think it's still related enough to be part of this discussion?) with APIs to display a sample of an actual valid response in the report by adding a step like:

And the response is like: _response_

Then during the execution the _response_ will be substituted with the actual JSON response or what ever.

What do you guys think of such approaches? Obviously this last variant doesn't actually test anything, but is there purely for documentational purposes. This is akin to the practices for displaying the id in the sense that it shows an actual sample of how the api responses just as the above shown approaches Matt brought up show an actual sample of a valid query and I like the way I can then just get the html output of the test run and provide it as a documentation for my api. I firmly believe that demonstration by example is a powerful tool what it comes to teaching, whether it be an api or something else.

[Seb Rose]

On 29 January 2014 20:17, Matt Hauck <matt...@gmail.com> wrote:

On Wednesday, January 29, 2014 12:41:11 AM UTC-8, Seb Rose wrote:

Matt,

Can you explain why you "want it to show up inside the text of the scenario definition"?

What value does the actual UUID this give the readers of the scenario beyond saying "the UUID of foo-resource"?

I suppose in the end it doesn't make a whole lot of difference, and it is just about the same to use something like "<fooResourceId>" in the url. What I really want to know is not so much how to get the string into the scenario -- but how to handle ids like these. Do you put them in the scenario (and modify the db to match that id); or do you put some generic text in the scenario (and keep the actual value in some hidden variable)? Or is there a better way? Is there a "best practices" for something like this?

The general rule of thumb is to "avoid incidental details". The actual ID is incidental - the important thing is that the ID requested is the same as the one that is returned when the resource is created. Use domain language to describe this in the scenario and hide the implementation in the code.

You could, for instance, pass the name "foo-resource" as an argument to your step definitions, storing the ID in a map when you create the resource and retrieving it on a get - thus freeing you up to deal with multiple, named resources without confusing the reader with impenetrable UUIDs

[Roberto Lo Giacco]

Il giorno mercoledì 29 gennaio 2014 22:58:48 UTC+1, Seb Rose ha scritto:

On 29 January 2014 20:17, Matt Hauck <matt...@gmail.com> wrote:

On Wednesday, January 29, 2014 12:41:11 AM UTC-8, Seb Rose wrote:

Matt,

Can you explain why you "want it to show up inside the text of the scenario definition"?

What value does the actual UUID this give the readers of the scenario beyond saying "the UUID of foo-resource"?

I suppose in the end it doesn't make a whole lot of difference, and it is just about the same to use something like "<fooResourceId>" in the url. What I really want to know is not so much how to get the string into the scenario -- but how to handle ids like these. Do you put them in the scenario (and modify the db to match that id); or do you put some generic text in the scenario (and keep the actual value in some hidden variable)? Or is there a better way? Is there a "best practices" for something like this?

The general rule of thumb is to "avoid incidental details". The actual ID is incidental - the important thing is that the ID requested is the same as the one that is returned when the resource is created. Use domain language to describe this in the scenario and hide the implementation in the code.

You could, for instance, pass the name "foo-resource" as an argument to your step definitions, storing the ID in a map when you create the resource and retrieving it on a get - thus freeing you up to deal with multiple, named resources without confusing the reader with impenetrable UUIDs

May I add that providing insight information on the resource ID format breaks the fundamental rule of REST to have totally transparent URIs?

Your REST API users shouldn't be building their URIs, you should provide them all the necessary links (RMM level 3) but, in case you are not yet at that stage, it would really help to move toward that if your users don't make any assumption on the URI format: whatever is the value returned within the ID field is what they have to append to the retrievel URL:

POST /foo <json-payload>

201 CREATED

<resource-id>

GET /foo/<resource-id>

200 OK

<resource-representation>

 

[Andrew Premdas]

--

Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

<controversial>

Personally I think putting url information in API tests is a bad idea. I see this as an implementation detail that is subject to alot of change. Requiring features to be re-written when implementation details change makes features much more expensive to live with. It also breaks DRY, as the detail is defined in (at least) two places.

Instead I think features for api's should you use business language focusing on why you are talking to the api and what you are expecting from it. In this way you can explore the mapping between natural language 'I want a foo' and your api GET /foo. The features talk about why you want a foo, and what you expect to get from the foo, step definitions deal with GET'ing a foo and the technical contents of the response.

So for me 

1. No URL's, id's in features

2. No json response hashes etc. in features

If you want to test at that level use RSpec, and test much more in isolation.

There is some more to this, things like thinking about version 2 of the API. Overall think about the cost of upkeep features like this impose on the API.

</contreversial>

HTH

Andrew

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Roberto Lo Giacco]

On Thu, Jan 30, 2014 at 3:32 PM, Andrew Premdas <apre...@gmail.com> wrote:

On 30 January 2014 13:41, Roberto Lo Giacco <rlog...@gmail.com> wrote:

Il giorno mercoledì 29 gennaio 2014 22:58:48 UTC+1, Seb Rose ha scritto:

On 29 January 2014 20:17, Matt Hauck <matt...@gmail.com> wrote:

On Wednesday, January 29, 2014 12:41:11 AM UTC-8, Seb Rose wrote:

Matt,

Can you explain why you "want it to show up inside the text of the scenario definition"?

What value does the actual UUID this give the readers of the scenario beyond saying "the UUID of foo-resource"?

I suppose in the end it doesn't make a whole lot of difference, and it is just about the same to use something like "<fooResourceId>" in the url. What I really want to know is not so much how to get the string into the scenario -- but how to handle ids like these. Do you put them in the scenario (and modify the db to match that id); or do you put some generic text in the scenario (and keep the actual value in some hidden variable)? Or is there a better way? Is there a "best practices" for something like this?

The general rule of thumb is to "avoid incidental details". The actual ID is incidental - the important thing is that the ID requested is the same as the one that is returned when the resource is created. Use domain language to describe this in the scenario and hide the implementation in the code.

You could, for instance, pass the name "foo-resource" as an argument to your step definitions, storing the ID in a map when you create the resource and retrieving it on a get - thus freeing you up to deal with multiple, named resources without confusing the reader with impenetrable UUIDs

May I add that providing insight information on the resource ID format breaks the fundamental rule of REST to have totally transparent URIs?

Your REST API users shouldn't be building their URIs, you should provide them all the necessary links (RMM level 3) but, in case you are not yet at that stage, it would really help to move toward that if your users don't make any assumption on the URI format: whatever is the value returned within the ID field is what they have to append to the retrievel URL:

POST /foo <json-payload>

201 CREATED

<resource-id>

GET /foo/<resource-id>

200 OK

<resource-representation>

<controversial>

Personally I think putting url information in API tests is a bad idea. I see this as an implementation detail that is subject to alot of change. Requiring features to be re-written when implementation details change makes features much more expensive to live with. It also breaks DRY, as the detail is defined in (at least) two places.

Instead I think features for api's should you use business language focusing on why you are talking to the api and what you are expecting from it. In this way you can explore the mapping between natural language 'I want a foo' and your api GET /foo. The features talk about why you want a foo, and what you expect to get from the foo, step definitions deal with GET'ing a foo and the technical contents of the response.

So for me 

1. No URL's, id's in features

2. No json response hashes etc. in features

If you want to test at that level use RSpec, and test much more in isolation.

There is some more to this, things like thinking about version 2 of the API. Overall think about the cost of upkeep features like this impose on the API.

</contreversial>

HTH

I would generally agree with you, but you still have to provide examples in order to be able to spec out your system. In this case the example might be expressed by using JSON or XML or any other data format which is clearly understood by the parties defining the specs.

On an end user product I would use terms like "Click on the start button", "Select the filename field" and so forth, but on a REST API you have a predefined terminology which is the base of the convention:

- resources and their identifiers (URI)

- HTTP protocol (methods, response codes, headers, etc...)

- content types

How would you spec out that when your REST service receives a request for /foo/123 with a Content-Type header value of XML the response must be the XML representation of resource foo with ID 123?

I would use:

Given a resource foo with id 123 exists with the following data

   | attribute name | attribute value |

   ...

When the system receives a GET request for resource foo with id 123

And the request content type is "application/xml"

Then the response code is 200 OK

And the response payload is

   ```<xml here>```

I wouldn't use a different terminology to abstract something like that in _this_ case, but it obviously depends on the parties involved on the spec definition.

[Matt Hauck]

Thanks all for the input. Convinced that placing the hard-coded id inside the scenario is not the best way to do this, but is better to just place an identifying name like "<fooId>". This is a cleaner implementation and also provides a clearer API specification (so that consumers of it don't get confused and actually try to use the actual values). 

<twoCents>

Regarding the "<controversial>" section above, I would have to disagree. I think it is very helpful to put URLs / JSON response hashes, etc in the feature spec when the spec is for a REST API. Saying "send a POST to 'the foo create resource page'" is not nearly as helpful as "send a POST to '/foo'". 

As for the JSON/XML response body, I think there is a way to do this without making it brittle. Instead of doing a string match on the json/xml, you can use smarter matchers to check for the relevant subset of the returned data. This way, your spec can include only the necessary json/xml data for that scenario and doesn't have to break when a new value is added that is not relevant to that scenario. 

</twoCents>

--
Posting rules: http://cukes.info/posting-rules.html
---

You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/yaAGaJ45XYU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.

[Andrew Premdas]

I wouldn't, not in a feature anyhow. Instead I would ask 

    Why would anyone want a foo?

    Who, or what, would want a foo?

    How would I know what foo to get? (this touches on the idea of navigability of the service)

    What is required to get a foo?

       id?

       authentication?

   

    If I require something, say an id, how do I get it, how do I know I've got the right one, how do I stop people using the wrong one,     what happens if someone uses the wrong one, why do I need an id, etc. etc.

I feel this is the sort of information that should be captured and expressed in features. 

If some of these questions were answered, then it would be possible to give foo context and meaning, and perhaps produce example scenarios that were more meaningful. 

Once you are 'stuck' in the implementation details of the service, it is really hard to see another way, particularly one that is so much more abstract; so I don't expect people to get this, hence the 'controversy'

I would use:

Given a resource foo with id 123 exists with the following data

   | attribute name | attribute value |

   ...

When the system receives a GET request for resource foo with id 123

And the request content type is "application/xml"

Then the response code is 200 OK

And the response payload is

   ```<xml here>```

I wouldn't use a different terminology to abstract something like that in _this_ case, but it obviously depends on the parties involved on the spec definition.

--

Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Andrew Premdas]

Well i wouldn't expect you to agree :), or to fully understand (yet): I'm certainly not saying 'send a Post to anything'. Part of the point I'm failing to make is that POST is an implementation detail, and implementation details do not belong in features. For some reason people seem to think that REST API's should be 'cuked' in a different way because they are an `API`. There is no reason an API should be cuked differently than anything else. Features should capture the what and why, not the how. If you can't capture the what and why, how do you know that implementing the API gives you any value?

And I did say it was controversial ;)

All best

Andrew

--
Posting rules: http://cukes.info/posting-rules.html
---

You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/yaAGaJ45XYU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Posting rules: http://cukes.info/posting-rules.html
---

You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Matt Hauck]

haha. thanks again for the thoughts.

-Matt

[George Dinwiddie]

Matt,

On 1/29/14, 12:15 AM, Matt Hauck wrote:
> I've run into this problem in the past and have run into it again so I
> thought I'd ask the group and see if anyone has a better solution for
> this issue than mine.
>
> So, I'm (again) using cucumber to spec a publicly facing backend REST
> API. I quite like how I am able to specify the exact inputs and outputs
> of the API in full text so that consumers of the API can see exactly
> what needs to be input and what exactly they can expect as output.
>
> Anyhoo, part of this is that I want to be able to explicitly list out
> the URL inside the step; something like:
>
> When I send a POST to the URL "/foo-resource/12"

When you talk with the business people, do they *really* ask for
software in terms of POSTs and URLs? Or do they have some other language
that they use to describe what they want to accomplish?

- George

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[Matt Hauck]

On Thursday, January 30, 2014 at 5:08 PM, George Dinwiddie wrote:

Matt,

On 1/29/14, 12:15 AM, Matt Hauck wrote:

I've run into this problem in the past and have run into it again so I

thought I'd ask the group and see if anyone has a better solution for

this issue than mine.

So, I'm (again) using cucumber to spec a publicly facing backend REST

API. I quite like how I am able to specify the exact inputs and outputs

of the API in full text so that consumers of the API can see exactly

what needs to be input and what exactly they can expect as output.

Anyhoo, part of this is that I want to be able to explicitly list out

the URL inside the step; something like:

When I send a POST to the URL "/foo-resource/12"

When you talk with the business people, do they *really* ask for

software in terms of POSTs and URLs? Or do they have some other language

that they use to describe what they want to accomplish?

- George

I am using cucumber to communicate the API usage to our QA, not exactly to the business people.

-Matt

 

--

----------------------------------------------------------------------

* George Dinwiddie * http://blog.gdinwiddie.com

Software Development http://www.idiacomputing.com

Consultant and Coach http://www.agilemaryland.org

----------------------------------------------------------------------

--

[George Dinwiddie]

Matt,

On 1/30/14, 8:18 PM, Matt Hauck wrote:
>
> On Thursday, January 30, 2014 at 5:08 PM, George Dinwiddie wrote:
>
>> Matt,
>>
>> On 1/29/14, 12:15 AM, Matt Hauck wrote:
>>> I've run into this problem in the past and have run into it again so I
>>> thought I'd ask the group and see if anyone has a better solution for
>>> this issue than mine.
>>>
>>> So, I'm (again) using cucumber to spec a publicly facing backend REST
>>> API. I quite like how I am able to specify the exact inputs and outputs
>>> of the API in full text so that consumers of the API can see exactly
>>> what needs to be input and what exactly they can expect as output.
>>>
>>> Anyhoo, part of this is that I want to be able to explicitly list out
>>> the URL inside the step; something like:
>>>
>>> When I send a POST to the URL "/foo-resource/12"
>>
>> When you talk with the business people, do they *really* ask for
>> software in terms of POSTs and URLs? Or do they have some other language
>> that they use to describe what they want to accomplish?
>>
>> - George
>>
> I am using cucumber to communicate the API usage to our QA, not exactly
> to the business people.

What is your role in this? And what will this person named "QA" do with it?

- George

[Roberto Lo Giacco]

On Thu, Jan 30, 2014 at 11:50 PM, Andrew Premdas <apre...@gmail.com> wrote:

On 30 January 2014 15:50, Roberto Lo Giacco <rlog...@gmail.com> wrote:

I would generally agree with you, but you still have to provide examples in order to be able to spec out your system. In this case the example might be expressed by using JSON or XML or any other data format which is clearly understood by the parties defining the specs.

On an end user product I would use terms like "Click on the start button", "Select the filename field" and so forth, but on a REST API you have a predefined terminology which is the base of the convention:

- resources and their identifiers (URI)

- HTTP protocol (methods, response codes, headers, etc...)

- content types

How would you spec out that when your REST service receives a request for /foo/123 with a Content-Type header value of XML the response must be the XML representation of resource foo with ID 123?

I wouldn't, not in a feature anyhow. Instead I would ask 

    Why would anyone want a foo?

Because they want to use that resource. Being an API, there might be many different reasons behind it. Let me explain: I have a temperature sensor on my Arduino, why do I want to get its value? Well you might say I want to know the room temperature, or the chip temperature or the board temperature... but I might be interested in getting a soil temperature to adjust accordingly the soil moisture sensor readings. What matters with regards to the API is I want to get the temperature sensor reading.

This is different when I say the user is able to push the leftmost button to move to the previous menu item: that is a UI, which is linked to an objective, something you give to the user as an option, within a limited set of choices.

I'm not trying to be dogmatic here, I'm just trying to reinforce my feeling that an API shouldn't limit it's possible usages "by design".

 

    Who, or what, would want a foo?

That's interesting: what if I answer as "the system implementing the API"? I mean, unless you want to differentiate your resource on a per authentication profile (I wouldn't do that as it prevents any caching system to step in and help your back end, I would rather provide different resources for different profiles) then it doesn't really matter who is requesting the resource.

 

    How would I know what foo to get? (this touches on the idea of navigability of the service)

Yes, this is something I would agree, and this is part of the API definition and what I believe it needs to be spec out, but by using examples....

 

    What is required to get a foo?

       id?

       authentication?

   

Correct, I agree with that. Where I don't agree is on how you would write that....

 

    If I require something, say an id, how do I get it, how do I know I've got the right one, how do I stop people using the wrong one,     what happens if someone uses the wrong one, why do I need an id, etc. etc.

I feel this is the sort of information that should be captured and expressed in features. 

Again, we agree on this, but how would you describe these with examples?

 

If some of these questions were answered, then it would be possible to give foo context and meaning, and perhaps produce example scenarios that were more meaningful. 

Once you are 'stuck' in the implementation details of the service, it is really hard to see another way, particularly one that is so much more abstract; so I don't expect people to get this, hence the 'controversy'

Here is where we strongly disagree, I believe because we have different points of view on what is "implementation details".

When we say GET/POST/PUT/DELETE for a web application that is an implementation detail (99% of the times) because the user is not submitting such request, he is clicking on a button, typing something or interacting with visual objects somehow and the HTTP method used to communicate that UI information to the back-end is an implementation detail... even the existence of the back-end is an implementation detail!

When you are interacting with a REST API, using a GET or a POST method *is not* anymore an implementation detail, that's one of the core distinguishing elements of REST concept! 

The same *does not* applies to the URL you are requesting if your REST API implements a RMM level 3, but if it's a level 2 the previous might not be true.

I believe you have to follow the usual rule of thumb: feature files should explicitly describe WHAT the api will do, not HOW it is going to be done.

Why don't we try to put up an example on this?

Suggestion: a REST API to calculate the distance between two locations capable of providing the response in XML or JSON using miles or kilometers.

Let's use a feature file to describe this.... How would you do that?

I would use:

Given a resource foo with id 123 exists with the following data

   | attribute name | attribute value |

   ...

When the system receives a GET request for resource foo with id 123

And the request content type is "application/xml"

Then the response code is 200 OK

And the response payload is

   ```<xml here>```

I wouldn't use a different terminology to abstract something like that in _this_ case, but it obviously depends on the parties involved on the spec definition.

--

Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

--
Posting rules: http://cukes.info/posting-rules.html
---

You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/yaAGaJ45XYU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.

[Andrew Premdas]

On 31 January 2014 11:09, Roberto Lo Giacco <rlog...@gmail.com> wrote:

On Thu, Jan 30, 2014 at 11:50 PM, Andrew Premdas <apre...@gmail.com> wrote:

On 30 January 2014 15:50, Roberto Lo Giacco <rlog...@gmail.com> wrote:

I would generally agree with you, but you still have to provide examples in order to be able to spec out your system. In this case the example might be expressed by using JSON or XML or any other data format which is clearly understood by the parties defining the specs.

On an end user product I would use terms like "Click on the start button", "Select the filename field" and so forth, but on a REST API you have a predefined terminology which is the base of the convention:

- resources and their identifiers (URI)

- HTTP protocol (methods, response codes, headers, etc...)

- content types

How would you spec out that when your REST service receives a request for /foo/123 with a Content-Type header value of XML the response must be the XML representation of resource foo with ID 123?

I wouldn't, not in a feature anyhow. Instead I would ask 

    Why would anyone want a foo?

Because they want to use that resource. Being an API, there might be many different reasons behind it. Let me explain: I have a temperature sensor on my Arduino, why do I want to get its value? Well you might say I want to know the room temperature, or the chip temperature or the board temperature... but I might be interested in getting a soil temperature to adjust accordingly the soil moisture sensor readings. What matters with regards to the API is I want to get the temperature sensor reading.

What matters (in the context of the feature) is that you want to get the temperature. How you get it is an implementation detail. So you could say

Given the temperature is 0C

When I get the temperature

Then the temperature should be 0C

Now please don't think this means that I'm saying you shouldn't have a spec somewhere for say TemperatureEnquirer, that talks about GET's and JSON. I'm just saying that in features you can (and IMO should) talk at a higher level 

This is different when I say the user is able to push the leftmost button to move to the previous menu item: that is a UI, which is linked to an objective, something you give to the user as an option, within a limited set of choices.

There is absolutely no difference, because I would never write a feature that talked about a user push the left most button ..., instead I would say

Given the temperature is 0C

When I get the temperature

Then the temperature should be 0C

Of course we need context about why you would get temperature, which is different, but that can be expressed in things like the path of the feature, its name and its coda

e.g.

api-features

    gettemp

 ui-features 

    gettemp

     

I'm not trying to be dogmatic here, I'm just trying to reinforce my feeling that an API shouldn't limit it's possible usages "by design".

 

    Who, or what, would want a foo?

That's interesting: what if I answer as "the system implementing the API"? I mean, unless you want to differentiate your resource on a per authentication profile (I wouldn't do that as it prevents any caching system to step in and help your back end, I would rather provide different resources for different profiles) then it doesn't really matter who is requesting the resource.

It does matter why you are building the api. One of the most productive ways to use Cucumber is to write a feature and decide not to build anything, because you can't find a good justification for the feature. You seem to have already decided what you are building before you have written the feature.

Your thinking:

Lets build a temperature api

Now lets write a feature to test it.

My thinking:

Lets write a feature to see if we should build a temperature api

 

    How would I know what foo to get? (this touches on the idea of navigability of the service)

Yes, this is something I would agree, and this is part of the API definition and what I believe it needs to be spec out, but by using examples....

 

    What is required to get a foo?

       id?

       authentication?

   

Correct, I agree with that. Where I don't agree is on how you would write that....

 

    If I require something, say an id, how do I get it, how do I know I've got the right one, how do I stop people using the wrong one,     what happens if someone uses the wrong one, why do I need an id, etc. etc.

I feel this is the sort of information that should be captured and expressed in features. 

Again, we agree on this, but how would you describe these with examples?

 

Feature: Provide temps for registered users

  As a user 

  I'm willing to pay extra for temperature information

  Because it allows me to save money on my heating costs

  Given I'm registered

  When I ask for temps 

  Then I should get temps

  Given I'm not registered

  When I ask for temps

  Then I should be given the opportunity to register

Now you can have a long hard think about how you would give an opportunity to register using the API.

  

If some of these questions were answered, then it would be possible to give foo context and meaning, and perhaps produce example scenarios that were more meaningful. 

Once you are 'stuck' in the implementation details of the service, it is really hard to see another way, particularly one that is so much more abstract; so I don't expect people to get this, hence the 'controversy'

Here is where we strongly disagree, I believe because we have different points of view on what is "implementation details".

When we say GET/POST/PUT/DELETE for a web application that is an implementation detail (99% of the times) because the user is not submitting such request, he is clicking on a button, typing something or interacting with visual objects somehow and the HTTP method used to communicate that UI information to the back-end is an implementation detail... even the existence of the back-end is an implementation detail!

When you are interacting with a REST API, using a GET or a POST method *is not* anymore an implementation detail, that's one of the core distinguishing elements of REST concept! 

The whole point of REST is to build API's that work in the same way as the web. If I want to read my mail in my browser I send a GET request to mail.google.com, if I want to read my mail via gmail's api I send a GET request to mail.google.com 

The same *does not* applies to the URL you are requesting if your REST API implements a RMM level 3, but if it's a level 2 the previous might not be true.

I believe you have to follow the usual rule of thumb: feature files should explicitly describe WHAT the api will do, not HOW it is going to be done.

Why don't we try to put up an example on this?

Suggestion: a REST API to calculate the distance between two locations capable of providing the response in XML or JSON using miles or kilometers.

Let's use a feature file to describe this.... How would you do that?

Why do you want this API. Again you are writing features (actually scenarios) to test the functionality of things you have already decided to develop. This is not Behaviour Driven Development, (well at least not BDD using Cucumber).  Instead try writing a feature that explores why you should build the distance calculator. You might find this is surprisingly difficult. In my experience writing scenarios is easy, the hard part of writing features is the coda. 

I suspect most cukers spend less than 5 minutes writing the coda of a feature, IMO if you haven't spent at least 15 minutes, and popped the why stack at least a couple of times then you aren't even close to doing BDD - and I have to say I'm guilty of this far too often.

Alot of this goes back to how the OP (Matt) is using Cucumber. He is using it to: 

    'communicate the API usage to our QA'  (Matt's words)

I think its interesting how this communication seems to be one way. IMO rspec is more suitable for this task.

If he was communicating with the business (bi-directional) then the language I'm suggesting is more appropriate. This would be closer to using Cucumber for its designed purpose as a tool for BDD.

Finally what I'm trying to do is show an alternative way to work with Cucumber and REST. I don't need to prove that my way is `better`.  I'm just providing a small contribution to open-source by presenting an alternative people can think about. If my language comes across as saying this is the ONE true way you should use Cucumber then please accept my apologies.

All best

Andrew

[Timo Lehto]

Great points everyone! I was first sceptical, but now I'm convinced that using traditional BDD/Cucumber techniques for API testing can make sense. I think the controversy around the subject is due to different kinds of uses for Cucumber. Normally and usually you do use it as it is intended, a tool for BDD development process. However, with APIs I think there is another possible use case (these use cases are not necessarily mutually exclusive).

As a programmer who wishes to use an API I'm not very satisfied with information given on this level of abstraction:

  Given there's a user

  When I ask for a user

  Then I get a user

So the other use is to generate documentation that can not rot, but is always up-to-date. Some say that Cucumber is not suitable for this, but I think it suits it much better than rspec for instance. Assuming you do not know who are going to use your API and with what kind of tools. Java, JavaScript and PHP coders might not be too fond of specifications written in Ruby. Cucumber allows me to use as natural language as possible so that I can control exactly which important details I want, but does not concern it self with the any details of how to actually build the query. Whether your using JavaScript or curl, is an incidental detail, but whether you are GETting or POSTing might result in a very different behaviour.

So all in all it's a different thing to use Cucumber as BDD tool to shape the project and test it. What about using it as a tool for writing testable documentation of how an API works? I am thinking that you could do both and perhaps the documentation should not concern its self with testing at all. Instead I'd write a bunch of features like this that do not actually assert anything other than that the fact that the API responds, but use the "Embed a substitution variable in the definition" hack to generate informative test reports. Do you consider it just insane to do something like this? Anybody have any better ideas or tools for the job?

In case some of you may not be aware of how the hack works and what the hell am I even talking about, allow me to illustrate it for you:

https://gist.github.com/anonymous/8743709

[Jeff Nyman]

On Friday, January 31, 2014 3:47:23 PM UTC-6, Timo Lehto wrote:

In case some of you may not be aware of how the hack works and what the hell am I even talking about, allow me to illustrate it for you:

https://gist.github.com/anonymous/8743709

For what it's worth, I think this is a great example. A lot of the above discussion shows (to me) the holes that the Cucumber community -- some of it anyway -- tries to pigeonhole itself into by insisting how features "must" be. In fact, the very notion of a "feature file" itself to me is mistake. If anything, it's a specification. (That's why I argued awhile back for the inclusion of Ability to be used as a synonym for Feature. I was very glad when that was accepted in to Gherkin.)

Like yourself, I often use specifications (via Gherkin) to describe details of how something works. At my current role, we just did a major project for an API wherein we were testing scenarios that dealt with IPv4 vs IPv6. Had I gone with the "you're cuking it wrong" crowd, our specifications -- not features! -- would have been so divorced from the details people wanted to see that they would have been not as useful. In our case, our business customers were our NOC (Network Operations Center) and our IT group. So we specified at the level of intent that they felt was useful, which just so happened to include implementation details. That ended up serving as requirements-as-examples, tests, and documentation. It also ended up serving as training material for apprentices.

The fact that tools like Cucumber can make variations like this possible, to handle the needs of different consumers, should be promoted as a strength. Instead you see the Cucumber community -- again, some of it -- constantly trying to tell people how they are "doing it wrong."

So, again, I think tools like Cucumber (and the slate of BDD tools that follow its approach) should be seen as a tool that can be molded to your needs, based on the fact that communication is something that can similarly be molded.

[Matt Wynne]

On 29 Jan 2014, at 05:15, Matt Hauck <matt...@gmail.com> wrote:

I've run into this problem in the past and have run into it again so I thought I'd ask the group and see if anyone has a better solution for this issue than mine.

So, I'm (again) using cucumber to spec a publicly facing backend REST API. I quite like how I am able to specify the exact inputs and outputs of the API in full text so that consumers of the API can see exactly what needs to be input and what exactly they can expect as output. 

Anyhoo, part of this is that I want to be able to explicitly list out the URL inside the step; something like:

When I send a POST to the URL "/foo-resource/12"

The difficulty with this is that I do not know the value of "12" at the time of writing the steps because that gets generated randomly by the database, and it could obviously change every time the test is run, depending on what order the tests are run in. (To make matters worse perhaps, our code doesn't even use the id value, it uses a java.util.UUID.randomUUID()). The point is, it is a value different each time the test is run, but I still want it to show up inside the text of the scenario definition.

I have come up with the following two ways to solve this in times past doing something similar with cucumber, but neither are entirely satisfying:

1) Embed a substitution variable in the definition

Use something like "/foo-resource/:foo-id" in the scenario definition, and then do a string replacement in the step definition that swaps out ":foo-id" for the actual foo-id that was captured at some other step definition when the foo-resource was actually created. yuck. this is hard to follow and it just isn't a clean way of doing it.

If you want to document the fact that the ID is in the URL, then I think this is the way to do it. I don't think it's yucky :)

2) Embed what the id should be in the creation step, and make the change in the database

In the step that creates the foo-resource, say something like: `When I create a foo-resource (with id "12")`, which follows the normal way of creating a foo-resource, and then sneaks into the database and updates the appropriate value to the expected id value. The problem with this is it requires backend manipulation doesn't follow the normal way of doing things and so is a bit funky.

It also ends up with quite a bit of clutter in your scenario. The fact that you want an ID of 12 is irrelevant detail - it's just there so you can build an URL later. I'd go with the other option.

This seems like a general enough problem that I'm hoping the community has a better answer than mine. Thanks for any help you may offer!

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

cheers,

Matt

--

http://mattwynne.net

email: ma...@mattwynne.net

twitter: @mattwynne

skype: mattwynne
google hangouts: matt....@gmail.com

tel: +447974430184

[Matt Wynne]

Good point Jeff.

There's a fine line here: sometimes people are asking for rope to hang themselves with, and it's nice of the people on the list to help keep them safe by warning them what they're getting into. However we need to bear in mind that old adage that there are no best practices, and remain curious and open for contexts where our existing experience doesn't fit.