Feature proposal: step data

[Jan Ahrens]

Hello *,

I'm currently thinking about a new Spock feature and want to know about
your opinion on it. It's a long post, because I added some code
examples. Any feedback is appreciated.

First what I'm trying to do: I want to write a Spock extension that
pretty prints the steps of each feature method to stdout, so that you
know what happens inside your feature methods.

For that purpose I'm using the FeatureInfo object that gets passed to my
IRunListener implementation and print the "blocks" field.

This works already pretty well, but there are cases, when the state
described in the blocks description string needs to be detailed further.

For that purpose I want to introduce the concept of "step-data" into Spock.

Here is an example, to show you what I want to do:

Spock test:
---- SNIP ----
def "test the filter feature"() {
given: "I have a list of my customers"
def customers = [['john', 23], ['jeff', 42]]

when: "I get the customers that have more than 30 credits"
def filtered = customers.grep { name, credits -> credits >= 30 }

then: "the list should be as follows"
filtered == [['jeff', 42]]
}
---- SNAP ----

Output of the test run on stdout:
---- SNIP ----
Given I have a list of my customers
| name | credits |
| john | 23 |
| jeff | 42 |
When I get the customers that have more than 30 credits
Then the list should be as follows
| name | credits |
| jeff | 42 |
---- SNAP ----

To achieve the above output, Spock must know about the data that is used
in the steps.

Here is my proposed change: Each block has the possibility, to specify a
Map of step data, that is stored within the BlockMetadata annotation.
Each key of that map maybe used as a variable.

Example in Spock:
--- SNIP ---
given: "I have a list of my customers"
data: [allCustomers: [['john', 23], ['jeff', 42]]

when: "i get the customers that have more than 30 credits"
def filtered = allCustomers.grep { name, credits -> credits >= 30 }

then: "the list should be as follows";
data: [richCustomers: ['jeff', 42]]
filtered == richCustomers
--- SNAP ---

What do you think about it? Is this a feature that would sound useful to
you?

Maybe I can talk with some of you about that idea on the GR8Conf next
week in Copenhagen.

Cheers,
Jan

[Bartosz Majsak]

I was actually wondering about quite similar feature. In brief - would
it be possible to extend IRunListener to have something like
beforeStep / afterStep hooks as well? Is it a fundamental change in
the framework? Asking since I'm quite a newcomer in Spock area.

Cheers,
Bartek

[Peter Niederwieser]

On May 18, 11:10 pm, Bartosz Majsak <maj...@gmail.com> wrote:
> In brief - would
> it be possible to extend IRunListener to have something like
> beforeStep / afterStep hooks as well?

Adding such hooks should be relatively simple as long as you don't
expect to be able to interact with the step implementation (like
changing the value of a local variable).

I had a long chat with Jan today (the original author of this thread).
He is doing exciting stuff with Spock, and I hope he'll soon talk more
about it on this list. However, it's quite clear that he needs more
than just beforeStep/afterStep hooks.

Do you have a concrete use case in mind for beforeStep/afterStep? If
yes, I'd love to hear about it.

Cheers,
Peter

[Bartosz Majsak]

Actually I do have a concrete use case. Let's say we have functional
tests of a web app written in BDD manner. Under the hood we use
WebDriver to emulate user's interaction. Now if we take a look at spec
as some sort of user manual/product demo we can intercept descriptions
of each step and display them through webdriver js executor as text
boxes during test execution. So we could have automated screencasts
for our app. It might be a crazy idea, but I find it really cool :)

Cheers,
Bartek

[Peter Niederwieser]

It's possible to enhance IRunListener like this, but it will need some work on Spock's AST transform to weave in code that triggers the notification. That's because in Spock, a step is *not* a method (not even in the underlying implementation). It would be good to prototype this and see how well it works. Contributions are welcome.

Cheers,

Peter

Am Donnerstag, 15. November 2012 17:37:32 UTC+1 schrieb John Smart:

Hi All,

I have a very concrete application of this. I'm working on adding Spock integration to Thucydides (http://www.thucydides.info). Basically, Thucydides keeps track of what methods were invoked during a test, in what order, and so forth. One obvious way for this to work would be to extend the IRunListener class with methods like the following:

  beforeStep(stepInfo)

  afterStep(stepInfo)

where the StepInfo would include the type (given, when, then, expect...) and the associated text (if any). 

Thoughts?

John

[Luke Daley]

On 16/01/2013, at 4:12 PM, Georgi Danov <georgi...@jumio.com> wrote:

> I vote for this as we use Spock and would like to use the reporting of Thucydides, but unfortunately am not good enough to do it myself

How/why would you use this?

Who would look at the reports and when?

>
> Cheers,
> G
>
> On Tuesday, January 15, 2013 11:02:52 AM UTC+1, John Smart wrote:
> Is anyone willing/able to take a look at this? AST transforms are not my speciality, but once this feature works, I think I can get the Thucydides integration running pretty easily, and Thucydides reporting for Spock tests would be a very cool feature.
>
> Thoughts?
>
> John.

> --
> You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/spockframework/-/juLvik-mAQIJ.
> To post to this group, send email to spockfr...@googlegroups.com.
> To unsubscribe from this group, send email to spockframewor...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/spockframework?hl=en.

[John Smart]

Thucydides works by recording and reporting on test execution. You can see some sample reports here: http://wakaleo.com/thucydides-sample-reports. Thucydides documents how the tests play out, making the acceptance testing process more transparent (see http://wakaleo.com/thucydides-sample-reports/5d1fad6cd0eb6bd3cc5fdccba7c66f50af7add2bfe663ff21f852ac0cd5302c7.html and http://wakaleo.com/thucydides-sample-reports/7c745f38e84ae671a475ac47b9390d42ddb0c8984ec9068b3d8705c797d3dbea.html for some examples).

The reports are mainly for acceptance tests, which have a stronger focus on communicating with stakeholders, though with Spock integration they might be used more generally. One of the nice things about Spock is the way the labels such as "given", "when" and "then" structure the tests in a more readable BDD-style. Folks are asking to use the Thucydides reports for Spock so that this can appear in the acceptance test reports, and so that some of the acceptance tests can be implemented using Spock. 

I think I would be able to do this with the IRunListener class, if it had methods like the following: 

   beforeStep(stepInfo) 

   afterStep(stepInfo) 

   

where the StepInfo would include the type (given, when, then, expect...) and the associated text (if any). 

Cheers,

- John.

[Peter Niederwieser]

@John: Is it crucial for Thucydides to have passed/failed/skipped information for individual steps? It might take a while until I get to implement this. (I once spiked it, but didn't get far enough.) Maybe we can find someone else who wants to drive this?

By the way, is it possible to use Geb (http://gebish.org) with Thucydides, or does one have to use WebDriver directly?

@Georgi: What exactly are your needs? If you just need a good high-level report, the upcoming spock-report might be for you. See here to get an idea (might take some time to load): http://htmlpreview.github.com/?https://github.com/spockframework/spock/blob/groovy-1.8/spock-report/src/main/resources/demoReport.html. Of course this tackles a narrower problem than Thucydides.

For the record, Jan (the original author of this thread) went on to create http://pease.github.com. Unfortunately we couldn't integrate this as-is, but at least it made for an excellent bachelor (or was it master) thesis.

Cheers,

Peter

[John Smart]

Hi Peter,

I think I could do without the passed/fail/skipped info for individual steps - the most important information for the reporting is really the given/when/then labels and any associated text. 

I haven't looked at Geb integration yet - at the moment, Thucydides needs to access the webdriver instance, but Geb integration is on my to-investigate list.

[Peter Niederwieser]

On Jan 16, 2013, at 6:45 PM, John Smart <john....@wakaleo.com> wrote:

> I think I could do without the passed/fail/skipped info for individual steps - the most important information for the reporting is really the given/when/then labels and any associated text.

That information is already available, via FeatureInfo.getBlocks(). Is this enough to get you going?

Cheers,
Peter

[John Smart]

Thanks, I wasn't aware of that. Unfortunately, I checked, and it's not quite enough, as I need to know not only what the blocks are, but when they start and finish. I need to know this so that Thucydides can create a corresponding step in the reporting structure, and nest the subsequent calls underneath.

Cheers,

John.

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.

To post to this group, send email to spockfr...@googlegroups.com.
To unsubscribe from this group, send email to spockframewor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/spockframework?hl=en.

--
___________________________________________________
John Smart | Wakaleo Consulting  |  +61 407 247 642
Optimizing your software development process
http://www.wakaleo.com  |  john....@wakaleo.com

[Peter Niederwieser]

On Jan 16, 2013, at 8:00 PM, John Smart <john....@wakaleo.com> wrote:

Thanks, I wasn't aware of that. Unfortunately, I checked, and it's not quite enough, as I need to know not only what the blocks are, but when they start and finish. I need to know this so that Thucydides can create a corresponding step in the reporting structure, and nest the subsequent calls underneath.

Which subsequent calls are you referring to? Would it be possible to create all report steps when the method starts? Unfortunately, providing the steps' start and end times means solving more or less the same problem as providing their passed/failed/skipped results.

Does Thucydides report all of results/exceptions/standard output/standard error, and does it report all of them at the step level?

Cheers,

Peter

[John Smart]

In acceptance tests (and some integration tests), a test will call other reusable "steps" (methods) - e.g. "logonAs(), openClientAccount()...) which combine lower-level operations (using page objects, for example). Thucydides records when these methods are called, in what order they are called, and whether any errors occur when they are called. For unit testing, the tests are normally simpler so this sort of breakdown is more rare. Looking at the code, I might be able to "cheat": create the "given", "when", "then" groups initially, and then (effectively) record all of the call results inside the last block. If Spock is being used for "unit-style" acceptance tests, this would probably be acceptable, but it would loose a lot of reporting power if Spock was being used for more higher-level acceptance tests. I'll see if I can make any headway with this approach, though in the medium term more sophisticated support would allow for much more powerful reporting.

Cheers,

John.

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To post to this group, send email to spockfr...@googlegroups.com.
To unsubscribe from this group, send email to spockframewor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/spockframework?hl=en.