BDD tool for Catch

[pepr]

I have started with implementation of a BDD tool for Catch. I will use it for my job. The first implementation language will be Python 3. The license will be the same as the Catch have -- the Boost one.

(To clarify my experience, I am very new to both Catch and to BDD. Anyway, I am seriously interested because of our new C++ project, and I like it.)

The idea is to have a features/ and testscatch/ or the like at the level of your src/ directory. If only the features/ exists, the testscatch/ will be created and filled with the generated Catch skeletons. If only the testscatch/ exist (i.e. also for older projects), the features/ files will be generated. If both exist, the synchronization happens.

I plan to base the xxx.feature files both on a non-programmer view (user story, scenarios, given when, then) but also on a programmer view (feature, test, sections with nesting) with possibility to mix the styles as you wish. Anyway, the xxx.feature files will not contain any code -- only the identification strings, possibly [tags], and some way to express nesting.

Would you be interested in such a tool?

[Mateusz Łoskot]

On 10 January 2014 08:15, pepr <pri...@atlas.cz> wrote:
> I have started with implementation of a BDD tool for Catch. I will use it
> for my job. The first implementation language will be Python 3. The license
> will be the same as the Catch have -- the Boost one.
>
> (To clarify my experience, I am very new to both Catch and to BDD. Anyway, I
> am seriously interested because of our new C++ project, and I like it.)
>
> The idea is to have a features/ and testscatch/ or the like at the level of
> your src/ directory. If only the features/ exists, the testscatch/ will be
> created and filled with the generated Catch skeletons. If only the
> testscatch/ exist (i.e. also for older projects), the features/ files will
> be generated. If both exist, the synchronization happens.

> [...]

I can't grasp what is the actual idea, what is the leading feature here,
what problem are you trying to solve.

So,

As a "BDD tool for Catch" user,
I want...<missing part from your post>
In order to... <missing part from your post>

Best regards,
--
Mateusz Łoskot, http://mateusz.loskot.net

[pepr]

On Friday, January 10, 2014 11:09:05 AM UTC+1, Mateusz Łoskot wrote:

On 10 January 2014 08:15, pepr <pri...@atlas.cz> wrote:
> [...]

I can't grasp what is the actual idea, what is the leading feature here,
what problem are you trying to solve.

Actually, I am asking...

> Would you be interested in such a tool?

It could be the case that there are already such tools for Catch and I did not find them.
I have found Cucumber and several articles that share user experience. It seems to me
that the tools are not perfect for Catch.

Consider this a start of collecting ideas, test cases, etc. Once I have something
workable, I will share the code via Github.

Petr

[Mateusz Łoskot]

On 10 January 2014 12:18, pepr <pri...@atlas.cz> wrote:
> On Friday, January 10, 2014 11:09:05 AM UTC+1, Mateusz Łoskot wrote:
>> On 10 January 2014 08:15, pepr <pri...@atlas.cz> wrote:
>
>> Would you be interested in such a tool?
>
> It could be the case that there are already such tools for Catch and I did
> not find them.
> I have found Cucumber and several articles that share user experience. It
> seems to me that the tools are not perfect for Catch.

You've mentioned Python.
Now, you say Cucumber.

This makes me think: http://pythonhosted.org/behave/

[Martin Moene]

> Would you be interested in such a tool?

Before we can answer the question about interest, we must first understand what your tool will enable one to do and why one would want to do that.

cheers,
Martin

[Phil Nash]

I'm not 100% clear what the proposed tool would do either, but I have some ideas and comments anyway:

First: Catch has a modicum of BDD support built in in the form of the SCENARIO, GIVEN, WHEN & THEN macros (as well as AND_WHEN and AND_THEN). These allow you to embed the stories in your test code in a way that maps onto the structure of the test code itself - and when you report on the runs the test headers read like a free form user story.

It appears what you're talking about is the higher level counterpart to that. AFAICS you can take that one of two ways: Either go "full Cucumber" and write a NLP that goes form the free-form stories to bits of native code that correspond to it. I've considered that approach and don't think it's a good fit for a C++-native test framework. That said if that's what you are talking about I'd be very interested in the outcome (and would actively support it, as an external tool).

The other possibility, which I think is more feasible - and appears to be what you're actually proposing, is that you maintain the free-form user stories as text files separately from the test code - but allow generation of test code skeleton (using the macros I mentioned earlier) from the text files, or of the text files from existing test code. I've considered this before to and had it in the back of my mind for a while. I would be very supportive of such a tool.

Such code generation in both directions would useful, natural, and not too big of a project. Where I think your proposal goes too far is in suggesting you should be able to keep them in sync if you maintain them independently. I think that would only be manageable in the "full Cucumber" version.

My suggestion there would be to allow freeform scenario files to be generated out of the code into different directories where they can be diff'ed with existing scenarios and any merges applied manually. I'd rule out merging regenerated skeleton code with existing test code.

Am I capturing that all correctly? Does my take make sense?

[pepr]

> [...] to maintain the free-form user stories as text files separately from the test code - but allow generation of test code skeleton (using the macros I mentioned earlier) from the text files, or of the text files from existing test code. I've considered this before to and had it in the back of my mind for a while. I would be very supportive of such a tool.

Yes, this is the first goal.

> Such code generation in both directions would useful, natural, and not too big of a project.

Yes, this is what I was thinking about.

> Where I think your proposal goes too far is in suggesting you should be able to keep them in sync if you maintain them independently.

My goal is at least to report the differences. It will be more apparent later whether the automatic synchronization is reasonable.

> I think that would only be manageable in the "full Cucumber" version.

Frankly, I do not know Ruby that much to be fluent in it. As you said, I prefer simple enough tools if they need not to be complex. I did not study Cucumber very much, but it seems to implement another "programming language" used in the text description. In my opinion, things should sometimes be more restrictive to be useful.

OK, I will be back when I have something runnable.

[pepr]

I need your opinion...

I am going to publish the t2f.py script that takes Catch tests, does lexical and syntactic parsing,
and produces the corresponding *.feature_new files. Actually, this is similar to what Catch produces
when processing the input:

SCENARIO( "name for scenario", "[optional tags]" ) {
    GIVEN( "some initial state" ) {
        // set up initial state

        WHEN( "an operation is performed" ) {
            // perform operation

            THEN( "we arrive at some expected state" ) {
                // assert expected state
            }
        }
    }
}

The Catch can display (see http://www.levelofindirection.com/journal/2013/6/28/catch-10.html):

-------------------------------------------------------------------------------
Scenario: name for scenario
     Given: some initial state
      When: an operation is performed
      Then: we arrive at some expected state
-------------------------------------------------------------------------------

The code I have now produces

scenario: name for scenario  [optional tags]
  given: some initial state
  when: an operation is performed
  then: we arrive at some expected state

Notice the [optional tags] in the output. I want the .feature files be as human readable as possible without anything that resembles programming. On the other hand, are the tags usefull for non-programmers? Would they be confusing or usefull. Or would it be better to use the form:

scenario: name for scenario
   tags: [optional tags]
   given: some initial state
   when: an operation is performed
   then: we arrive at some expected state

What is your opinion?

[pepr]

Please, have a look at https://github.com/pepr/BDDtool that implements `f2c.py` script for conversion of  `feature/*.feature` files to the `catch/*.catch` skeletons. See the `README.md` at the GitHub.

For those more deeply interested, have a look at `pyunittests/` for what inputs are tested for conversion to syntax trees. I am open to any suggestion.

The tool (probably named `c2f.py`) for the reverse conversion is to be published soon.

[pepr]

Hi Phil,

The f2c.py script from https://github.com/pepr/BDDtool does the transformation of the ./features/*.feature files to ./tests/*.h files for Catch if the file does not exist, yet. If the .h file does exist, the ./tests/*.skeleton files are generated (for diff comparison, if needed).

I need some feedback for further development. Please, if you find some time, try it.


On Friday, January 10, 2014 6:17:28 PM UTC+1, Phil Nash wrote:

[...] The other possibility, which I think is more feasible - and appears to be what you're actually proposing, is that you maintain the free-form user stories as text files separately from the test code - but allow generation of test code skeleton (using the macros I mentioned earlier) from the text files,

 
This should already work -- see the f2c.py

or of the text files from existing test code. I've considered this before to and had it in the back of my mind for a while. I would be very supportive of such a tool.

There is the c2f.py script for the reversed process. However, it fails with more complex Catch header files.
 

Such code generation in both directions would useful, natural, and not too big of a project. Where I think your proposal goes too far is in suggesting you should be able to keep them in sync if you maintain them independently. I think that would only be manageable in the "full Cucumber" version.

Both f2c.py and c2f.py do use the other py scripts for lexical and syntactic parsing. The result of the syntactic parser is a syntax tree that should be the same in both cases. (c2f.py already works for skeletons, but not for Catch sources with more code inside). When c2.py works, the syntax trees can be compared to detect whether the .feature file and the .h file are related.

The f2c.py is capable to consume also the "and" parts of the BDD scenario description. My observation is that there probably is a need for the AND_GIVEN alias of the SECTION. Also, the BDD descriptions sometimes use the "but" concatenations of the sections (in the sense of AND, i.e. nesting in Catch). There probably should also be BUT_XXXX aliases for the section. The reason is that it enhances the Catch identifier readability if the original .feature text started like "but some condition hodls...". The f2c.py transforms the missing AND_GIVEN to GIVEN, and the BUT_WHEN to AND_WHEN. However, in such case the process cannot be reversed -- the syntax tree changes as it is not possible to get the lost BUT information from the generated Catch sources.

The scripts are still very raw; anyway, I am already using them for my project. I have to confirm that writing the .feature files first is rather nice way to think about the final solution. Easy to change drastically because no "precious programming hours" were spent to discover problems later.

Have a nice time,
    Petr