Should failing acceptance tests be checked in?

[vishy]

So, we write acceptance first. Now, it will take some time to make the
test pass. So, how to treat them w.r.t to source control? Should we
check-in the acceptance test? If yes, should Continous Integration run
them?

[Steve Freeman]

Check them in, but either into a separate place or marked up so that they're not part of the main build. Then you move them into the build when they start passing.

S.

[Gishu Pillai]

+1. Check them into source control. This ensures
- your changes are not lost
- developers can additionally run your tests before they check in the corresponding implementation.

I'd recommend tagging them with an [Ignore("Implementation pending")] so that they do not cause CI builds to fail and are visible (yellow) too. The yellows should turn to green before each iteration ends.

[Graeme Foster]

If you're using a source control system that makes branching fast and easy, then they just go straight into your feature branch until the feature branch is ready to be merged back into the mainline, at which point the tests should all be passing.
--
Graeme Foster

[vishy]

thanks for the suggestions

[Colin Vipurs]

+1 for checking in.

Not sure what tool you use, but the one we roll with (JBehave) has the
notion of "pending" tests, which do not fail the build, so these can
quite happily sit in trunk. They act as a nice marker for work
outstanding as well.

--
Maybe she awoke to see the roommate's boyfriend swinging from the
chandelier wearing a boar's head.

Something which you, I, and everyone else would call "Tuesday", of course.

[Chris Parsons]

On 20 May 2011 10:34, Colin Vipurs <zodi...@gmail.com> wrote:
> +1 for checking in.
>
> Not sure what tool you use, but the one we roll with (JBehave) has the
> notion of "pending" tests, which do not fail the build, so these can
> quite happily sit in trunk.  They act as a nice marker for work
> outstanding as well.

In cucumber, you can tag the feature with "@wip" which excludes by
default from a normal build. I've used this before and found it quite
useful. I tried only to check in pending tests (with a suitable
comment if possible), not failing tests.

Chris

> On Fri, May 20, 2011 at 9:43 AM, vishy <vishal...@gmail.com> wrote:
>> thanks for the suggestions
>>
>> On May 20, 1:33 pm, Graeme Foster <grae...@gmail.com> wrote:
>>> If you're using a source control system that makes branching fast and easy,
>>> then they just go straight into your feature branch until the feature branch
>>> is ready to be merged back into the mainline, at which point the tests
>>> should all be passing.
>>> --
>>> Graeme Foster
>
>
>
> --
> Maybe she awoke to see the roommate's boyfriend swinging from the
> chandelier wearing a boar's head.
>
> Something which you, I, and everyone else would call "Tuesday", of course.
>

--
Chris Parsons
chr...@rsons.org
http://pa.rsons.org
http://twitter.com/chrismdp
http://linkedin.com/in/chrisparsons

--
Chris Parsons
chr...@rsons.org
http://pa.rsons.org
http://twitter.com/chrismdp
http://linkedin.com/in/chrisparsons

[J. B. Rainsberger]

I use these rules, when checking in code:

1. Customer/acceptance tests don't have to pass 100%, because they
describe progress, not correctness.
2. Programmer/micro/unit/design tests must pass 100%, because they
help us avoid making mistakes.

I usually end up separating my tests into three major categories:

1. Customer tests
2. Fast programmer tests
3. Slow programmer tests (system tests, performance tests, and so on)

I build each category separately. In Java terms, they're separate
source folders with separate target folders.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca ::
http://blog.thecodewhisperer.com
Diaspar Software Services :: http://www.diasparsoftware.com
Author, JUnit Recipes
2005 Gordon Pask Award for contribution to Agile practice :: Agile
2010: Learn. Practice. Explore.

[tdenley]

Definitely check them in.

We use https://github.com/ttsui/pending in our codebase. This works
like the @wip that Chris mentions, but has the added bonus of still
running your test and failing it if it starts passing, reminding you
to remove the annotation

On May 20, 1:27 pm, Chris Parsons <chri...@rsons.org> wrote:

> chri...@rsons.orghttp://pa.rsons.orghttp://twitter.com/chrismdphttp://linkedin.com/in/chrisparsons
>
> --
> Chris Parsons
> chri...@rsons.orghttp://pa.rsons.orghttp://twitter.com/chrismdphttp://linkedin.com/in/chrisparsons

[Matt Wynne]

On 22 May 2011, at 08:58, tdenley wrote:

> Definitely check them in.
>
> We use https://github.com/ttsui/pending in our codebase. This works
> like the @wip that Chris mentions, but has the added bonus of still
> running your test and failing it if it starts passing, reminding you
> to remove the annotation

Yeah, Cucumber does a similar thing if you pass it the --wip switch.

cheers,
Matt

--
Freelance programmer & coach
Founder, http://relishapp.com
+44(0)7974430184 | http://twitter.com/mattwynne

[Steven Smith]

Some acceptance test and BDD frameworks support the notion of "not
implemented" which is different from "broken". You should certainly
check these in, but they should not break the build until they are
implemented (at which point they either succeed or fail).

Steve

--
Steve Smith
http://SteveSmithBlog.com/
http://twitter.com/ardalis

[Andrew Premdas]

There are two operations here, putting the code under revision control, and publishing the code to your team. I don't think you should publish broken tests of any sort ever, its very disruptive and frankly rather rude. I do think you should put every thing under revision control ASAP (if I'm being good I do this every 5 minutes or so). If your source control practices don't support these separate operations then you should change them e.g. use Git in front of SVN

All best

Andrew

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Nat Pryce]

I wrote something similar for Python's Nose framework:
https://gist.github.com/997195

--Nat

--
http://www.natpryce.com

[Uberto Barbini]

I'm wondering if a kind of @wip annotation would be useful for Junit as well.
I don't want to break the green/red circle but let's say you wrote a
test that cannot possibly work for a while (until you finish you other
code).
Now your options are 3 basically: you can put on @ignore on it,
commenting out the assertions or just delete the test and put it as
todo on your task list.

I know also in the book there are examples like that, but after have
tried @wip in cucumber I found it most rewarding.
Maybe it's also possible to implement it with @Rules in Junit, I
haven't look at it yet.

Any thoughts?

Uberto

[tdenley]

This has been done -- see: https://github.com/ttsui/pending

On May 29, 8:37 am, Uberto Barbini <ube...@ubiland.net> wrote:
> I'm wondering if a kind of @wip annotation would be useful for Junit as well.
> I don't want to break the green/red circle but let's say you wrote a
> test that cannot possibly work for a while (until you finish you other
> code).
> Now your options are 3 basically:  you can put on @ignore on it,
> commenting out the assertions or just delete the test and put it as
> todo on your task list.
>
> I know also in the book there are examples like that, but after have
> tried @wip in cucumber I found it most rewarding.
> Maybe it's also possible to implement it with @Rules in Junit, I
> haven't look at it yet.
>
> Any thoughts?
>
> Uberto
>
>
>
>
>
>
>
> On Mon, May 23, 2011 at 4:50 PM, Steven Smith <ssmith.li...@gmail.com> wrote:
> > Some acceptance test and BDD frameworks support the notion of "not
> > implemented" which is different from "broken".  You should certainly
> > check these in, but they should not break the build until they are
> > implemented (at which point they either succeed or fail).
>
> > Steve
>

[J. B. Rainsberger]

Uberto Barbini wrote:

> I'm wondering if a kind of @wip annotation would be useful for Junit as well.
> I don't want to break the green/red circle but let's say you wrote a
> test that cannot possibly work for a while (until you finish you other
> code).
> Now your options are 3 basically: you can put on @ignore on it,
> commenting out the assertions or just delete the test and put it as
> todo on your task list.
>
> I know also in the book there are examples like that, but after have
> tried @wip in cucumber I found it most rewarding.
> Maybe it's also possible to implement it with @Rules in Junit, I
> haven't look at it yet.
>
> Any thoughts?

I've started using the @Ignore tag in order to be able to write my "to
do" list directly in with my tests. I don't mind it, although I prefer
to have my to-dos on paper, where I can more easily throw it away when I
change my mind.
--
J. B. Rainsberger :: http://www.jbrains.ca ::
http://www.thecodewhisperer.com

[Steve Freeman]

don't forget you can add a message to @ignore. And you can tune your IDE to list your own todo markup. 

S 

Steve Freeman

http://www.m3p.co.uk/blog

Written on a phone, so please allow for typos and short content.  

On 29 May 2011, at 15:29, "J. B. Rainsberger" <jbrain...@gmail.com> wrote:

[Uberto Barbini]

well I didn't know you can add a message to your todos, that's nice.

Anyway the nice part of @wip is that it actually runs your tests and
raise an exception in case they where failing (to remember you to
remove @wip)

I hope they'll include in junit then

Uberto

[Steve Freeman]

On 29 May 2011, at 21:57, Uberto Barbini wrote:
> Anyway the nice part of @wip is that it actually runs your tests and
> raise an exception in case they where failing (to remember you to
> remove @wip)

good point.

S.