Micro Testing Framework

[Charlie Poole]

Hi All,

Some of you may know that I'm winding down my many-year association with the NUnit project so that new people can carry on. I'm looking for new things to do in the open source world, probably smaller things, more along the size of what one guy can do alone.

Something I have noticed over the years is how a piece of software can grow in size and complexity, taking on this and that function at the request of one or another person. That's certainly the case with NUnit.

I'm thinking of creating a test framework tailored for micro-tests and micro-tests alone. I'd leverage my experience with NUnit and do the first implementation in C# and I'd add some ideas I have not gotten to implement in NUnit as well. My thoughts about such a framework so far are...

* It should have everything you need for micro-tests

* It should have nothing that leads you away from microtests

* It should leverage features of each language in it's implementation, rather than substituting it's own conventions

* It should be self-contained, requiring no other runner to execute tests

I'm looking for folks who are willing to chime in with ideas and this seems like a great forum to start. If it generates too much traffic, we can take it elsewhere.

I'm particularly interested in the features that fall under the first two bullet points. What do you think is essential for a framework aimed at micro-tests? What do you think should definitely be excluded?

My thinking is to come up with a spec and a sample implementation in C#. Other folks might pitch in with other implementations. I imagine the scope of this to be small enough for one or two people to do in a few months of intense work or a year at a more laid-back pace.

Charlie

[Ron Jeffries]

Hi Charlie!

> On Aug 6, 2017, at 7:57 PM, Charlie Poole <cha...@nunit.org> wrote:
>
> * It should have everything you need for micro-tests
> * It should have nothing that leads you away from microtests

I'm certainly all for this, though I am at this moment quite without ideas. Can you think of an example of something in NUnit that leads away? (I suspect it's something I never used and don't even know about.)

Thanks!

R

[Charlie Poole]

Hi Ron,

Some examples could be...

 * File assertions

 * Ordering of tests

 * Shared state across tests

 * hierarchical setup in various forms

Some of these, of course, may also be used in ways that do no harm.

Charlie

R

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[J. B. Rainsberger]

On Sun, Aug 6, 2017 at 8:57 PM, Charlie Poole <cha...@nunit.org> wrote:

 

I'm thinking of creating a test framework tailored for micro-tests and micro-tests alone. I'd leverage my experience with NUnit and do the first implementation in C# and I'd add some ideas I have not gotten to implement in NUnit as well. My thoughts about such a framework so far are...
* It should have everything you need for micro-tests

* It should have nothing that leads you away from microtests

* It should leverage features of each language in it's implementation, rather than substituting it's own conventions

* It should be self-contained, requiring no other runner to execute tests

I'm looking for folks who are willing to chime in with ideas and this seems like a great forum to start. If it generates too much traffic, we can take it elsewhere.

I'm particularly interested in the features that fall under the first two bullet points. What do you think is essential for a framework aimed at micro-tests? What do you think should definitely be excluded?

Definitely include:

* data-oriented tests in the style of Spock/Fit over the Parameterized Test Case pattern

* direct support for Contract Tests: I can combine standard tests with my test subject factories to “generate” the same test run for multiple implementations of the same interface

Definitely exclude:

* arbitrary grouping of tests (“tags”, “categories”…) — if I want to have multiple groups of independently-run tests, I should work for it

* tearDown() (you should never need it!)

That’s just off the top of my head. I’m curious to see whether my subconscious thinks of anything else particularly salient.

It sounds like a good project!

-- 

J. B. (Joe) Rainsberger :: http://testdrivendevelopment.training :: http://www.jbrains.ca ::
http://blog.thecodewhisperer.com

[Michael Hill]

Can you use some more words, Joe?

what's a data-oriented test in the style of spock?

what's a contract test in the context you envision?

Virus-free. www.avast.com

--

[Jeff Langr]

Greetings Charlie--

I particularly appreciate that creating multiple test fixtures is available to me through NUnit (via nested classes), but JUnit (prior to 5) doesn't directly support that. Having to spread related tests across multiple source files is awkward. (Maybe there's solution that doesn't appear quite as cluttered/constrainted as nested classes--something that allows for a bit better contextual organization, and doesn't demand encoding all context in the name of each test.)

I agree with JB on teardown.

I'd also add: eliminate the old-school assertEquals, and offer only fluent assertions.
On a similar but more personal bent, also eliminate assertion failure messages.

I wonder if there's a way to encourage single behavior tests (i.e. tests that push asserts in the direction of 1-per-test).

This sort of interest always makes me wonder what a new, TDD-focused language might look like.

Regards,
Jeff

J. B. Rainsberger wrote:

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[Michael Hill]

Jeff...

I live in Java. As far as the assertions go, I also prefer "new style" to the old. To me, that means AssertJ. The extensibility here is just tremendous and very easy to do.

Is that what you're thinking of?

Hill

Virus-free. www.avast.com

J. B. Rainsberger wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[Jeff Langr]

Yeah, AssertJ is good, Hamcrest is good, etc. Eliminate the ability to do less-literary assertions.

Michael Hill wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[Michael Hill]

i am no friend of hamcrest. that was the one i was trying to remember its name. :)

Michael Hill wrote:

J. B. Rainsberger wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Langr]

Gotcha. Yeah I hear you. I suspect that is an implementation problem more than an intent problem?

--
Sent from Mail.Ru app for Android

Tuesday, 08 August 2017, 06:29PM -04:00 from Michael Hill GeePa...@geepawhill.org:

Michael Hill wrote:

J. B. Rainsberger wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[Allen Holub]

Hi Charlie,

My main issue with testing is durability. I want to be able to change class definitions, for example, without having to change the tests. I can help enforce that in Java by putting the test in a different package than the classes I'm testing --- at least that guarantees that my test won't access private or package-level stuff. Be nice if NUnit could optionally enforce something similar for C#.

-Allen

_______________________

Allen @ Holub .com
@allenholub
+1 (510) 859-3620  (Skype: allenholub)
http://holub.com
http://linkedin.com/in/allenholub

J. B. Rainsberger wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[James Grenning]

Hi Charlie

One thing that comes to mind is that any test should fail on the first behavior that is wrong. I spend my time in C and C++. I suggest to client that use GoogleTest to not use the macros that report errors but let the test case continue. That may be helpful for AT level tests but not TDD level unit tests. For micro-testing running on is not helpful.

James

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[James Grenning]

Why is teardown a problem? Is that because OO languages' object destruction should handle the cleanup?

--

[Michael Hill]

good point. tho we mean in the current test, right? i still want to run the rest of the  microtests, generally, i.e. the rest of the sequences that start with a setUp()?

Virus-free. www.avast.com

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[Michael Hill]

i wondered, too. OO certainly includes java, and t'aint no destructors there.

Virus-free. www.avast.com

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[James Grenning]

Fluent assertions? I suspect these are what you get when you gather and refactor assertions. Not that only one primitive thing is checked, but that logically one thing is checked.

[James Grenning]

Under the hood in garbage collection there is something that destroys objects.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[James Grenning]

yes, run all the tests but only continue in a test until something wrong happens.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[Michael Hill]

yes, notoriously and purposefully unforceable. when the gc finalizes, you finalize, and your finalization can be run. but you can't force the gc to finalize. in c++ terms, there's a auto-delete function called stochastically, but no direct delete.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Langr]

Hi James,

Some might call them matcher-based assertions? Dunno. The generic name I've heard most is "fluent assertions." in .Net, there is:

http://fluentassertions.com/documentation.html#coding-by-example

and in Java, Hill mentioned AssertJ, which uses that phrase as well:

http://joel-costigliola.github.io/assertj/

... basically, to produce assertions that read left-to-right in a very literary fashion.

So, possibly--you're referring to helper methods; with a good fluent assertion library, you can quickly create custom assertions that combine logical groupings--but the main purpose is for the phrasing.

GoogleTest supports them through their matchers--I think the same ones that are used for argument matchers for GoogleMock. (It's been a while though since I've had to look at that.)

Jeff

[Jeff Langr]

+1 on that, mostly.

GoogleTest supports EXPECT_EQ and I think promotes it more; it continues execution on failure. ASSERT_EQ does not.

I've found some rare cases where it would have been or was useful to have something like EXPECT_EQ--for example, I'm asserting a handful of populated attributes. But for that, maybe just support an overloaded form, something you could quickly change to using a regex replace.

[James Grenning]

Cool, thanks

[Jeff Langr]

Yes, and I think more to the point, most folks end up using tear downs to relinquish connections, i.e. they are writing integration tests.

I do recall a few rare cases where I wanted / needed tear downs in Java. One specifically: the test is going to trigger code that logs or otherwise writes to sysout, and so in setup I suppress it, and in teardown restore the pointer to sysout. I suppose this is a design issue--maybe one should be injecting print writers--but sometimes 3rd party stuff does things you don't want to, and the overhead of wrapping it all may seem like overkill otherwise.

[Jeff Langr]

Hi Charlie,

This discussion triggers another thought, which is: I like the way JUnit promotes unit tests, particularly with its isolation (e.g. creating a new instance *per test*, unlike, say, NUnit), and inability to directly order test execution. But I also appreciate that JUnit provides ways and/or hooks to support these sorts of things if you really need them. (Opinionated is fine, authoritarian / closed is not as fine, IMO.)

I'd probably rather not use two different tools for writing unit and integration tests, though I like the idea of one tool streamlined for unit tests.

What do you think about a C# framework that makes it easiest to write unit tests, and demands a little of additional effort / isolation for the integration tests?

Something like, the test fails if it's not in a class explicitly marked as [Integration], and if the test runner perceives it as such (hmm... execution time? interacts at all with system io?)... A test marked as [Integration] can use things like teardown and other integration test trappings.

Or not, just my later night thoughts. And now my mind is mulling over the potential positives of just insisting on two tools.

Name suggestion, in any case: NReallyUnit

Regards.
Jeff

[Charlie Poole]

Hi All,

Wow! Great thread! Thanks for the feedback. I'm mostly just listening for now, but I'll ask questions for clarity. Here's one: regarding teardown, can someone spell out why it's bad for micro-tests? Would you eliminate Setup as well?

I'm trying to collect and categorize what's said. Seems like some things are general to any kind of micro test, slightly fatter test, integration test or acceptance test, like having a decent syntax. Other things are specific to micro testing, like leaving out warnings that continue execution after the error. Maybe a third category is features specific to the particular language of implementation. I'll start writing something and add stuff as this continues.

Charlie

Jeff Langr wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[Jeff Langr]

Hi Charlie--

This is my thought on why tear downs in unit tests should be rare--they're usually an indicator of the compulsion to clean up some tests that dealt with the integrated world--delete a file, close a connection, clean a table, etc.

Jeff

[Steve Freeman]

What is your beef with hamcrest? Asking for a friend...

[Charlie Poole]

In .NET, I'm thinking that the rare cases where cleanup is needed can be handled by having the test class implement IDisposable. That's an expected thing on the platform and removes the need to have teardown at all. It's a tad bit more work than teardown, but that could be a good thing.

Charlie

Jeff Langr wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[James Grenning]

Heavens forbid if there is a static variable, teardown could reset it. Yes, static is bad. Seems like micro-testing should not make that impossible.

I'd like to see a micro-test fail if a static is not restored. A class of these failures is detected by CppUTest when an allocated object is not deleted and intentionally left in a static. The standard library is prone to this. A singleton has this as part of its definition. (yeah, I know. Don't use singletons.)

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[rob....@agileinstitute.com]

Joe, would you be willing to give an example of how you've used...

* direct support for Contract Tests: I can combine standard tests with my test subject factories to “generate” the same test run for multiple implementations of the same interface

This one has me befuddled. Different classes would implement the interface differently, so what could we test that we hadn't otherwise tested elsewhere? I could see the need for something like this (or at least what *I think* you're suggesting) if I'm going to have to test different classes on different platforms, but for the same behavior. But as I type that I just can't imagine doing that with a "test generator" - it smells funny to me whenever a microtest is generated. 

Then I thought, "Well, what about testing an abstraction factory that changes based on platform?" (Uh, sorry, an "Abstract Factory" Design Pattern, all praise to the Gang of Four! ;-) But I can think of other ways to microtest that, too.

* tearDown() (you should never need it!)

Can I get an "Amen!"?

Thanks,

Rob

[Allen Holub]

Maybe I’m just dense, but if you don’t have teardown, how do you undo side effects (e.g. file creation or mods) of the tests? Even micro tests occasionally have side effects and the boy-scout rule definitely applies.

-Allen
_______________________
Allen @ Holub .com
@allenholub
+1 (510) 859-3620

http://linkedin.com/in/allenholub

[rob....@agileinstitute.com]

Ditto. I like fluent assertions. Not sure where you get those besides an archaic copy of the Hamcrest JAR...? Last time I needed fluent assertions, I had to pull Hamcrest out of the trash bin. Why???

[rob....@agileinstitute.com]

Oh! AssertJ! <headdesk> <grumble grumble>

I must go rewrite a sh*tload of sample code...

On Tuesday, August 8, 2017 at 3:11:58 PM UTC-7, Michael Hill wrote:

Jeff...

I live in Java. As far as the assertions go, I also prefer "new style" to the old. To me, that means AssertJ. The extensibility here is just tremendous and very easy to do.

Is that what you're thinking of?

Hill

Virus-free. www.avast.com

On Tue, Aug 8, 2017 at 6:02 PM, Jeff Langr <jjl...@gmail.com> wrote:

Greetings Charlie--

I particularly appreciate that creating multiple test fixtures is available to me through NUnit (via nested classes), but JUnit (prior to 5) doesn't directly support that. Having to spread related tests across multiple source files is awkward. (Maybe there's solution that doesn't appear quite as cluttered/constrainted as nested classes--something that allows for a bit better contextual organization, and doesn't demand encoding all context in the name of each test.)

I agree with JB on teardown.

I'd also add: eliminate the old-school assertEquals, and offer only fluent assertions.
On a similar but more personal bent, also eliminate assertion failure messages.

I wonder if there's a way to encourage single behavior tests (i.e. tests that push asserts in the direction of 1-per-test).

This sort of interest always makes me wonder what a new, TDD-focused language might look like.

Regards,
Jeff

J. B. Rainsberger wrote:

On Sun, Aug 6, 2017 at 8:57 PM, Charlie Poole <cha...@nunit.org> wrote:

 

I'm thinking of creating a test framework tailored for micro-tests and micro-tests alone. I'd leverage my experience with NUnit and do the first implementation in C# and I'd add some ideas I have not gotten to implement in NUnit as well. My thoughts about such a framework so far are...
* It should have everything you need for micro-tests

* It should have nothing that leads you away from microtests

* It should leverage features of each language in it's implementation, rather than substituting it's own conventions

* It should be self-contained, requiring no other runner to execute tests

I'm looking for folks who are willing to chime in with ideas and this seems like a great forum to start. If it generates too much traffic, we can take it elsewhere.

I'm particularly interested in the features that fall under the first two bullet points. What do you think is essential for a framework aimed at micro-tests? What do you think should definitely be excluded?

Definitely include:

* data-oriented tests in the style of Spock/Fit over the Parameterized Test Case pattern

* direct support for Contract Tests: I can combine standard tests with my test subject factories to “generate” the same test run for multiple implementations of the same interface

Definitely exclude:

* arbitrary grouping of tests (“tags”, “categories”…) — if I want to have multiple groups of independently-run tests, I should work for it

* tearDown() (you should never need it!)

That’s just off the top of my head. I’m curious to see whether my subconscious thinks of anything else particularly salient.

It sounds like a good project!

-- 

J. B. (Joe) Rainsberger :: http://testdrivendevelopment.training :: http://www.jbrains.ca ::
http://blog.thecodewhisperer.com

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[Rob Myers]

The idea there is that if we stick to microtesting, nothing needs to be torn down.

BUT, you bring up a pet peeve of mine regarding xUnit (the .Net framework, not the family of frameworks): They chose to remove any form of setup, suggesting (accurately) that it was often being used poorly.

True, but some of us use it correctly, in small test classes or describe blocks, and to reduce a particular flavor of duplication; and I don’t think taking away a useful tool because others are misusing it is a great product strategy.

In Charlie’s case, in any language besides C++, I’m not sure where I would ever use teardown when microtesting.

You received this message because you are subscribed to a topic in the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lonely-coaches-sodality/KxHTCCF_OVs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lonely-coaches-so...@googlegroups.com.

[Mark Levison]

On Wed, Aug 9, 2017 at 4:42 PM, <rob....@agileinstitute.com> wrote:

Oh! AssertJ! <headdesk> <grumble grumble>

I must go rewrite a sh*tload of sample code...

​or move to JUnit 5 which builds them in.

Cheers

Mark​

[Rob Myers]

I must not have been able to find the right import. I'll check my notes but I recall struggling with writing a simple readable test in junit 5. 

Rob Myers

Principal Agile Instructor & Coach

Rob....@agileforall.com

+1.707.704.9371

Twitter: @agilecoach

http://www.agileforall.com/author/rmyers/

--

---

You received this message because you are subscribed to a topic in the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lonely-coaches-sodality/KxHTCCF_OVs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lonely-coaches-so...@googlegroups.com.

[Ron Jeffries]

Here is a use of setup/teardown in the thing I’m currently working on (in Ruby):

  def setup

    @test_startup_folder = Dir.pwd

  end

  def teardown

    Dir.chdir(@test_startup_folder)

  end

The Dir object is set when the tests start running, I guess to the folder the tests are in or the project folder or whatever the hell Ruby and the test framework decide.

Our code uses the Dir object to read out various folder contents so as to move them around, FTP them, and the like. We need that to be consistent for all the tests, even if a preceding test might have changed it. So we save and restore it.

Elsewhere we need sample folders to test, for example whether we moved the right files. So we remove them, create them empty, test them. In that case we used only setup. I’d say I use setup nearly always, because tests have commonality, and teardown nearly never.

Similarly but much less frequently, I want a suite-level setup and still less frequently maybe I’ve used a suite-tear down. If I did it would be for setting up some resource, I suppose.

This topic and some of the other stuff that has gone by makes me think that triggering something at test and suite start and end is probably pretty close to essential. I’d rather be wrong, tho.

Ron Jeffries

ronjeffries.com

Impossible is not a fact. It is an opinion.  -- Muhammad Ali

[Ron Jeffries]

I tend to agree …

On Aug 9, 2017, at 4:36 PM, Allen Holub <al...@holub.com> wrote:

Maybe I’m just dense, but if you don’t have teardown, how do you undo side effects (e.g. file creation or mods) of the tests? Even micro tests occasionally have side effects and the boy-scout rule definitely applies.

Ron Jeffries

ronjeffries.com

Sometimes I give myself admirable advice, but I am incapable of taking it.

-- Mary Wortley Montagu

[Ron Jeffries]

I find fluent assertions to be somewhat easy to read, in that they are sort of like a sentence. My profession is writing and reading programming languages, but even so I find reading, say, Java, easier than writing some DSL someone pulled out of their um brain and wrote in Java. DSLs are only useful to the people who write and live with them all the time. They are rather opaque to everyone else.

I find it nearly impossible to understand how fluent assertions could possibly work, because you have to understand the return type of every method you’ve never seen before. So it distracts me to look at them

I find them nearly impossible to write new ones because even though someone can say .Be(11) it’s not clear if you can say .Be().Bigger().Than().A(breadbox) or not. The fluent “vocabulary” is huge and all one mostly needs is “assertEqual()”/

On Aug 9, 2017, at 4:38 PM, rob....@agileinstitute.com wrote:

Ditto. I like fluent assertions. Not sure where you get those besides an archaic copy of the Hamcrest JAR...? Last time I needed fluent assertions, I had to pull Hamcrest out of the trash bin. Why???

Ron Jeffries
ronjeffries.com
Apparently, “semantics” doesn’t mean what I think it means.

[Ron Jeffries]

Hmm …

On Aug 9, 2017, at 4:50 PM, Rob Myers <rob....@agileinstitute.com> wrote:

The idea there is that if we stick to microtesting, nothing needs to be torn down.

I think I want to challenge part of this idea. Mind you, I am all for an incredibly simple stripped down tool.

However, microtesting isn’t the only testing one does. I’d think one would move rather smoothly up and down the size scale. Micro micro micro mini micro mini moderate … 

I ‘m not convinced we should have a tool so limited that every time our ideas expand a bit we need to switch tools. 

This makes me think that perhaps the “micro” notion isn’t quite the thing … but I’ve very sure that very simple is good.

Some way of being open for extension perhaps?

Ron Jeffries

ronjeffries.com

I don't necessarily agree with everything I say. -- Marshall McLuhan

[Adrian Howard]

On 10 Aug 2017, at 12:40, Ron Jeffries <ronje...@acm.org> wrote:

[snip]

However, microtesting isn’t the only testing one does. I’d think one would move rather smoothly up and down the size scale. Micro micro micro mini micro mini moderate … 

I ‘m not convinced we should have a tool so limited that every time our ideas expand a bit we need to switch tools. 

This makes me think that perhaps the “micro” notion isn’t quite the thing … but I’ve very sure that very simple is good.

[snip]

This has reminded me of how much I like Perl’s testing infrastructure.

It’s built around a very simple streaming protocol (TAP https://testanything.org) with test producers producing TAP, and test consumers consuming it in various ways.

Which means you can have a test suite built up of multiple kinds of test framework/methodology depending on what best suits your particular testing problem…:

* really simple plain assertion like Test::More (https://metacpan.org/pod/Test::More)

* xUnit-ish style like Test::Class::Moose (https://metacpan.org/pod/Test::Class::Moose)

* BDD style like Test::BDD (https://metacpan.org/pod/Test::BDD::Cucumber)

* declarative style like Test Spec https://metacpan.org/pod/Test::Spec

etc. Plus it’s easy to hand-roll your own custom test producer to make some DSL for your system’s own particular testing issues.

And because everything outputs TAP all the standard test runners, build tools, etc. can consume it.

Adrian

-- 

adr...@quietstars.com / +44 (0)7752 419080 / @adrianh / quietstars.com

(CSSTWP.com the product team certification programme you can trust! ;-)

[James Grenning]

I don't know that 'nothing needs to be torn down'.

Lets say for micro testing you create fake concurrency mechanisms, like mutex.acquire and mutex.release. I'd like to make sure that every test ends in a state with all mutex's released. I bet this group could dream up many other things like this. open/close for example.

James

---

James Grenning - Author of TDD for Embedded C - wingman-sw.com/tddec
wingman-sw.com
wingman-sw.com/blog
twitter.com/jwgrenning
facebook.com/wingman.sw
Cell: +1 847-630-0998

[Charlie Poole]

Hi James,

Interesting. You are using teardown there as a common final step for all tests. NUnit itself doesn't handle this in an entirely friendly way, treating every assertion failure in teardown as an unhandled exception. Of course, you still get a report of the error, which seems to satisfy most people.

I made a proposal at one time for a PostTest attribute to allow for common "tails" on every test, separate from teardown, but it never got much traction.

The fact is that setup and teardown both have several different functions that we usually mix together in discussion.

1. To reduce duplication in the tests.

2. To initialize, clean up resources.

3. For teardown only, to have some code that is guaranteed to run after an exception.

In micro-tests, (1) remains completely useful even as the need for (2) and (3) may be reduced.

BTW, I'm using setup / teardown here to mean "some code that runs first / last". It might not be a pair of specially named or marked methods as it is in most frameworks.

Charlie

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to a topic in the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lonely-coaches-sodality/KxHTCCF_OVs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[Charlie Poole]

Hi Rob,

A word in favor of the xUnit guys... they took away SetUp, but gave you the constructor for the class, since xUnit creates a new class for each test method. So the function is still there although there is room for argument as to which approach expresses things most clearly.

As it happens, I'm leaning toward the use of the constructor in this hypothetical new framework. It allows developers to use a familiar idiom to initialize the test. Downside is that it doesn't work for static methods.

Charlie

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---

You received this message because you are subscribed to a topic in the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lonely-coaches-sodality/KxHTCCF_OVs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[James Grenning]

setup/teardown mean pre-test and post-test for me too.

CppUTest as a generalized plugin idea. Where each plugin has a pre-test and post-test action.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[Michael Hill]

on balance, i think i do want what junit calls @Before/@After. i very rarely use @After, but the border is gray, and sometimes i don't write the best possible test, e.g. one that doesn't use files. instead, i write more or less desperately "at least this *is* a test" tests. in those cases, cleanup is sometimes required.

as far as constructor/destructor vs named methods, i'd say that depends entirely on the nature of the language. in java, it must be methods. in c++, constructor/destructor work fine and are more natural, assuming the fixture is re-created with each test method.

cheers,

hill

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to a topic in the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lonely-coaches-sodality/KxHTCCF_OVs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ron Jeffries]

Yes, this seems quite wise to me. The re-use of the instance is probably a violation of some principle or other.

On Aug 10, 2017, at 11:08 AM, Charlie Poole <nuni...@gmail.com> wrote:

As it happens, I'm leaning toward the use of the constructor in this hypothetical new framework. It allows developers to use a familiar idiom to initialize the test. Downside is that it doesn't work for static methods.

Ron Jeffries
ronjeffries.com
Sometimes you just have to stop holding on with both hands, both feet, and your tail, to get someplace better. 

Of course you might plummet to the earth and die, but probably not: you were made for this.

[Michael Hill]

back to the fit/spock thing.

in llewellyn's approval tests model, it's very easy to specify a single setup/exercise/probe block, then tell the 'puter here's an array of data, where each row in the array is the arguments to that block, including expectations.

this definitely has some advantages, especially if i can easily give my rows names.

i don't normally work this way, but i do see it as a possible way to work, and i'd imagine there are techniques lurking in the normal practice of it that i don't yet have.

seeya,

hill

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Charlie Poole]

Hi Hill,

Does the @after include the provision that it always runs, even if an exception is thrown?

Charlie

[Michael Hill]

truth to tell: beats hell outta me.

i think it should? if not, then before|throwing-test| means that a second test that expects to start from scratch won't.

so, let me go stronger: YES. :)

[Ron Jeffries]

…

On Aug 10, 2017, at 2:10 PM, Charlie Poole <nuni...@gmail.com> wrote:

Does the @after include the provision that it always runs, even if an exception is thrown?

I’d think yes. And it might be interesting if the results were somehow accessible to it, via that TAP thing or some similar deal.

Simplicity plus extensibility. Be cool ...

Ron Jeffries

ronjeffries.com

Before I start arguing with somebody, I try to save them a little time by letting them know that I’m usually right about stuff.

— Andy Richter

[Allen Holub]

Yes, @After runs even if @Before or the test method throws an exception in JUnit.

I'm very much on the side of including this or some similar mechanism. Though it's absolutely true that the test should be standalone, objects do function in a context, and sometimes you have to test in that context, and the context is global or otherwise persistent. Injecting a million mocks solely to make the object testable in isolation unnecessarily clutters up the code. That is, If I can avoid it, I don't want to make the actual code more complex (potentially introducing bugs and always impacting maintainability) solely to make the object testable. I, at least, really need @After or some similar mechanism. to clean up after the occasional test that has side effects. Without it, I have to put the clean up into a script that invokes the test, and I'd much rather have all the test stuff in one place.

-Allen

_______________________

Allen @ Holub .com
@allenholub

+1 (510) 859-3620  (Skype: allenholub)
http://holub.com
http://linkedin.com/in/allenholub

[Allen Holub]

Also, related to the @After-and-exception issue, JUnit has an attribute on the @Test annotation that both signals an error if the exception is not tossed and absorbs the exception toss. It’s pretty handy:

@Test(expected = IndexOutOfBoundsException.class)  
public void empty() {  
     new ArrayList<Object>().get(0);  
}

[Ron Jeffries]

…

On Aug 10, 2017, at 2:32 PM, Allen Holub <al...@holub.com> wrote:

Injecting a million mocks solely to make the object testable in isolation unnecessarily clutters up the code. That is, If I can avoid it, I don't want to make the actual code more complex (potentially introducing bugs and always impacting maintainability) solely to make the object testable. I, at least, really need @After or some similar mechanism. to clean up after the occasional test that has side effects. Without it, I have to put the clean up into a script that invokes the test, and I'd much rather have all the test stuff in one place.

What if, though, there was some simple-yet-general mechanism like injection that allowed these contextual objects to be created and destroyed at various intervals, in the test, rather than in the subject code? Imagine that there are only certain times one cares about, e.g. before and after ...

• Any tests run;
• A suite runs (if there are suites. Maybe I mean “file”)
• A test runs

And maybe I can append a block / closure to the list for that time, and the block / closure accepts a defined parameter, that is, the test framework itself, or an object accessing it, so that I can interrogate the situation and maybe have the framework remember an item in a key-value kind of way.

Ron Jeffries

ronjeffries.com

If it is more than you need, it is waste. -- Andy Seidl

[Mark Levison]

Charlie - decloaking for just a second. 

Avoid: Ordering or anything else that makes dependancies possible.

Reasoning - I'm subscribed to the JUnit5 Github PR requests - the "benefit" of having been a project sponsor. I see a number of requests for ordering all of which go against the intention of Unit (let alone Micro) Testing.

BTW Thanks for all of those years of work on NUnit. I haven't worked with .NET for nearly 10yrs but really appreciated NUnit at the time.

Cheers

Mark

[Eric Anderson]

For several years, I have used the basic constructs in JUnit / NUnit to write context-specification tests so that I can separate assertions from common test setup. I setup the context and perform all actions in the Setup method. Any state needed for the assertions is stored in fields, and then each "logical" assertion gets its own test method. I generally follow this structure when I'm writing unit tests, integration tests, and even full-system web tests. Over the years, when new co-workers have seen the tests, they tend to appreciate the readability that these tests provide.

So for me, even a lightweight test framework needs to accommodate context-specification. NOTE: This is different from supporting Gherkin (Given-When-Then) syntax which I personally find to be exceptionally clunky and technically misleading. I don't ever use Gherkin unless absolutely forced to by some client / boss -- even then, I still explain the technical and communication problems that Gherkin tends to create before I consent to "do it their way."

Eric

[Allen Holub]

Ron,

That *might* be better. I’m hesitant because of my experience with Java’s Mockito/Powermock systems, which I’m using only rarely now because it turned out to take less time to write the mock/spy in Java than it did to define it in the mocking framework. The framework also effectively introduces a whole new language that I’d rather not have to remember. It was usually easier just to write classes, though I do still use the framework occasionally. So, I wouldn’t want the “fix” to introduce more complexity as a side effect. I’m also hesitant to trust an injection-by-magic system because of my experience with Spring. Sometimes the bug is in the injection system, and I really don’t want that additional unknown in a test. I’m happier when I can see everything. That, admittedly, might just be a foible of mine.

-Allen

[Michael Hill]

Eric...

Can you give us a decent snippet or a link to one? I'm not getting what you say in the abstract.

Thanks,

GeePaw

[Michael Hill]

I, too, hate most forms of magic injection. Spring in particular, the kudzu of Java, creates quite magical code and tests, provided only that you what you wished to do is exactly what the sorcerors wanted you to do.

And I, too, use automocking frameworks only in legacy where i have no testability at all without them, and usually only in the "pin it down" phase.

[Eric Anderson]

Here is a test from a system I wrote for displaying build status via Z-Wave switches. The system can monitor multiple builds and display them on multiple groups of (red-yellow-green) lights. The large use case is a large team spread across a large floor -- so, multiple displays for the same build on different parts of the floor. 

In the gist below, I was testing the operation of moving a specific switch (one light) between groups.

https://gist.github.com/Vector241-Eric/8d01474a09e3ccf6a4571e64f1648c6e

The model is setup and the actions are taken in the [SetUp]. Assertions are made in individual [Test] methods. I happen to use a fluent assertion framework too. My focus in testing has always been to create the most readable tests possible. At one point, I used a test structure similar to this in Java, then used reflection to scrape out all the language into a report I would discuss with the project manager. That worked pretty well also.

Eric

[Michael Hill]

i see. so from my point of view, what's different here is that each test object only has a single action in it.

i'm used to: before [action assert] after, where the brackets are the test method. you're doing before-action [assert] after.

my gut reaction is to wonder if that won't multiply to quite a few test objects for a rich API or an API with complex datasets.

[Eric Anderson]

I have never cared for tests in which I had to study multiple locations to determine what's going on in the test -- in other words, "Why did this fail?" So, I attempt to keep my tests as simple as possible. Part of this for me is to keep the tests as contextualized as possible. Over time, I have come to view the arrange and the act as part of the context of the test. I don't care a bunch about reusing the arrange across different contexts. I prefer to keep my tests less DRY.

If the test setup gets too long, then I will stop a evaluate why. Sometimes, it's because the SUT interface is too complicated. Sometimes it's because the data setup is rather large as required by the specific context. In the case of the former, I evaluate simplifying the SUT. In the case of the former, I will consider employing a setup helper that allows me to express the essence of the setup in a more condensed, semantic way -- often this is something that encapsulates the models which need to be setup. 

This whole structure was a intentional progression away from Given-When-Then. I want the context expressed in the most clear concise language possible.

[Gerard Meszaros]

Sounds like a parameterized test method. Most xUnit frameworks support this directly or through extensions. Some give easier semantics while others make it more/too complicated. But no reason why xUnit tests should be run multiple examples through the same given/when/then or AAA logic.

Gerard

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 
Gerard Meszaros
Lean/Agile Coach/Mentor/Trainer
http://www.gerardmeszaros.com
http://xunittraining.com  
1-403-827-2967
Twitter: @gerardmes
LinkedIn: gerardmeszaros

Author of the Jolt Productivity Award winning book "xUnit Test Patterns - Refactoring Test Code." Learn more at http://xunitpatterns.com/index.html

[J. B. Rainsberger]

On Tue, Aug 8, 2017 at 5:56 PM, Michael Hill <GeePa...@geepawhill.org> wrote:

 

what's a data-oriented test in the style of spock?

https://www.testwithspring.com/lesson/writing-parameterized-tests-with-spock-framework/

Search the page for “data tables”. I don’t care about the format, as long as it’s dead easy. Spock does this really well:

* define a test like any other

* treat table columns like variables in the test

* “unroll” — optionally run each row in the table as a single test

what's a contract test in the context you envision?

A test plus a factory method that I can “inherit”. In JUnit, we literally inherit them by subclassing (it’s the one time :) ). In RSpec, there are shared example groups which, if I remember correctly, are mixins with tests that expect factory methods. The factory method provides the subject of the test.

Example in JUnit:

https://github.com/jbrains/WBITDD-20161123/blob/client-first/src/test/java/ca/jbrains/pos/test/FindPriceInCatalogContract.java 

and its subclass/implementation

https://github.com/jbrains/WBITDD-20161123/blob/client-first/src/test/java/ca/jbrains/pos/test/FindPriceInMemoryCatalogTest.java

-- 

J. B. (Joe) Rainsberger :: http://testdrivendevelopment.training :: http://www.jbrains.ca ::
http://blog.thecodewhisperer.com

[J. B. Rainsberger]

On Wed, Aug 9, 2017 at 5:32 PM, <rob....@agileinstitute.com> wrote:

 

This one has me befuddled. Different classes would implement the interface differently, so what could we test that we hadn't otherwise tested elsewhere? I could see the need for something like this (or at least what *I think* you're suggesting) if I'm going to have to test different classes on different platforms, but for the same behavior. But as I type that I just can't imagine doing that with a "test generator" - it smells funny to me whenever a microtest is generated. 

That’s why I put “generate” in quotes; a Contract Test is a template for standard tests to run on multiple implementations to check that they respect the common contract. Classic example is Object.equals() in Java, but it can be any interface. How to check, for example, that an implementation of (Java) List behaves like a List? 

Then I thought, "Well, what about testing an abstraction factory that changes based on platform?" (Uh, sorry, an "Abstract Factory" Design Pattern, all praise to the Gang of Four! ;-) But I can think of other ways to microtest that, too.

Factories usually have a contract simple enough that the type checker checks it without extra tests, but imagine that an Abstract Factory method had the additional constraint that “created object X always needs to have property Y”. A Contract Test would express that for the Abstract Factory without reference to a specific implementation of the Factory.

-- 

[J. B. Rainsberger]

On Tue, Aug 8, 2017 at 11:30 PM, James Grenning <ja...@grenning.net> wrote:

Why is teardown a problem? Is that because OO languages' object destruction should handle the cleanup?

It’s not that teardown is a *problem*, but rather that it should be unnecessary. If every test runs in a separate runtime context (in an OO language, its own object), then there should be no need to clean up. Needing to clean up probably (always?) makes it “not a microtest”. It might subtly encourage integrated tests by making it too easy to clean up.

-- 

[J. B. Rainsberger]

On Wed, Aug 9, 2017 at 2:59 AM, Charlie Poole <nuni...@gmail.com> wrote:

 

Wow! Great thread! Thanks for the feedback. I'm mostly just listening for now, but I'll ask questions for clarity. Here's one: regarding teardown, can someone spell out why it's bad for micro-tests? Would you eliminate Setup as well?

Teardown isn’t *bad*, but merely unnecessary. I would retain Setup, because I organize tests by fixture and use Setup to initialize the common fixture. Setup helps me remove duplication.

-- 

[Mark Levison]

On Aug 11, 2017 4:54 PM, "J. B. Rainsberger" <m...@jbrains.ca> wrote:

On Tue, Aug 8, 2017 at 11:30 PM, James Grenning <ja...@grenning.net> wrote:

Why is teardown a problem? Is that because OO languages' object destruction should handle the cleanup?

It’s not that teardown is a *problem*, but rather that it should be unnecessary. If every test runs in a separate runtime context (in an OO language, its own object), then there should be no need to clean up. Needing to clean up probably (always?) makes it “not a microtest”. It might subtly encourage integrated tests by making it too easy to clean up.

How about a teardown with 1 ms sleep? A few won't hurt but many will slow your tests down. 

Cheers 

Mark 

[Ron Jeffries]

All,

On Aug 11, 2017, at 11:29 AM, J. B. Rainsberger <m...@jbrains.ca> wrote:

It’s not that teardown is a *problem*, but rather that it should be unnecessary. If every test runs in a separate runtime context (in an OO language, its own object), then there should be no need to clean up. Needing to clean up probably (always?) makes it “not a microtest”. It might subtly encourage integrated tests by making it too easy to clean up.

I totally get the idea of simplicity in the framework. I don’t get the idea of limiting its capability. What’s the upside to that?

Ron Jeffries

ronjeffries.com

Impossible is not a fact. It is an opinion.  -- Muhammad Ali

[Allen Holub]

I totally get the idea of simplicity in the framework. I don’t get the idea of limiting its capability. What’s the upside to that?

I strongly agree with Ron. I just hate it when a framework tries to impose a philosophy on me. It's usually the case that I'll eventually have to do something that an opinionated framework makes impossible. I find that I tend to abandon opinionated systems much earlier than not (even when I've written them :-)).

_______________________

Allen @ Holub .com
@allenholub

[Jon Kern]

70 messages… geesh “You guys should get a room.” Oh. Wait. This is the room.

This last exchange makes me want to say: Welcome to the world of being a Product Owner. 

You can be opinionated and decide to settle on a vision and limits for your micro testing product.

Or allow in everyone’s wishes and water down the original intent.

I think the micro test framework should refuse to run after hours or on weekends (or whatever the user’s schedule is). And, if a test takes too long, flag it and refuse to give it much of the CPU/bandwidth on subsequent runs — until it "shapes up and flies right!" Play favoritism for fast tests.

Jon Kern

blog: http://technicaldebt.com

twitter: http://twitter.com/JonKernPA

github: https://github.com/JonKernPA

[Ron Jeffries]

Jon,

On Aug 11, 2017, at 8:28 PM, Jon Kern <jonk...@gmail.com> wrote:

You can be opinionated and decide to settle on a vision and limits for your micro testing product.

Or allow in everyone’s wishes and water down the original intent.

A very simple program could be quite open for extension, with a few hooks at critical points before/after.

Ron Jeffries

ronjeffries.com

Everything that needs to be said has already been said.

But since no one was listening, everything must be said again. -- Andre Gide

[Charlie Poole]

Hi Jon,

Seriously, I've never found it very easy to set the limits cleanly. If you go with a philosophy of "The users get want they want", most of us eventually end up feeling that the product has gotten out of hand and go work on something simpler. That's what Jim did when he started xunit, for example, and what I did when I set aside NUnit to make NUnitLite. But then the simple thing, if you keep at it long enough, begins to get complicated again.

Charlie

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[Charlie Poole]

Hi Ron,

Yeah, that's what I'm thinking. The trick is to pick those few hooks. Otherwise, you start adding them everywhre.

Mostly, this  whole thread comes out of the fact that I feel like working on something simple for a while.

Charlie

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[Eric Anderson]

Sounds like a normal progression for both a skilled software developer and a successful product. 

Sent with fat fingers and an autocorrect with a mind of its own.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[Jeff Langr]

Hi Charlie,

Right. I'm not sure about the history of NUnit, but it looks like support for test ordering, as one example, was added at a later point. I appreciate that 17 or so years later, the JUnit folks have managed to resist adding things like this (I believe 5 will not have it).

One reason to not eliminate certain features: If we add "legacy rescue" into the mix, there are some good rationales for teardown, such as the sysout / log  suppression & reenabling example. If I'm trying to transition legacy code characterization tests over time into micro tests, I'd hope not have to "move them" to use another framework.

Regards,
Jeff

Charlie Poole wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[Charlie Poole]

Hi Ron,

Interesting comment about fluent assertions... easy to read but hard to write.

What do you use to edit? Most people seem to rely on the presence of some sort of prompting from an IDE (Intellisense in VS) to tell them what they are allowed to type next. I suspect that the use of a really complex fluent language may even require that sort of prompting.

Charlie

On Thu, Aug 10, 2017 at 4:35 AM, Ron Jeffries <ronje...@acm.org> wrote:

I find fluent assertions to be somewhat easy to read, in that they are sort of like a sentence. My profession is writing and reading programming languages, but even so I find reading, say, Java, easier than writing some DSL someone pulled out of their um brain and wrote in Java. DSLs are only useful to the people who write and live with them all the time. They are rather opaque to everyone else.

I find it nearly impossible to understand how fluent assertions could possibly work, because you have to understand the return type of every method you’ve never seen before. So it distracts me to look at them

I find them nearly impossible to write new ones because even though someone can say .Be(11) it’s not clear if you can say .Be().Bigger().Than().A(breadbox) or not. The fluent “vocabulary” is huge and all one mostly needs is “assertEqual()”/

On Aug 9, 2017, at 4:38 PM, rob....@agileinstitute.com wrote:

Ditto. I like fluent assertions. Not sure where you get those besides an archaic copy of the Hamcrest JAR...? Last time I needed fluent assertions, I had to pull Hamcrest out of the trash bin. Why???

Ron Jeffries
ronjeffries.com
Apparently, “semantics” doesn’t mean what I think it means.

[Ron Jeffries]

Hi Charlie,

On Aug 12, 2017, at 7:44 PM, Charlie Poole <nuni...@gmail.com> wrote:

Interesting comment about fluent assertions... easy to read but hard to write.

What do you use to edit? Most people seem to rely on the presence of some sort of prompting from an IDE (Intellisense in VS) to tell them what they are allowed to type next. I suspect that the use of a really complex fluent language may even require that sort of prompting.

Yes … and I think that’s kind of a sign. Type dot, get 131 possibilities. And, as I mentioned, to me the syntax just doesn’t make sense anyway.

And then ten seconds later you’re writing real Java or C# again.

But, hey, horses for courses, I guess. Amazing what I can do with assertEqual though. :)

[James Grenning]

On 11 Aug 2017, at 16:04, Mark Levison wrote:

>
> How about a teardown with 1 ms sleep? A few won't hurt but many will
> slow
> your tests down.

Certainly there are misuses of teardown, as here are misuses of beer. I
am not for outlawing either.

>
> Cheers
> Mark

>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Lonely Coaches Sodality" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to lonely-coaches-so...@googlegroups.com.

[Nigel Thorne]

Charlie, 

I often find I write and read test cases bottom up... or inside out. 

"The employee will be docked 3 weeks pay"

"When the employer rejects a timesheet"

"Given the employee has been paid 3 weeks in advance"

or 

"When the employer rejects a timesheet"

"THEN The employee will be docked 3 weeks pay"

"GIVEN the employee had been paid 3 weeks in advance"

I know you are talking about Micro tests, but in TDD I often start with the assertion and work back. 

I think all I'm saying is. We should look at the order of Arrange|Act|Assert and check it's the most readable way to document behaviour. 

PS. love the idea of passing a readable table of data to the test. 

And J.B. Rainsberger: Thanks for mentioning Spock. I've not seen that before. Interesting syntax. 

 

---

"Man, I'm going to have so many chickens when this lot hatch!"

On Mon, Aug 7, 2017 at 6:36 PM, Charlie Poole <cha...@nunit.org> wrote:

Hi Ron,

Some examples could be...

 * File assertions

 * Ordering of tests

 * Shared state across tests

 * hierarchical setup in various forms

Some of these, of course, may also be used in ways that do no harm.

Charlie

On Sun, Aug 6, 2017 at 11:46 PM, Ron Jeffries <ronjeff...@gmail.com> wrote:

Hi Charlie!

> On Aug 6, 2017, at 7:57 PM, Charlie Poole <cha...@nunit.org> wrote:
>
> * It should have everything you need for micro-tests
> * It should have nothing that leads you away from microtests

I'm certainly all for this,  though I am at this moment quite without ideas. Can you think of an example of something in NUnit that leads away? (I suspect it's something I never used and don't even know about.)

Thanks!

R

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.

[J. B. Rainsberger]

On Fri, Aug 11, 2017 at 8:45 PM, Ron Jeffries <ronje...@acm.org> wrote:

 

I totally get the idea of simplicity in the framework. I don’t get the idea of limiting its capability. What’s the upside to that?

Well, it’s one of the things that Charlie explicitly asked for ideas on, so I gave one. To quote him: "It should have nothing that leads you away from microtests.” I think that teardown leads one away from microtests. That point is clearly debatable, and so we debated it. I’m happy with the result.

-- 

[Charlie Poole]

If you look at my post giving the conclusions I drew from the discussion, I think you'll see that I took the "simplicity rather than restriction" notion to heart!

--

[J. B. Rainsberger]

On Wed, Aug 23, 2017 at 1:00 PM, Charlie Poole <nuni...@gmail.com> wrote:

 

If you look at my post giving the conclusions I drew from the discussion, I think you'll see that I took the "simplicity rather than restriction" notion to heart!

I have no doubt of this!

-- 

[Nigel Thorne]

What about... any test taking over X time auto-fail.

---

"Man, I'm going to have so many chickens when this lot hatch!"

--

[Charlie Poole]

Interesting idea. For general testing, I think no timeout makes sense as the default, but for a micro test it should only take some relatively small amount of time. Has to be at least three or four times the minimum resolution of the timer however or you'll get irregular results.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Allen Holub]

I wrote a heartbeat for a distributed microservices system a few years back, the test for which needed to prove that activity occurred every minute. We did jigger the clock, but the timeout isn't necessarily short.

-Allen
___________________________________
Allen Holub
al...@holub.com
@allenholub
+1 (510) 859-3620 (skype ID: allenholub)

linkedin: http://linkedin.com/in/allenholub

[Ron Jeffries]

What core facility would be required to allow the programmer to “simply" place a convenient start-timer call in setup, that would upon timer elapsing, fire an interrupt / exception which could be fielded as desired?

The languages I use most have timer-tick events built in.

This makes me wonder how to expose general exception handling up to the programmer level, rather than handle such things inside the framework. That is, framework doesn’t do anything with exceptions and events, treat them as fail or pass or whatever, just passes them to the programmer, who has set up whatever’s to be done in his handlers.

Then, I’d suppose, there’d be well-known handlers in a library that one could reference and use, but one could write one’s own.

On Aug 27, 2017, at 12:01 AM, Charlie Poole <nuni...@gmail.com> wrote:

Interesting idea. For general testing, I think no timeout makes sense as the default, but for a micro test it should only take some relatively small amount of time. Has to be at least three or four times the minimum resolution of the timer however or you'll get irregular results.

On Sat, Aug 26, 2017 at 5:22 PM, Nigel Thorne <ni...@nigelthorne.com> wrote:

What about... any test taking over X time auto-fail.

Ron Jeffries

ronjeffries.com
Master your instrument, master the music, then forget all that shit and just play. -- Charlie Parker

[Nigel Thorne]

How about... Blurring the lines where tests end, so they are async.. arrange, act, assert, timeout are all callbacks,

Then I can write tests how I read them.. "when" "then” "given". :)

Timeouts default callback is fail. 

Optional training wheels feature that checks the number of "ifs" encounter during execution of an act callback. If it's greater than some value, fail the test. 

This would be a proxy for npath complexity, but the test covering the complex path would fail, not the while method. 

[Ron Jeffries]

mmm i think i like the callback idea a lot

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-so...@googlegroups.com.

[Nigel Thorne]

Also... any test failing in the "arrange" should be marked as 'blocked' not 'failed'. The focus of the test was never reached. Another test should be failing at that point. This way only the test focused on the bug will 'fail', giving you a laser pointer focused on the actual error. 

I think the same should be true for the 'act' if you haven't got some type of "expect an exception to be thrown" type behaviour. This is less clear to me however. 

---

"Man, I'm going to have so many chickens when this lot hatch!"

On Mon, Aug 28, 2017 at 7:51 AM, Ron Jeffries <ronjeff...@gmail.com> wrote:

mmm i think i like the callback idea a lot

On Aug 27, 2017, at 5:31 PM, Nigel Thorne <nigel....@gmail.com> wrote:

How about... Blurring the lines where tests end, so they are async.. arrange, act, assert, timeout are all callbacks,

Then I can write tests how I read them.. "when" "then” "given". :)

Timeouts default callback is fail. 

Optional training wheels feature that checks the number of "ifs" encounter during execution of an act callback. If it's greater than some value, fail the test. 

This would be a proxy for npath complexity, but the test covering the complex path would fail, not the while method. 

On 27 Aug. 2017 8:05 pm, "Ron Jeffries" <ronje...@acm.org> wrote:

What core facility would be required to allow the programmer to “simply" place a convenient start-timer call in setup, that would upon timer elapsing, fire an interrupt / exception which could be fielded as desired?

The languages I use most have timer-tick events built in.

This makes me wonder how to expose general exception handling up to the programmer level, rather than handle such things inside the framework. That is, framework doesn’t do anything with exceptions and events, treat them as fail or pass or whatever, just passes them to the programmer, who has set up whatever’s to be done in his handlers.

Then, I’d suppose, there’d be well-known handlers in a library that one could reference and use, but one could write one’s own.

On Aug 27, 2017, at 12:01 AM, Charlie Poole <nuni...@gmail.com> wrote:

Interesting idea. For general testing, I think no timeout makes sense as the default, but for a micro test it should only take some relatively small amount of time. Has to be at least three or four times the minimum resolution of the timer however or you'll get irregular results.

On Sat, Aug 26, 2017 at 5:22 PM, Nigel Thorne <ni...@nigelthorne.com> wrote:

What about... any test taking over X time auto-fail.

Ron Jeffries

ronjeffries.com
Master your instrument, master the music, then forget all that shit and just play. -- Charlie Parker

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Lonely Coaches Sodality" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lonely-coaches-sodality+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Feathers]

Wish I'd seen this thread earlier. So many great ideas. I agree strongly with the notion of no individual tearDown.

I took a stab at the minimal framework thing years ago with CppUnitLite, which was a single header file for C++. It had the basics of test execution and no setup or teardown. In C++ you can using RAII instead, but I didn't want to support setUp/tearDown natively. I just learned that CppUnitLite was forked years ago and it changed from being one file to this whole fucking ecosystem: https://github.com/smikes/CppUnitLite That brings me back to the notion of a seedwork, which is a framework that is meant to be minimal - and unchangeable: http://www.artima.com/weblogs/viewpost.jsp?thread=8826 Charlie, I'm not sure whether that's what you want or not.

To me, there are a couple of very interesting design choices in this space. One is 'minimal framework you never have to change', another is 'opinionated framework which forces you to work to do something ill-advised', and another is 'framework that actively prevents you from doing bad things.' I hear you, Allen, on the problems with the latter, but I think that if people actively choose that sort of thing and have the option of writing other tests in another framework that they run along side, it's workable.

I know this is not your direction, Charlie, but expanding on the 'framework that prevents you from doing bad things' (just because it hasn't been elaborated much) there are a  couple of interesting directions beyond the timeouts and measures of complexity that were just mentioned. One is to have the framework actively prevent IO ( https://github.com/fmedio/ashcroft ). Another I haven't seen done is to carry that further and allow no dependencies at compile time or run time on code that is not in a particular set of packages/assemblies. This can lean development toward what were once called POJOs and provide separation of domains (here's our code, and there's the framework we use). If there's something that would prevent reflective access as well, that could be a nice addition.

[J. B. Rainsberger]

On Tue, Aug 29, 2017 at 3:11 AM, Nigel Thorne <ni...@nigelthorne.com> wrote:

 

Also... any test failing in the "arrange" should be marked as 'blocked' not 'failed'. The focus of the test was never reached. Another test should be failing at that point. This way only the test focused on the bug will 'fail', giving you a laser pointer focused on the actual error. 

I like this on its face and also in order to discourage the practice I see of extracting common assertions into “arrange” (setup()). I like the impulse of removing duplication, but not this way. :)

I think the same should be true for the 'act' if you haven't got some type of "expect an exception to be thrown" type behaviour. This is less clear to me however. 

I love love love that I can not think about unexpected exceptions from “act” in JUnit. My code template marks all tests with “throws Exception”. I teach this to everyone. We struggled for years with “it is a failure or an error”? When “act” throws an exception, it’s just a failure; when “arrange” throws an exception, it’s a precondition violation. I’ve generally seen it cause more trouble than it saves to try to differentiate more finely than that.

-- 

[J. B. Rainsberger]

On Tue, Aug 29, 2017 at 10:48 AM, Michael Feathers <michael....@gmail.com> wrote:

 

Wish I'd seen this thread earlier. So many great ideas. I agree strongly with the notion of no individual tearDown.

On the one hand, vindication; on the other hand, I see Feathers’ posts on Facebook and have to wonder about HIM being the one to agree with me on this. :P

-- 

[Michael Feathers]

Great minds think alike ;)

--

[Nigel Thorne]

Michael's post triggered a thought ...

I'd like to explore the idea of putting the test code in the same assembly as the code under test. 

I just like the idea of a dll being self contained. 

I currently don't because I don't want my production assemblies referencing 3rd party test assemblies... but even that doesn't seem that important. 

Also we could design the runners so they require conventions, not inheritance. If any class that ducktypes to a specific interface is treated like a test-fixture, then I don't need to couple my code to a test framework at all. If returning a list of strings was good enough to represent the 'errrors', then you wouldn't need an assertion library...  It feels like there is something here.... but I can't factor out the value yet.   

[Rob Myers]

Hi all Lonely Coaches! 

[First, apologies for resuscitating this thread if it's now obsolete. I feel like I might be picking at an old scab...  During the switch from Agile Institute to Agile for All I managed to miss changing the e-mail address for LCS. Today I woke up in a cold sweat thinking "What happened to LCS?!" ;-]

I wanted to clarify a couple of my opinions based on replies that I didn't see until now. I think it started with @AfterEach. Like JB, I see it as a smell suggesting the test is not a unit-test (not necessarily a *bad* smell, just a defining smell). James G. pointed out a great example of its usefulness, and I think it was Ron who suggested that the framework should be flexible enough to handle various types of testing. 

And I agree with all that, so to clarify my opinions:

1. I agree that features like @BeforeEach and @AfterEach should be available in these tools. What drives me nuts are the wizards that include all the options in a template, thus encouraging inexperienced developers to develop bad habits (e.g., VisualStudio.NET used to do this for MSTest templates. It would even provide a default constructor. A seriously bad place to put anything, from a readability standpoint, because you would have had to go research the tool to figure out whether the test class instance was reused or re-instantiated for each test!) 

I think we need to remember there are always folks who can craft elegant code with really simple tools, and those who will use every feature of a tool, initially, because they think they're "supposed to." (Or because "...a wizard did it!")

2. If we want a tool that can be used for a variety of purposes, so teams don't have to switch tools, should we not be looking more closely at the Cucumber family? Gherkin, to rule them all, and in the glue-code, bind them? 

I admit I'm resistant to the idea of using Cukes for microtesting--for a variety of reasons--but I've watched teams really blur the dividing line with great success. And, when done well, Cukes are readable by a broader audience. By the way, those great teams would still often "drop into" a TDD cycle using their microtesting tool. What's interesting to me are the reasons why they felt Gherkin was not appropriate for those tests.  

Consider: There's either a cognitive context switch between Cucumber and _Unit, or there's a cognitive context switch between Gherkin and the development language.  I'm not going to guess (or dictate to teams) which one is harder. Good teams seem to find their own thresholds. 

It's been a little humbling (aw, who am I kidding?! It's been VERY HUMBLING) to no longer be able to show off my mad developer skills in coaching and training circumstances. I find I'm focusing more, these days, on coaching/mentoring overall good test-first "patterns" that apply to all the tools and languages; Gherkin, _Unit, Jasmine...

Maybe the guideline for the tool builders is this: Tools can't--and shouldn't even try to--dictate technique. 

And, for the practitioner: tools aren't a substitute for good technique.

3. [And I can't recall what part of the thread prompted #3] I also appreciate tools like Jasmine/Mocha, and now JUnit5, which allow embedded describe() blocks, each with their own beforeEach(). Great for reducing duplication in "given" and even "when". 

At first it felt a little too clever, like I was using a Composite Pattern to reduce duplication across only two levels. But I did end up, quickly and naturally, with three levels of describe() blocks, and when I re-read the specs, they are quite nice. 

The big win, as far as I can tell, is that the beforeEach() block is always very close to the small handful of specs that require it. Very readable, particularly to those who grok JavaScript. (I'm still playing with the similar JUnit5 feature, but I'm seeing similar results so far.)

Good to be back!

Rob

@agilecoach [thankfully my Twitter handle hasn't needed to changed...yet]

[nuni...@gmail.com]

Hi Mike,

For my part, I wish I had replied to your post sooner... oh well.

I used CppUnitLite in some of my C++ coaching work. I had read your blog post about it and I used it as you describe, as a core which the team modified only as needed. It has an interesting effect on a team to work that way. Teams whose primary job is to build a framework sometimes tend to gold-plate that framework, imagining that users will want various things. I never saw that happen with the teams that were __using__ CppUnitLite. They had other priorities... like actually getting their software out.

I named NUnitLite as a hat-tip to CppUnitLite and followed up the same way with TC-Lite. TC-Lite will be simpler than NUnitLite but probably a bit more complex than CpUnitLite, which I think is inevitable if it is productized.

I'm leaning toward the "make it very difficult" to do bad things approach by the way. Actively preventing them doesn't appeal to me. As a tool builder, I don't think I have a contract with my users to __make__ them do anything. However, I can definitely be clear about what I will and won't support.

Sometime "soon" I'll do a little article on what I put in and what I left out.

Charlie