Questions about Testing Frameworks

[Paulmichael Blasucci]

At this point, there are many different tools for testing. Though it seems FsCheck is in a bit of a niche. Meanwhile, are there any serious feelings (i.e. anecdotal evidence) about choosing between Unquote, NUnit, and xUnit? Also, what's the story with FsUnit? Further more, are mocks considered essential or more of a special-case tool? I'm asking as part of an effort to gather up a collection of recommended practices for F#. Options (especially from professional scenarios) are highly welcomed. Thanks!

[Anton Tayanovskyy]

There's the unbeatable Fuchu - https://github.com/mausch/Fuchu --A

> --
> --
> You received this message because you are subscribed to the Google
> Groups "FSharp Open Source Community" group.
> To post to this group, send email to fsharp-o...@googlegroups.com
> To unsubscribe from this group, send email to
> fsharp-opensou...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/fsharp-opensource?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "FSharp Open Source Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to fsharp-opensou...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Kind Regards,
Anton Tayanovskyy

WebSharper™ - type-safe JavaScript in F#
http://intellifactory.com

[Mauricio Scheffer]

My two cents:

Test frameworks like xUnit, NUnit, etc provide two different sets of features: assertions and test organization. Test organization using attributes is quite inflexible compared to modeling tests as first-class values, which is the approach I took with Fuchu. I wrote about first-class tests in this series of posts:

 * http://bugsquash.blogspot.com/2012/05/first-class-tests-in-mbunit.html

 * http://bugsquash.blogspot.com/2012/05/f-dsl-for-mbunit.html

 * http://bugsquash.blogspot.com/2012/06/fuchu-functional-test-library-for-net.html

Anton also wrote about this before (which I seemed to have missed while writing my posts): http://t0yv0.blogspot.com/2011/08/minimalistic-unit-testsing-in-f.html , you can see this in action in the WebSharper tests (e.g. https://github.com/intellifactory/websharper/tree/master/IntelliFactory.JavaScript.Tests )

Assertions: FsUnit and Unquote fall into this category. 

I used to use FsUnit, but then I came to generally dislike DSLs that want to look like English as they are more verbose for no real benefit or even actually obfuscate things.

Unquote looks interesting, unfortunately I don't have any first-hand experience with it. Also I'm always a bit wary of expression evaluators, i.e. what if I feed it some expression it doesn't support?

Anyway I tend to be pretty minimal about assertions, and sometimes a pattern match is just simpler than calling an assert function.

Mocks are mostly useful if you have code with side-effects. Since side-effects should be minimized, mocks shouldn't be common. You might want to check out Phil Trelford's Foq ( http://foq.codeplex.com/ ), which is sort of like Moq but for F#. The Foq homepage also has links to lots of articles about mocking.

I've written about my thoughts on mocks in http://bugsquash.blogspot.com/2012/05/moroco-minimal-mocking-library-for-c.html. In a nutshell, I don't think mocking libraries are worth their complexity, especially since F# has object expressions.

FsCheck is pretty great, and I'm constantly feeling I'm underutilizing it. I don't think it's as niche as it may seem, it just takes some time getting used to. Even if you can't think of any meaningful properties, you can use it to check for corner cases you might have missed in your code which throw unwanted exceptions. Model checking is also very useful (I blogged about it in http://bugsquash.blogspot.com/2013/06/optimizing-with-help-of-fscheck.html ). Jack Fox has a great blogpost about case classification and model testing with FsCheck: http://jackfoxy.com/gaining-fscheck-fluency-through-transparency/

For BDD-style testing there's Phil Trelford's TickSpec ( http://tickspec.codeplex.com/ ) and Steffen's NaturalSpec ( https://github.com/forki/NaturalSpec ). I'm generally not a fan of BDD but maybe that's because I've never worked in an environment where non-technical stakeholders could (or wanted to) write specs in Gherkin.

I hope I didn't miss any general-purpose F# testing tools! Anyway, none of this is very specific of F# or even .NET, e.g. FsCheck and Fuchu are usable in C#/VB.NET and are based on concepts borrowed from Haskell. And there's a project similar to Unquote for C# ( https://github.com/PowerAssert/PowerAssert.Net ), and both are based on a testing library for Groovy. Also regardless of the language, mocks are mocks, BDD is BDD, randomized testing, fluent interfaces, first-class tests are the same thing, just that some have a slightly specific implementation/execution in F#, but in general, "recommended practices for F# testing" isn't very meaningful IMHO.

Cheers,

Mauricio

[Phillip Trelford]

What Mauricio said :) and...

Using an attributed based .Net unit test framework like NUnit or xUnit is particularly useful when building in a continuous integration environment like TeamCity.

Fluent assertion libraries are a matter of taste, FsUnit and Unquote are both great if that's your thing.

I've found mocking libraries useful in mixed C# and F# projects, or when testings C# from F#, and you want to mock larger interfaces. For smaller interfaces object expressions are sufficient.

You can mock functions via partial application, no frameworks, see http://trelford.com/blog/post/Stubs.aspx

[Paulmichael Blasucci]

Anton, Mauricio, Phil,

Thanks for the feedback. I suspected this might be a highly subjective topic. :-) 

In the end, I'm creating a sample solution. So, I'll probably just use NUnit (to show calling a test-runner from FAKE) and FsCheck (because I think it's cool). That ought to be sufficient for demonstration purposes.

Cheers!

[Jack Fox]

What everyone said, and...

1) I use NUnit because it seems to be the most common test framework, I like it's external runner, and I've gotten accustomed to it.

2) I use FsUnit for one big thing: piping and fluent style so I don't have to write clunky assert statements.

3) I find Unquote fascinating, but I have yet to have a real need for it. Still, I put it in my test project template. http://visualstudiogallery.msdn.microsoft.com/a52388eb-e1d3-4900-a25a-d18c8d23a1f3

4) FsCheck is my big "go to" tool. Always feel like I don't know it well enough. I plan to publish more usage examples in my project template in a couple of weeks. Really needs full API documentation so it's not such a research project for new (and even experienced) users. (Yeah, this is easy to point out, but where's the action on my part...well earliest would be late December. I'm also wondering how far Tomas is going in standardizing API docs through FSharp.Formatting.)

5) I've overlooked Fuchu. I need to visit it.

6) Canopy is interesting... Very well-documented API. Still looking for the right combination of needs to use it.

7) TickSpec...always feel like I'm right around the corner from diving into it. Want to publish examples in my template. Like FsCheck needs great docs.

6) Personally uninterested in tools geared towards testing C#.

[Paulmichael Blasucci]

Thanks, Jack. Good feedback. Also, re: FSharp.Formatting, it is making huge gains (based on commits from ~10 hours ago) in terms of being an excellent tool for API docs. (And it's already a great tool for narrative explanations). 

[Paul Blair]

On Wednesday, October 23, 2013 2:37:57 AM UTC-4, Mauricio Scheffer wrote:

Mocks are mostly useful if you have code with side-effects. Since side-effects should be minimized, mocks shouldn't be common. 

I'd like to offer a different perspective on this. One of the interesting frameworks available for Clojure is midje, a functional mock testing library by Brian Marick. His approach is influenced by a general desire to do test-driven development by Abelson and Sussman's idea of "programming by wishful thinking." In SICP Abelson and Sussman often talk about approaching design by starting with the problem you wish to solve, and designing the solution by assuming that you have certain functions at your disposal that will perform computations at a different layer of abstraction. In midje's TDD approach, the tests supply mock implementations for the functions you wish you had, so that you don't have to implement those functions before you test your high-level code. This allows the developer to start with the problem they are trying to solve and work inwards by implementing the details.

Midje's approach is designed to work with functions whether or not they have side-effects. The rationale has more to do with a strategy for how to design, implement and test a complex computation that has to be built up of many parts, not all of which can be implemented at the same time.

I've toyed with the idea of trying to implement midje in F#, but so far I'm not having much luck with taking on any side projects.