Asynchronous support

[Matt Ellis]

Hi folks. We're using SpecFlow against a Silverlight app, and it's
working rather nicely. But we're running a custom build. Some of the
smaller changes have already made there way into the master, and I've
just issued a pull request with the others, but there's a really big
change that we require that I'd love to see integrated into the
official build - support for asynchronous tests.

Since Silverlight is inherently asynchronous (stuff like web downloads
can only be done asynchronously, and we need tests to be able to wait
for this to complete, and everything's blocking on the UI thread and
it all gets a bit hairy).

There already is an AsynchSilverlight branch, but it hasn't been
touched since Aug, and never made it into master. We've taken this and
built on it, and it's now something we're happy with.

I'd love to get this into the official build, if only that we don't
have to maintain a separate build and keep it in sync with master
(which it's not right now!). Is this something you'd like to see in
the official build? If so, how do we go about getting it in?

I'm a little wary of just sending a pull request, because it's quite a
large change, and while it's only supported by the silverlight runner
(and framework), it affects the main codebase (the implementation
should support other frameworks, but this hasn't been attempted).

Would it be helpful if I describe the approach and the changes?

Thanks
Matt

[Gáspár Nagy]

Yes! Let's solve this problem, that is pending for such a while.

If you can clean up the solution, please do so and send a pull
request. Please also send a brief description with it.

We can still refactor the solution later, when we better see how this
could be supported by other frameworks.

Thx,
Gaspar

[Matt Ellis]

Cool. I should be able to get a pull request sorted this week. Is
there a tag for the new 1.6 release? I'd like to build against that so
I can start reducing the custom bits in our private build - would like
to be able to get it down to just the generator and silverlight
runtime.

Also, I doubt this needs to be said, but just to be crystal clear -
this change is about asynchronous support, not concurrency. This won't
add the means to run tests in parallel.

Cheers
Matt

[Gáspár Nagy]

The tags should be there now. I'm looking forward to merge that in. I
did not plan to touch the generator part in the upcoming weeks, so the
merge should be easy.

[Town]

Hi Gáspár / Matt,

Is there any news on when asynchronous support might be available?
We've reached something of an impasse trying to get our tests to work
when waits for service calls are required - we're using Silverlight
and Caliburn.Micro.

I'm new to BDD but SpecFlow looks like an ideal fit for what we want
to do, if we can just get around this async issue!

Cheers,
Chris

[Matt Ellis]

Sorry, no progress lately (busy at work, plus off on holiday this week). We badly want to upgrade at work, too, so we get the goodness of improved intellisense. I'll put the effort in and get it into a pull request by the end of next week at the latest.

That said, I've been having second thoughts about the API. It's very imperative and very much based on the Silverlight unit testing framework. Something based on iterators or the reactive framework's schedulers might be a better choice. But I'll get the current design checked in and we can take it from there.

[nitro52]

Hey,
I've spent the last week trying to come up with a acceptance testing
solution for our silverlight project. What we discovered is that if
you do a web call to a service on a worker thread it is marshalled to
the UI Thread, waiting the main thread would lock it so the response
from the network call was being block creating a deadlock. After a lot
of research it would seem that the Silverlight Unit Test framework
gets around this with a queuing type system but it wasn't going to be
compatible with specflow definitions. I was going to try and reproduce
the behaviour but i found this post and it seems like you guys are
pretty close to a solution. How is this going? is there a branch i
could download to do a some testing with? if so can you briefly
explain how this should work? I did download the branch that you guys
seem to be working on but i can't see an issue that describes how it
is meant to work.

Regards
Ryan

On Apr 28, 6:27 am, Matt Ellis <m.t.el...@gmail.com> wrote:
> Sorry, no progress lately (busy at work, plus off on holiday this week). We badly want to upgrade at work, too, so we get the goodness of improved intellisense. I'll put the effort in and get it into a pull request by the end of next week at the latest.
>
> That said, I've been having second thoughts about the API. It's very imperative and very much based on the Silverlight unit testing framework. Something based on iterators or the reactive framework's schedulers might be a better choice. But I'll get the current design checked in and we can take it from there.
>

[Matt Ellis]

Hi folks. Sorry for the delay in this, but I've just pushed the
changes to my fork in github (http://github.com/citizenmatt/SpecFlow).
It's on the Async2 branch, and is built on top of the 1.6.1 release
(meaning it doesn't include any of the code that's been merged into
master since the 1.6.1 release). I'll get a pull request together
shortly.

There's nothing special you need to do to build, and there's a
Runtime.Silverlight.Async silverlight test project with a couple of
scenarios to exercise the async stuff. It updates both the generator
and the runtime, but there should be no breaking changes with existing
feature files. The simplest thing is to update all of the dlls in your
Program Files\TechTalk\SpecFlow folder, and reference
TechTalk.SpecFlow.Silverlight3.dll in your silverlight project.

Using the async stuff is very straightforward, but it's probably best
to give a little background first.

The async support is NOT parallel support. Your steps will still
execute serially, one after another. The async support here allows
your test to give up control flow, and allow other operations to
execute before continuing.

The support is also runtime specific. For Silverlight, the test
framework will run your tests on the UI thread, which can cause
deadlocks if you want to use web requests (requests and responses are
bizarrely marshalled onto the UI thread). Async support allows your
test to give control back to the runtime to let other operations to
continue before you get called back again.

For the most part, this is handled transparently. Each SpecFlow step
is run in its own async context - as a separate work item. After each
step, control is given back to the system, and other async operations
have the opportunity to continue or complete.

When you want to wait for an async operation to be complete, or to
sleep for a while, call the Enqueue methods on AsyncContext. You can
get an instance of AsyncContext via AsyncContext.Current (I tend to
wrap these in an AsynchronousSteps base class that derives from Steps.
Might add that, actually)

* EnqueueDelay will perform a non-blocking sleep. e.g.
context.EnqueueDelay(TimeSpan.FromSeconds(2))
* EnqueueConditional will perform a non-blocking wait until a
condition is true. e.g. context.EnqueueConditional(() => flag == true)
* EnqueueCallback will schedule a simple action to be performed after
the currently enqueued actions are done. e.g.
context.EnqueueCallback(() => doSomething());

When you enqueue a work item, it gets added to the queue to be
executed after the current step, BUT YOUR STEP WILL CONTINUE
EXECUTING!

E.g.

public void When_I_do_something()
{
// e.g. a web request
StartSomethingAsync();

// Enqueue a non-blocking wait for the async op to complete
EnqueueConditional(() => myAsyncSomethingHasCompleted == true);

// WARNING! The operation hasn't completed yet!
// UseResultOfAsyncOperation();

// Enqueue an action for the result. This will only get called
once
// myAsyncSomethingHasCompleted is true
EnqueueCallback(() => UseResultOfAsyncOperation());
}

Similarly, if you call a step from within a step definition, then it
isn't called immediately, it is enqueued to be run at the end of the
currently executing step (and in fact, after any steps that have
already been enqueued).

To summarise:

1. Control flow is relinquished after each step
2. You can sleep using EnqueueDelay
3. You can perform a non-blocking wait until something is true using
EnqueueConditional
4. You can enqueue further work items using EnqueueCallback
5. Given/When/Then are enqueued as new async work items
6. Any code written after a call to EnqueueXXX or Given/When/Then will
execute BEFORE the enqueued work items

Let me know how you get on. It's not as complex as it might seem,
honest.

Thanks
Matt

[nitro52]

Thanks i'll check this out today and tomorrow if if i run out of
time.
Just had a couple of thoughts that i'm not sure are covered here yet.

1. Is there a way to enter a timeout for your tests. you wouldn't want
something going wrong and never getting a call back from a service
because your other tests will never get run.
2. Will this work for Windows Phone? The same principles probably
apply. thinking about it would you have the same sort of issues in
something like WPF where you are making a Async call to a web service?
could the same system be a generic Async Solution?

[Matt Ellis]

1. Timeouts aren't handled out of the box. The silverlight testing
framework handles them using mstest's TimeoutAttribute, and checking
the timeout in between each async work item. But this can't be applied
to the SpecFlow tests. You can get a similar affect by rolling your
own. The predicate passed to EnqueueConditional is called multiple
times until it returns true. It's straight forward to add a timeout in
there (and I've done so with our AsynchronousSteps base class). It's
probably worth adding an overload to allow passing in a predicate and
a timeout, although (and I'll look to see if the AsyncSteps base
class should be included, too).

2. There's no reason this shouldn't work on the phone - I just didn't
add the async context implementation to the phone runtime project.
Since they're using the same testing toolkit it should just work. I'll
add it and test it (although that will probably be next week
sometime). It actually is a (fairly) generic async solution, with most
of the implementation in the base runtime, but it does require support
from the runtime provider and the underlying testing framework.

Cheers
Matt

> > > > >>>>> large change, and while it's...
>
> read more »

[nitro52]

Ah that's true it does relay of the underlying Silverlight async, at
least having the support for Silverlight and WP7 would be good. I was
to busy to take a look at it today so i'll take a look tomorrow. From
what i read it sounds similar to the stuff i read about the async
process in the silverlight testing framework so i think i understand
how it works. Is there an easy way to build the specflow dll's into
the same output folder like what you would see in the installation
folder or do you just have to manually copy them?

> ...
>
> read more »

[Matt Ellis]

I just copied them manually, which isn't great. Should have included a
batch file in the pull request!

And yes, the implementation of async is VERY much like the Silverlight
stuff, 'cos that's the only framework I know that either has or needs
async support. So the concepts (and method names...) are just copied
over.

> > > > > > >>> I can start reducing the custom bits in...
>
> read more »

[nitro52]

I noticed my code behind wasn't being generated correctly because in
the sample you changed the provider to use an Async specific one. Will
this be merged into the silverlight provider? I had to change the
config file to

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<section name="specFlow"
type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler,
TechTalk.SpecFlow"/>
</configSections>
<specFlow>
<!-- <unitTestProvider name="MsTest.Silverlight" /> -->
<unitTestProvider name="mstest.silverlight4.async"/>
</specFlow>
</configuration>

I'm also working on a blog post documenting the process so if anything
changes can you let me know. The blog post is
http://rburnham.wordpress.com/2011/05/13/testing-silverlight-asynchronous-code-with-specflow/

I've created a small spiking project and it all seems to work fine.
After i finish documenting it i'll try it on our actual silverlight
project to test it with some actual web calls. Its easy to use which
is good. Just a matter of understanding the queuing system.

> > > > > > > > I'm new to BDD but SpecFlow looks like an ideal fit for what we want...
>
> read more »

[Matthew Ellis]

Great blog post!

It's a shame this is so complex to explain, because using it is actually pretty easy, and most of the time you can ignore calling into the AsyncContext. Each step is already enqueued to run asynchronously, so you only need to touch the AsyncContext if you're waiting for something, or if you have a much more complex step definition. Looking at the last code snipped in your blog post (the step definitions written to use the async stuff) I'd write it like this:

[Binding]

public class StepDefinitions

{

  [Given(@"I calculate")]

  public void GivenICalculate()

  {

    Calculator.Calculate();

    AsyncContext.Current.EnqueueConditional(() => Calculator.Result > 0);

  }

  [Then(@"the result should be 120")]

  public void ThenTheResultShouldBe120()

  {

    Assert.AreEqual(120, Calculator.Result);

  }

}

Which has much less async noise in it. We've found that unless you're actually waiting for an async operation to complete, you (generally) don't need to add any calls to Enqueue. Essentially, the framework is doing the equivalent of calling each step inside a call to EnqueueCallback (this is over-simplifying, but it's close enough).

I think it might take a few attempts to figure out the right level to pitch the docs at. While the support is modelled on the Silverlight unit testing framework's implementation, it shouldn't be required to understand the Silverlight model to understand the SpecFlow implementation.

It boils down to this: SpecFlow will queue each step to run asynchronously. So each step is a discrete operation that is guaranteed to execute after the previous step and before the next step, but with the possibility of other asynchronous operations (such as UI or web callbacks) to run in between each step. The AsyncContext is there primarily to allow you to wait for one of these other async operations to complete (EnqueueConditional), but it's also to allow for more complex async scenarios (steps) where you need to queue multiple operations, conditions or delays.

That said, nice blog post - you can't beat a good example.

(Oh yeah, and I forgot about having to change the generator in the config - sorry!)

Thanks

Matt

[nitro52]

Ah thats good to know, in the Silverlight framework I'm pretty sure
you needed to enqueue anything after an async call, i'll give this a
go tomorrow. It sounds like you would only need to call an enqueue
method within a step definition that has already enqueued another
async process. I'll have to amend the blog post, maybe i'll split it
in two. The main reason for that doc was for my understanding of how
it all works. after having a go at it I'm sure we can simplify it.

One last thing, After you mentioned the wrapper class i could see how
something Step.EnqueueCallback() could be nice and short but its not
to bad the way it is.

> On 16 May 2011 04:01, nitro52 <nitr...@iinet.net.au> wrote:
>
>
>
>
>
>
>
> > I noticed my code behind wasn't being generated correctly because in
> > the sample you changed the provider to use an Async specific one. Will
> > this be merged into the silverlight provider? I had to change the
> > config file to
>
> > <?xml version="1.0" encoding="utf-8" ?>
> > <configuration>
> >  <configSections>
> >    <section name="specFlow"
> > type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler,
> > TechTalk.SpecFlow"/>
> >  </configSections>
> >  <specFlow>
> >    <!-- <unitTestProvider name="MsTest.Silverlight" /> -->
> >    <unitTestProvider name="mstest.silverlight4.async"/>
> >  </specFlow>
> > </configuration>
>
> > I'm also working on a blog post documenting the process so if anything
> > changes can you let me know. The blog post is
>

> >http://rburnham.wordpress.com/2011/05/13/testing-silverlight-asynchro...

> ...
>
> read more »

[Matthew Ellis]

In the Silverlight framework, you do need to enqueue everything after an async call, but because every step definition is already enqueued, the SpecFlow framework is doing this for you.

"It sounds like you would only need to call an enqueue
method within a step definition that has already enqueued another
async process"

Yep. Exactly.

The base class is pretty useful - I'll dig it out tomorrow and add it to the pull request.

[nitro52]

I've updated the blog to reflect the way it specflow executes the step
definitions and added a diagram of it. Feel free to use any content
from it.

On May 16, 10:25 pm, Matthew Ellis <m.t.el...@gmail.com> wrote:
> In the Silverlight framework, you do need to enqueue everything after an
> async call, but because every step definition is already enqueued, the
> SpecFlow framework is doing this for you.
>
> "It sounds like you would only need to call an enqueue
> method within a step definition that has already enqueued another
> async process"
>
> Yep. Exactly.
>
> The base class is pretty useful - I'll dig it out tomorrow and add it to the
> pull request.
>

> ...
>
> read more »

[Gáspár Nagy]

This is really a great post. (Now I understand much better this
feature. :)

Sorry for the delay in merging the pull requests. I was just very
busy. I hope to do it soon.

Br,
Gaspar

> ...
>
> read more »

[Jimmy Li]

Matt,

I've downloaded the latest Async2 branch, built the solution, and I
see the tests in AsyncSteps.feature.cs are decorated with
[Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute()].
But those tests are not detected by MSTest (in Test > Windows > Test
View > it says "Currently no tests are available to display in Test
View"). What can I do for MSTest to see those tests?

Thanks
-Jimmy

> ...
>
> read more »

[nitro52]

I beleive this is the way silverlight unit tests show in Visual
Studio. The Silverlight Unit tests are not really supported the same
way a normal .net unit test are. The Silverlight unit test project is
actually just a silverlight application, all the unit tests are
executed within that application. You just need to run the project as
though you would run a normal silverlight application. If its hosted
by an asp.net page you may need to set the default page to the
silverlight unit test project. I've read ways that the normal
silverlight unit test framework can be run via command line but i'm
yet to test this with specflow.

> > > > > > > > > > > 1. Is there a way to...
>
> read more »

[Stuart Caborn]

We run our tests using StatLight which provides a command line wrapper
over the silverlight unit test framework. It works well with specflow
tests and has a teamcity mode which allows CI integration.

So you will lose your IDE support but you do gain a continuous test
runner on the command line which is kind of nice.

Stu

[Gáspár Nagy]

FYI: I have merged the pull requests to the AsyncSilverlight2 branch
(Feature_AsyncSilverlight2 on the build server).

The default merging caused some troubles, so I had to merge the
changes manually. Please start from this merged branch if you need to
change anything. (I also did some minor code structure changes.)

The test is currently not added to the solution, because the build
server cannot build it. I'll fix it later.

On May 19, 8:55 am, Stuart Caborn <stoofe...@gmail.com> wrote:
> We run our tests using StatLight which provides a command line wrapper
> over the silverlight unit test framework.  It works well with specflow
> tests and has a teamcity mode which allows CI integration.
>
> So you will lose your IDE support but you do gain a continuous test
> runner on the command line which is kind of nice.
>
> Stu
>

> ...
>
> read more »

[Gáspár Nagy]

While merging, I was thinking over this feature once more. I have two
smaller ideas to discuss.

1. I'm a bit concerned, that the "async" aspect is combined with the
unit test provider (i.e. you need to use the name
"mstest.silverlight3.async"). While I see that the async has a
relation to the unit test provider, I more feel it orthogonal.
Especially since the possibility of the async execution is now built
in into the core runtime, so hopefully we can make the normal .NET
test runners also async-compatible.

If I need to find an analogy for such orthogonal feature, the
"rowtest" comes to my mind. Some providers support row test, some not;
sometimes you want to use row tests, sometimes not. This would fit to
async as well imho. Following this, maybe the "async" should be rather
an explicit setting in the generator config.

2. The other thing that made me thinking was the "AsyncContext". It is
maybe the name, but it took me some time to realize what it is for
(and especially what is the relation of it to the scenariocontext). My
first idea was to put it "under" the scenario context
(ScenarioContext.Current.Async.EnqueueCallback(...)) but that feels
too long. An other way would be to not provide it as a static
accessor, but one could inject it to the step definition, but this is
maybe too hard to document.
My final alternative for the asynchcontext was to define it as a set
of extension methods for the scenario context (in a namespace
TechTalk.SpecFlow.Async or so). This way, if you have the namespace
using for TechTalk.SpecFlow.Async, you could write:
ScenarioContext.Current.EnqueueCallback(...) which is better
describing the intention imho (i.e. you want to to enqueue a callback
for the current scenario).

What do you think?

[nitro52]

I still like the idea of wrapping it in a step class so its like
Step.EnqueueCallback()
I guess it makes sense to add it to the scenario context though.

[Matt Ellis]

I don't see any reason why the async support couldn't be a feature of
the normal generators, just like the row test support. The only thing
to note is that it's all or nothing - all of your features will be
generated as async, you can't pick and choose which ones (the same as
with row tests).

The name for AsyncContext is really due to FeatureContext and
ScenarioContext, in that these are the main entry points for the
public API you use in your tests - if you need to do something, you go
to one of these Context classes. Similarly, to do Async stuff, you go
to the AsyncContext class. I can't find a good argument for or against
putting it on to ScenarioContext, but my preference is to have a
separate object. We already extend ScenarioContext in our tests, and I
feel maybe adding these methods would start to make it too much of a
dumping ground? Fairly weak argument, I know...

Also, I'd like to evolve this so that AsyncContext wasn't the main
interface. I can see how it would instead be the low-level API that a
nicer API would use. For example, maybe steps could return
IEnumerable<something> so that they can do something async, yield
return a thing (not sure what, though) and the framework would use the
AsyncContext functionality (wherever it would live) to enqueue
callbacks and conditionals. Then taking this further, it could be
integrated with the new C# async stuff. Then the AsyncContext would be
more hidden away.

On May 19, 4:20 pm, Gáspár Nagy <gaspar.n...@gmail.com> wrote:
> While merging, I was thinking over this feature once more. I have two
> smaller ideas to discuss.
>
> 1. I'm a bit concerned, that the "async" aspect is combined with the
> unit test provider (i.e. you need to use the name

> "mstest.silverlight3.async"). While I see that theasynchas a

> relation to the unit test provider, I more feel it orthogonal.

> Especially since the possibility of theasyncexecution is now built

> in into the core runtime, so hopefully we can make the normal .NET
> test runners alsoasync-compatible.
>
> If I need to find an analogy for such orthogonal feature, the
> "rowtest" comes to my mind. Some providers support row test, some not;

> sometimes you want to use row tests, sometimes not. This would fit toasyncas well imho. Following this, maybe the "async" should be rather

> an explicit setting in the generator config.
>
> 2. The other thing that made me thinking was the "AsyncContext". It is
> maybe the name, but it took me some time to realize what it is for
> (and especially what is the relation of it to the scenariocontext). My
> first idea was to put it "under" the scenario context
> (ScenarioContext.Current.Async.EnqueueCallback(...)) but that feels
> too long. An other way would be to not provide it as a static
> accessor, but one could inject it to the step definition, but this is
> maybe too hard to document.
> My final alternative for the asynchcontext was to define it as a set
> of extension methods for the scenario context (in a namespace

> TechTalk.SpecFlow.Asyncor so). This way, if you have the namespace

[Matt Ellis]

I've just had a look at our AsynchronousSteps base class. I'd
forgotten that it's doing a bit more than just calling into the
AsyncContext methods. There are a couple of convenience methods that
enqueue a reactive framework IObservable, and enqueues a wait until it
gets called back.

But the really interesting methods are EnqueueCallback and
EnqueueConditional.

The big problem we found with the async stuff was that when an
enqueued operation threw an exception, the stack trace didn't have any
record of the currently executing step, making it hard to figure out
the context. (The test method enqueues each step, which get run by the
scheduler (Silverlight's Dispatcher) AFTER the test method has
completed. If a step throws an exception, it doesn't have the test
method in the stack. Similarly, if the step enqueues any operations,
they execute after the step is finished, and they won't have the step
in the call stack.) To get around this, our Enqueue helpers grab the
current stack and creates a new delegate that gets enqueued. This
delegate catches any exceptions and wraps them in a new
EnqueuedOperationsException that presents the captured stack trace
(which includes the step information) and has the real exception as
the InnerException.

On top of that, the enqueue conditional has an overload that takes a
timeout, and throws a TimeoutException if the condition isn't met
before then. The method without the overload has a default timeout of
1 minute.

We added this as user code, rather than into the framework because it
makes some assumptions that might not best suite all users, e.g. the
default timeout of 1 minute. Works for us, but perhaps you'd like it
to be higher? And the stack trace capturing and munging is tailored to
our needs (we also strip out non-user code frames based on our
namespace).

Do you think they'd be appropriate in the main codebase?

On May 19, 10:37 pm, nitro52 <nitr...@iinet.net.au> wrote:
> I still like the idea of wrapping it in a step class so its like
> Step.EnqueueCallback()
> I guess it makes sense to add it to the scenario context though.
> On May 20, 1:20 am, Gáspár Nagy <gaspar.n...@gmail.com> wrote:
>
>
>
>
>
>
>
> > While merging, I was thinking over this feature once more. I have two
> > smaller ideas to discuss.
>
> > 1. I'm a bit concerned, that the "async" aspect is combined with the
> > unit test provider (i.e. you need to use the name

> > "mstest.silverlight3.async"). While I see that theasynchas a

> > relation to the unit test provider, I more feel it orthogonal.

> > Especially since the possibility of theasyncexecution is now built

> > in into the core runtime, so hopefully we can make the normal .NET
> > test runners alsoasync-compatible.
>
> > If I need to find an analogy for such orthogonal feature, the
> > "rowtest" comes to my mind. Some providers support row test, some not;
> > sometimes you want to use row tests, sometimes not. This would fit to

> >asyncas well imho. Following this, maybe the "async" should be rather

> > an explicit setting in the generator config.
>
> > 2. The other thing that made me thinking was the "AsyncContext". It is
> > maybe the name, but it took me some time to realize what it is for
> > (and especially what is the relation of it to the scenariocontext). My
> > first idea was to put it "under" the scenario context
> > (ScenarioContext.Current.Async.EnqueueCallback(...)) but that feels
> > too long. An other way would be to not provide it as a static
> > accessor, but one could inject it to the step definition, but this is
> > maybe too hard to document.
> > My final alternative for the asynchcontext was to define it as a set
> > of extension methods for the scenario context (in a namespace

> > TechTalk.SpecFlow.Asyncor so). This way, if you have the namespace

[Gáspár Nagy]

all-or-nothing aspect: I guess this is also like that with the special
unit test provider. I was also thinking on how we could eliminate
this, but if you don't want to make an extra partial class for each
async generated test (which i suppose), the only chance for making it
more granular is to use a special tag on the scenario (@asyncSteps?).
To be honest, I already feel guilty that we have hard-wired the
@ignore attribute, so I don't really like to have another special
attribute. Do we need this per-feature granularity in the
configuration?

If we go for the config setting, how should it be called? My
suggestion is "allowAsynchronousStepExecution" or
"allowAsynchronousTesting". What do you think?

AsyncContext: ok, I see the point for the separated object. Maybe my
problem is the name and that we have one more static accessor. The
"Step" class is also not too bad, but it does not say anything about
async and it implies a lot of static methods.

Let me list what we have already
1. we have Scenario/FeatureContext and these are kind of property bags
2. we have ITestRunner, which is a kind of execution engine (has the
GWT methods), though it is usually not called by the bindings directly
3. we have a "Steps" base class that exposes he test runner methods
4. in the SL testing framework, this is a base class (inside
SilverlightTest)

and we have the AsyncContext that have helper methods (e.g.
EnqueueCallback(...)) to execute step statements asynchronously...

I don't know. I think the most "safe" would be to have a base class
"AsyncSteps" as the primary API (for now) and maybe an
IAsyncStepExecutor (IAsyncStepRunner) interface as low-level API, that
you can use through feature injection (you have to make a ctor
parameter of type IAsyncStepExecutor in the step binding classes if
you don't want to derive from AsyncSteps.

Any other ideas?

(sorry for being so picky, but it is so hard to change an API after it
is once released. just tell me to stop if i go to your nerves.)

[Gáspár Nagy]

any ideas?

[Matthew Ellis]

Sorry for the late reply...

I agree on not-using-an-attribute. I don't like having to annotate a scenario that is supposed to be business oriented with something which is very much implementation specific. So, yeah, I think the best bet with this is (like row tests) everything's async or not.

As for config, while it fits with the other parameters, we're not really *allowing* to run steps asynchronously. It's more like enabling you to, or generating support for it. So maybe we should go with something like "enableAsyncSupport" or "enableAsynchronousSteps" (async is easier to spell!) or "generateAsyncTests"?

API-wise, an AsyncSteps base class is my favourite approach, and is the one we're actually using in our Silverlight project at work. But it shouldn't be the only method - I don't think it's a good idea to insist that everyone derives from a base class. I'm still fine with the AsyncContext class, to be honest. I'm less inclined to use feature injection because it's less discoverable - you have to know to set up your constructor right before you can get at it.

(And don't worry about talking this over, I'm with you - let's get the API into a good shape before it's released and frozen!)

[nitro52]

One structure i have seen before and in fact have started using is to
place your step definitions inside a class that resembles what that
step is doing, for example it relates to contacts so the class would
be ContactsStepDefinitions. This makes re using steps between features
easier to manage how ever in this case it doesn't make a lot of sense
to inherit from an AsyncBase class just for one of those steps. I'm
all for making the call to the enqueue methods shorter in some way
though.

[nitro52]

How is this going, what you guys were talking about was a bit over my
head but i was just wondering if you'd sorted it out. Has it been
merged into the main specflow branch?

[Gáspár Nagy]

From my point, the answer is quite profane... i was on vacation :) but
from next week on, i'll warm up this topic. It is not merged to the
main branch yet.

[Matthew Ellis]

Sounds good. We've been heads down finishing off the project we needed this for. Would be happy to pick this back up again.

Thanks

Matt

[evifra]

"So, yeah, I think the best bet with this
is (like row tests) everything's async or not. "

I like that idea, in my current project, we are working in Silverlight
(so most of the methods being called are async) and what we are doing
now is we encapsulate the all of our steps definition in
AsyncContext.Current.EnqueueCallback and
AsyncContext.Current.EnqueueConditional..

But if we make some settings adjustment in the config, (that we are on
Async), the generator will wrap the "Given", "When", "Then" in the
test methods with EnqueueCallbacks and EnqueueCondition, so that
there is no tedious work of enques inside the step definition,

I was thinking of a pseudocode something like this
AsyncContext.Current.EnqueueCallback(() => { testRunner.Given("There
is a task that I marked completed"); });
AsyncContext.Current.EnqueueConditional(NotDoneWithTheGivenStep("There
is a task that I marked completed"));
AsyncContext.Current.EnqueueCallback(() => { testRunner.When("I click
on the \"Complete Task\" button"); });
AsyncContext.Current.EnqueueConditional(NotDoneWithTheWhenStep("I
click on the \"Complete Task\" button"));

On Jun 2, 6:34 am, Matthew Ellis <m.t.el...@gmail.com> wrote:
> Sorry for the late reply...
>
> I agree on not-using-an-attribute. I don't like having to annotate a
> scenario that is supposed to be business oriented with something which is
> very much implementation specific. So, yeah, I think the best bet with this
> is (like row tests) everything's async or not.
>
> As for config, while it fits with the other parameters, we're not really
> *allowing* to run steps asynchronously. It's more like enabling you to, or
> generating support for it. So maybe we should go with something like
> "enableAsyncSupport" or "enableAsynchronousSteps" (async is easier to
> spell!) or "generateAsyncTests"?
>
> API-wise, an AsyncSteps base class is my favourite approach, and is the one
> we're actually using in our Silverlight project at work. But it shouldn't be
> the only method - I don't think it's a good idea to insist that everyone
> derives from a base class. I'm still fine with the AsyncContext class, to be
> honest. I'm less inclined to use feature injection because it's less
> discoverable - you have to know to set up your constructor right before you
> can get at it.
>
> (And don't worry about talking this over, I'm with you - let's get the API
> into a good shape before it's released and frozen!)
>

[Matt Ellis]

If I understand correctly, the code already does this. When generating with the async generator, all Given, When and Then steps called from the generated file are automatically enqueued, and each step is run asynchronously. If you use the Steps base class and call steps using the Given When and Then methods in the base class, these are also enqueued for you.

You only need to enqueue conditionals and callbacks which you want to "wait" on within a single step.

[Gáspár Nagy]

I did a bigger refactoring on the AsyncSilverlight2 branch to support
the mentioned things:
- configure the async behavior separately
<unitTestProvider name="mstest.silverlight3"/>
<generator generateAsyncTests="true"/>
- have a AsyncSteps base class
- keep the AsyncContext, but only as an accessor for the base
infrastructure
- currenly all these API are in the TechTalk.SpecFlow.Async namespace
(we can reconsider this once more)

Finally it was quite a big change and I was only able to do minimal
smoke testing (not with the real SL testing framework).

So if you have a little time, please download the build from
build.specflow.org (Feature_AsyncSilverlight2) and test it a little
bit.

Let me know if there is any problems, maybe we can do a release early
next week.

Br,
Gaspar

On Jul 8, 12:33 am, Matthew Ellis <m.t.el...@gmail.com> wrote:
> Sounds good. We've been heads down finishing off the project we needed this
> for. Would be happy to pick this back up again.
>
> Thanks
> Matt
>

[Stuart Caborn]

We are a bit stretched this week but we can probably test migrate to
your build early next week.

Stu

[Gáspár Nagy]

I think that's fine. I'll also do some testing in the meanwhile and
probably merge the branch back to the trunk (to perform other changes
with an easier integration).

I have also started to create a documentation page at:
https://github.com/techtalk/SpecFlow/wiki/Testing-Silverlight-Asynchronous-Code

Please feel free to extend/fix it.

@Ryan: could you please extract some parts from your post to the
documentation?

On Jul 13, 6:36 pm, Stuart Caborn <stoofe...@gmail.com> wrote:
> We are a bit stretched this week but we can probably test migrate to
> your build early next week.
>
> Stu
>