Testing thread concurrency with Spock

[groovybayo]

Sorry to double post this, but I figured it might be better to give
the full picture here, rather than directing to stackoverflow as I did
in my previous post

So here it goes...

Is there a spock equivalent of TestNG's @Test(threadPoolSize=n) that
will allow me test the execution of a test, with multiple threads
concurrently?

Basically, given a specification like so...

class SampleSpec extends Specification {
def "test concurrent access"(){
setup:
//do complex logic
expect:
//assert complex logic
}

}
What I want is a way to do this in spock, but with multiple threads
spawned concurrently to execute the test method. In TestNG, I could
easily achieve this by doing

@Test(threadPoolSize=10)
public void testMethod(){
//do complex logic and assertion

}

Thanks in advance.

[Peter Niederwieser]

See my answer on Stack Overflow.

Cheers,
Peter

[Glen]

I also have had the need for some concurrency testing so I tried to
implement this myself and have been banging my head on this for
awhile. I'm wondering if you could check my work as I'm having a hard
time trying to follow where it's getting caught up in the Spock code.

Here is my extension so far:

import org.spockframework.runtime.extension.ExtensionAnnotation;

import java.lang.annotation.*;

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD})
@ExtensionAnnotation(ConcurrentExtension.class)
public @interface Concurrent {
int value();
}


import
org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension;
import org.spockframework.runtime.model.FeatureInfo;

public class ConcurrentExtension extends
AbstractAnnotationDrivenExtension<Concurrent> {
@Override
public void visitFeatureAnnotation(Concurrent concurrent,
FeatureInfo feature) {
feature.getFeatureMethod().addInterceptor(new
ConcurrentInterceptor(concurrent));
}
}

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;

public class ConcurrentInterceptor implements IMethodInterceptor {
private final Concurrent concurrent;

public ConcurrentInterceptor(Concurrent concurrent) {
this.concurrent = concurrent;
}

public void intercept(final IMethodInvocation invocation) throws
Throwable {
List<Future<Throwable>> futures = new
ArrayList<Future<Throwable>>();

ExecutorService executor =
Executors.newFixedThreadPool(concurrent.value());

for (int i = 0; i < concurrent.value(); i++) {
futures.add(executor.submit(new Callable<Throwable>() {
public Throwable call() throws Exception {
try {
invocation.proceed();
} catch (Throwable t) {
return t;
}
return null;
}
}));
}

for (Future<Throwable> future : futures) {
Throwable throwable = future.get();
if (throwable != null) {
throw throwable;
}
}
}
}

Here is an example spec I've tried:

import com.entero.spock.Concurrent
import spock.lang.Specification

class ConcurrentSpecification extends Specification {
@Concurrent(2)
def "test"() {
when:
def a = "foo"
then:
a == "foo"
}
}

I'm not sure I have the interceptor quite right but it blows up in a
weird spot:

org.spockframework.util.UnreachableCodeError: verifyInteractions
at
org.spockframework.mock.DefaultInteractionScope.verifyInteractions(DefaultInteractionScope.java:
48)
at
org.spockframework.mock.MockController.leaveScope(MockController.java:
76)
at ConcurrentSpecification.test(ConcurrentSpecification.groovy:12)

I have also seen this:

java.util.NoSuchElementException
at java.util.LinkedList.remove(LinkedList.java:788)
at java.util.LinkedList.removeFirst(LinkedList.java:134)
at
org.spockframework.mock.MockController.leaveScope(MockController.java:
75)
at ConcurrentSpecification.test(ConcurrentSpecification.groovy:12)

I'm kind of at a loss since I don't really understand what the
MockController is trying to do. Thanks for any guidance you can
provide and apologies for my ignorance :)

[Peter Niederwieser]

Nice try! Unfortunately, I don't think it's currently possible to
write such an extension. One reason is that there is an assumption
that each feature method invocation happens on a new spec instance.

What you could do though is to write a class that's used from within a
feature method to run a block of code (closure) multiple times
concurrently. That's more flexible than an extension and almost as
elegant.

Let us know if you come up with something.

Cheers,
Peter

[groovybayo]

Peter - I'll sure appreciate some guidance as well. I did submit a
pull request a while back (https://github.com/spockframework/spock/
pull/2). My implementation just spawned a new thread in the
interceptor. I didn't get much time to work further on it, but I was
hoping to somehow introduce a cyclic barrier or the like to make sure
all the threads are kicked off at the same time to ensure true
concurrent nature.

That seemed to work for me. Taking thread dumps showed the right
number of threads were kicked off.

Now the question is, is that a wrong way of implementing it?

Please let me know.

I'll also start thinking about the using closures like you suggested
to Glen.

Thanks

[Glen]

Ah ok I was afraid that it would have to have a new spec instance for
each thread, in that case likely support would have to be added in the
runner class (Sputnik).

So for the idea of using a closure, I was kind of hoping I could still
use the when/then blocks, i.e.:

class ConcurrentSpecification extends Specification {
def test = {

when:
def a = "foo"
then:
a == "foo"
}

def "test concurrent"() {
// do something with test closure
}
}

Was that what you were thinking? Or are the when/then blocks only
available in the feature methods themselves?

Thanks,
Glen

[Glen]

Ok more of a brain dump here, so assuming I can't use the when/then
blocks I've come up with a mostly generic solution below, including
the CyclicBarrier idea from groovybayo. I think this should satisfy
my needs, what do you guys think?


class ConcurrentSpecification extends Specification {

def concurrent(int count, Closure closure) {
def values = []
def futures = []

ExecutorService executor = Executors.newFixedThreadPool(count)
CyclicBarrier barrier = new CyclicBarrier(count)

for (int i = 0; i < count; i++) {
futures.add(executor.submit(new Callable() {
public def call() throws Exception {
barrier.await()
closure.call()
}
}))
}

for (Future future: futures) {
try {
def value = future.get()
values << value
} catch (ExecutionException e) {
values << e.cause
}
}

return values
}

def "test concurrent"() {
def test = {
"foo"
}

when:
def results = concurrent(10, test)

then:
results == "foo,".multiply(10).split(',')

[groovybayo]

Glen - This almost looks complete to me. One question I have is what
happens when I want to test something that returns more than 1 value?
e.g. different methods of a service method for example

Service {
int method1()
List method2()
}

With this approach, I will I go about testing this?

def "sample test"(){
def test {
def s = new Service()
def intValue = s.method1()
def listValue = s.method2()
}

when:
def result = concurrent(2, test)

then:
//need to assert for intValue and listValue here
}

Also, how will this work with @Unroll? Will that still work without
jumping hoops?

Cheers

[Glen]

Typically in groovy you'd just return it as a map:

  def test {       def s = new Service()       def intValue =
s.method1()       def listValue = s.method2()

      [intValue: intValue, listValue: listValue]
  }
...
then:
result.intValue == blah

I'm not sure about the @Unroll, that's new to me I gotta go read about
it.

[groovybayo]

That should work then. What I was mentioning about unroll was if this
approach will still allow you to specify in the where clause e.g. this
example taken from spock's homepage


class HelloSpock extends spock.lang.Specification {
def "length of Spock's and his friends' names"() {
expect:
name.size() == length

where:
name | length
"Spock" | 5
"Kirk" | 4
"Scotty" | 6
}
}


But I don't see why not now that I think about it. Will have to try
and find out.

Good work. I am still not clear on my Peter said an extension might be
impossible to write. I refined the earlier code I submitted (you can
check it out @ http://goo.gl/Xr5qu) and that works just fine. I don't
encounter the same errors you were reporting earlier. The only wrinkle
to the code is the

executor.awaitTermination(1, TimeUnit.MINUTES);

in the intercept() method. Without that, executor seems to shutdown
the spawned threads prematurely. Need to find a way to remove that.

Maybe you understand what he was talking about and can explain.

Thanks

PS: One reason I like the closure approach is that you can start using
it right away. With the extension route, one has to wait for when
Peter gets time to merge it into the branch.

> ...
>
> read more »

[Glen]

I tried the unroll/where block and it seems to work, cool!

@Unroll("#arg")
def "test concurrent where"() {
expect:
concurrent(10, {
arg
}) == result

where:
arg | result
"foo" | "foo,".multiply(10).split(',')
"bar" | "bar,".multiply(10).split(',')
}

I'm not sure I'll bother with the extension now, it's odd you don't
get the exception I was getting though since like Peter was saying,
feature methods can only be run once per spec instance.

On Nov 17, 1:17 pm, groovybayo <groovyb...@gmail.com> wrote:
> That should work then. What I was mentioning about unroll was if this
> approach will still allow you to specify in the where clause e.g. this
> example taken from spock's homepage
>
> class HelloSpock extends spock.lang.Specification {
>     def "length of Spock's and his friends' names"() {
>         expect:
>         name.size() == length
>
>         where:
>         name     | length
>         "Spock"  | 5
>         "Kirk"   | 4
>         "Scotty" | 6
>     }
>
> }
>
> But I don't see why not now that I think about it. Will have to try
> and find out.
>
> Good work. I am still not clear on my Peter said an extension might be
> impossible to write. I refined the earlier code I submitted (you can

> check it out @http://goo.gl/Xr5qu) and that works just fine. I don't

> ...
>
> read more »

[groovybayo]

Peter - pls can you shed some light in this? This is mainly for my
understanding and curiousity.

I just want to know why this works and I don't hit the same roadblock
Glen encountered.

http://goo.gl/Xr5qu

Thanks in advance.

> ...
>
> read more »