Unit Testing Recommendations

[Kieron Wilkinson]

Hi,

I'm writing acceptance tests of ScalaSTM, and I'd like to be able to specify something like an execution plan for isolation phenomena in a rather serial fashion, and execute it on ScalaSTM. For example, something like:

get(txn1, ref)

set(txn2, ref, value)

get(txn1, ref)

// test for retry/failure

I realise this rather goes against the grain of what ScalaSTM is trying to achieve in it's programming model, but when writing many unit tests something like the above would be a whole lot clearer. I can then more easily encode some of the classic isolation tests like with bank account in a very straight-forward way.

I suppose I might be able to achieve the above with a lot of CountDownLatch & Thread code, but it seems I need to do a lot of thread sleeping to get what I expect (perhaps because of some back-off used to avoid unnecessary retries), and that is going to add up, as well as no doubt being a bit flakey.

I had a look at some of the unit tests for ScalaSTM, like the write skew one. That was interesting for sure, but seems more like a battery of tests, rather than the many different logic ones that I am trying to create. Is there a simpler/recommended way of doing something like above? Maybe there is even a way of avoiding threads completely (where the transactions are separated by their context instead)?

Thanks,

Kieron

[Nathan Bronson]

Kieron,

This is a tricky topic, because one of the advantages of centralizing the optimistic retry logic into the STM (as opposed to hand-coding it) is that we can apply sophisticated optimizations to improve the average and worst-case behavior. For example, ScalaSTM has lock-less (invisible) reads for scalability and performance in the average case, but falls back to read locks to guarantee forward progress even if other threads are constantly performing writes. The resulting schedule is an implementation detail, so encoding it in a unit test will make it brittle.

Despite the theoretical difficulties, we could probably make this work anyway. We could use a BlockContext to detect blocking and fail the test immediately if execution wasn't going according to plan, and we could centralize the execution by having a non-transactional coordinator thread that sends and receives notifications from the transactional threads. Since the tests would have to perfectly predict the execution schedule of the STM, it seems useful if we could run it in a recording mode, check to see if the schedule is okay, and then encode that in the test. I will try to write the code tonight.

So far, my preferred approach has been to write tests where the transaction bodies can detect the anomalous phenomena, and then run them repeatedly in a stress test. The WriteSkewTest you mentioned is one of these. The write skew anomaly is when the reads and the writes of the transaction serialize at different points, so the code at the core of that test preserves an invariant between its read and write sets that will be violated if the serialization points differ. These tests cover a smaller set of visible behaviors, but are more robust over time and do a better job of stressing things like the deadlock detection and ScalaSTM's use of the JMM. The FlipperTest does something similar by assigning each thread a bit in a word, so there would actually be no need for atomicity if we were performing thread-safe single bit reads and writes. That means the final result of the computation should be the same for a sequential and a parallel execution, which we check.

 - Nathan

--

---
You received this message because you are subscribed to the Google Groups "Scala STM Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-stm-expert-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Nathan Grasso Bronson
ngbr...@gmail.com

[Kieron Wilkinson]

Hi Nathan,

Thanks so much for the comprehensive answer. Given what you've said, I'm slowly come round to the fact that the 100 or so tests I have already written (originally to find a suitable key-value store with serializable transaction isolation) are fairly useless in an STM environment. :)

I do see your point, and indeed I very much don't want to bother with all the effort to get flakey tests anyway. I'm much more interested in encoding a set of requirements where it must not fail and the behaviour I'll be depending on, rather than the particular properties of an implementation. I guess that is just going to be more like the stress tests you already have rather than the specific ones I currently have. With that in mind, please don't go to the trouble writing up the recording facility on my account.

I guess my challenge now is try and find a way to set things up with minimal threading boilerplate that may otherwise hide the intent of my tests. ScalaSTM with your CCSTM implementation seems to offer exactly what I am looking for so far, so it's certainly worth the effort.

There is already a good set of tests in there I know, but when I'm finished, I'll chuck my tests your way anyway if you're interested. Just in case you find any useful.

Thanks again,
Kieron