How to approach functional tests in segregated systems

[Augusto]

Hi Everyone,

I have what I consider a simple question for some people, but I can't
find a good answer. I'm implementing some automatic functional tests
in an environenment that has about 20 applications talking to
eachother using queues and/or restful interfaces.

The problem I'm facing is that I added some test to one of the
application, lets call it Payments, and they all looked very nice and
passed, but one of the changes on the Payments application added an
extra field in a messages that is send to another application, lets
call it CRM, and the CRM discarded the message, which meant that the
applications started to become out of sync.

What I'm wondering is how to test these dependencies between systems.
One option (that I dislike from my gut, but no idea why) is to connect
to the other system and verify that the record was updated. Most
applications have a restful interface and also provide clients, so is
not a lot of work. But I have the strong feeling that doing this will
make the test more brittle. One of the reasons why I don't like this,
is that most of the applications talk to another 4-10 other
applications, so it's not just checking another app... and since lots
of things are asyncronous, it adds a bit of uncertainity to the test.

I would greatly appreciate any thoughts :)


Thanks,
Augusto

[Anders Dinsen]

Hello Agusto

This is not a simple question, and the answer will not be simple either. I see your problem as an integration test problem, and suggest you start by dividing your tests into two conceptual layers:

1. Functional tests testing the individual applications in isolation,
2. Functional tests verifying critical integrated problems

In the latter case, you /will/ have to carry out operations in one application, verifying results in other applications. That's the nature of the problem! And yes, you do risk adding uncertanity to the test which you will have to try to mitigate, but it's better to have a test with a risk of providing a false positive than to leave integrated functionality untested.

Your problems could also indicate you're facing fundamentally flawed system design. Are all these queues and asynch interfaces really necessary?

Cheers,
Anders
http://blog.asym.dk/

--
Venlig hilsen/best regards

Anders Dinsen
ASYM APS

a...@asym.dk
+45.2818.4925

Espegårdsvej 20A
DK-2880 Bagsværd

[Jake Benilov]

Hi Augusto,

You might want to look into adopting consumer driven contract tests
(http://www.infoq.com/articles/consumer-driven-contracts). My current
client has been applying this approach fairly effectively for defining
executable contracts between services and catching breakages during
development.

regards,

Jake

[Augusto]

Hi Anders & Jake,

Thanks a lot for your responses and sorry for the extremely late reply.

We decided to take Anders approach for the time being, so we now have an extra set of functional tests that focus on testing that the external systems get updated properly. This worked fine for the simple cases, but we're having some issues as some of the external systems transform the information we send, and we don't want to duplicate the transformation logic in our tests. Also because that transformation is well beyond the scope of the test and the system we're trying to test. This means that we might need to relax the tests a bit.

Jake, thanks a lot for the reference to consumer-driven contracts. I hope we can put this into practice... but I'm quite sure that it will cause the same type of issues, not because this approach is flawed, but bacause we'll have the same core problem, communication. I can really, really see teams not communicating that there was an update to the contracts, so the consumer and producer would be working on different versions without knowing it. This is partially because the teams are distributed across the globe :s.


Thanks again for your suggestions!
Augusto