Black-box testing

[Dana Scheider]

Hi everybody,

A colleague just asked me a question I found interesting and wanted to see if anybody had better answers than what we came up with.

Suppose we are black-box testing an API endpoint that sends a different greeting depending on time of day. Early in the day, the greeting is "Good morning", and later it is "Good evening". We want to test this business-critical functionality. Here are the ideas we came up with:

1. Test the method that generates the greeting at the unit level and use Cucumber only to test that the greeting is sent

2. Expose a special endpoint on the API to set system time that is used only for testing.

Both these solutions have significant drawbacks and I was curious how others would solve this problem.

Cheers,
Dana

[Andrew Premdas]

Write scenarios that express what your business wants.

Given it is early in the day

When I request a greeting

Then I should see the morning greeting

Given it is late in the day
When I request a greeting
Then I should see the evening greeting

Now your only problem is a technical one which is to implement the Givens.

An interesting question is whose time are you interested in, the clients or the servers. So you might want to change the language to reflect that.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Thomas Sundberg]

Hi!

Your issue is that you are not in control of time and therefore can't
be sure what to expect when you execute the call.

This leads me to think that verifying the message, using Cucumber
instead of a unit test, should be done under the hood and not through
an API call. It is interesting for the business that the correct
message is returned so they might want to see the Gherkin executed.
But in order to be sure that you get the correct message back, you
need to specify the current time for the system under test.

If it is important that the API call is verified, I would complement
the verification above with a verification that sees a message and
don't care about which message. This would verify that the system is
able to answer something that is correct. I.e. the wiring of the
application is OK.


If it isn't possible to do the verification as above, a combination of
white and black box, you need a way to take control over time before
the API call through an endpoint. I would not be happy with it since
that would have to change the time for the entire application. You
can't be sure that your actual API call will be served with the same
thread or similar that you just set the time for.

I guess this is one of those cases where automated black box testing
is complicated due to circumstances you are not in control over. Time
passes and we can't be sure what time an execution will take place.


A twist that Andrews response triggered for me is to force the caller
to supply it's local time. This would solve the problem with users in
different timezones and not rely on server time. It would also
magically solve your problem since you are delegating the
responsibility of handling time to the caller.

Cheers,
Thomas

--
Thomas Sundberg
M. Sc. in Computer Science

Mobile: +46 70 767 33 15
Blog: http://www.thinkcode.se/blog
Twitter: @thomassundberg

Better software through faster feedback

[Andrew Premdas]

Just to add to Thomas's stuff.

If you are working on API/Server time, your Given can just go and change the server time on your test server, which should not be a problem.

But this particular scenario makes much more sense as a client time scenario.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George Dinwiddie]

Dana,

The clock is an external system to the system we are writing. Therefore
I recommend using the Adapter Pattern for accessing it. This allows us
to substitute a different adapter to use a controllable clock for
testing. The sample chapter of https://leanpub.com/EvolutionaryAnatomy/
describes this in more detail.

I would certainly test this at the unit level with direct injection of
the desired adapter as part of building the functionality.

For acceptance testing, I might test this at the code API (called from
the official endpoint, which I'm presuming might be a webservice) to
make injecting the adapter easier.

Alternatively, I might have the test alter the configuration of the
service so that it uses the controllable clock instead of system time.

- George

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------