Re: How to write specs that will not pass first time, but during/after some grace period?

[Myron Marston]

On Feb 25, 5:00 am, Peter Mounce <p...@neverrunwithscissors.com>
wrote:
> Hi there
>
> I'm trying to write some tests for infrastructure launched into and running
> within AWS.  Things like "home page should return 200 OK" to begin with -
> end-to-end tests against just the public interface to kick the tyres.
>
> I want to kick off the test suite in parallel with a deployment, so that
> the test suite is red, then goes green as various resources come into
> service.
>
> I'd like to know whether there is an extension point so that I can wrap
> `it` methods inside a `retryable` type structure, so that rspec does not
> report the specs as failed until a particular number of retries have
> occurred within a particular grace period.  So, ideally, each `it` would
> try itself and report back whether it passed or failed; if it passes,
> remove itself from the not-passing-yet list.  If it failed, go to the next
> not-passing-yet test.  I wouldn't want the first test to block the rest of
> the suite from being attempted - so, breadth-first iterations.
>
> If there isn't that, I'd like to know whether it would be easier to create
> one, vs create my own rspec runner that would do it at either a) the
> individual spec basis, or b) the suite basis.
>
> The issue really is that I have a long running `before :all` (like,
> ~25min), in the form of a series of shell commands, to execute, then run
> tests against the result.  That, combined with the fact that when the shell
> commands finish, it's not guaranteed that the infrastructure that has been
> spun up is actually ready for business, vs merely extant.  Creating
> something that is event-driven would be better, but not something I can
> invest in immediately.
>
> Any help gratefully received!
>
> Regards
> Pete

RSpec isn't really designed for what you're trying to do, so it may be
easier in the end to do this outside of RSpec than in it. That said,
I helped Sean Cribbs add a little "sometimes" extension to the riak-
ruby-client spec suite that handles retries up to a set number of
times. You might find it useful as a starting point for ideas:

https://gist.github.com/seancribbs/1718985
https://github.com/basho/riak-ruby-client/commit/16b4b2f3cce3689ad82b99754fd676954674f5f5

HTH,
Myron

[Irish]

At last years RailsConf there was a presention by the Major League Gaming guys that discussed how they write Cucumber scenarios to test their services and infrastructure.  Not sure about your specific timeout feature, but it may be helpful into how you could go about implementing those kind of tests.

http://confreaks.com/videos/895-railsconf2012-stack-smashing

[Peter Mounce]

Yeah, I figured.  What I'm after is to arrive at a set of continuously-running specs that I can use to indicate live environment status - similar to what things like nagios and monit and ipmonitor allow, but with the flexibility of being able to define scenarios rather than more simple ICMP or http 200 OK type checks.

Thanks for the links, I'll investigate.

It sounds like I might simply (heh) want to create my own test runner, to run as a daemon, watching and adding to checks that are added to a particular directory...?  I'm thinking that on each example, I'd want to emit telemetry to my monitoring system (statsd+graphite), and then my status dashboard can go green/amber/red depending on that.  Sound feasible...?