End to End Testing Best Practices?

[Jon Gordon]

Hi everyone,

I'm quite new to RSpec, and I have used it mainly for unit-testing. Lately, a need for a small number of end-to-end tests became relevant. When writing test-cases, I'm trying to stub all dependencies, but because that's not an option when doing integration tests, I need some help to understand what's the proper way to do things. Here's couple of questions:

1. The test requires an IP for remote machine (which is not local and sadly can not be). Obviously, I shouldn't supply the IP inside the spec file. The simple way is reading an external YML file with the IP (that will get created automatically during the CI process with the right IP for example) and populate the IP directly from it. But, I was checking couple of big project that uses rspec, and I never seen an external configuration file, so I'm thinking perhaps there is a better way of doing it

2. If indeed YML file is the right answer, I'm not sure if reading from the YML file every spec file (that uses this service) is the right thing to do? Shouldn't I be using hooks instead for that?

3. The test-object is a REST service, and some of the requests require big json object. I have two options:
    a. I can create the json object in the spec file itself (which makes all information visible to you from the spec file itself, but clutters the spec)
    b. Creating an external default fixture (which is basically a json file), read from it during the spec, and re-write the values that are relevant for the specific tests.

Thank you!

[Jon Rowe]

Hi Jon

A couple of tips, firstly you can stub out your external dependencies for an end to end test, it just depends on the level of integration you want, it’s equally fine to do what you propose. For injecting your endpoint (IP, hostname or otherwise) you have a couple of ways of doing it, the simplest is to use environment variables e.g. `ENV[‘API_ENDPOINT’]`, or you can build yourself a config system like you mention. The reason why you don’t see big projects using external configuration files is it is usually done at the app level rather than in rspec.

If you chose to go down the config file route, xml, yml or otherwise, you’d be better off loading it in a spec_helper or other such support file, and assigning it somewhere.

Personally I would go with json fixture files for static json, or a generator method if it needs to be dynamic.

Cheers.

Jon

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk

--
You received this message because you are subscribed to the Google Groups "rspec" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rspec+un...@googlegroups.com.
To post to this group, send email to rs...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/61ac9ade-1045-4211-80d3-441ef01ae7cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Xavier Shay]

Obligatory plug for https://www.pluralsight.com/courses/rspec-ruby-application-testing which touches on some of the themes you're asking about :)

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/3FF6FCF2018A482CBDC70C02BAFFB643%40jonrowe.co.uk.

[Jon Gordon]

Thank you fro the reply Jon. You right, even while doing end to end test, I find myself able to stub 'some' of the things I don't need.
Environment variables  is great for CI (easier to do compared to change a YML file), but for daily development - static YML file feels easier to me. Is there any convention what the configuration file should be named/where?
What do you mean in the 'app'? the app runs remotely, even if the information is extract-able - I still need to know the IP of the service don't I?

Great, in that case - I will keep a 'template' of the json in a file, and when I'll need to change small amount of values (2-3), I can just parse the JSON into hash object and do those changes.
Thank you :)

[Jon Gordon]

Thanks Xavier :)
I will be checking this course!

Is there perhaps an open-source project with end-to-end spec tests you can recommend (REST tests are preferred, not Capybara)? something to get a reference from?
Thank you.

[Myron Marston]

My upcoming book, Effective Testing with RSpec 3, has an example of building a JSON API using end-to-end acceptance tests, isolated unit tests, and integration tests.  It might fit what you're looking for better since you mentioned you're looking for examples of end-to-end testing of REST services.

The code for the book is all online, as well.

All that said, Xavier's screen cast is very good, and I definitely recommend it, particularly if you do better with videos than printed materials.

Myron

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.

To post to this group, send email to rs...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/28f3f239-1515-437b-b011-82b2dd163502%40googlegroups.com.

[Jon Gordon]

Hi Myron,

I will definitely check the book - looks like it's perfect for RSpec beginner. Also, thanks for sharing the example online - that's alone can give me a good starting base :)

I will ask another question while posting -  for unit-testing I'm using Faker gem to fake common strings. However, in end to end tests, we work against a database - so even if I'm using the 'unique' method, I'm running out of unique strings after a while. I'm using 'securerandom' to generate random number, but wondering if there's a better approach for that.

Thanks!

[Myron Marston]

In general, if you need absolutely unique strings, `SecureRandom.uuid` is a good way to get one.  UUIDs are universally unique, after all :).

That said, the fact that you are running out of unique random strings from faker is concerning.  All tests should work off of a "clean slate" in your database, either by wrapping each test in a rolled-back transaction, or by truncating your DB tables.  Are you "leaking" DB records between tests?

Myron

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.

To post to this group, send email to rs...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/c297c4c9-5225-47d9-a6e2-80f461bd1226%40googlegroups.com.

[Jon Gordon]

Not in Unit-tests of-course, but It seems like the only option in real end-to-end testing. The system is quite complex, as it's basically a set of couple micro-services. Each has it's own unique database (Can be Postgress, Casanda, Oracle...). In a single End to End test, all databases are populated with information. Clearing tables between each test can take time, and is quite complex. The CI process does re-start the containers at the very start of the test-run (so it's like restarting them to a fresh state), but not during tests.

if I'll take the list of music instruments in the Faker gem for example, It only has around 10 options. So even if I use the unique flag - It will run out of options after 10 test-cases.  I guess use 'msuic instruments' in one spec file, and 'cat-names' on the other to avoid it, but that means I need to 'remember' what pool of string I already used in previous tests, and that's feel even worse for me.

Thanks.



but I thought that there no better way around it. Because

[Jon Gordon]

Hi again :-)

So I tried to write couple of RSpec test since we last talked, I'll mention again I'm writing couple end to end tests to verify all micro-services are up and running. Please consider the following example. It's an authentication service, that can create users. Before user is being created, a schema for the user-type needs to be created on a different micro-service:

RSpec.describe 'Authentication' do
  subject { AuthenticationService.new }
  let(:user_schema) { SchemaService.new }

  context 'When creating a user that does not exists ' do
    it 'response with status code 200 (success)' do
      schema_name = SecureRandom.uuid.to_s
      user_schema.create_schema(schema_name)
      payload = JSON.parse(File.read('spec/acceptence/fixtures/feature.json'))
      payload['schema_name'] = schema_name

      response = subject.create_user(user: 'my_user', payload: payload)

      expect(response.code).to eq 200
    end

    it 'creates a new user' do
      schema_name = SecureRandom.uuid.to_s
      user_schema.create_schema(schema_name)     
      payload = JSON.parse(File.read('spec/acceptence/fixtures/feature.json'))
      payload['schema_name'] = schema_name

      subject.create_user(user: 'my_user', payload: payload)

      response = subject.get_user_by_category(category: payload['category'])
      remote_entity = JSON.parse(response.body)

      expect(payload.to_json).to eq(
        remote_entity['list'][unique_value]
      )
    end
  end
end 

To keep it dry, I should be moving the whole schema creation into a before block:

before
  schema_name = SecureRandom.uuid.to
  user_schema.create_schema(schema_name)
end

Before makes sense to me over let here, because it's an action. However, because the schema is more of a 'pre-condition', I can create it just once, and avoid multiple schema in my database. Therefore, before(:all) seems like a better option.

before(:all)
  schema_name = SecureRandom.uuid.to_s
  user_schema.create_schema(schema_name)
end

Now that problem is that when I create user in my examples, I NEED to schema name, so I need to share context between the before and it block. It makes sense for me to do it like so:

let(:schema_name) { SecureRandom.uuid.to_s }

before(:all)
  user_schema.create_schema(schema_name)
end

Then I can create easily create user in my example like so:

payload = JSON.parse(File.read('spec/acceptence/fixtures/feature.json'))
payload['schema_name'] = schema_name

subject.create_user(user: 'my_user', payload: payload)


Alas, this will not work. As it not allowed by RSpec to use  a let value inside a before(:all) block. So I need to hold a string that can be used in both the it and before block. It can solved it by defining a Constant or using Instance variable but both methods feels reek to me. I mentioned I don't have access to the database (as those are remote machines and they don't expose the ip for that database) so I can't truncate the db information and avoid the before block here.

Thanks!

[Jon Rowe]

>  As it not allowed by RSpec to use  a let value inside a before(:all) block. 

This is for good reason as it’s a bad idea to share test stare, you could assign a constant but it would be better to refactor your code base not to depend on an external db for each test like this.

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/48d88387-8e71-49a5-b25a-850a79fe4181%40googlegroups.com.

[Jon Gordon]

Refactor how? the system consists of 4 micro-services. In a single sanity scenario, they are all being called (some sort of a smoke test to verify all can communicate with each other when spun up). The database is being written on the last micro-service in line, it's code that being handled by another group. I can ask them to give me an API to mock the db, but that still requires me during setup to access the mock, name the entity and then use that name in the example block. As I don't communicate here with objects like I do with unit-testings (instead I'm using REST API) I see no way how I can achieve that?

Thanks!

[Myron Marston]

before(:all) hooks require special care and we usually recommend you avoid them. Most RSpec tooling assumes a per-example lifecycle for resources (such as DB transactions, test doubles, and let memoization), and before(:all) hooks operate outside that lifecycle. If you’re not careful to manage the resources explicitly you’re likely to experience problems from using :all hooks. For this reason we don’t provide syntactic sugar for them (such as a let construct), but it’s pretty trivial to create your own construct if you want:

module BeforeAllConstructs
  def set(name, &block)
    attr_reader name
    before(:context) { instance_variable_set("@#{name}", block.call }
  end
end

RSpec.configure do |config|
  config.extend BeforeAllConstructs
end

With this in place, you could use set in place of let to have a construct like let designed for :all/:context hooks.

That said, in your case, I wouldn’t recommend you take that route. It doesn’t seem like having two separate examples here provides you any clear benefit, and the fact you are considering using a before(:all) hook indicates you want to do some computation once, and then make multiple assertions about that. Instead, I’d recommend you combine the two examples, but then use :aggregate_failures so that you get a list of all failures (and not just the first one). Here’s how you could do that:

RSpec.describe 'Authentication' do
  subject { AuthenticationService.new }
  let(:user_schema) { SchemaService.new }

  context 'When creating a user that does not exists ' do

    it 'creates a new user', :aggregate_failures do

      schema_name = SecureRandom.uuid.to_s
      user_schema.create_schema(schema_name)      
      payload = JSON.parse(File.read('spec/acceptence/fixtures/feature.json'))
      payload['schema_name'] = schema_name

      response = subject.create_user(user: 'my_user', payload: payload)

      expect(response.code).to eq 200

      response = subject.get_user_by_category(category: payload['category'])
      remote_entity = JSON.parse(response.body)

      expect(payload.to_json).to eq(
        remote_entity['list'][unique_value]
      )
    end
  end
end

HTH,
Myron

​

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.

To post to this group, send email to rs...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/3ae74ab4-f4fd-4f17-b87f-4256656b57d4%40googlegroups.com.

[Jon Gordon]

Hello again Myron :)

I'm guessing that's why I couldn't find many before(:all) examples online. I wasn't aware of aggregate_failures, that's pretty cool feature! I always tried to follow the 'single assertion per test' principle. However, the example above is small portion of a bigger Spec. In the same Spec file I will also try to delete an entity (which require me to create an entity first), get a list of entities (create at least 2 entities) and expect exception to get raised when I'm trying to create an entity that I already created. All of those require a working schema. I can't group all up them under a single aggregate_failures test. So sadly, I will have to use instance variable, or to wrap a set block like the example you shared above (unless you have another idea)

But I'm wondering, let's say I could control every point in the process and design this whole spec from scratch, having the option to do whatever I want on each of the remote micro-services. What would be the proper way of addressing it? Doesn't feel like the tests above are something specials, more like a standard REST service testing. Here's the option I can think about:

1. Create the schema over and over each test with 'before' block like so :

let(schema_name) { SecureRandom.uuid.to }

before
  user_schema.create_schema(schema_name)
end

it 'my test' do
  puts schema_name
end


This will be slower as schema will getting created every-test, and I only really need a single one as the schema is just a dependency and no the main focus of the test. But - perhaps clearer to read and avoid sharing context with before(:all).

2. Pre-populate the remote test database with default user schema with default name. I can either create it directly on the database when I spin the environment or from spec_helper . I can even store the default values in YML file, create an object from it (config) and use it on test.

 payload = JSON.parse(File.read('spec/acceptence/fixtures/feature.json'))
 payload['schema_name'] = config(:default_schema_name)

This however makes the test less clear because not all information is being exposed when you read just the spec file. I would like to do it the 'proper' way - as I can probably talk with the other teams in the future - makes those test better align with the standards.
Thanks for the help again!

[Myron Marston]

I don’t have sufficient information to tell you what I’d do in this situation, but here are some general principles to help you think through the tradeoffs here.

• Whether or not I’d create a user schema per example really depends on how long that operation takes. If it’s a pretty quick operation (e.g. milliseconds) I’d probably just make one per example; since these are end-to-end tests they’re expected to be slow and doing something like a before(:context) hook caries a high maintenance cost since it has so many caveats compared to a typical before(:example) hook.
• I personally don’t follow the “one expectation per example” rule at all. It was a useful corrective to poorly written tests that contained tons of expectations and were hard to debug, but bears a high cost (much higher than I’m willing to pay) due to all the time wasted repeating setup. In fast, isolated unit tests, I do follow a principle I’d call “one behavior per example” (where a single behavior can often be specified with a single expectation, but not always). I care a lot about keeping a fast, snappy test suite so in slower integration or acceptance tests I write much courser-grained tests that may encapsulate a whole workflow.
• I tend to use :aggregate_failures for all my integration and acceptance tests.
• If a user schema can safely be re-used in many examples, and creation of the schema was slow enough to warrant sharing it among multiple examples, I’d probably use a different approach than a before(:context) hook. Instead, I’d probably do something like this:

# spec/spec_helper.rb
module UserSchemaCache
  def self.schema
    @schema ||= SecureRandom.uuid.tap do |schema_name|
      SchemaService.new.create_schema(schema_name)
    end
  end
end

RSpec.configure do |config|
  config.when_first_matching_example_defined(:needs_schema) do
    UserSchemaCache.schema
  end
end

# some_spec.rb

RSpec.describe 'Authentication', :needs_schema do
  # ...
end

That allows you to use the same user schema among all your end-to-end examples, incurring the cost of creating it only once. The when_first_matching_example_defined hook is only invoked if there’s an example matching :needs_schema, so there’s no cost when running your other specs.

HTH,
Myron

​

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.

To post to this group, send email to rs...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/adfd947f-875e-47cf-91e5-a95264f7aedc%40googlegroups.com.

[Jon Gordon]

Hello Myron, thanks yet again for the detailed answer :)

Well, I will time it. So a rule of a thumb would be anything below 500ms can go inside the test iself?

Also, you mentioned your following 'one behaviour per example'. If I'm correct, 'it` is a behaviour per example. So I'm assuming that in integration tests/acceptance tests you always have on it block per context with multiple assertions in it?
I see what you mean. Specific on the example needs :needs_schema is much more elegant then using instance variable that is harder to decipher. Schema creation however is not enough, as I need the name of the scehma. Will the above let me use "UserSchemCache.schema" to get the schema name?

Thanks again!

[Myron Marston]

Well, I will time it. So a rule of a thumb would be anything below 500ms can go inside the test iself?

The rule of thumb for what’s too long depends on what you can tolerate, how much value the tests provide, etc.

Also, you mentioned your following ‘one behaviour per example’. If I’m correct, it is a behaviour per example.

The it method certainly lends itself to describing a single behavior, but there’s nothing stopping you from describing what would normally be considered more than one behavior: e.g. it "persists the expense, returns it when queried with the expense date, and includes it in the date's expense total".

It’s all a matter of trade offs, what you’re trying to achieve, how fast you need your test suite to run, etc.

So I’m assuming that in integration tests/acceptance tests you always have on it block per context with multiple assertions in it?

Nope. Integration and acceptance tests will still have multiple examples in an example group. All I’m saying is that in an integration or acceptance test, I’m keenly aware of the cost of repeating slow setup many times, and so I don’t necessarily strive for “one behavior per example”. I compare the costs of the tests to the value I'm getting out of them, and look for ways to cut the cost of the tests (e.g. by cutting down the time it takes to run them). For example, I’ve worked on JSON APIs that are basically read-only APIs, but do tons of data processing to prepare the data set to be queried. In this kind of situation, my end-to-end tests aren’t limited to one JSON endpoint per test, because my test time is dominated by how long it takes to prepare the dataset. If the data prep time is 10 seconds, and hitting a JSON endpoint takes 100ms, it makes a big difference to group lots of related JSON endpoints into a single end-to-end test that spends 10 seconds building the dataset once, and then hits lots of JSON endpoints to demonstrate they work in an end-to-end manner, without paying a 10 second per endpoint cost.

Schema creation however is not enough, as I need the name of the scehma. Will the above let me use “UserSchemCache.schema” to get the schema name?

You can cache or assign the schema name in the when_first_matching_example_defined hook as well. The UserSchemaCache module is just a normal ruby module, with nothing RSpec-specific in it, so do whatever you need to do there to compute, store, and expose whatever state you need.

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.

To post to this group, send email to rs...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/0f5083dc-5a77-44ee-af4f-44a34d8a4e56%40googlegroups.com.

​

[Jon Gordon]

Fair enough. So efficiency is the key word here over conventions. Fair enough. You gave me couple of good point. I will go back to the code and have a look how I can apply those correctly. I thank you again Myron - this, the book and the code examples you shared were very educational for me. I'll let everything sink in and see how it goes :)

​