RTD - Meteor Test Runner

[Sam Hatoum]

I've created a project page that has a screencast of RTD in action. You take a look here

And for anyone that missed out on the TDD with Meteor talk at the Meteor Devshop 4, you can see annotated slides here

[Arunoda Susiripala]

This is really nice. Awesome stuff.

On Thu, Jun 6, 2013 at 2:07 PM, Sam Hatoum <hat...@gmail.com> wrote:

I've created a project page that has a screencast of RTD in action. You take a look here

And for anyone that missed out on the TDD with Meteor talk at the Meteor Devshop 4, you can see annotated slides here

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Arunoda Susiripala

@arunoda

https://github.com/arunoda

http://www.linkedin.com/in/arunoda

[Sam Hatoum]

Thanks!

[Sam Hatoum]

We should investigate integrating Laika into the runner. That way anyone wanting to do unit/integration/acceptance tests will have the option to.

On Thursday, June 6, 2013 2:11:22 AM UTC-7, Arunoda Susiripala wrote:

[Arunoda Susiripala]

That's really cool. would like to help you.

[Arunoda Susiripala]

I  didn't try rtd. kind a busy. Will do it tomorrow. Seems you guys have done a pretty amazing work.

[Sam Hatoum]

Thanks again Arunoda.

Let's talk after you've tried it and explore options with Laika. Two hands are better than one :)

[Abigail Watson]

Hi Sam, 
Thanks for putting this detailed information together.  Got a question or two for you, before I dive into things.  

First of all, let me just say that I've been burned by Selenium before, and while I'm trying to keep an open mind, there's also a bit of pessimism and dismay when I read through your blog at what's necessary to do end-to-end acceptance tests with Selenium and Meteor.  The stubs and prototypes and promises are already making me sad.  :(

At any rate, you stated in your post on unit testing that:

4.  A prototype containing all the collection methods as per the Meteor documentation. This will allow you to add spies with frameworks such as Jasmine or Mocha which expect the methods to exist before spying.

So, here's my question:  could you speak a bit to RTD's interoperability with Mocha, and how RTD is (or is not) compatible with mocha-web?

https://atmosphere.meteor.com/package/mocha-web

In particular, I'm interested in knowing about code reuse and upgrade paths.  Mocha-web works great on getting unit-tests into your app with a simple mrt install mocha-web, and presto!  You got a page with unit tests in your app, that you can show users your app is passing.  But, it obviously doesn't use Selenium.  

So, I'm thinking it would be real nice to be able to write unit tests in Mocha-Web, then eventually port them over to something like RTD when the time comes for Selenium.  Any thoughts on this?
Thanks in advance for your thoughts,

Abigail

[Sam Hatoum]

Hey Abigail

I've been there with Selenium, and oftentimes I've thought I'd rather be at the dentist. But the safety net has always been worth the effort. 

I've just had a look at mocha-web and from what I understand, it basically circumvents the Meteor dependancy problem by running unit tests in an integrated environment. I'm personally against this approach for unit testing as it does not create true isolation of units. I would therefore call mocha-web an integration testing approach that allows unit testing through discipline. But I digress!

Since mocha-web is installed via a package, it should not interfere with RTD and they should work harmoniously. RTD is basically a runner. It combines;

karma-runner for unit tests only - stubs are used here

selenium-server + jasmine-node + webdriver  for acceptnace tests - no stubs are used here

a modified version of istanbul-middleware that works with Meteor for instrumentation and code coverage

If you want to use Mocha instead of Jasmine for unit tests, you can do so by editing the karma.conf file. (I will be extracting a configuration file for RTD soon)

If you want to use Mocha with webdriver, you'd have to change jasmine-node to perhaps just a plain node command (I should probably also extract the acceptance test runner command)

So if you're comfortable with using mocha-web and it's working for you, there's no need to change to use the stubbing approach. It should work out of the box with RTD. You'll actually get test coverage reports from RTD for free (not working for coffee yet - needs work).

Thinking about it some more, I can see how to get better integration of mocha-web into RTD. Consider the following:

RTD separates test code form the main app code. The philosophy there is that test code should never interfere with the main app code, and prevention is better than the cure

It creates a paralel mirror application to do integration testing on

It synchronises the main app code into the mirror app on every save (delta's using rsync)

it does <extra steps> here (like instrumenting the code)

I'm thinking the tests that you would normally write for mocha-web can now live outside the main app code and moved into the test directory, and in the <extra steps> you would copy the test code on top of the mirror app, and also modify the packages file to include mocha-web. The result would be your main code is pure app code.

I'd be more than happy to work with you on getting the above working and wired up, as the more use-cases RTD can cover, the better it is for the community.

Cheers

Sam

--

[Abigail Watson]

Thanks for the detailed reply Sam. 

That definitely gives me enough to get started with.  Before diving into questions of integrating Mocha-Web, however, I'm going to make a fork of my project, and see how RTD works as it's currently implemented, without any modifications.   After going through your screencasts and slideshows in a bit more detail, I'm feeling a bit more confident with the stubs and prototypes and promises.  They're really not anything I haven't done elsewhere, so if I keep my initial tests simple, I think I should be able to step through things.

I'm going to have to table this for a week or two, as I've got to do a road-trip next week to Chicago and then New York.  But then I'll be revisiting it in earnest afterwards, and may very well be sending you the occasional question.  

I'd also give a +1 to the idea of Cucumber support, in so far as it would be great to be able to define business level behavior descriptions in the way that Cucumber provides, then define the steps using Jasmine/Mocha/Chai, and then farm the tests out to Phantom and Selenium.  That's a really powerful feature from a business perspective and a regulatory compliance perspective.  Something that I'd love to see in a Meteor Test Runner.

Thanks again!
Abigail

[Abigail Watson]

Hi Sam,
Well, looks like my OCD got the better of me, and I spent much of last night getting RTD installed.  Here's some feedback on where the documentation was lacking for me, and some initial thoughts on RDT as a tool.  So far, it looks like it has all the right pieces of the testing puzzle going on.  Looking forward to stepping through the blog and tutorial next...

Abigail

- When moving contents to the app directory, don't forget to include the .meteor file!

- package.json and README.md should stay in root directory.  Final result looked like the following for me:

• README.md
• app
• build
• node_modules
• package.json
• test

- After running the following:

sudo npm i -g karma phantomjs selenium-webdriver grunt-cli jasmine-node istanbul chai mocha

- I needed to run this, which was quite confusing and not documented:

cd appRoot 

npm init

npm install grunt

 

cd test/rtd

npm install grunt-bg-shell

npm install grunt-contrib-watch  

npm install grunt-zip

npm install request

npm install hash_file

npm install freeport

 

- Karma Runner is still titled 'Testacular' in the Browser tab.  Karma Runner is a much better name.  Maybe that should be updated?

- Also, it would be quite nice to have a home page with links to the various pages involved (Meteor, Mirror, Karma, Selenium, Coverage).  Which is the Karma runner on port 9100, right?  What exactly *is* the main page?

- Right now, it's not intuitive why we're running grunt in order to launch RTD.  A bash script with an alias might really help tie things together.  

- Would also be great to have some guidance on how to get from having the Karma runner up and running to being able to follow the instructions in your blog.  For example, it might be great to have some default instructions on the Karma Runner page on how to create your first unit test.  Something basic along the lines of  'Congrats!  Karma Runner is Working!  To create your first unit test, go to test/unit, create a file called test.js, add the following code `… unit test code… ` and check out the Karma runner on port 9100.  

[Sam Hatoum]

That's fantastic feedback Abigail! Thank you very much for taking the time to put that together. It's all very valid and I'll get onto those. 

Please see some inline responses:

- package.json and README.md should stay in root directory.  Final result looked like the following for me:

• README.md
• app
• build
• node_modules
• package.json
• test

You're right about the README.md, but it's odd that there's a packages.json on the project root. I don't have that or node_modules on my setup. Is this your own package file or the one from RTD? 

Have a look at the package structure on the example app here 

- After running the following:

sudo npm i -g karma phantomjs selenium-webdriver grunt-cli jasmine-node istanbul chai mocha

- I needed to run this, which was quite confusing and not documented:

cd appRoot 

npm init

npm install grunt

 

cd test/rtd

npm install grunt-bg-shell

npm install grunt-contrib-watch  

npm install grunt-zip

npm install request

npm install hash_file

npm install freeport

All of those (grunt, grunt-bg-shell, grunt-contrib...) are all in the packages.json inside RTD. They should have been installed when you ran this line from the instructions:

$ git submodule add g...@github.com:xolvio/rtd.git ./test/rtd; cd test/rtd; npm install; 

Perhaps I should write a migration script that would automate the move.

- Karma Runner is still titled 'Testacular' in the Browser tab.  Karma Runner is a much better name.  Maybe that should be updated?

Haha! That's probably because the Karma guys haven't renamed that yet. Not sure I have much control over that.

 

- Also, it would be quite nice to have a home page with links to the various pages involved (Meteor, Mirror, Karma, Selenium, Coverage).  Which is the Karma runner on port 9100, right?  What exactly *is* the main page?

Excellent idea. There isn't actually a main page to check stuff at the moment, as all the reporting happens in the console. I have the console captured inside my IDE (WebStorm) and that works, however I like the idea of creating a main page that stays open and updates with reports.

 

- Right now, it's not intuitive why we're running grunt in order to launch RTD.  A bash script with an alias might really help tie things together.  

Agreed. I'll get on that.

 

- Would also be great to have some guidance on how to get from having the Karma runner up and running to being able to follow the instructions in your blog.  For example, it might be great to have some default instructions on the Karma Runner page on how to create your first unit test.  Something basic along the lines of  'Congrats!  Karma Runner is Working!  To create your first unit test, go to test/unit, create a file called test.js, add the following code `… unit test code… ` and check out the Karma runner on port 9100.  

Another good idea I'll get onto. So far, I have setup an example here (https://github.com/xolvio/real-time-development-with-meteor) that exemplifies RTD usage. I'll look at making that and RTD a bit clearer and more integrated.

[Stefano Diem]

RTD just killed the only reason i had to not use Meteor instead of a more conventional Rails Application ( by allowing me to do TDD and make sure that code quality high and thing are covered ). Well, there are still some edge cases, but then again it made that much difference to me. Thank you very much!

Abraços,

Stefano Diem Benatti

phone: 55 11 9343 0994 skype:  teonimesic

[Sam Hatoum]

Thank you for your feedback Stefano, I'm very happy you're finding it useful 

[Kevin Smith]

First, awesome work! Like the direction this is going.

A feature I would love to see is support for notifications via libnotify / growl. I find this a great help when using Guard for TDD on Rails projects.

Look forward to seeing how this develops!

[Sam Hatoum]

Thanks Kevin. Your wish has been granted! Growl support is now in there, get the latest and run npm install inside RTD :)

New Instructions have been posted here in case you need to install terminal-notifier

http://xolvio.github.io/rtd/

Cheers

Sam

--

[Kevin Smith]

Fantastic! Shortly after posting that request I realised I shouldn't be so lazy and go implement it myself, but you did it so fast you beat me to it. :D

Anyway, I submitted a pull request to add libnotify on top of your Growl support since I use linux ;)

[Sam Hatoum]

Many thanks, pushed and updated instructions for Linux.

[Abigail Watson]

So, RDT seems to have added ~1000 new files to my project, between selenium, phantom, jasmine, istanbul, grunt, and everything else.

Question:  Would it make sense to separate the application test harness into it's own repository, and to include the actual app as a git submodule?   I don't particularly like working with submodules, since the dependencies tend to be brittle.  But this seems like it might actually be the right time and place to use one.  

Anybody else try this approach?  Thoughts?

--
You received this message because you are subscribed to a topic in the Google Groups "meteor-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-talk/3QEhNXq2Rb4/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to meteor-talk...@googlegroups.com.

[Sam Hatoum]

Hey Abigail

I think having the submodule works for when somebody else is maintaining code that you also contribute to, which is perfect for RTD. I did consider a package but opted for a submodule for that reason. I also have meteoric-sh setup like this in my project for doing deployments to AWS.

My experience of setting up projects has taught me to always have everything needed to setup/code/test/deploy under one tree as it makes developer ramp-up easier and the project portable build scalability (think multiple cloud agents agents).

With the above two points in mind, I personally wouldn't separate the main source code as I see test and app code as tightly coupled and should always be maintained in tandem.

The place I can see where that might be required is if you use some git-based deployment, such as Heroku. Test code of course should never be deployed.

Sam

[Kevin Smith]

Hey,

I noticed some issues tonight after updating Meteorite.  Meteorite (post 0.6.0?) now symlinks packages into the /packages directory, and RTD tries to load them when executing unit tests which throws a bunch of errors as no templates etc in the packages are actually defined.

Adding '/packages/**/*' to the exclude array in karma.conf.js resolved the issue for me, and I think that's fine since packages can and should be tested separately with Tinytest.

I was also wondering if there are any plans to include stubs for objects in commonly used packages like Accounts and Meteor.Router?

[Sam Hatoum]

Thanks for highlighting that. Agree that packages are an external dependency and therefore are better assumed to be tested. I'll add that in soon.

The plan stub wise is to ensure unit tests can load all code without having any undefined errors. So if you ever use something like Account.xxx, this is a key place to stub. What to do after that, is another matter. Because now there's the option of either doing some smart stubbing, or just placeholders that allow mocking using spies. As a general rule of thumb, I use these two rules:

1 - Anything that requires code to be isolated so that it can be run as a single unit, such as the fireEvent method on the Template stub, requires smart stubbing.

2 - Anything that requires assertion only, such as <collection>.find, just needs a place holder and Jasmine/Mocha spies will allow you to unit test.

We've been stubbing things as we need them and would very much welcome pull requests for anything you find that gives an 'undefined' error. More than happy to help/discuss stubbing vs mocking as it won't always be so black and white as the two rules above.

Looking at the Accounts methods, I would say all of those are simple placeholder stubs, which is very easy. And looking (very quickly) at Meteor.Router (this page), it looks like Meteor.Router.filters would require an isolation mechanism so you can say something like:

// SETUP

// spy on/fake Meteor.loggingIn and to return some response

// EXECUTE

var result = Meteor.Router.applyFilter('checkLoggedIn')

// VERIFY

// verify that result === 'loading' or whatever

So the RTD stub would need to provide the applyFilter to isolate units.

I didn't intend this email to be so long, but I got carried away so I guess I'll turn it into a blog post at some point :)

Sam

--

[Pieter Devolder]

Hi Sam,

Just to share my setup roadmap to get RTD working on a Virtualbox Ubuntu 12.04 setup.

I generally did the same steps as Abigail however, I had to 

• remove the references to "[process.platform]" in the Gruntfile.js on my system (I haven't researched thoroughly though why this had to be done)
  
• download, install and refer to the 'linux32' ChromeDriverOs...

Hope this helps in any way.

great effort by the way. RTD is fast helping in getting up to speed with TDD in general, although wrapping my head around promises and deferred objects is not easy for a first-timer...

thanks,

Pieter

[Sam Hatoum]

Thanks for sharing. I should use process.arch to also check for 64/32 bit and get the right chrome. Another improvement to add to the list!