Behat tests on modules

[Anselm Christophersen]

As some of you might have noticed, I’ve come to love Behat, and it’s integration in SilverStripe.
Though it can be a mess to set up, and there’s even some controversy in the core group whether it’s the right tool (https://github.com/silverstripe-labs/silverstripe-behat-extension/issues/62), I really feel it helps to make sure that my frontend and backend code work as expected when performing updates.
Lately it’s helped me to identify this issue: https://github.com/silverstripe/silverstripe-cms/issues/1398
I also wrote some behat tests to identify this issue, which showed to have been solved when I updated composer, and hence my behat tests passed:
https://github.com/sheadawson/silverstripe-blocks/pull/68#issuecomment-194494709

In that issue I was suggesting to add my tests to the module, but then it occurred to me that adding those tests would probably only make sense if the module was integrated with a CI server that’s capable of running Behat.
I checked, and saw that the CMS module is running Behat on Travis:
https://travis-ci.org/silverstripe/silverstripe-cms/jobs/114629894

My question is now, is Travis set up so it’s actually running a headless browser to run those tests? To my knowledge you’d have to be using PhantomJS to achieve that. And, if so, are there any instructions on how to set this up?

I’d love to make it easier for modules to include Behat tests, and have them integrated with CI. I started up a document on this: https://gist.github.com/anselmdk/ee38e089080e3c0b4b88
I’d like to add instructions on it on how to configure that CI server, and ultimately contribute it to the SilverStripe docs. Given of course that people think this is a good idea - which probably depends on whether the core group feels like staying with Behat. Thoughts?


Anselm

[Marcus Nyeholt]

I personally have never been able to get into Behat properly; the time investment to get things working for relatively simple tests was prohibitive to the point that I couldn't continue pursuing it to get over that initial hurdle - though I've heard that once you get through that barrier, things get much simpler. Instead, I've used selenium directly with some wrappers around common CMS tasks (ie ->login(), ->createUser(), ->createPage(), ->fillOutForm()). Anyway, rather than this be a thread of A vs B, given I'm not strongly for either side, some things I've used for the actual question - "Travis set up so it’s actually running a headless browser to run those tests? "

For executing selenium, I run it using xvfb; the browser instance that selenium starts for running tests inherits this environment, so executes in a fully headless manner

/usr/bin/xvfb-run -e /var/log/selenium/xvfb-selenium.log -n 10 -s "-screen 10 1024x768x8" java -jar /home/marcus/programs/selenium-server-standalone-2.52.0.jar -port 4444 -log /var/log/selenium/server.log

I can't remember how the underlying drivers work for behat, but I imagine it shouldn't be overly difficult to do something similar. https://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-a-GUI has some info on getting things started for travis.

Anselm

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Visit this group at https://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.

[Anselm Christophersen]

That’s a great start, Marcus,

regarding your issues, I remember it being a bit of a pain to get it running, but I actually think the main part was getting Selenium and the browser working. The SilverStripe installer comes bundled with configuration file that should get you going (https://github.com/silverstripe/silverstripe-installer/blob/3.2/behat.yml).

The big plus for me for using the built-in Behat library, is that, as mentioned, it let’s you start testing the CMS without no, or close to no actual coding.