|GSOC 2012 - Student wanted for "Improve Testing Framework"||Ingo Schommer||3/26/12 2:30 AM|
I'm still looking for a student to take on a fairly central piece in
the future development of the CMS interface: The behaviour testing framework.
Its a great opportunity to work with cutting edge tech in the PHP space,
as well as adding a useful skill to your portfolio - employers
love a documented personal focus on software quality.
When starting with this, I've also found it quite rewarding as a programmer
to shift gears from lowlevel implementation tests to describing business
functionality - it gets you more into a client-focused mode of thinking,
again a great asset for your career.
Alright, enough sales pitch - anybody keen? ;)
P.S.: Ryan Dao, if you can still be swayed to switch from your ecommerce
project idea, let me know :)
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Sayak Sarkar||3/26/12 2:43 AM|
My name is Sayak Sarkar and I am a 2nd semester M.Sc.(Computer
I am a regular FOSS contributor and also do some Bengali translations
Also I am proficient with web content management systems such as
I've also got a good grasp on the basic functioning of Content
I take a keen interest in exploring through open source content
I'm interested in taking up the "Improve Behaviour Testing Framework
However, I am certain that with some guidance I will be able to grasp
I would be very grateful if you could give me some suggestions on how
Awaiting your suggestion,
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Ingo Schommer||3/26/12 3:04 AM|
Hello Sayak, thanks for your interest! :)
In terms of getting started, I'd suggest you check out this blog post,
which guides you through the various steps required:
(haven't followed them myself, so not sure how accurate they are)
In terms of both you and us gaining confidence that the project is a good fit,
it would be ideal if you get to a point where you can log in to the CMS
via PHP-Webdriver controlled by Cuke4PHP, and maybe save a page.
Do you think that's realistic to achieve during the application period?
We have gotten this far already, so its more of an excercise for you.
The code required is currently not open source, but thats just a formality
(it would form the basis for your project work). Here's a gist to give
you an idea of our approach: https://gist.github.com/2204261
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Ingo Schommer||3/26/12 4:08 PM|
Hello again, we've just opensourced the project I talked about:
Its more of a proof of concept, and due to heavy development on SS3
most of the actual CMS behaviour tests currently fail - this could
be a nice area for you to get started as well: Try to fix those tests,
and submit us pull requests :)
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Sayak Sarkar||3/26/12 4:13 PM|
Thanks for the update, I've been working on understanding PHP BDD and
> --> https://groups.google.com/d/msg/silverstripe-dev/-/QfNtSizzHJkJ.
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Ingo Schommer||3/31/12 8:07 AM|
Hello Sayak, how you're getting on with Cuke? Anything I can help with? 6 days until the application deadline :) - All the best to India, Ingo
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Sayak Sarkar||3/31/12 12:44 PM|
Sorry, I couldn't update my status with Cuke earlier. My laptop's hard
disk crashed so had been a bit busy with recovery and stuff. I did go
through Cuke, and hope to submit a draft within Monday evening, there
are some areas which I am still going through and trying to smooth out
before submitting a plan, hence the delay.
In the meantime I've already forked the repositories and am trying to
On Sat, Mar 31, 2012 at 8:37 PM, Ingo Schommer <ingo.s...@gmail.com> wrote:
> --> https://groups.google.com/d/msg/silverstripe-dev/-/CtUWLqTt7bAJ.
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Sayak Sarkar||4/1/12 10:37 AM|
Still trying to get around the examples, though I have to confess, I'm
still not very comfortable with cucumber implementation.though I have
understood the basic idea behind creating testing frameworks.
I'm trying to try out some sample test cases, but haven't yet been
quite successful yet. Once, I sort out the the problem I would draft a
proposal and submit it. Of course if I'm unable to sort out the issues
within a couple of days, I would ask for your help. ;-)
|Re: GSOC 2012 - Student wanted for "Improve Testing Framework"||Michał Ochman||4/1/12 1:13 PM|
My name is Michał and I'll be applying to GSoC for this project. I'm
in my last year of MSc studies in Computer Science and since I have
been collaborating with Wojtek Szkutnik's Imagomedia on SilverStripe
projects for over a year, I have decided to help out in open source
development before it's too late for me ;-)
For the last 5+ years I was involved in PHP development for some big
companies such as Betfair, Ladbrokes and Bwin.com. While we did mostly
WordPress and custom PHP development, but in my free time I've been
enjoying playing around with SilverStripe. I also did lots of
and frontend. One of our last projects was Agile-driven and since we
have used all best practices of TDD development such as unit testing,
acceptance tests and other various testing techniques, I feel quite
comfortable with developing the testing framework. I have done most of
my TDD projects in Python, which involved various testing libraries
(mostly PyUnit), techniques (using mocks, factories, testing all
layers of the apps from frontend to http communication) and
technologies (lettuce, splinter /a python open source library for
webdriver/), as well as handling CI intergration (Jenkins).
I'll try to do the code test today and have a closer look at the state
of current SilverStripe unit tests - maybe I'll find some place for
improvements for the core tests as well.
> On Sun, Apr 1, 2012 at 1:14 AM, Sayak Sarkar <sayak.bugsm...@gmail.com> wrote:
|Re: [silverstripe-dev] Re: GSOC 2012 - Student wanted for "Improve Testing Framework"||Ingo Schommer||4/2/12 1:07 AM|
Hey Michal, that sounds very promising!
Same advice as to Sayak, please try to get
our proof of concept implementation going on your machine as a first step :)
Just to clarify, this project is about high-level behaviour (automated) testing
rather than unit testing, although I'm sure there's going to be some overlap
and duplication, so looking at the unit tests won't hurt.
On Sunday, 1 April 2012 at 10:13 PM, Michał Ochman wrote:
> Hello!> companies such as Betfair, Ladbrokes and Bwin.com (http://Bwin.com). While we did mostly
> On Apr 1, 7:37 pm, Sayak Sarkar <sayak.bugsm...@gmail.com (http://gmail.com)> wrote:
> > On Sun, Apr 1, 2012 at 1:14 AM, Sayak Sarkar <sayak.bugsm...@gmail.com (http://gmail.com)> wrote:
> > > On Sat, Mar 31, 2012 at 8:37 PM, Ingo Schommer <ingo.schom...@gmail.com (http://gmail.com)> wrote:> > > > To post to this group, send email to silverst...@googlegroups.com (mailto:silverst...@googlegroups.com).
> > > > silverstripe-d...@googlegroups.com (mailto:email@example.com).
> --> To post to this group, send email to silverst...@googlegroups.com (mailto:silverst...@googlegroups.com).
> To unsubscribe from this group, send email to silverstripe-d...@googlegroups.com (mailto:firstname.lastname@example.org).
|Re: [silverstripe-dev] Re: GSOC 2012 - Student wanted for "Improve Testing Framework"||Michał Ochman||4/4/12 9:59 PM|
Obviously it's about high level behaviour, I was just pointing out that I will have a look at the unit tests since it's one of the best ways of getting to know the code base.
I have managed to set up the testing environment and have a few remarks:
First of all, as a general note - in an ideal world, we would have a separate command to set up a test environment, then run the selenium server, then execute the tests - this way we could simulate the production setup and make sure that all will run well after deploying the code. Unfortunately, since it's almost not doable in PHP, we'll have to make the best out of it and just make a simple script for checking/running the selenium server and then running the tests - to make it a one-click operation.
Also, I'll have to check how it works with cuke4php, but I know that in bigger testing environments it's handy to separate testing modules and include only the really needed functions - when I did similiar things in Python, we had a common_steps file with the most popular steps and module-specific step libraries. From what I see, there is a lot of features to introduce - form handling, all kinds of JS stuff etc - looks like an interesting job to do!
One last question - do core SilverStripe developers use some kind of CI environment right now? If not, it would be very handy to set up a test runner which would execute unit tests and selenium tests on every commit (I would suggest jenkins). We'd just need to set up a headless environment for selenium tests (they need some kind of X simulation to run the "browser tests").
As a last remark, it would be very nice to easily integrate with possible selenium tests included by module/widget developers - this way the runner would include not only the core tests but also tests introduced by installed plugins.
Sorry for my disordered train of thought, I was just writing down all things that came to my mind to keep them in mind on a later stage ;-) I will start polishing my GSoC application right now to make sure to get it to you on time. If you have any remarks, please let me know - I'll be glad to iterate over my ideas to make the summer project as exciting as possible.
Looking forward to getting involved in SilverStripe development!
|Re: [silverstripe-dev] Re: GSOC 2012 - Student wanted for "Improve Testing Framework"||Ingo Schommer||4/5/12 1:54 AM|
Agreed that it should be "one click" if possible, and that we need improved install docs- Pull requests welcome :)
SilverStripe the company is 99% on OSX for dev machines, so our preferences tend to show in these early stage projects.
Ideally its not an either-or decision, one motivation of using WebDriver is its cross browser support.
We'll have to see how realistic it is to use a behaviour test suite as cross browser smoke tests,
vs. how much work it is to deal with the various quirks (as you mentioned).
If you look at the class structure, we're already trying to build layers of "common steps",
and I think its crucial for developer adoption to have a clearly structured library of reuseable steps.
Otherwise we'll have devs implementing the same steps all over again, leading to a perceived
lack of productivity on the tool.
We're using buildbot (buildbot.silverstripe.org), but mostly migrated to TeamCity (http://teamcity.silverstripe.com, there's a guest login for the open source stuff).
The idea is to run all our CI on the same environment, including the behavioural tests. Lower priority for the GSOC project iself, although it would be handy.
Headless environments: Not sure if its worth the trouble, as I imagine debugging problems (DOM/JS/CSS) gets quite difficult. Do you have
any experience using headless browsers in non-trivial app tests?
Yeah, we follow that philisophy with our PHPUnit runs as well, so its just natural to extend it to behavioural tests as well.
All the best
|Re: [silverstripe-dev] Re: GSOC 2012 - Student wanted for "Improve Testing Framework"||Michał Ochman||4/12/12 1:37 AM|
Moving the discussion from Google Melange here :)
Ingo wrote: Completely agree with the lack of fixture support in the current proof of concept, that'll be one of the first things to fix in the GSOC project to enable meaningful testing. Have you looked at the way we generate fixtures through YAML in SilverStripe? Good find with Phactory, but I think we're 2/3 there with the SapphireTest+YamlFixture classes already. One of the base assumptions in the proof of concept is that the steps can use the underlying SilverStripe PHP framework, so we can generate fixtures programmatically, and even share them between PHPUnit and Cuke tests.
I went through the YAML fixtures feature and gave it some more thought during the last few days. I agree that YAML fixtures are very useful in some cases, but from what I see and understand, you always have to generate them first. It's not a problem when you test something like things described at http://doc.silverstripe.org/sapphire/en/topics/testing/create-sapphire-test , but I think that there are some cases where some kinds of Factories may come very useful. Obviously, we don't have to decide on this until I'm proved right or wrong during the development phase, but I believe that we'll end up using both, anyway. I wrote down a couple of reasons for why Factories may be better than Fixtures for testing:
After all, fixtures are pre-defined sets of data. You can import some parts of them but in general, one fixture fits only one purpose. A Factory is much more versatile - if you want to generate a hundred objects and don't care about their properties, you just do ObjectFactory::create() a hundred times. However, if you want to generate objects with specific attributes, you don't have to use another factory - you just need to specify an argument like in ObjectFactory::create(color=yellow) to get a hundred of yellow objects. This way, you can test every behaviour related to an object, using just one tool and don't have to generate fixtures on a per-case basis
2. Code readability
Using fixtures moves part of the testing implementation to an external file. This means, that to fully understand the tests you need to browser through the test files and then to fixtures files. The purpose of tests is to test specific behaviour, so that when something breaks we can browse the tests code and see what's happening there as fast as possible. Factories provide clean code integrated with tests, while fixtures require extra attention from developers.
Imagine that you want to change some part of the tests. If you re-use fixtures in multiple tests, it's highly possible that some of the tests will break when you change a specific fixture. Factories provide a solution which make tests fully independent from other tests. Also, if there is more than one developer working on tests using fixtures, there is no easy way to know whether a fixture is used in just one test or a hundred. Again, factories fix that ;-)
Since implementing the use of Phactory or any other Factory library would be pretty straightforward, I think that this isn't much of a problem - if we decide that Factories would be useful in the testing process, I can implement them as part of my GSoC task, and if for some reason we decide to struggle with a fixture-only testing system, we can just go on with what we have. My point is that factories are very easy to implement and would be probably a huge boost to the testing process, while fixtures would be still very useful for pre-filling the database with some standard records which are not test-specific.
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Ingo Schommer||4/12/12 7:33 AM|
On 12/04/2012, at 10:37 AM, Michał Ochman wrote:
> Moving the discussion from Google Melange here :)
> I think that there are some cases where some kinds of Factories may come very useful. Obviously, we don't have to decide on this until I'm proved right or wrong during the development phase, but I believe that we'll end up using both, anyway.Agreed, we're actually already using crude fixture generators in our projects, usually just custom code in SapphireTest->setUp().
There's really no way around this once you get into non-trivial fixtures (lots of repetition or large dependency graph).
On a recent project, we generated half-completed orders based on certain product criteria,
to simulate taking them through the booking process.
We also dynamically generated SOAP XML fixtures based on the data submitted to the mock -
both quite bespoke generators which would be hard to generalise into a factory class, right?
>Can you provide a specific example where testing core CMS functionality would become unmaintainable with YAML fixtures?
I guess something like ModelAdmin pagination would need 20+ records to work, and you'd need a bit of variation to test the effectiveness of search.
I'd suggest that we timebox this to a couple of days proof of concept - I don't want us to get stuck in something that's only marginally related to the GSOC project.
Specifically, how much value we can get out of Phactory against using something more bespoke, but closer to our ORM definitions. What do you think?
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Michał Ochman||4/13/12 3:31 AM|
Actually, I don't think that there is anything that you can do with fixtures that you can't do with factories. In the worst case scenario, you can use factories the same way that you would use fixtures, but from inside the code. Factories handle dependencies real well and you can both use sub-factories and not have to worry about the details, or specify custom dependencies to make sure that you get exactly what you need.
As for maintainability, the whole idea of YAML fixtures seems like kind of a workaround rather than a proper solution - maybe I'm just used to factories, but it seems like the simpliest way to handle multiple object generation. There is a lot of discussions about similiar issues in Rails and most of them speak in favour of factories. I think that I'll just take a day or two to make a proof of concept at the beginning of GSoC or sometime in May and if it doesn't take off - well, it won't make much harm.
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Ingo Schommer||4/13/12 5:41 AM|
If you look in our core unit tests, the fixtures are usually quite small,
and for the amount of characters used quite expressive, particularly
when it comes to relation aliasing (Parent: =>SiteTree.my-parent).
Even if you ignore any PHP APIs, just using PHP's array() notation
will make this less readable than YAML. This is one of our longer ones:
Even at this length and level of repetition, I find it fairly easy to follow.
I think there's a tension between readability and maintainability here,
but my point is that it needs to be decided based on project requirements.
Then again, I've never worked with a generic factory API.
Had a look at a popular Rails one, and can definitely see its appeal:
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Sam Minnée||4/13/12 9:43 PM|
Ingo, I know that for projects with more complex domain models (say a booking engine, as a hypothetical example ;-) ) Factories can help abstract away the fixture plumbing. Mark Stephens might also have some opinions base on his current project.
I don't think we have a sufficiently complex domain model in core to make much of a difference, but as a tool in the arsenal it could be useful.
|Re: [silverstripe-dev] GSOC 2012 - Student wanted for "Improve Testing Framework"||Michał Ochman||4/18/12 1:07 AM|
Introducing some basic factories will not be much of a hassle so it won't interfere with the project and I will be able to implement some basic proof of concept. I already started going through Phactory docs to have a head start.