[scala-ide-dev] Automated testing for the Scala-eclipse-IDE by Squish

[Slartibartfast]

Hello together,

I´ve got excellent news for the scala-eclipse-IDE community:

Thanks to the support from Froglogic (http://www.froglogic.com ) we
have now licenses for one of the leading commercial automated GUI
testing tools.
I personally know Squish [2] since 2007, when I saw a fascinating live
presentation by Reginald Stadlbauer [3]. Since then Squish evolved
even more. In the industry project I´m currently working at (> 10 man
years already invested) we´ve thoroughly evaluated many commercial GUI
testing tools in the market and came to the solution that Squish is
the best!

Now we have Squish to test the Scala-eclipse-plugin!

I personally think that the importance of automated (GUI) testing can
hardly be overestimated.
With the help of Squish we can assert a high quality of released
versions in the long term. To achieve this we need a set of good unit
tests.
At the moment Miles and I have licences for Squish and I´m just
starting to setup everything together with Miles.

A few words about myself:
I´m a software architect working for a software development and
consulting company near Zurich ( http://www.zuehlke.com/ ). I´ve deep
knowledge of Java since 1998 and am actively developing Java
applications. My first contact with Scala was in 2009 during which I
had the chance to spend a full week to discover Scala.

As I’ll be working on the Scala GUI testing mainly voluntary in my
free time, please be patient if I cannot answer all questions and
respond to discussions immediately.

My initial plan to test the Scala eclipse plugin is as follows:
- The Scala-IDE-Testing team (currently only me, volunteers welcome!)
will develop tests based on Squish
- Miles creates a new release candidate and deploys all artifacts to a
local update Site <LUS>
- Miles invokes a single test script <test-scala-ide.bat> which will
do the following
+ Extract a eclipse installation (without scala) as downloaded
from eclipse.org
+ Invoke eclipse.exe
+ Download and install the Scala-IDE from the local update Site
<LUS>
+ Perform automated tests (to be written and continuously
extended)
+ Create a test report
- Based on this test report, which is hopefully mainly green, Miles
can decide whether to deploy the release candidate to the global
accessible update site (likely by copying files from the local update
site) or not yet.

Please note that this is just an initial plan and subject to change
based on future findings and discussions.

If you have any suggestions, questions or comments feel free to
contact me, either on this mailing list or by email scala-ide-
te...@bulitta.com .

Cheers,
Chris


Links:
[1] http://www.froglogic.com
[2] http://www.froglogic.com/pg?id=Products&category=squish&sub=editions&subsub=java
[3] http://www.java-forum-stuttgart.de/jfs/2007/folien/F3.pdf

[Miles Sabin]

On Sun, May 2, 2010 at 3:13 PM, Slartibartfast <goo...@bulitta.com> wrote:
> I´ve got excellent news for the scala-eclipse-IDE community:

Thanks Chris! This is indeed excellent news :-)

Please feel free to create new pages in the developer area of the Wiki
to support the project,

http://scala-ide.assembla.com/wiki/show/scala-ide/Developer_Documentation

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype: milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Matt Russell]

Hi Chris,

I very much support the idea of automated GUI testing for the Scala
IDE, thanks for taking the initiative on this!

My only concern would be, as an open source project, how readily can
contributors get access to these testing tools? For example, if
someone contributes a bug fix or new feature, it would be good if they
created tests for it, and to be able to run the other tests locally to
reduce the risk of having broken something. Is Froglogic willing to
make Squish licenses available to any contributor to the project?

-- Matt

On May 2, 3:13 pm, Slartibartfast <goo...@bulitta.com> wrote:
> Hello together,
>
> I´ve got excellent news for the scala-eclipse-IDE community:
>

> Thanks to the support from Froglogic (http://www.froglogic.com) we

> have now licenses for one of the leading commercial automated GUI
> testing tools.
> I personally know Squish [2] since 2007, when I saw a fascinating live
> presentation by Reginald Stadlbauer [3]. Since then Squish evolved
> even more. In the industry project I´m currently working at (> 10 man
> years already invested) we´ve thoroughly evaluated many commercial GUI
> testing tools in the market and came to the solution that Squish is
> the best!
>
> Now we have Squish to test the Scala-eclipse-plugin!
>
> I personally think that the importance of automated (GUI) testing can
> hardly be overestimated.
> With the help of Squish we can assert a high quality of released
> versions in the long term. To achieve this we need a set of good unit
> tests.
> At the moment Miles and I have licences for Squish and I´m just
> starting to setup everything together with Miles.
>
> A few words about myself:
> I´m a software architect working for a software development and

> consulting company near Zurich (http://www.zuehlke.com/). I´ve deep

> t...@bulitta.com .

>
> Cheers,
>   Chris
>
> Links:
> [1]http://www.froglogic.com

> [2]http://www.froglogic.com/pg?id=Products&category=squish⊂=editions&...
> [3]http://www.java-forum-stuttgart.de/jfs/2007/folien/F3.pdf

[Slartibartfast]

Hi Matt,

> Is Froglogic willing to
> make Squish licenses available to any contributor to the project?

I can´t tell how many licenses we can get in total.
I think there are possibilities to increase the number of licences by
a few (not hundreds) for other future members of the test-team, but we
have to ask Froglogic.

In the future the process might be as follows:
- Anyone makes a contribution to the IDE and verbally describes test
cases.
- The test team records these testcases in a prioritized backlog to be
implemented by member of the test team (having a licence of squish).
- A member of the test team implements the testcases.
- Simple maintenance of a testcase can be done by everybody as squish
testcases are simple ASCII file, e.g. written in phyton, JavaScript or
Perl

Alternatively we can think of templates of commonly used testcases
like refactorings in the form:
- <inputFile>.scala
- refactoring , e.g. rename "a" --> "b" or extract method "x"
between lines 56...78
- <expectedOutputFile>.scala

Unfortunately I don´t know any other GUI testing tool which is as
powerful as Squish (especially no open source one).

Cheers,
Chris

[Matt Russell]

Hi Chris,

I hope you don't mind if we could explore this a little further. I
dislike being negative, and I'd like to emphasise that I 100% support
the idea of automated GUI testing, and if we have to use a proprietary
tool to do so, then it's much better than not doing it at all. And it
is very generous of Froglogic to offer licences to the project.

To cut to the point: how much advantage is there to using Squish over
the alternatives? I would argue there is a cost in using a proprietary
tool in that it creates a barrier to participation. One of the current
goals of the project is to remove those barriers and get more people
contributing -- that was the focus of Miles's presentation at
ScalaDays, and, as I understand it, one of the motivations for
rehosting the project at Assembla. If the advantages from using Squish
are only slight, then perhaps it makes sense to use a less powerful
open source tool instead, but with more people able to get involved?

Just a bit of background on where I'm coming from: at my company, we
do GUI testing with an in-house testing framework tailor-made for our
application. Test scripts are written in Java against a high-level API
containing all the sorts of actions you can perform in our GUI. The
framework is implemented using a low-level library to manipulate GUI
widgets (Swing in our case). Our process is that developers create
test scripts to demonstrate both bug fixes and new features. We've
found it very advantageous for developers to be able to directly
create and run the tests, not just our testing team. (We evaluated
Squish at one point; I wasn't involved, and I'm not sure why we didn't
use it in the end.)

-- Matt

[Slartibartfast]

Hi Matt,

no worries, it good to discuss these things initially.

I personally am not (yet) an expert in GUI testing and have not yet
done this in any project.
I´ve only watched several presentations at conferences talking about
this topic, including the presentation about squish.

My personal resume of these conferences has been that GUI testing can
involve a really huge amount of work, especially when it comes to the
details.
With squish I had the feeling that the initial and overall maintenance
effort is the best.
I personally would not dare to tailor a GUI testing framework for the
Scala-IDE.

The testmanager in our project has done the evaluation of the GUI
testing tools and also for him Squish was best suited (I´ll ask him
tomorrow what his reasons have been in detail to use squish).
Sure the drawback is that not everybody can write GUI tests using
squish.
But as long as the committers willing to write UI tests are not too
numerous I´m quite optimistic that those who want to perform tests
using squish can get licences.

From my point of view the usage of Squish does not forbid other
contributors to use other GUI testing tools to test their
functionality.
Hence the core functionality could be tested using Squish while in
parallel other functionality can still be tested using other GUI
testing frameworks.

Therefore I´d recommend to setup the (squish)-gui-tests as separate
module, so that it can be executed separately and does not infer with
the build of the plugin itself.

What would be your recommendation to do?
Do you have any other open-source GUI testing tool at hand, which you
´d recommend to use?

-- Chris

[Miles Sabin]

On Sun, May 2, 2010 at 4:52 PM, Matt Russell
<mattru...@googlemail.com> wrote:
> I very much support the idea of automated GUI testing for the Scala
> IDE, thanks for taking the initiative on this!
>
> My only concern would be, as an open source project, how readily can
> contributors get access to these testing tools? For example, if
> someone contributes a bug fix or new feature, it would be good if they
> created tests for it, and to be able to run the other tests locally to
> reduce the risk of having broken something. Is Froglogic willing to
> make Squish licenses available to any contributor to the project?

It's not either/or: if people want to use other GUI testing tools
(SWTBot is the obvious free and open source candidate) then they can.
And if additional Squish licenses aren't available when we need them
then SWTBot tests would just fine.

When I spoke to Chris at Scala Days he expressed a very strong
preference for Froglogic's tools over SWTBot, based on experience with
both. Seeing as he's volunteering to get going on constructing tests
and has contacted Froglogic and obtained free licenses, I don't really
see any objections.

Obviously the tests themselves aren't part of the software that ships,
so a testing framework not being open source isn't an issue directly.
It's tempting to see the situation as analogous to Linus's
controversial use of BitKeeper in the pre-git days, but I think not
really, because whilst for any given project you have to agree on a
single VCS, there's no comparable pressure to agree on any single
testing framework ... we can let a hundred flowers bloom.

It'd be good if Reginald would make a comment on what his approach to
allocating license is likely to be ...

Cheers,


Miles

--
Miles Sabin

tel: +44 7813 944 528
gtalk: mi...@milessabin.com

[Joachim Hofer]

As I've got a lot of experience with GUI testing (Swing, not SWT,
though), here's my lengthy 2 cents on the matter. - in short: GUI
testing is not as easy as it might look at first. However, it can be
very effective in finding bugs.

From what I've seen of SWTBot (admittedly, that's not very much), it
doesn't look very feasible... - a tool like Squish or QFTest can do a
lot more.

In the company I'm working for, we've set up a project build on the
Hudson CI server that runs the unit tests (TestNG) and the integration
tests (also TestNG) immediately after every commit. Additionally, each
night, the GUI test automation via QFTest runs. And yes, although
QFTest is quite fast, it takes all night (I'm talking about ~10000
basic test steps here, which is not really a large number of tests).

So here's my experience with this:
- GUI testing is slow. You can't just run it during your standard
build. Especially because you can't do anything else while it's
running (it "uses your keyboard and moves your mouse").
- GUI testing is not easy to integrate into your build. Even with
commercial tool support. Often, build servers run in headless mode,
but you can't test GUIs in headless mode. Thus, you have to run X
servers or virtual machines and you have to take care that they don't
crash/hang and all that...
- GUI tests break easily. You have to be very careful to get them to
be robust against even small changes of the GUI. Also, they are
fragile when it comes to slight differences in platform-specific gui
behavior (focus behavior, for example).
- GUI test results are often not easy to interprete. It's often not
immediately obvious whether the automation is broken or the code under
test is broken. And if the code is broken, it's not as easy to find
out where it's broken as it is with unit/integration tests. In
addition to the log files QFTest generates for us, and although these
already contain screenshots and stuff, we're also capturing the whole
nightly GUI test run on video for that reason.
- GUI tests are system-level tests. You better have unit and/or
integration tests in place first.
+ Often, use cases you want to test can be most easily modeled by GUI
tests.
+ GUI tests find many bugs, and are especially good at finding
difficult bugs.
+ When GUI testing, you don't need much insight into the code under
test. It's black box testing and would therefore be easy on people not
knowing about the internals of the Scala IDE.

Now don't get me wrong. I'm very much in support of GUI testing
despite all the listed disadvantages, simply because it can be so
effective, and it's great that the Scala IDE now has this option
thanks to Squish.

But seeing that there are nearly no unit tests (especially no PDE
tests) against the Scala IDE in place at the moment, I think we could
get more results faster and with lower entry barrier there. PDE tests
offer powerful capabilities, too: you can easily use the JDT
programatically from within them. - See my few simple PDE tests
exercising the Java Search Engine programatically against the Scala
IDE. It should also be relatively easy to integrate PDE tests into the
build, so that every contributing developer can run them all the
time. :)

Of course, it all depends on who contributes what... and imho, the
more options there are concerning how to contribute, the better.

Cheers,
Joachim

[Miles Sabin]

Reginald isn't subscribed to scala-ide-dev, so I'm forwarding his response ...


---------- Forwarded message ----------
From: Reginald Stadlbauer <reg...@froglogic.com>
Date: Tue, May 4, 2010 at 5:18 AM
Subject: Re: [scala-ide-dev] Re: Automated testing for the
Scala-eclipse-IDE by Squish
To: Miles Sabin <mi...@milessabin.com>
Cc: scala-...@googlegroups.com


Hello everybody,

On 03.05.10 22:51, Miles Sabin wrote:
>
> On Sun, May 2, 2010 at 4:52 PM, Matt Russell
> <mattru...@googlemail.com>  wrote:
>
>>
>> I very much support the idea of automated GUI testing for the Scala
>> IDE, thanks for taking the initiative on this!
>>
>> My only concern would be, as an open source project, how readily can
>> contributors get access to these testing tools? For example, if
>> someone contributes a bug fix or new feature, it would be good if they
>> created tests for it, and to be able to run the other tests locally to
>> reduce the risk of having broken something. Is Froglogic willing to
>> make Squish licenses available to any contributor to the project?
>>

[...]

> It'd be good if Reginald would make a comment on what his approach to
> allocating license is likely to be ...
>

Not sure if 'm allowed to post to scala-ide-dev but let's try :-)

Before answering the question, I'd like to mention that I myself, the
other founder for froglogic (Harri) as well as several of our
engineers come from an open source background (KDE developers of the
early days) and several are still active KDE conributors.

So we very well understand how open source projects work and are very
supportive of open source software.

That said, I can also understand the concerns of using a commercial
tool in an open source project. But as Miles stated, GUI test
automation is not a core part of the project and can be seen as
optional. It is similar to when KDE got free licenses of Purify and
Insure to do memory debugging on platforms where there was no
valgrind, or Coverity to do code checks.

To answer the question regarding additional, free licenses: We can
certainly give out more licenses to contributors as need arises. But
we need to ensure that this is really used for the Scala IDE project
by maybe having someone from the Scala IDE core team confirming us
that the requester is a contributor.

If demand would rise a lot we could create an edition of Squish Java,
similar to Squish/KDE, which works only for the Scala IDE and can the
be distributed more easily. But I guess time will tell...

Best Regards / Mit freundlichen Grüßen
Reginald Stadlbauer

--
Reginald Stadlbauer - co-founder&  CEO
froglogic GmbH - Automated UI and Web Testing
reg...@froglogic.com - +49 40 78897137

Pflichtangaben gemäß § 35a GmbHG / § 125a HGB unter
http://www.froglogic.com/corp

[Miles Sabin]

On Tue, May 4, 2010 at 8:39 AM, Joachim Hofer <jmhofer...@johoop.de> wrote:
> So here's my experience with this:

<snip/>

This tallies with my experience as well. In my experience the
potential fragility of tests means that you need a flexible tool and a
dedicated test maintainer ... for me that speaks very much in favour
of letting the people volunteering to do the work choose the tools
they feel are best suited to the job.

> But seeing that there are nearly no unit tests (especially no PDE
> tests) against the Scala IDE in place at the moment, I think we could
> get more results faster and with lower entry barrier there.

<snip/>

> Of course, it all depends on who contributes what... and imho, the
> more options there are concerning how to contribute, the better.

Again, I don't see this as either/or ... the great thing about the few
weeks since launching scala-ide.org is that we have new contributors
stepping forward to make progress in both of these areas.

I think that's wonderful ... thanks!

[Miles Sabin]

On Tue, May 4, 2010 at 9:19 AM, Miles Sabin <mi...@milessabin.com> wrote:
> To answer the question regarding additional, free licenses: We can
> certainly give out more licenses to contributors as need arises. But
> we need to ensure that this is really used for the Scala IDE project
> by maybe having someone from the Scala IDE core team confirming us
> that the requester is a contributor.
>
> If demand would rise a lot we could create an edition of Squish Java,
> similar to Squish/KDE, which works only for the Scala IDE and can the
> be distributed more easily. But I guess time will tell...

Matt (and everyone else) ... does this address your concerns adequately?

Cheers,


Miles

[Matt Russell]

On May 4, 9:31 am, Miles Sabin <mi...@milessabin.com> wrote:

> letting the people volunteering to do the work choose the tools they
> feel are best suited to the job.

I very much agree with this.

> Matt (and everyone else) ... does this address your concerns adequately?

Yes, it does for me. Thanks again to Froglogic for its support.

-- Matt

[Slartibartfast]

Hello everybody,

thanks very much for this fruitful and objective discussions about GUI
testing and tools.
Especially I want to thank Reginald for his statement about possible
additional licenses if needed, Joachim for his detailed experiences
with GUI tests and Miles for the coordination.

For me the essence of these discussions is, that we should take care
for the following things when introducing GUI tests:
- GUI-Test have definitively to be independent from the productive
code
--> we should definitely put them in a separate git
repository
- It should be easily possible for other contributors to introduce
other GUI tests based other tools like SWTbot, PDEtest, QFTest, …

Therefore I´d recommend to set up a separate git repository like
this:
- scala-plugin-test (or scala-ide.4 ?)
|- squish
|- SWTbot
|- PDEtest
|- toolXtest
|- toolYtest

There could be a single build.xml under
- scala-plugin-test (or scala-ide.4 ?)
|- build.xml

which checks for available licences and executes the respective tests.
Therefore under each tool submodule the could be a build.xml to
execute the tests based on the specific submodule:

- scala-plugin-test (or scala-ide.4 ?)
|- <toolSubModule>
|- build.xml

e.g.
- scala-plugin-test (or scala-ide.4 ?)
|- build.xml <<--- executes tests from all submodules
|- squish
|- build.xml <<--- executes all tests based on squish
|- SWTbot
|- build.xml <<--- executes all tests based on SWTbot
...

@All: do you think this is a sensible way to do it?

Cheers,
Chris

[Joachim Hofer]

This sounds sensible to me.

The PDE tests should definitely have their own directory outside of
the plugin itself because they define a fragment for the plugin (they
currently live at http://github.com/jmhofer/scala-plugin-test while
the IDE still moves from svn to git, but I can move them over to
Assembla at any time).

Probably, the other pure unit tests should move from inside the plugin
to an outside directory too, so that the Scala IDE doesn't "inherit"
any dependencies of them, either (currently, they only depend on JUnit
afaik, but as soon as someone wants to use something like a mocking
framework, we'll run into that problem).

However, I'm not sure if we should start building a complex Ant build
structure right now, when Miles wants to switch to Maven/Tycho anyway?

Cheers,
Joachim

[Miles Sabin]

On Tue, May 4, 2010 at 9:18 PM, Slartibartfast <goo...@bulitta.com> wrote:
> For me the essence of these discussions is, that we should take care
> for the following things when introducing GUI tests:
> -  GUI-Test have definitively to be independent from the productive
> code
>   --> we should definitely put them in a separate git repository
> -  It should be easily possible for other contributors to introduce
> other GUI tests based other tools like SWTbot, PDEtest, QFTest, …

Agreed.

This all sounds sensible. I'd recommend starting with something as
simple as possible though.

A couple of things to bear in mind. The main IDE Eclipse project repos
are going to be reorganized so that all the Eclipse projects are
contained in a single git repo, as described here,

http://scala-ide.assembla.com/wiki/show/scala-ide/Proposed_Repository_Reorganization

which means that scala-ide.n are going away. I think the best way
setup for this on Assembla would be to create a separate Assembla
space for this, and then add that space to the main Scala IDE for
Eclipse "project portfolio" ... it's a bit fiddly, but this does seem
to be the way that Assembla wants us to work.

The other thing is that, as Joachim mentioned we'll be moving to a
Tycho/Maven 3.0 based build very soon. I don't think that should stop
you putting together Ant scripts right now, but we might have to
review the way things are done later.

[Miles Sabin]

On Wed, May 5, 2010 at 8:21 AM, Joachim Hofer <jmhofer...@johoop.de> wrote:
> The PDE tests should definitely have their own directory outside of
> the plugin itself because they define a fragment for the plugin (they
> currently live at http://github.com/jmhofer/scala-plugin-test while
> the IDE still moves from svn to git, but I can move them over to
> Assembla at any time).

I think that this will get easier once the repository reorganization
is done ... then you can add the PDE test projects to the main IDE
repository as a sibling project.

[Slartibartfast]

Hello together,

that´s perfectly fine for me.
Until the repo-reorganisation is finished, I won´t check-in anything,
work locally and play around with Squish.

To have the advantage of atomic commits, I think we should put
everything into ONE git repository and separate everything using maven
modules, e.g.

<git-repo-root>
|-- pom.xml
|-- <productive-module-1, e.g. scala-plugin>
|-- pom.xml
|-- src
|-- main
|-- test <-- non-GUI-tests
|-- <productive-module-2, e.g. scala-plugin-aspects>
|-- pom.xml
|-- src
|-- main
|-- test <-- non-GUI-tests
|-- <productive-module-3, e.g. scala-library>
|-- pom.xml
|-- src
|-- main
|-- test <-- non-GUI-tests
|-- ...
|-- pom.xml
|-- <gui-test-module-1, e.g. scala-gui-tests-squish>
|-- pom.xml
|-- src
|-- test <-- GUI-tests based on Squish
|-- <gui-test-module-2, e.g. scala-gui-tests-pde>
|-- pom.xml
|-- src
|-- test <-- GUI-tests based on Squish

The root pom could have different profiles depending on what you want
to do, e.g.
- 1 profile for everything except of GUI tests
- 1 profile for everything including GUI tests
- 1 profile for GUI tests only

As an alternative we could also go for a nested module structure, e.g.

<git-repo-root>
|-- pom.xml
|-- <productive-module-1>
|-- <productive-module-2>
|-- <productive-module-3>
|-- <gui-tests>
|-- pom.xml
|-- <gui-test-module-1, e.g. scala-gui-tests-squish>
|-- pom.xml
|-- src
|-- test <-- GUI-tests based on Squish
|-- <gui-test-module-2, e.g. scala-gui-tests-pde>
|-- pom.xml
|-- src
|-- test <-- GUI-tests based on Squish

The 2nd option would allow us to invoke a mvn test on module "gui-
tests" without the need of having a profile for the gui-test in the
root-pom.

What do you think?

Cheers,
Chris


On 5 Mai, 09:58, Miles Sabin <mi...@milessabin.com> wrote:

[Miles Sabin]

On Thu, May 6, 2010 at 8:53 PM, Slartibartfast <goo...@bulitta.com> wrote:
> that´s perfectly fine for me.
> Until the repo-reorganisation is finished, I won´t check-in anything,
> work locally and play around with Squish.

Sounds good ... hopefully this won't take too much longer.

> To have the advantage of atomic commits, I think we should put
> everything into ONE git repository

Yes, on reflection it does make sense to be able to make atomic
commits of UI changes and the corresponding tests, so that does mean
that the tests should live in the same git repo as the other IDE
projects.

> and separate everything using maven modules, e.g.

How would that get along with Tycho?

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com

[Slartibartfast]

Hello Miles,

I think this will go pretty well with tycho.
After thinking a little bit more about it, I think it could be best to
go for an option using 2 separate maven modules:

- 1 module "scala-ide" for everything except of the GUI tests
- 1 module "gui-tests" for the GUI tests

In maven this could look like this:

<git-repo-root>/pom.xml <-- parent module of everything
<git-repo-root>/scala-ide/pom.xml <-- contains all productive
plugins, dependencies, ...
<git-repo-root>/gui-tests/pom.xml <-- contains all GUI tests for the
scala-IDE

The reason for this is, that not everybody working with mvn might be
as familiar as required with profiles:
http://maven.apache.org/guides/introduction/introduction-to-profiles.html

Very likely the simplest thing to start with is without profiles and
segment the modules to be built using a plain-old folder/(sub)module
structure.
Profiles still can be added later, e.g. depending on the build-task to
be performed.

When separating the productive modules from gui-test modules like this
would allow everybody to build the ide pretty simple:

cd <git-repo-root>/scala-ide
mvn install

--> this would build, perform-non-gui-tests and install everything
needed to create the update-site.

The submodules of
<git-repo-root>/scala-ide/pom.xml
would be the already existing modules:
<git-repo-root>
|- scala-ide
|- scala-plugin
|- scala-library
|- scala-plugin-aspects

Each having it´s own pom.xml, i.e.
<git-repo-root>/scala-ide/pom.xml
will be the parent pom of
<git-repo-root>/scala-ide/scala-plugin/pom.xml
<git-repo-root>/scala-ide/scala-library/pom.xml
<git-repo-root>/scala-ide/scala-plugin-aspects/pom.xml


The build result of module scala-ide likely is something like

<git-repo-root>/scala-ide/scala-ide-plugin/target/update-site/...

To invoke the gui tests ("downloading" and installing the IDE from the
previously built local update site) someone could do the following:

cd <git-repo-root>/gui-tests
mvn test

--> this would run all GUI tests located in submodules like

<git-repo-root>
|- gui-tests
|- squish-tests
|- pde-tests
|- SWTbot-tests

Each having it´s own pom.xml, i.e.
<git-repo-root>/gui-tests/pom.xml
will be the parent pom of
<git-repo-root>/gui-tests/squish-tests/pom.xml
<git-repo-root>/gui-tests/pde-tests/pom.xml
<git-repo-root>/gui-tests/SWTbot-tests/pom.xml


If someone does not have a licence of Squish, he can just invoke
individual GUI tests like this:

cd <git-repo-root>/gui-tests/pde-tests
mvn test

Later we can still add mvn profiles in
<git-repo-root>/gui-tests/pom.xml
which might automatically select the tests to be performed depending
on the licence status.

Summary: This is the full directory structure I´d suggest:

<git-repo-root>
|- scala-ide
|- pom.xml
|- scala-plugin
|- src
|- main
|- test <<-- non-gui-tests only
|- scala-library
|- main
|- test <<-- non-gui-tests only
|- scala-plugin-aspects
|- main
|- test <<-- non-gui-tests only
|- gui-tests
|- pom.xml
|- squish-tests
|- pom.xml
|- src
|- test <<-- gui-tests only
|- pde-tests
|- pom.xml
|- src
|- test <<-- gui-tests only
|- SWTbot-tests
|- pom.xml
|- src
|- test <<-- gui-tests only

Do you already have plans how the directory structure after the repo-
reorg will look like?

Cheers,
Chris

[Joachim Hofer]

You're assuming that PDE tests are gui tests... - I wouldn't call them
that. They're more like unit tests.

And also, I'd really suggest separating all the unit tests from the
plugin module, too, and put them into their own plugin fragment sub-
project. I know that that's not the standard Maven way of organizing
things, but it's the recommended Eclipse plugin development way, as
far as I know.

[Martin Gamwell Dawids]

On May 9, 9:22 pm, Joachim Hofer <jmhofer.git...@johoop.de> wrote:
> And also, I'd really suggest separating all the unit tests from the
> plugin module, too, and put them into their own plugin fragment sub-
> project. I know that that's not the standard Maven way of organizing
> things, but it's the recommended Eclipse plugin development way, as
> far as I know.

Yes, that is correct.

In Eclipse/OSGi tests are placed in fragment bundles (which are able
to access the internals of their host bundles). When running the
tests, the bundles and fragment bundles are loaded into an OSGi
container together with all the bundles they depend on (resolved from
the target platform). The test methods are then run from the test
fragment bundles.

By using fragment bundles, it's also possible to avoid bringing test
dependencies into the "real" bundle. Maven accomplishes this in a
different way.

Tycho is able to execute tests from a Maven build in the same way that
tests are executed from inside Eclipse. The only way to really benefit
from using Tycho to build Eclipse projects is to follow the Eclipse
conventions for organising projects.

Regards,
Michael Werner-Gram and Martin Gamwell Dawids

[Jin Mingjian]

Firstly, I am not very familiar with maven. For my experiences, in the OSGI world, the benefits of  maven declared by maveners seemly do not  exist. One thing is, the dependencies between bundles(modules) is resolved natively by OSGI. In PDE build, the dependencies will be checked. In the integration test, the runtime dependencies will be checked as well. 

I help one Chinese commercial(100+ plugins) Eclipse products to do CI things with Hudson+ant+PDE(Eclipse)  at the early of this year. All KISS things principles the CI of this products working fine. The complex dependencies makes simple things more complex. When one complex system meets one bug, it is hard to fix it.(I found one PDE regression bug about headless building, even it is the Eclipse 3.6's PDE).

Just some thoughts for reference:)

regards,

Jin

2010/5/10 Martin Gamwell Dawids <m...@maconomy.com>

[Martin Gamwell Dawids]

On May 10, 10:33 am, Jin Mingjian <jin....@gmail.com> wrote:
> Firstly, I am not very familiar with maven. For my experiences, in the OSGI
> world, the benefits of  maven declared by maveners seemly do not  exist. One
> thing is, the dependencies between bundles(modules) is resolved natively by
> OSGI. In PDE build, the dependencies will be checked. In the integration
> test, the runtime dependencies will be checked as well.

When using Tycho with Maven things are done in the same way.
Dependencies are resolved against the target platform as with PDE and
with integration tests runtime dependencies are checked.

IMHO, the benefit of using Maven + Tycho instead of Ant + PDE is that
scripting PDE from Ant is not easy and when you add Scala to the mix
you will find that PDE compiles only Java sources and puts you
uncompiled Scala sources in the JARs. (This is actually a problem for
anything else but Java, and not only Scala.) It is possible to make
PDE build Scala using something called custom callbacks. However,
reporting errors from the Scala compiler in your CI builds is neither
straightforward nor easy.

I would definitely recommend building using Maven + Tycho. We use
Maven, Tycho and Hudson for our CI builds and integration testing and
it works fine. (We have also added a custom Scala compiler plug-in to
enforce OSGi import/export constraints since the Scala compiler does
not support this out of the box.)

Regards,
Martin

[Jin Mingjian]

ADJT has done some works about PDE+ApsectJ+Java[1]. Every technology has its pros and cons, and no problem can not be solved. If Tycho is nice, no problem. I just point out some thoughts:) For my thoughts in the building of Eclipse-based products, I still think the best practice is still PDE. It is simple, powerful and eaten by Eclipse development team itself.

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=303960

2010/5/10 Martin Gamwell Dawids <m...@maconomy.com>

On May 10, 10:33 am, Jin Mingjian <jin....@gmail.com> wrote:

[Miles Sabin]

On Mon, May 10, 2010 at 3:33 PM, Jin Mingjian <jin...@gmail.com> wrote:
> ADJT has done some works about PDE+ApsectJ+Java[1]. Every technology has its
> pros and cons, and no problem can not be solved. If Tycho is nice, no
> problem. I just point out some thoughts:) For my thoughts in the building
> of Eclipse-based products, I still think the best practice is still PDE. It
> is simple, powerful and eaten by Eclipse development team itself.
>
> [1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=303960

I have that more or less up and running, but in truth it's still pretty clunky.

The consensus at EclipseCon this year is that PDE build is as near as
anything deprecated, so even if it has been the case in the past, I'm
pretty sure it's no longer true that PDE is the best practice. Of the
candidates for replacing PDE build I think that Tycho looks the most
promising right now ... though I would be very grateful to hear other
opinions on this.

[Martin Gamwell Dawids]

I forgot to mention that unlike Ant+PDE, Maven+Tycho integrates nicely
with Hudson. This is also true even though Tycho requires Maven 3
which is currently not released (but in beta) and therefore not
provided as embedded build in Hudson. (Hudson starts Maven 3 as an
external process.) The integration is expected to be even better (as
with Maven 2) when Maven 3 is released and embedded in Hudson.

Regards,
Martin

[Joachim Hofer]

> The consensus at EclipseCon this year is that PDE build is as near as
> anything deprecated, so even if it has been the case in the past, I'm
> pretty sure it's no longer true that PDE is the best practice. Of the
> candidates for replacing PDE build I think that Tycho looks the most
> promising right now ... though I would be very grateful to hear other
> opinions on this.

I haven't tried it out myself yet, but there's now a plugin for sbt
called "bnd4sbt" (see http://wiki.github.com/weiglewilczek/bnd4sbt/)
for generating OSGi bundles from sbt.

This, to me, looks promising (after fighting with the Maven/Tycho
installation for a whole evening)... - of course, sbt won't generate
an update site for you or anything.

Cheers,
Joachim

[Stefan Langer]

That is something that can be fixed by creating a plugin for
generating update sites. Any takers? ;) I'd really like to have this
as I'm planning to building sbt-eclipse-integration using sbt in order
to eat my own dogfood.

-Stefan

2010/5/11 Joachim Hofer <jmhofer...@johoop.de>:

[Jin Mingjian]

great job, very promising, I will try it  definitely;)

2010/5/11 Stefan Langer <mailto...@googlemail.com>