Why are we using Bamboo?

[Santiago Suarez Ordoñez]

Been trying to work with bamboo for the last few days and I got stuck on the following issues.  I'm hoping someone will be able to solve them. Otherwise, I propose getting rid of Bamboo as soon as we can.

1) When builds hung, bamboo is able to detect it, but after spending over 15 minutes in the UI, I can't find a configuration that will allow me to set bamboo to kill them. I see this as either a lack of functionality or a terrible job in usability.
2) I'm willing to set Bamboo to send emails, so that we know when things break. It doesn't have it's own SMTP server and we have to set one ourselves, which is annoying considering CI servers rely so much in email notifications that they should have figured this out already.
3) Having failed in #2, tried setting my own gmail as the smtp server so that bamboo can just use it as a proof of concept. It doesn't just fail somehow but it shows me the most cryptic error message I've seen in a while. This is the message:

An unexpected error has occurred: The mail settings you entered were not valid. Error thrown was: com.sun.mail.smtp.SMTPSendFailedException: 530 5.7.0 Must issue a STARTTLS command first. g28sm2822287qck.39 .

I'd be interested in hearing wether someone finds something useful in it...
4) Having failed in #3, I decided to try to set an IRC bot to notify us on #selenium. I've failed on that too, as IRC is not an option in the notifications section and there's no easy way to get new plugins from within bamboo itself...

After spending over 3 hours spread in the last 3 days without any real progress getting our build to a decent point, I declare bamboo is the worst CI server I've used so far (this is my personal opinion and not of the rest of the team).

Now my question is: Considering this, plus the fact that bamboo agains the project's philosophy by not even being open source, why are we bearing with it? I'd like to hear some good rationale in this and possible solutions to the current state of things from people.

Now here's my proposed solution: I propose myself as the maintainer of a CI system based in Jenkins. This doesn't make me happy and probably some people will disagree, but having our current CI testing situation into consideration (we don't really have one), seems like a clear improvement to what we have so far and I'm willing to make the sacrifice for the sake of the project and future releases. This doesn't just mean I'll spend the needed time to set it up, but also that once that's done, I'll dedicate up to 4 hours a week making sure we can rely on a decent testing environment that will make sure we don't fuck things up when we release.

Thoughts?
Santi

[Daniel Wagner-Hall]

On 9 April 2011 21:06, Santiago Suarez Ordoñez <sant...@gmail.com> wrote:
> 3) Having failed in #2, tried setting my own gmail as the smtp server so
> that bamboo can just use it as a proof of concept. It doesn't just fail
> somehow but it shows me the most cryptic error message I've seen in a while.
> This is the message:
>>
>> An unexpected error has occurred: The mail settings you entered were not
>> valid. Error thrown was: com.sun.mail.smtp.SMTPSendFailedException: 530
>> 5.7.0 Must issue a STARTTLS command first. g28sm2822287qck.39 .
>
> I'd be interested in hearing wether someone finds something useful in it...

Gmail forces TLS. Bamboo doesn't support TLS other than using JNDI.
And it doesn't support custom JNDI unless you're explicitly running in
a servlet container, rather than the standalone app (which we do).
Which is awesome.

I've set Bamboo up to use a mailserver I own (we should maybe set up
Postfix for openqa.org, maybe even on the xserve itself, but for now
this one will at least send email) - email should work now, if it's
enabled for any builds for any users...

> After spending over 3 hours spread in the last 3 days without any real
> progress getting our build to a decent point, I declare bamboo is the worst
> CI server I've used so far (this is my personal opinion and not of the rest
> of the team).

I've used worse! But it's definitely up there! (I'm disappointed I've
not yet found one that does everything I want it to... Though I've not
tried Go yet, I really should give that a go...)

> Now my question is: Considering this, plus the fact that bamboo agains the
> project's philosophy by not even being open source, why are we bearing with
> it? I'd like to hear some good rationale in this and possible solutions to
> the current state of things from people.
>
> Now here's my proposed solution: I propose myself as the maintainer of a CI
> system based in Jenkins. This doesn't make me happy and probably some people
> will disagree, but having our current CI testing situation into
> consideration (we don't really have one), seems like a clear improvement to
> what we have so far and I'm willing to make the sacrifice for the sake of
> the project and future releases. This doesn't just mean I'll spend the
> needed time to set it up, but also that once that's done, I'll dedicate up
> to 4 hours a week making sure we can rely on a decent testing environment
> that will make sure we don't fuck things up when we release.

IIRC we used to have a Jenkins (Hudson) set up, and we moved to
Bamboo? (Or maybe we just had a very old Bamboo install which we
upgraded) - certainly I had my own Hudson instance set up at some
point because I was fed up for Bamboo, which was less painful than
Bamboo. I'd be massively in favour of someone actually looking after
the CI, and any steps we can take toward this being easier are
definitely good. So +1 for that!

[Kristian Rosenvold]

Atlassian managed to bork the UI *seriously* for the 2.7 release, which
makes 2.7 among the all-time worst build systems I've seen in terms of
UI and usability.

They obviously understood they'd gone wrong because 3.0 (3.0.2) got a
real rework of the UI and it's real smooth - probably the best again. We
should upgrade (and that was problem free for us when we did it)

Bamboo has a stuck build detection algorithm that can be tweaked
from the Adiminstration/Build Monitoring tab. The only problem I've seen
with this algorithm is that if *all* your builds get stuck I think it
gets fooled in the end.

As for the philosophical bit, I'm not sure what you're referring to. We
can use whatever tools we consider best for the job. I use non-free
software when developing free software, some of it I even pay for
although most of the nice guys give OSS licenses. I don't need to like
Richard Stallman even though I work several OSS projects.

In my day job we burned through almost *all* the available CI systems
and all the "free" ones ended up being the most expensive ones in
practical use. That's not just the man-hour bit, but also the amount of
energy you want to spend on tinkering with your CI as well has having
half your project tinkering with it.

I think the main selection criteria for a CI for selenium should be the
*stability* and that it *works* (features are nice too, but I really
assume most will fit in one way or another). Unfortunately we have /more
than enough/ trouble with stability of our own stuff and most of it is
our own doing.

In terms of CI I think the real issues that need to be addressed is
getting our own stuff to be sufficiently stable to run on *any* CI at
all. If I type "./go" on my machine, i'll have transient build failures
>50% of the builds, at the moments I think I have a 90% transient
failure rate. (Sinc I run a pure 64 bit stack I generally get a lot
more trouble)

Additionally deploy this in a ton of different vm's
running a ton of different platforms, well it sounds like a
recipe for instability for me.

I know at my day job we dropped all virtualization and
basically removed all the fluff to get 24/7 stability, we reboot
all the remotes every third night. Every
component we add that only has "99%" uptime brings the total uptime
down, which was the reason we ditched grid (a looong time ago)

If I was to flame-bait, the current oracle/sonatype hudson probably
offers a much higher stability promise than jenkins. But that would just
be teasing.
http://gojko.net/2011/04/05/how-is-it-even-possible-code-to-be-this-bad/
is a good read no matter what.

Bamboo gets the job done, and once you get off 2.7 it does it
quite nicely too. It just seems like we have a crapload of other
CI-related issues that deserve attention much more.

Kristian

[Tarun K]

@Santi,

I did not have chance to read all the messages in this thread. But doc
build is taking eternal to get completed.
Paul made a couple of edits and asked me offline to see as his changes
were not reflected on site.
I aborted the previous build, one more was initiated and it just keeps
building endlessly.

Job is available here - http://xserve.openqa.org:8085/browse/WEB-SEHQ-JOB1-98

Thanks
Tarun K

[Patrick Lightbody]

Yeah, not interested in changing build systems again (we had Hudson prior to Bamboo). Let's just get it upgraded to 3.0.x, clean up our VMs, buy more hardware (or move some stuff to the cloud), and get our build & tests passing reliably. Easy, right? :)

--
Patrick Lightbody
+1 (415) 830-5488

> --
> You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
> To post to this group, send email to selenium-...@googlegroups.com.
> To unsubscribe from this group, send email to selenium-develo...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/selenium-developers?hl=en.
>

[Kristian Rosenvold]

As can be seen from
http://xserve.openqa.org:8085/build/admin/edit/defaultBuildRequirement.action?buildKey=WEB-SEHQ-JOB1

No agent can run the job, therefore the rather long wait. It will
probably go on waiting until the "Website" agent is brought online.

Kristian

[Simon Stewart]

The reason for moving off Hudson is that it was wildly flaky and would
drop distributed nodes on a regular basis. Bamboo, for all the current
problems, is a step up from it.

What would Make My Day would be a CI system that understood how to
spin up snapshots of VMs and run things on those. As an aside, we
should find a way to engage the devops community. This is exactly the
sort of thing that they'd rock at.

Simon

[Adam Goucher]

Patrick, does the current key work with 3.0.1? I could try to upgrade it
again this week. (Knowing now that it is not enough to just backup the
filesystem, but there is a database involved too.)

-adam
> Yeah, not interested in changing build systems again (we had Hudson prior to Bamboo). Let's just get it upgraded to 3.0.x, clean up our VMs, buy more hardware (or move some stuff to the cloud), and get our build& tests passing reliably. Easy, right? :)

[Kristian Rosenvold]

I think you have to get a new key, at least we did. Our upgrade from
2.7->3 was smooth.

Kristian

[Adam Goucher]

> What would Make My Day would be a CI system that understood how to
> spin up snapshots of VMs and run things on those. As an aside, we
> should find a way to engage the devops community. This is exactly the
> sort of thing that they'd rock at.

Right now we are using VMWare Fusion on the XServe to dish out VMs. One
thing we have discussed is to move some of the VMs to commodity-ish PC
hardware at which point we'll have another/newer problem, but for now
the solution might be to play with
http://www.vmware.com/pdf/vix162_vmrun_command.pdf which would seem to
allow for this.

-adam

[Patrick Lightbody]

If the key doesn't work, we can get a new one easily and in a pinch we can use a 30 day trial key.

--
Patrick Lightbody
+1 (415) 830-5488

[Aditya Ivaturi]

> What would Make My Day would be a CI system that understood how to
> spin up  snapshots of VMs and run things on those.

What we did in-house might be a little relevant, but it does involve
some infrastructure costs. We use VMware Lab Manager to create
multiple templates for various OS (both 32 & 64 bit). Note that
templates in Lab Manager are not just snapshots and that means I can
use a single copy of XP & spin up as many as I want (pretty much
instantaneously). And that means I can run parallel builds of the same
type, if I want to. Whenever our (custom) build server gets a job
request, it "deploys" the appropriate template, completes the build &
destroys the deployed VM. BTW, this same infrastructure is also used
to carry out automation tests, which can get kicked off right after a
successful build. This is, in principle, similar to what Eran
presented at Selenium conference.

--aditya

[Patrick Lightbody]

We have budget to spend money on the cloud (EC2) or on additional
hardware and software. We can also probably request donations from
companies for hardware and software. So if someone wants to take on
this project, just give us a shopping list and I'll start making it
happen :)

Patrick

[Aditya Ivaturi]

OK, let me take a look at this. I have built Selenium server on OS X
alone, so I have to spend time on other platforms to figure out all
the various builds. I'm aware of these docs, please let me know if
there are others I should be aware of:

https://code.google.com/p/selenium/wiki/BuildingWebDriver
https://code.google.com/p/selenium/wiki/CrazyFunBuild
https://code.google.com/p/selenium/wiki/BuildReadme

--aditya

On Apr 17, 9:10 pm, Patrick Lightbody <patr...@lightbody.net> wrote:
> We have budget to spend money on the cloud (EC2) or on additional
> hardware and software. We can also probably request donations from
> companies for hardware and software. So if someone wants to take on
> this project, just give us a shopping list and I'll start making it
> happen :)
>
> Patrick
>

[Adam Goucher]

The problem we're facing is not necessarily how to build it -- but
creating, and maintaining the infrastructure. Including fully automated
configuration management, monitoring and snapshot refreshing.

-adam

> OK, let me take a look at this. I have built Selenium server on OS X
> alone, so I have to spend time on other platforms to figure out all
> the various builds. I'm aware of these docs, please let me know if
> there are others I should be aware of:
>
> https://code.google.com/p/selenium/wiki/BuildingWebDriver
> https://code.google.com/p/selenium/wiki/CrazyFunBuild
> https://code.google.com/p/selenium/wiki/BuildReadme
>
> --aditya
>
> On Apr 17, 9:10 pm, Patrick Lightbody<patr...@lightbody.net> wrote:
>> We have budget to spend money on the cloud (EC2) or on additional
>> hardware and software. We can also probably request donations from
>> companies for hardware and software. So if someone wants to take on
>> this project, just give us a shopping list and I'll start making it
>> happen :)
>>
>> Patrick
>>
>> On Mon, Apr 11, 2011 at 9:51 AM, Aditya Ivaturi<ivat...@gmail.com> wrote:
>>>> What would Make My Day would be a CI system that understood how to
>>>> spin up snapshots of VMs and run things on those.
>>> What we did in-house might be a little relevant, but it does involve
>>> some infrastructure costs. We use VMware Lab Manager to create

>>> multiple templates for various OS (both 32& 64 bit). Note that

>>> templates in Lab Manager are not just snapshots and that means I can

>>> use a single copy of XP& spin up as many as I want (pretty much

[Aditya Ivaturi]

Yes, I understand that it not build itself that you guys worried
about. But regardless, I still need to be familiar with what is
involved in a build to look in to the appropriate infrastructure
needs.

--aditya

[Alexei Barantsev]

Hi,

Where is the CI server now?

http://xserve.openqa.org:8085/ is not available

Regards,

--

Alexei Barantsev

Software-Testing.Ru

[Patrick Lightbody]

That should be it. Anyone know if it's down (I'm on the train right now so I can't check)?

--

Patrick Lightbody

+1 (415) 830-5488

[David Burns]

I have started speaking to Julian Simpson who is really keen on helping. Since he, and I, are currently on holiday I haven't taken it further yet. Once he has had a look at things I will get more people involved in the emails.

David

David Burns
URL: http://www.theautomatedtester.co.uk/

[Ivo Grootjes]

Hi,

I'm also willing to help. If i look at the
http://xserve.openqa.org:8085/start.action?selectedTab=currentTab it
seems like the last build ran about 2 weeks ago! Locally i've setup a
CI build which only runs the firefox tests and it seems like 8 tests
are failing. I think it will really help alot to get at least the
firefox tests run on each commit. I'm willing to help get this done
untill a better solution is found. If anyone is interested in my help
please contact me.


On Apr 27, 9:20 pm, David Burns <david.bu...@theautomatedtester.co.uk>
wrote:

> I have started speaking to Julian Simpson who is really keen on helping.
> Since he, and I, are currently on holiday I haven't taken it further yet.
> Once he has had a look at things I will get more people involved in the
> emails.
>
> David
>
> David Burns
> URL:http://www.theautomatedtester.co.uk/
>

> On Wed, Apr 27, 2011 at 4:07 PM, Patrick Lightbody <patr...@lightbody.net>wrote:
>
>
>
>
>
>
>
> > That should be it. Anyone know if it's down (I'm on the train right now so
> > I can't check)?
>
> >  --
> > Patrick Lightbody
> > +1 (415) 830-5488
>
> > On Apr 26, 2011, at 11:42 AM, Alexei Barantsev wrote:
>
> > Hi,
>
> > Where is the CI server now?

> >http://xserve.openqa.org:8085/is not available

[Ivo Grootjes]

Any progress on this?

On May 13, 2:02 pm, Ivo Grootjes <ivo...@gmail.com> wrote:
> Hi,
>

> I'm also willing to help. If i look at thehttp://xserve.openqa.org:8085/start.action?selectedTab=currentTab it

> seems like the last build ran about 2 weeks ago! Locally i've setup a
> CI build which only runs the firefox tests and it seems like 8 tests
> are failing. I think it will really help alot to get at least the
> firefox tests run on each commit. I'm willing to help get this done
> untill a better solution is found. If anyone is interested in my help
> please contact me.
>
> On Apr 27, 9:20 pm, David Burns <david.bu...@theautomatedtester.co.uk>
> wrote:
>
>
>
>
>
>
>
> > I have started speaking to Julian Simpson who is really keen on helping.
> > Since he, and I, are currently on holiday I haven't taken it further yet.
> > Once he has had a look at things I will get more people involved in the
> > emails.
>
> > David
>
> > David Burns
> > URL:http://www.theautomatedtester.co.uk/
>
> > On Wed, Apr 27, 2011 at 4:07 PM, Patrick Lightbody <patr...@lightbody.net>wrote:
>
> > > That should be it. Anyone know if it's down (I'm on the train right now so
> > > I can't check)?
>
> > >  --
> > > Patrick Lightbody
> > > +1 (415) 830-5488
>
> > > On Apr 26, 2011, at 11:42 AM, Alexei Barantsev wrote:
>
> > > Hi,
>
> > > Where is the CI server now?

> > >http://xserve.openqa.org:8085/isnot available

[Ivo Grootjes]

Hi,

Any update on the new buildserver setup? Is having a buildserver seen
as something important or is it just a nice to have kind of thing? It
seems the current buildserver hasn't build anything for over a month.

Kind regards,

Ivo Grootjes

[Adam Goucher]

Conceptually the former. In reality, the project has been released the
last dozen releases off of people's personal/work machines so its really
the latter.

[Simon Stewart]

Having a clean build server is vital. I'm deeply frustrated by how
hard it's been to make releases, and with the pace as fast as it is
it's something that's becoming increasingly important. So far,
however, we've yet to find a build system that can cope with the
demands that we'd place upon it (in particular, spinning up multiple
virtual machines and collecting results off them) so, as Adam says,
we've been building and testing on people's desktops.

*sigh*

Simon

[Ivo Grootjes]

While spinning up multiple virtual machines sounds like a cool thing to do it's probably the thing that's blocking having a stable buildserver right now. I suggest to start off simple. A couple of builds per day is more then enough anyways and it would be a huge improvement of the current situation. Just set up 3 machines, linux, mac, and windows kinda like what the current buildserver looks like right now. Make sure hanging builds are killed (by the buildserver) and make sure the buildserver is actively maintained. 

I'm volunteering. I'm willing to dive into the problems of bamboo and fix them but I prefer to setup TeamCity because it's just easier to set up and it works better (I've set up selenium builds on my laptop with win64+linux agents). Teamcity can also be setup to spin off Amazon EC2 instances very easily which could be looked into after the 3 buildagents seem to work properly.

Up to you! :) I'm willing to invest some time.

2011/7/21 Simon Stewart <simon.m...@gmail.com>

[Simon Stewart]

The reason for the VMs isn't just coolness: it's to provide a
consistent environment. One of the main causes of instability on the
current build server is wedged tests and discarded browser instances.

Simon

[Andy Tinkham]

I have no idea if this is relevant, but on a local rubyists mailing list today, someone was asking about travis-ci (https://github.com/travis-ci/travis-ci). In a quick glance, it seemed like many of the concepts I've seen in this thread are involved there, too, so I thought I'd share the link in case it was relevant.

Andy

[Jari Bakken]

On Thu, Jul 21, 2011 at 5:11 PM, Simon Stewart
<simon.m...@gmail.com> wrote:
> The reason for the VMs isn't just coolness: it's to provide a
> consistent environment. One of the main causes of instability on the
> current build server is wedged tests and discarded browser instances.
>

Although a clean VM for each test run would certainly be preferable,
we could probably do better than we have so far by adding more
aggressive cleanup scripts. Other projects with large and complex
builds (e.g. Chromium) manage to run decent CI without VMs (though
with a lot of babysitting).

If Ivo wants to donate his time to work with the existing
infrastructure, I'm not going to stop him :)

[David Burns]

I have emailed Julian Simpson (A.K.A Build Doctor) today to get that thread restarted. I will let you know how that goes.

David

David Burns

URL: http://www.theautomatedtester.co.uk/

--

[Simon Stewart]

On Thu, Jul 21, 2011 at 10:56 PM, Jari Bakken <jari....@gmail.com> wrote:

I'd actively encourage it :) I was just trying to explain the
reasoning behind some of the constraints we have.

Simon