Private build farm

[Daniel Stonier]

Hi all,

About to set up a private build farm here and while I know there is some activity with Mike's buildbot-ros and recent activity on the osrf buildfarm side (particularly the docker announcements), I'd like to understand where this is headed. Sometimes hard catching the pulse of things living across the pond!

Q) Are there differences between the two approaches?

Q) Are they developing in parallel to an equivalent goal - private build farms? Or is it a case of having a working solution in one place (buildbot-ros) until the other arrived (osrf docker buildfarm)?

Q) Comments from anyone using one or the other?

Hopefully this will trigger some discussion :)

Daniel.

PS Mike, I cc'd you as I'm only 99.99% sure you must be on this list...and I'd like to hear your thoughts!

[Michael Ferguson]

I am on this list -- although not at that email :)

RE: state of buildbot-ros: We use several instances of buildbot-ros at Fetch Robotics. I know there are at least 2-3 other decently sized entities using it as well. While there isn't a huge amount of activity, the project is maintained. Earlier this summer we added support for Pull Request builders (which we use extensively at Fetch), and I merged a pull request just yesterday to fix some documentation/install issues.

RE: difference in approaches: I started buildbot-ros mainly because the old Jenkins farm was so difficult to setup, and was not at all ready for building non-open-source code. I'm not sure of the status of the OSRF docker farm, I know it has made forward movement, but I haven't had the time to try it out. I suppose at some point, if it is easy enough to setup, stable enough to depend on, and supports the required workflows, we might end up some day using the docker farm. That said, I can't imagine it ever being a 0-effort move, and given the sheer number of things that have to get done each day at Fetch, switching from our current farm to a new one is fairly low priority task.

The OSRF approach is certainly more scalable (buildbot-ros is best for teams building 1-15 repositories, some of the Web UIs look terrible with more than 15 repos, but you can continue adding more repos. Buildbot has a hard-ish limit of 1000 different kinds of jobs (or roughly 250 repos if you have docbuild, testbuild, debbuild and pull request builders for each).

-Fergs

--
You received this message because you are subscribed to the Google Groups "ros-sig-buildfarm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildf...@googlegroups.com.
To post to this group, send email to ros-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/CAP6aRqZe8ZM4SOdXFC7V9z%3DaQp%2B86gq63W7edgtn1i-Qp0wvTQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Nikolaus Demmel]

Hi,

so our colleagues at Deepfield (CC Dejan) have been working together with OSRF to make setting up the docker-based farm easier and they have been using it for a while. There will be a presentation about the state of that at ROSCon, so maybe you might want to wait for that to get some condensed info without searching through list archives and wiki pages. Else I'm sure Dejan can give you a head's up even before.

Cheers,
Niko

[Alexandre Vincent]

Hi everyone,

So I have previous experience in setting up CI infrastructure for non ROS software, and I have been trying buildbot-ros and ros_buildfarm to help Daniel make a choice for a private buildfarm.

So far I managed to get :
- a jenkins master for ros_buildfarm running on my machine, with a default job. Running some of them failed though and I realized It looks like I need quite a complex slave setup to get it to run... Didn't get around to do it yet.
- a build job of ecl_core with builbot master and 1 slave, still a bit broken though ( some untrusted dependency ), but mostly working ( it took me about 2 days to setup for the record ).
I used jenkins a lot before so I m probably biased ( even thought I definitely prefer python to java ), and I am not completely familiar with the release process of ROS packages, so please take the following with a grain of salt.

But anyway I thought I ll just try to explain my first experience & impressions about both, hoping it can somehow relate to what people are experiencing when they start thinking about setting up their continuous integration setup with ROS :

In both cases I need to have some custom configuration ( rosdistro + build config ), and I need to get it right.

In both cases, I will need a custom apt repository for my custom deb packages, which I haven't done yet, so there is some unknown there in my case.

=> I want to go with a simple build server setup so that I can test all this configuration, without worrying about the build process and machine setup, at least as a first step.

Buildbot-ros :
+ Setup looks simple and quick enough to do for a few packages.
- It is based on buildbot which I need to learn...

- buildbot is not used by OSRF as far as I know -> not sure about its future with ROS.

- Maybe cannot scale when later we have more packages to build.

- I ll probably have to maintain the build process myself ?

Ros_buildfarm :
+ Based on jenkins which I already know.
+ Used by OSRF -> probably good to take as standard template

+ Will scale up fine if we ever need to.

+ build process will likely be maintained by OSRF, so I probably don't need to worry about it.

- Not obvious how to run it in a simple setup (only one machine). Seems it needs complex deployment on multiple machines.

Problems :
=> A continuous integration setup is a long term investment, yet the one that is probably good for long term doesn't seem to match what I need right now, and the time I can afford to set it up.

=> If I go one direction, changing later when it becomes needed, will probably be painful :

  - configuration is different ( buildbot-ros still use one rosdistro repo / ros_buildfarm uses rosdistro + config repo )

  - different job configuration ( probably have to rewrite all the jobs config )

  - different software and frontend ( so what I ll learn with one might not be usable with the other - and the users will be confused )

I ll keep investigating these whenever I can get some time, over the next few weeks, but I wanted to share my first impressions before I forget them.

Please, let me know if you think I am mistaken about any of my previous assumptions.

Cheers,
--

AlexV

--

You received this message because you are subscribed to the Google Groups "ros-sig-buildfarm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildf...@googlegroups.com.
To post to this group, send email to ros-sig-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/fce80cdc-cc3b-4b20-91cc-8e7696ee75ac%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--
AlexV

[Dirk Thomas]

If the following ticket in the buidlfarm_deployment repo would get addressed (https://github.com/ros-infrastructure/buildfarm_deployment/issues/42) you could provision all necessary infrastructure for the ros buildfarm on a single machine / vm.

But such a setup would likely only be useful for testing or very small environments.

Once it gets beyond testing or building a handful of packages you really want to scale to multiple machines.

Cheers,

- Dirk

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/CAFt-VfJQYJdVw20xmgPGoX6%2Bn_ttn3sCDNLyt5_fq06bK9KX9w%40mail.gmail.com.

[Daniel Stonier]

On 24 September 2015 at 22:37, Michael Ferguson <mfe...@gmail.com> wrote:

I am on this list -- although not at that email :)

RE: state of buildbot-ros: We use several instances of buildbot-ros at Fetch Robotics. I know there are at least 2-3 other decently sized entities using it as well. While there isn't a huge amount of activity, the project is maintained. Earlier this summer we added support for Pull Request builders (which we use extensively at Fetch), and I merged a pull request just yesterday to fix some documentation/install issues.

RE: difference in approaches: I started buildbot-ros mainly because the old Jenkins farm was so difficult to setup, and was not at all ready for building non-open-source code. I'm not sure of the status of the OSRF docker farm, I know it has made forward movement, but I haven't had the time to try it out. I suppose at some point, if it is easy enough to setup, stable enough to depend on, and supports the required workflows, we might end up some day using the docker farm. That said, I can't imagine it ever being a 0-effort move, and given the sheer number of things that have to get done each day at Fetch, switching from our current farm to a new one is fairly low priority task.

The OSRF approach is certainly more scalable (buildbot-ros is best for teams building 1-15 repositories, some of the Web UIs look terrible with more than 15 repos, but you can continue adding more repos. Buildbot has a hard-ish limit of 1000 different kinds of jobs (or roughly 250 repos if you have docbuild, testbuild, debbuild and pull request builders for each).

-Fergs

Thanks for the information Mike. Always great to get a few sentences capturing a big picture to better choose your direction before blundering off.

I was hoping to drop into the testing/feedback loop for the OSRF buildfarm but we ran into quite a few minor issues and I don't have the dedicated resources for properly engaging in this loop right now. So for now, we've got buildbot up and running and that should see us through the next six months of milestones, tests and demos. 

I cannot express how great it is to have a resource like this readily available - thanks Mike! Also Tully, Dirk and co....it's great you are moving towards filling this need in a proper way. Long term that's where I would like to be - and am hoping that I'll have the need to scale up...along with dedicated with the required cycles to jump into this loop next year.

Cheers,

Daniel.