Custom rosdistro setup

[Daniel Di Marco]

Hi all,

we have done the buildfarm deployment process (i.e. jenkins master/slave and repo servers are configured and running), and now I've tried to follow this document:

https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/custom_rosdistro.rst

We want to go for a "add-on" fork of distribution. How do we go about setting up such a thing?

I assume we'll need to create a git repository which contains some yaml files such as https://github.com/ros/rosdistro, but I can't see how exactly.

Can you provide me with the workflow (or point me to the right documentation)?

Many thanks,

Daniel

[Dirk Thomas]

Hi Daniel,

you need to do the following steps:

https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/custom_rosdistro.rst#add-on-fork

* fork the https://github.com/ros/rosdistro repository

* add a distributions file containing your custom repositories (or being empty at first)

* list that additional distribution file after the "normal" one in the index.yaml

https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/custom_rosdistro.rst#import-ros-packages-which-are-not-built-by-the-custom-build-farm

* import the current ROS packages

* whenever you want to sync with the official ROS distribution

  * merge in changes from the upstream rosdistro repo

  * run the import package again

https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/custom_rosdistro.rst#use-the-forked-rosdep-database-or-keep-using-the-official-one

* decide to use the official rosdep or the one from your forked repo

https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/custom_rosdistro.rst#mandatory-changes-when-using-a-custom-rosdistro-database

* update reference to custom cache

* update build farm configuration

* everybody must set ROSDISTRO_INDEX_URL

I hope that helps.

If you have any further questions please let us know.

If you think the docs should be improved please consider providing a PR once you have managed to finish your setup to make it easier for others.

Cheers,

- Dirk

--
You received this message because you are subscribed to the Google Groups "ros-sig-buildfarm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildf...@googlegroups.com.
To post to this group, send email to ros-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/9cad3c37-a23d-4929-aacc-c0928a54d147%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Ferguson]

I had actually been putting off setting up a custom rosdistro for some time, but since it looked pretty well documented now, I figured I would try it out today. I'll be opening a PR shortly with some updates to docs, but in the meantime, these were my gotchas:

 * When using add-ons distribution files, you have to be using rosdistro format 3 (REP-143), otherwise you get a somewhat non-intuitive error from rosdep/bloom

 * When using rosdistro format 3, all distribution entries need to be made into lists, otherwise you get an error about "wrong type of key" (although no indicator which distribution the error is with)

 * When using rosdistro format 3, if the second distribution file has packages with no "release" entry, rosdep update barfs (ticket here https://github.com/ros-infrastructure/rosdistro/issues/49)

-Fergs

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/CA%2Bp5Lf-E8Fb2eJWAZMbWw_TDFu9N%3D6nJ_buhKviS7-63SXNwxA%40mail.gmail.com.

[Michael Ferguson]

Right after I made a PR, I realized that REP-143 is mentioned, but only on the front page README (https://github.com/ros-infrastructure/ros_buildfarm/blob/master/README.md)

[Pangercic Dejan (CR/RTC1.1-NA)]

Hi guys, 

we yesterday tried to set up an add-on fork of the build farm and hit the wall with the following issues:

* jenkinsapi pkg was not in the documentation and thus not installed

* importing rosdistro packages from official ros_build farm by calling import_upstream jenkins script is listed in the documentation way before we actually call generate_all_jobs.py and thus create import_upstream jenkins job.

* generate_all_jobs.py can not deal with the private mode of our github enterprise (where login is required to e.g. download content from github) such as our equivalent of https://raw.githubusercontent.com/YOUR_FORK/ros_buildfarm_config/master/index.yaml. 

Since we didn’t want to compromise our security we hosted above index.yam in the repo machine - hope this is ok.

* import indigo.yaml - where shall we put this file?

* jenkins credentials are not recognised. This is our jenkins.ini:

https://gist.github.com/anonymous/28c7dd03077f5b8da2c6

Upon Tully’s suggestion we changed hostname to DEFAULT and this got us over the credentials error, however then there is one further error:

https://gist.github.com/anonymous/66ee4e2a98fcc3e7fbd4

Suggestions would be very appreciate to get us through the last error which is a show stopper for us right now.

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildfarm+unsubscribe@googlegroups.com.
To post to this group, send email to ros-sig-buildfarm@googlegroups.com.

[Pangercic Dejan (CR/RTC1.1-NA)]

Follow-up to my own problem below:

Problem was that we didn’t set up our environment from the virtualenv (https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/environment.rst). Additionally 2 more issues popped up:

- jenkinsapi has to be version 0.2.26 (and not 0.2.27 which installs currently by default)

- several scripts that are called by generate_all_jobs.py have broken links to various misc, status, doc, release, etc. folders. Current work around is to create sim links and Tully already filled the ticket.

Then we ran the import_upstream jenkins job which pulls in all the system dependencies from an official ROS build farm. After that this job has to be executed with the import_indigo.yaml (https://github.com/ros-infrastructure/ros_buildfarm/blob/01743db70c219e0d4077ea24287a57a24c77d154/doc/custom_rosdistro.rst#import-ros-packages-which-are-not-built-by-the-custom-build-farm) file as an argument. This file can be placed in the repo machine, inside jenkins-slave workspace (/home/jenkins-slave/workspace/import_upstream/import_indigo.yaml) and the argument to the jenkins job is passed in as follows: $WORKSPACE/import_indigo.yaml.

After that rosdep (https://github.com/ros-infrastructure/ros_buildfarm/blob/01743db70c219e0d4077ea24287a57a24c77d154/doc/custom_rosdistro.rst#import-ros-packages-which-are-not-built-by-the-custom-build-farm) needs to be set up. If you want to use an official rosdep from official build farm you need to use Mike Purvis’ trick: https://github.com/mikepurvis/rosdep-generator such that the private packages resolve.

Deepfield specific:

- We need to remove Indigo and Jade from our builds and add deepfield-indigo instead: https://github.int.bosppaa.com/deepfield-experimental/ros_buildfarm_config/blob/master/index.yaml

- we need to remove all the currently imported jenkins jobs and re-import them

- then perform import_upstream twice again and in theory we should be able to start the release process with bloom. The latter step though was never tested with github enterprise, so possible surprises not excluded:).

D.

[Brian Gerkey]

Thanks for the detailed reports. Are these notes being captured
somewhere in the ros_buildfarm docs?

>>>> an email to ros-sig-buildf...@googlegroups.com.
>>>> To post to this group, send email to ros-sig-...@googlegroups.com.

>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/ros-sig-buildfarm/9cad3c37-a23d-4929-aacc-c0928a54d147%40googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>

> --
> You received this message because you are subscribed to the Google Groups
> "ros-sig-buildfarm" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ros-sig-buildf...@googlegroups.com.
> To post to this group, send email to ros-sig-...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ros-sig-buildfarm/bafe24d4-1b5b-4099-a89e-e9e731de6563%40googlegroups.com.

[Pangercic Dejan (CR/RTC1.1-NA)]

Not yet. We are still working on the deployment and there are still some steps to go. Once we have all sorted out (preferably by Monday) we will stall fresh and document every single step with the right sequencing.

>>>> an email to ros-sig-buildfarm+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to ros-sig-buildfarm@googlegroups.com.

>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/ros-sig-buildfarm/9cad3c37-a23d-4929-aacc-c0928a54d147%40googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
> --
> You received this message because you are subscribed to the Google Groups
> "ros-sig-buildfarm" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ros-sig-buildfarm+unsubscribe@googlegroups.com.
> To post to this group, send email to ros-sig-buildfarm@googlegroups.com.

[Brian Gerkey]

Sounds good, thanks!

On Thu, Apr 30, 2015 at 11:50 AM, Pangercic Dejan (CR/RTC1.1-NA)

>> >>>> an email to ros-sig-buildf...@googlegroups.com.

>> >>>> To post to this group, send email to

>> >>>> ros-sig-...@googlegroups.com.

>> >>>> To view this discussion on the web visit
>> >>>>
>> >>>> https://groups.google.com/d/msgid/ros-sig-buildfarm/9cad3c37-a23d-4929-aacc-c0928a54d147%40googlegroups.com.
>> >>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>>
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "ros-sig-buildfarm" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to ros-sig-buildf...@googlegroups.com.
>> > To post to this group, send email to ros-sig-...@googlegroups.com.

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/ros-sig-buildfarm/bafe24d4-1b5b-4099-a89e-e9e731de6563%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "ros-sig-buildfarm" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ros-sig-buildf...@googlegroups.com.
> To post to this group, send email to ros-sig-...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ros-sig-buildfarm/7bbd585a-bb75-4562-815a-769d325b53a2%40googlegroups.com.

[Dirk Thomas]

Regarding the problem with jenkinsapi: I have reported this upstream and the latest release (0.2.28) should not have that issue anymore.

Cheers,

- Dirk

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/CAGmW%2BQa5G_GbVdjWM4UZjEu0Ej0Jf3NyFnwAz3mW5c%3DwD24YwQ%40mail.gmail.com.