[Discourse.ros.org] [Buildfarm] Big workspace management

[Rayman]

Rayman August 19

Hey, I have a question about how to manage the workspace of a large collection of packages (~50/100). My main question is how to setup this workspace. We have a large number of unreleased packages and a few forked versions of official packages. Can I use rosinstall_generator to recursively check out the source repositories of my custom packages? I tried a forked rosdistro but apparently rosinstall_generator does not check out dependencies of unreleased packages. Is there an other way?

---

Visit Topic or reply to this email to respond.

To unsubscribe from these emails, click here.

[Andriy]

Andriy August 19

Hey Rayman, In our "Shadow's Build Tools" we put all source code dependencies for repository in the file repository.rosinstall. During the build system will pull all those repositories from Git into local workspace and if those repositories have repository.rosinstall it will do it for them as well for several levels. It is using wstool feature merge rosinstall files. So you may reuse this approach or even give a try to the whole "Shadow's Build Tools" solution which is really easy to setup.

[Dirk Thomas]

dirk-thomas August 19

Hi Rayman, the information about the dependencies of a package comes from its manifest. If your packages refer to names which are not part of the rosdistro the tool can't resolve the source repository for them. And for repositories which are not released no packages and therefore no manifests and recursive dependencies are known. I would suggest to put all your custom repos (including the forks) in a .rosinstall file (optionally with the distributed approach Andriy mentioned) and fill the dependencies using rosdep from binary packages.

---

Visit Topic or reply to this email to respond.

---

In Reply To

Andriy August 19

Hey Rayman, In our "Shadow's Build Tools" we put all source code dependencies for repository in the file repository.rosinstall. During the build system will pull all those repositories from Git into local workspace and if those repositories have repository.rosinstall it will do it for them as well…

[Rayman]

Rayman August 19

Thanks for the quick response. I'll give the distributed solution of @Andriy a try. Just a quick question, the manifests rosinstall_generator needs are cached in the in the <distro>-cache.yaml.gz file generated by rosdistro_build_cache? I'm trying to figure out how the whole system works.

[Dirk Thomas]

dirk-thomas August 19

Correct, a Jenkins job regenerates the cache every 5 minutes. It makes all manifests of released packages available without having to fetch them from each release repo. This is the data the rosinstall_generator as well as the build farm mainly operate on. The cache only contains the manifests of released packages since only for them it is clear when they have changed. Basically when the version number of the release changed. For source repositories it would require to constantly check the repository content since it can change without any change to the rosdistro files.

---

Visit Topic or reply to this email to respond.

---

In Reply To

[Mike Purvis]

Mike_Purvis August 19

Hi Rayman, Presently rosinstall_generator only works for released packages, but there are pending PRs on rosdistro which enable it to work with source repos as well, please check it out if interested: https://github.com/ros-infrastructure/rosdistro/pull/84 Clearpath is primarily deploying our ROS software now using rosinstall_generator-based bundles; you can see a small prototype of this idea here if it's of interest: https://github.com/mikepurvis/ros-bundling

[Rayman]

Rayman August 20

Nice, that patch would really help my use case! I could probably do the following define a custom ros distro with our source repositories use rosinstall_generator to resolve dependencies and use wstool to check out all these source repos finally use rosdep to resolve the standard ros depdendencies I hope I can find time next week to try out your patch.

---

Visit Topic or reply to this email to respond.

---

In Reply To

Mike_Purvis August 19

Hi Rayman, Presently rosinstall_generator only works for released packages, but there are pending PRs on rosdistro which enable it to work with source repos as well, please check it out if interested: https://github.com/ros-infrastructure/rosdistro/pull/84 Clearpath is primarily deploying our RO…