jenkins github access token not used?

[Manuel Werlberger]

Sorry, yet another question for setting up a private buildfarm...

When calling `generate_all_jobs.py` with the private address to the master/index.yaml without token in the URL I get an HTTP error because the file is of course not accessible. When I add a token than when another script refers to `doc-independent-build.yaml` there is no token used. Is it possible to configure a 'global' token for the ros_buildfarm when generating the jenkins jobs?

Thanks again.

Best,

 Manuel

[Dirk Thomas]

Hi Manuel,

since we don't use a token for the official ROS build farm the `generate_all_jobs.py` script likely doesn't handle that case yet. It would be great if you could create a pull request for this to pass along the token to make your use case work.

Thanks,

- Dirk

--
You received this message because you are subscribed to the Google Groups "ros-sig-buildfarm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildf...@googlegroups.com.
To post to this group, send email to ros-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/2520c0f3-6fbe-4fd0-a4d4-b3a94cb3265f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Manuel Werlberger]

Hi Dirk,

I'm in the progress of adapting all the python scripts so that you can use github tokens as well as the github api for accessing the files in the end. I will file a pull request as soon as I have a decently working version so we can also test this extensively.

In general, I always thought that the scripts are prepared to setup a private buildfarm which also implied to me to use private repositories. But for example having also a private rosdistro and config repo needs quite some changes. I also ran into some other problems which I hope it is ok that I keep posting them to the list. I hope to have a private buildfarm including virtual machine support up and running soon so that I can share the setup instructions.

Best,

 Manuel

[Tully Foote]

HI Manuel, 

We have several users already using instances with private rosdistros and configs. Most have had requirements to use internally hosted resources for which they have used other access control mechanisms. We'd definitely like to extend/improve support to for using github with authentification as the private repository for the rosdistro and buildfarm config. We'd love to see what you're doing and answer questions that might make things easier for you. 

Tully

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/d042e288-9ea5-401d-9308-c16e36eacf53%40googlegroups.com.

[Manuel Werlberger]

Thanks. And I will file a pull request as soon as I have everything running and do some cleanup in my changes. I just saw today that bloom handles github access via api calls nicely and I'm wondering if it makes sense to use common scripts to fetch files and information from github?

Best,

 Manuel

[Tully Foote]

It does make sense sometimes, however we don't want to tie our infrastructure specifically to github. For example with bloom the only github specific calls are at the very end to make the pull-request to rosdistro. Everything else is generic git and thus hosting independent. 

There are a lot of people using bitbucket, and we don't want to lock people out of hosting their own content either. 

You'll find there are a few examples of places where we detect github in the url and take a shortcut using their APIs but there is always a fallback mode for other hosting. 

Tully

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/c693490c-5708-4a19-adb1-6dc43b486ab9%40googlegroups.com.

[Manuel Werlberger]

Good point. I'll try to have both options included and configurable.

--
You received this message because you are subscribed to a topic in the Google Groups "ros-sig-buildfarm" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ros-sig-buildfarm/MXf5OLwp8Qg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ros-sig-buildf...@googlegroups.com.

To post to this group, send email to ros-sig-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/CAM7qi7X_Jow6Lx7VCoAgC%2BAmq-BerGqxTonhtiBNRPGDdcHuqw%40mail.gmail.com.

[lu...@savioke.com]

Hi Manuel,

Were you able to get the support for Github private repos working? It would a very valuable contribution!

Best,

Lucas

[Manuel Werlberger]

Hi Lucas,

Yes we have it working but it is not too clean and nice. I'll try to clean it up during the week and create a pull request with the changes.

Best,

 Manuel

[Lucas Chiesa]

Hi Manual,

Looking forward for it, thanks. It will be a great addition!

Best,

Lucas

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildfarm/a9141c54-4b00-462e-99eb-937a1ed4d651%40googlegroups.com.

[Tom Moore]

I would also be interested in this. I assumed that the credentials::git-fetch-ssh:: settings would let me work with private repos, as long as they were listed in the distribution file with the GitHub SSH URL. However, I am getting this error for the rosdistro cache job after adding the credentials:

docker run --rm --cidfile=/var/lib/jenkins/jobs/test-indigo_rosdistro-cache/workspace/docker_generate_rosdistro_cache/docker.cid --net=host -v /var/lib/jenkins/jobs/test-indigo_rosdistro-cache/workspace/rosdistro_cache:/tmp/rosdistro_cache -v /var/lib/jenkins/.ssh/known_hosts:/etc/ssh/ssh_known_hosts:ro -v :/tmp/ssh_auth_sock -e SSH_AUTH_SOCK=/tmp/ssh_auth_sock rosdistro_cache_generation 13:21:02 docker: Error response from daemon: Invalid volume spec ":/tmp/ssh_auth_sock": Invalid volume specification: ':/tmp/ssh_auth_sock'.

Looking at the error, it looks like the issue is in the part that I highlighted. I checked out the code that's generating the job, and it looks to me like it's exporting the SSH_AUTH_SOCK variable after it's using it:

https://github.com/ros-infrastructure/ros_buildfarm/blob/master/ros_buildfarm/templates/misc/rosdistro_cache_job.xml.em#L84

This can also be seen above. Am I misunderstanding what's going on there?

[Tom Moore]

Ah, I see the original question was more about accessing a private master/index.yaml. I got around this by running my own instance of GitLab on a server that is only accessible from the build farm machines. I then added "forks" of the buildfarm config repo and rosdistro to my GitLab server. My issue is that once I release a package via bloom, the rosdistro cache job can't pull it down from GitHub over https (private repo), and I get the error below when I try adding GH ssh credentials and changing the repor URL in the distribution file to SSH instead.