How to add a private repository to a pipeline when gocd-server runs in a Docker container ?

[gls]

Hello !

I have a private repository on gitlab.com that I am trying to add to a pipeline in gocd but I cannot find the way to use the passwordless ssh connection.

Here is the config part in my docker-compose.yaml :

gocd-server:

  restart: always

  container_name: gocd-server

  image: gocd/gocd-server:latest

  ports:

    - "18153:8153"

    - "18154:8154"

  volumes:

    - $SSH_AUTH_SOCK:/ssh-agent

  env_file: .gocd-server_env

I found this configuration here https://gist.github.com/d11wtq/8699521

I can successfully test the connection in the pipeline if I change the URL to https://login:password@gitlab... but I don't want to do that.

my key inside the container is then the key my host is using which is fine because the key is in the list of allowed keys in gitlab.

Unfortunately, when I type git ls-remote inside the container, I am asked for the username and password. I can find my personal key if I type ssh-add -L inside the docker container.

Here is the error :

--- ERROR ---
STDERR: fatal: could not read Username for 'https://gitlab.com': No such device or address
---

Any idea what I am doing wrong ?

Thank you very much !

[Ketan Padegaonkar]

You're using an HTTPS url with an SSH key, you probably mean `g...@gitlab.com:....`

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gilles Habran]

I tried that but I get this error :

--- ERROR ---
STDERR: Host key verification failed.
STDERR: fatal: Could not read from remote repository.
STDERR: 
STDERR: Please make sure you have the correct access rights
STDERR: and the repository exists.
---

That's why I thought that I could only create a pipeline with https.

The key is the same in ssh-add -L (in the docker container) and in gitlab (unless I am mistaken of course). I only use one key.

--
You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/7ugcNNrdOYE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.

[Gilles Habran]

I tried a few things and I think the problem is with the go user.

So I connect to my docker container using "docker exec -it gocd-server bash"

Once I am inside as root, I tried the following :

1) ssh -T g...@gitlab.com and received a success response

2) I checked in /root/.ssh/known_hosts and I have a few lines inside 

3) I tried (as root) : git ls-remote g...@gitlab.com... and it works fine.

Now I try with the user go and I get this :

go@340cfb5c5682:~/.ssh$ ssh -T g...@gitlab.com

The authenticity of host 'gitlab.com (104.210.2.228)' can't be established.

ECDSA key fingerprint is f1:d0:fb:46:73:7a:70:92:5a:ab:5d:ef:43:e2:1c:35.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'gitlab.com,104.210.2.228' (ECDSA) to the list of known hosts.

Permission denied (publickey).

Any idea ? :)

Thank you.

[Gilles Habran]

Hi,

I have had no luck for a solution today. Is there a way to ask root instead of "go" to access private repository for the pipelines ? The "go" user is terrible to deal with in Docker. I don't have any problems to configure gocd in a real environment but I can't do it with Docker.

The "go" user has a different UID (unless lucky) than my host so even if I create directories, put real SSH keys, known hosts, ... and so on, the go user in the docker cannot use those because of the different UID even if I have a go user  on the host and in docker.

I can only reach my private repository with root in the docker container.

Any idea about a possible configuration with the "go" user to allow it to access ssh informations in the Docker container ?

Thanks.

[Bobby]

You are most likely not using an ssh-agent, or you haven't created a ~/.ssh/config file that says Host uses Key :

Host gitlabaddress

Hostname gitlabaddress.fqdn

User gitlabuser

IdentityFile /path/to/ssh/key/that/has/access

Do this as a regular user, not root user.

[Fredrik Wendt]

Run `ssh -T -vvv g...@gitlab.com` and make sure the ssh client is offering the right key. I've been bitten by this a couple of times.

/ Fredrik

+46 702 778511

[Chris Forbes]

I'm trying to set up GoCD for the first time on Linux (CentOS 7) with the official GoCD Server Docker container. Documentation for this is very scarce as I'm sure you are aware. I'm having a lot of difficulties getting the server to use the SSH key correctly. I've tried a number of suggested solutions with no luck.

Some of the things I have tried:

• Mount the instance ssh folder as a volume to `/var/go/.ssh`
• Create a `config` file in the container to specifically target the `id_rsa` file
• Run `ssh -T -vvv git@github.com` on the instance and the container to validate the key

But still, clicking "Check Connection" results in the error:

--- ERROR ---
STDERR: Host key verification failed.
STDERR: fatal: Could not read from remote repository.
STDERR: 
STDERR: Please make sure you have the correct access rights
STDERR: and the repository exists.
---

Could someone that has done this be so kind as to list the steps required to make this work? Or point me at a detailed blog post/write up of how to do this? We are keen to give GoCD a try, but being unable to use Git with SSH is a deal breaker.

[Jay]

Hi,

Make sure you've properly chowned those directories and files. Keep in mind that you have to either run those commands you've mentioned as the "go" user, or establish those files in that .ssh folder and then chown them with "go:go". If you're running inside a docker container, as I believe you are, it's pretty easy to set up this process as an entry-point script. I also believe you'll always have to chown that .ssh directory so that the container go user can access them. I believe it's also pretty safe to pre-populate your git servers public ssh host keys and simply cat them into the known_hosts file. This way you don't have to accept them and these will rarely change. I've already got a working solution in place, so I'll provide some highlights here, since that's all I can do.

* gather your git servers public ssh keys with command "ssh-keyscan -p 22 [host | addrlist namelist] ...". Make sure you use any git server aliases here.

* the go server home can be figured out programatically with this command: GO_HOME=$( getent passwd "go" | cut -d: -f6 )

* create folder ${GO_HOME}/.ssh with chmod 700 and chown go:go

* generate a pair of keys if they don't already exist: ssh-keygen -q -t rsa -C "gocd-server" -N '' -f "$GO_ID_RSA_FILE"

* add your pub key to gitlab

* cat your git servers pub ssh keys into ${GO_HOME}/.ssh/known_hosts, chmod 0644 the file

* chown the entire /var/go, or just /var/go/.ssh if you prefer

All these steps should work with the .ssh folder bind-mounted.

Hope you get something working soon.

Jay

[Chris Forbes]

Thank you for your response, Jay.

I followed your steps to ensure I hadn't missed anything. Unfortunately, it still isn't working. I'll try once more going from scratch and see if anything plays out differently.

Chris

[Zabil C M]

Hey Chris,

 If you have docker running on your machine with the "$HOME/.ssh" folder setup up to connect successfully to your authenticated git repo. 

 You can quickly test this by creating a data volume container 

 

 $ docker create --name authenticated -v $HOME/.ssh:/var/go/.ssh  gocd/gocd-server:16.10.0 

 and using the data volume container while running your GoCD server container

 $ docker run --volumes-from authenticated -p 8154:8154 -p 8153:8153 gocd/gocd-server:16.10.0

 Your issue might be with the ssh keys set up.

 

 To make it simpler, create data containers from the right ".ssh" after verifying that it works outside of docker and use them while running the server or agents.

--

You received this message because you are subscribed to the Google Groups "go-cd" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscribe@googlegroups.com.

[kshar]

There is an open issue on Docker Github Issues . The permissions on the /var/go/.ssh aren't getting set even in my case and I believe it's a problem with docker. The permissions for a user created in a parent container (go:go) are being forgotten or not applied for some reason. 

If anybody has been able to solve this problem please let me know. I have been struggling with the go user permissions hell for the past 2 days too.

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.

[Alvaro Hernandez]

Hi,
I had the same issue, I accessed to my goserver docker using 
docker exec -it --user go gocdserver /bin/bash

The important part is to create the ssh key with the go user.
so I generated my ssh key and added it to my bitbucket.
Now it's working :)