GitHub Clone to Different Local Directory

51 views
Skip to first unread message

eric....@gmail.com

unread,
Mar 26, 2021, 12:54:17 PM3/26/21
to Jenkins Users
OK, so this is kind of complex so hang with me.  We're moving from StarTeam to Github and I'm trying to reproduce what I'm doing in StarTeam with Github.  StarTeam was easy because I owned the repository machine as well as administrated the tool.  With Github, we're hosted.  So I'm admin on the project but can't create an RSA Token on the machine for easy access.  So I had to play around to make a personal access token work.  In order to make that access token work, I had to run the checkout job on a different node so that it was running as a user that lived in Github as well (access token's namesake).  So when this job gets called from the jenkins job, I want it to clone to the calling job's workspace (/opt/jenkins/workspace/<project_name>).  Well since in order to authenticate, it lives in it's own shell, the workspace for this guy, and where it wants to clone to, is /home/<username>/<project_name>.  All that for my question:  how can I specify what folder to checkout to?  I tried to use "checkout to specific local branch" but it fails saying "is not a valid branch name".  Well, yeah, branch is referring to branch not folder, lol.  In StarTeam this is easy, you just specify working folder.  Any help?  Thanks!

Mark Waite

unread,
Mar 26, 2021, 6:44:42 PM3/26/21
to Jenkins Users
If you're using pipeline, change to that directory with the dir('/users/username/projectname') { } wrapper around the checkout step;

If you're using a freestyle job, use the git plugin extension 'checkout to a subdirectory'

On Fri, Mar 26, 2021 at 10:54 AM eric....@gmail.com <eric....@gmail.com> wrote:
OK, so this is kind of complex so hang with me.  We're moving from StarTeam to Github and I'm trying to reproduce what I'm doing in StarTeam with Github.  StarTeam was easy because I owned the repository machine as well as administrated the tool.  With Github, we're hosted.  So I'm admin on the project but can't create an RSA Token on the machine for easy access.  So I had to play around to make a personal access token work.  In order to make that access token work, I had to run the checkout job on a different node so that it was running as a user that lived in Github as well (access token's namesake).  So when this job gets called from the jenkins job, I want it to clone to the calling job's workspace (/opt/jenkins/workspace/<project_name>).  Well since in order to authenticate, it lives in it's own shell, the workspace for this guy, and where it wants to clone to, is /home/<username>/<project_name>.  All that for my question:  how can I specify what folder to checkout to?  I tried to use "checkout to specific local branch" but it fails saying "is not a valid branch name".  Well, yeah, branch is referring to branch not folder, lol.  In StarTeam this is easy, you just specify working folder.  Any help?  Thanks!

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/128374b8-63d5-4b8c-86bb-5f214f5e68d2n%40googlegroups.com.

eric.fetzer

unread,
Mar 26, 2021, 10:00:05 PM3/26/21
to jenkins...@googlegroups.com
Thanks for the response Mark..  I'm using freestyle.  Doesn't checkout to a subdirectory only take relative paths?



Sent from my T-Mobile 4G LTE Device
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/Ob42FqU-0UY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtG6VTjt8F7Vj5pJW13VQJTfxCqRuZuDHjj5zZTa3jJ9%3Dg%40mail.gmail.com.

Mark Waite

unread,
Mar 26, 2021, 10:49:53 PM3/26/21
to Jenkins Users
Good point.  How about the external workspace plugin from GSoC 2016?  https://plugins.jenkins.io/external-workspace-manager/

eric.fetzer

unread,
Mar 26, 2021, 11:05:44 PM3/26/21
to jenkins...@googlegroups.com
Thanks Mark, I'll check it out.  Has to beat the hokey workaround I worked up.  :)

Björn Pedersen

unread,
Mar 29, 2021, 5:26:47 AM3/29/21
to Jenkins Users
Did you try to use credentials in jenkins to store the token  instead of  fiddling  with token on the machine?

Then you could do everything in one job instead of  doing some copy-magic (which would  horribly break once you start to use more nodes).

Björn

eric....@gmail.com

unread,
Mar 29, 2021, 9:14:19 AM3/29/21
to Jenkins Users
Yes Bjorn, that's the thread I used to figure out how to use the access token.  Unfortunately, the jenkins plugin isn't smart enough to pass a username, it uses the username it runs as.  So my master node is running as the user "jenkins" so it tries to authenticate to GitHub Enterprise as jenkins:<myToken which of course fails.  So I created a different node to run as my user, which thankfully, is the same username as I have in GitHub Enterprise.  Smoke and mirrors...

Eric Fetzer

unread,
Mar 29, 2021, 9:22:43 AM3/29/21
to jenkins...@googlegroups.com
Where I got the info to make this work, Bjorn, is from the thread that your thread referenced:

https://stackoverflow.com/questions/56809684/git-clone-in-jenkins-with-personal-access-token-idles-forever/61104603#61104603

It was a bit painful, but at least it gets it done.  What I've implemented is passing the workspace to the checkout, checking out as my user to my user workspace (home directory), moving the files checked out to the jenkins workspace, using sudo to chown the folder and files to be owned by the jenkins user.  It's very crude but it works.  I'm going to take a look at Mark's option above, but at least I found something that will work...

Dirk Heinrichs

unread,
Mar 29, 2021, 9:40:35 AM3/29/21
to jenkins...@googlegroups.com
Am Montag, den 29.03.2021, 07:22 -0600 schrieb Eric Fetzer:

It was a bit painful, but at least it gets it done.  What I've implemented is passing the workspace to the checkout, checking out as my user to my user workspace (home directory), moving the files checked out to the jenkins workspace, using sudo to chown the folder and files to be owned by the jenkins user.  It's very crude but it works.  I'm going to take a look at Mark's option above, but at least I found something that will work...

We've been doing GitHub checkouts from Jenkins for quite some time (until we moved the other way round, to a self-hosted GitLab), and didn't have any problems. However, we used SSH keys, which can be added to Jenkins Credentials and then used by the plain Git plugin in any Freestyle or Pipeline job.

HTH...

Dirk
-- 
Dirk Heinrichs
Senior Systems Engineer, Delivery Pipeline
OpenText ™ Discovery | Recommind
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, Registernummer HRB 10646
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
signature.asc

Eric Fetzer

unread,
Mar 29, 2021, 9:45:26 AM3/29/21
to jenkins...@googlegroups.com
Thanks for your reply Dirk!  I'm a unix guy and that would have been my first choice, however I don't have access to the GitHub OS, only my particular repositories.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/Ob42FqU-0UY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

Dirk Heinrichs

unread,
Mar 29, 2021, 9:52:10 AM3/29/21
to jenkins...@googlegroups.com
Am Montag, den 29.03.2021, 07:44 -0600 schrieb Eric Fetzer:

Thanks for your reply Dirk!  I'm a unix guy and that would have been my first choice, however I don't have access to the GitHub OS, only my particular repositories.

I don't understand. Nobody has this kind of access. But everybody can add SSH keys to their GH account (using the web UI).

Bye...
signature.asc

Eric Fetzer

unread,
Mar 29, 2021, 10:09:52 AM3/29/21
to jenkins...@googlegroups.com
Ohhhh, duh, I get it now.  I create the key with the jenkins user and add id_rsa.pub to the GitHub account for access.  I was somehow thinking I would need a key on the GitHub side.  Feeling kind of foolish now, lol.  Thanks Dirk!

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/Ob42FqU-0UY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

Eric Fetzer

unread,
Mar 29, 2021, 12:38:49 PM3/29/21
to jenkins...@googlegroups.com
Hmmm, thought that was going to work but it didn't.  I followed everything to a T at:  https://mohitgoyal.co/2017/02/27/configuring-ssh-authentication-between-github-and-jenkins/
The output shows:

Building on master in workspace /var/lib/jenkins/workspace/Git-Checkout
using credential JenkinsSSHKey
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://myGitRepo.git # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
No valid HEAD. Skipping the resetting
 > git clean -fdx # timeout=10
Fetching upstream changes from https://myGitRepo.git
 > git --version # timeout=10
using GIT_SSH to set credentials 

So I see it's using the creds.  Then when it tries to do the fetch, it says:

 > git fetch --tags --progress https://myGitRepo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://myGitRepo.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167)
	at hudson.scm.SCM.checkout(SCM.java:505)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1206)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:637)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:509)
	at hudson.model.Run.execute(Run.java:1907)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://myGitRepo.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Password authentication is not available for Git operations.
remote: You must use a personal access token or SSH key.

Mark Waite

unread,
Mar 29, 2021, 1:23:28 PM3/29/21
to Jenkins Users
You can't use ssh authentication with an https repository URL.  When using ssh authentication, you need to use an ssh repository URL.  When using username / password or username / token, you must use an HTTPS (or HTTP) URL.

Mark Waite

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAByBicb7irzVcbgHO0vV39EE7f515%3DVCkcHqrYE5eefkjbZT%3Dg%40mail.gmail.com.

eric....@gmail.com

unread,
Mar 29, 2021, 2:07:01 PM3/29/21
to Jenkins Users
Thanks Mark!  I believe I'm one step closer but it's still not working.  I'm now getting:

Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git@myURLRepo:myUser/myProject.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: ssh: connect to host myURLRepo port 22: Connection timed out fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

eric....@gmail.com

unread,
Mar 29, 2021, 3:26:56 PM3/29/21
to Jenkins Users
The only thing I can guess is that ssh is getting a question when he attempts to connect wanting to be added to the known_hosts file.  Wondering if maybe there's a way to establish this if this is indeed the issue?

Jérôme Godbout

unread,
Mar 29, 2021, 4:06:33 PM3/29/21
to jenkins...@googlegroups.com

I do add the domain and hsot to my known host file before to avoid that nasty confirm:

 

sh("ssh -R ${domain} -f \".ssh/known_hosts\"")

sh("ssh -t rsa ${domain} >> \".ssh/known_hosts\"")

 

that should do the trick and avoid that prompt when later used. Use the domain of the repos url domain.

 

Jérôme Godbout, B. Ing.


Software / Firmware Team Lead
O: (418) 682-3636 ext.: 114  

C: (581) 777-0050 
godb...@dimonoff.com

signature_168530195

dimonoff.com

1015 Avenue Wilfrid-Pelletier, 

Québec, QC G1W 0C4, 4e étage

 

 

From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> on behalf of eric....@gmail.com <eric....@gmail.com>
Date: Monday, March 29, 2021 at 3:27 PM
To: Jenkins Users <jenkins...@googlegroups.com>
Subject: Re: GitHub Clone to Different Local Directory

Dirk Heinrichs

unread,
Mar 30, 2021, 1:59:36 AM3/30/21
to jenkins...@googlegroups.com
Am Montag, den 29.03.2021, 11:07 -0700 schrieb eric....@gmail.com:

Please make sure you have the correct access rights and the repository exists.

Oh, yes, forgot that: On the host(s) that does/do the cloning, you need to either
  • connect to g...@github.com once per SSH on the console and as the user who runs Jenkins, to add the GH host key to that users known_hosts file, or
  • add a host block to that users ~/.ssh/config file like:
    host github
           Hostname github.com
           User git
           IdentityFile ~/.ssh/my_github_key
           ServerAliveInterval 15
           
    StrictHostKeyChecking no

In the second case, you also don't need to store the credentials in Jenkins, since you can use the alias "github" directly in your URLs, like "github:path/to/repo" (instead of "g...@github.com:..."). If you're reluctant to skip host key verification, you can also omit the last line and combine the two methods.
signature.asc

Dirk Heinrichs

unread,
Mar 30, 2021, 2:10:15 AM3/30/21
to jenkins...@googlegroups.com
Am Dienstag, den 30.03.2021, 05:59 +0000 schrieb 'Dirk Heinrichs' via Jenkins Users:
Am Montag, den 29.03.2021, 11:07 -0700 schrieb eric....@gmail.com:

Please make sure you have the correct access rights and the repository exists.

Oh, yes, forgot that: On the host(s) that does/do the cloning, you need to either
  • connect to g...@github.com once per SSH on the console and as the user who runs Jenkins, to add the GH host key to that users known_hosts file, or
  • add a host block to that users ~/.ssh/config file like:
    host github
           Hostname github.com
           User git
           IdentityFile ~/.ssh/my_github_key
           ServerAliveInterval 15
           
    StrictHostKeyChecking no

In the second case, you also don't need to store the credentials in Jenkins, since you can use the alias "github" directly in your URLs, like "github:path/to/repo" (instead of "g...@github.com:..."). If you're reluctant to skip host key verification, you can also omit the last line and combine the two methods.

If you are running many Jenkins agents you can of course do the first step once on one of them and then copy the GH host key into either the user's ~/.ssh/.known_hosts or the global /etc/ssh/ssh_known_hosts file on the other ones.
signature.asc

Eric Fetzer

unread,
Mar 30, 2021, 10:15:50 AM3/30/21
to jenkins...@googlegroups.com
So the url is g...@a.b.c.d and I tried to use your first command with a.b.c.d for domain and got "bad remote forwarding specification".  I tried it with b.c.d and c.d as well resulting in the same error.  Any ideas?  Thanks!

Jérôme Godbout

unread,
Mar 30, 2021, 11:44:47 AM3/30/21
to jenkins...@googlegroups.com

Oups my bad, in my hurry I forgot to change the right exe, I keep those into variable and I badly refilled them here:

 

sh("ssh-keygen -R ${domain} -f \".ssh/known_hosts\"")

sh("ssh-keyscan -t rsa ${domain} >> \".ssh/known_hosts\"")

The first one removes any old entry, the second one adds the record to the file.

 

Jérôme Godbout, B. Ing.


Software / Firmware Team Lead
O: (418) 682-3636 ext.: 114  

C: (581) 777-0050 
godb...@dimonoff.com

signature_796898478

Eric Fetzer

unread,
Mar 30, 2021, 1:18:36 PM3/30/21
to jenkins...@googlegroups.com
For this one I get "Host 'myDomain' not found in .ssh/known_hosts".  Not sure what I should be using for domain.  Seems it should be github.com in the example, so if my connection is g...@myGitProject.mydomain.com:/my/project, I would put "myGitProject.mydomain.com" in there.  Is that correct?

Jérôme Godbout

unread,
Mar 30, 2021, 1:26:02 PM3/30/21
to jenkins...@googlegroups.com

Do not check if the first command succeed, it remove previous entry if any. If none was present the command will fail.

 

Do the following:

 

String domain = “github.com”;

sh("ssh-keygen -R ${domain} -f \".ssh/known_hosts\"", returnStatus: true);

sh("ssh-keyscan -t rsa ${domain} >> \".ssh/known_hosts\"")

Now that should do it, you do not really care if the remove hit something or not, just make sure you do not have an old invalid entry.

 

Jérôme Godbout, B. Ing.


Software / Firmware Team Lead
O: (418) 682-3636 ext.: 114  

C: (581) 777-0050 
godb...@dimonoff.com

signature_1235056834

Eric Fetzer

unread,
Mar 30, 2021, 1:57:33 PM3/30/21
to jenkins...@googlegroups.com
Holy cow, this is NOT my day.  So I did it once and it appeared to work.  I vi'd the known_hosts file and it was at the bottom but it looked like it was transposed wrong (instead of a.b.c.d, it had b.a.c.d).  So I deleted that line and redid the entry.  It no longer adds to my known_hosts file even though it throws no error.  I'm totally lost as to what is happening here...

Eric Fetzer

unread,
Mar 30, 2021, 5:03:24 PM3/30/21
to jenkins...@googlegroups.com
On second thought, I don't think this ever worked at all.  I had attempted to add a line to the known_hosts file yesterday, that was probably what I saw and thought it worked but transposed.  The transposition was me.  I'm giving up on the rsa thing, going back to my clunky way to do it.  Thanks for trying!

Dirk Heinrichs

unread,
Mar 31, 2021, 2:10:21 AM3/31/21
to jenkins...@googlegroups.com
Am Montag, den 29.03.2021, 12:26 -0700 schrieb eric....@gmail.com:

The only thing I can guess is that ssh is getting a question when he attempts to connect wanting to be added to the known_hosts file.  Wondering if maybe there's a way to establish this if this is indeed the issue?

Yes, of course it does, and yes, there is (see my other two replies from yesterday). SSH verifies that it indeed connects to the correct remote host (github.com in this case), by comparing the HOST key of the remote host it has stored in the local user's (or machine's) known_hosts file. On the first connection, it asks the user to confirm the remote host key (unless the user has certain options set in her ~/.ssh/config to either skip the verification completely or accept new keys w/o asking). And this is what's happening here.

So, to get out of this, you need to login to your Jenkins machine once (as the user running Jenkins), run "ssh g...@github.com" and confirm the host key. Or add "StrictHostKeyChecking no" or, if your SSH is recent enough, "StrictHostKeyChecking accept-new" to that user's ~/.ssh/config (or machine's /etc/ssh/ssh_config).

HTH...
signature.asc

Eric Fetzer

unread,
Mar 31, 2021, 10:18:23 AM3/31/21
to jenkins...@googlegroups.com
Hi Dirk, sorry I didn't respond earlier, but I did try these things.  They just didn't work.  Questioning whether I didn't run into a firewall rule.  Even though they allow us to add an SSH Key in GitHub, that doesn't mean they've opened port 22 on the server.  When I try to ssh g...@github.com, it ends up "Connection timed out".  I would kind of expect a refusal if the firewall were slapping it down, but not sure.  I've also turned strictHostKeyChecking off and that also produced failure from the cloning.  This is a gov repository so I'm guessing that it is, indeed, a firewall rule.  Thanks for all the help!  Guess I should be thankful that 443 is open to me...

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/Ob42FqU-0UY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

Dirk Heinrichs

unread,
Apr 1, 2021, 1:31:39 AM4/1/21
to jenkins...@googlegroups.com
Am Mittwoch, den 31.03.2021, 08:17 -0600 schrieb Eric Fetzer:

Hi Dirk, sorry I didn't respond earlier, but I did try these things.  They just didn't work.  Questioning whether I didn't run into a firewall rule.  Even though they allow us to add an SSH Key in GitHub, that doesn't mean they've opened port 22 on the server.  When I try to ssh g...@github.com, it ends up "Connection timed out".  I would kind of expect a refusal if the firewall were slapping it down, but not sure.

Nope. This is exactly what you see if it's blocked by FW. If you could reach github.com with SSH (on port 22), but they didn't have the port open (no daemon listening), you'd get a "Connection refused" right away.

Bye...
signature.asc
Reply all
Reply to author
Forward
0 new messages