Jenkinsfile SCM Checkout stage, how to match remote pull request head with local build head

[Amila Gunathilake]

Hi all,

I'm having the below issue.

I was change my code to use GIT Reference repository in Jenkinsfile code such as below.

So I made a new pull request in GIT-HUB for the purpose of merge these changes into my master branch.

But when the Pull-Request running I saw in my builder machine (workspace) its not downloading (fetching) the relevant Remote Pull-Request Head/Branch/ because in my workspace (builder-machine) local branch always checkout with the master branch and fetching from the master branch and not checkout for the correct Pull Request Head. 

eg: Above screenshot is the way that my local workspace's .git > HEAD file looks like in the Pull Request I created after above code changes. And as you can see in the HEAD file it's automatically checkout in to the master branch instead of checking out into the relevant Pull-Request hash (tag). 

• But below is the correct screenshot I captured from any of other Pull Requests, which should be the correct way to show the checkout branch or the HEAD.

I hope you will understand my situation here? Please let me know if you guys need further information.

I throughly suggest that I want some code to replace above Yellow color highlighted sections. 

So please could anyone please give some advice regarding this issue ?  

[Mark Waite]

The value of 'branches' that you're using is incorrect.  In the triggers section you're declaring that if the env.BRANCH_NAME is master, then perform builds on a specific schedule.  Use the same env.BRANCH_NAME in the branches section by changing from:

branches: [[name: 'origin/* ]]

to:

branches: [[name: env.BRANCH_NAME]]

If the env.BRANCH_NAME is defined for the trigger case, then it should also be defined and available to use when defining the branch name.

The duplicate value of refspec will likely prevent the checkout from detecting any changes in your repository.  Change from:

refspec: "+refs/heads/*:refs/remotes/origin/*",

refspec: "+refs/heads/pr/*:refs/remotes/origin/pr/*",

 

to:

refspec: "+refs/heads/*:refs/remotes/origin/* +refs/heads/pr/*:refs/remotes/origin/pr/*",

 

The syntax you're using provides a value for refspec then replaces it with a different value for refspec.  The corrected syntax provides the default refspec plus the "pr" namespace refspec as a single argument so that both will be used.

Mark Waite

[Amila Gunathilake]

Hi Mark,

After I did your changes Jenkins log/console-output gives below error message.  







So as per the above error after your requested changes its not downloading code from remote repo.  Please see below screenshots of Builder node's workspace folder, which you can have any idea. 

 

In above Workspace folder as you can see there are no any other my remote repository files, only having the .git folder.  Which is because of the above error Couldn't find any revision to build

I think Couldn't find any revision to build error coming from after we changed branch name into "branches: [[name: env.BRANCH_NAME]]"

Please suggest any other ideas.  

Thanks
Amila

[Amila Gunathilake]

Hi Mark,

After I did your changes Jenkins log/console-output gives below error message.  






So as per the above error after your requested changes its not downloading code from remote repo.  Please see below screenshots of Builder node's workspace folder, which you can have any idea. 

 

In above Workspace folder as you can see there are no any other my remote repository files, only having the .git folder.  Which is because of the above error Couldn't find any revision to build

I think Couldn't find any revision to build error coming from after we changed branch name into "branches: [[name: env.BRANCH_NAME]]" 

Please suggest any other ideas.  

Thanks
Amila

[Mark Waite]

As far as I can tell, you want to:

• Use declarative pipeline for clarity of the pipeline
• Skip the default checkout so that you can tune the performance of checkout
• Perform your own checkout using a reference repository
• Use the same Jenkinsfile on branches and on pull requests so that pull requests can be merged into target branches easily
• Checkout in the workspace to a branch with the same name as the branch on the git server

Based on those assumptions, I created a sample job and a pull request that targets the master branch and meets those requirements.  When a pull request is being evaluated, the changes on the pull request are detected and used.  When the master branch is being evaluated, its changes are detected and used.

Please refer to the sample job and the techniques used in the sample job.  Note that you will need the most recent release of the git plugin for references to the scm variable.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d69b5c1c-e4f8-42e4-ba75-084ada0b653a%40googlegroups.com.

[Amila Gunathilake]

Hi Mark,

Thanks for your reply and your sample job really helped me.  Here my New Jenkinsfile checkout stage looks like. 

I had successfully build the Pull Request this time, when I'm building the PR below is the Console Output log I had from Jenkins.  

But after I merged my Pull Request into the master branch I'm just getting below Console output for checkout stage without the message "using Reference Repository"

Also after merge into master branch and after building in master branch "alternates" file also missing from the workspace path - 'workspace\master411813f6\.git\objects\info' 

is it everything okay?   I cannot find weather reference repository is using or not ?  I'm 100% sure that my reference repository exists in build server. 

Thanks
Amila

On Friday, February 21, 2020 at 12:00:49 AM UTC-5, Mark Waite wrote:

As far as I can tell, you want to:

• Use declarative pipeline for clarity of the pipeline
• Skip the default checkout so that you can tune the performance of checkout
• Perform your own checkout using a reference repository
• Use the same Jenkinsfile on branches and on pull requests so that pull requests can be merged into target branches easily
• Checkout in the workspace to a branch with the same name as the branch on the git server

Based on those assumptions, I created a sample job and a pull request that targets the master branch and meets those requirements.  When a pull request is being evaluated, the changes on the pull request are detected and used.  When the master branch is being evaluated, its changes are detected and used.

Please refer to the sample job and the techniques used in the sample job.  Note that you will need the most recent release of the git plugin for references to the scm variable.

Mark Waite

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

[Amila Gunathilake]

Hi Mark,

This is the new master branch Jenkins Console output I'm getting.  There is no message says that I'm using GIT reference repository like-wise my previous console output in Pull Request. 





So what will be the reason for this ?  Is everything okay with this console out-put ? 


Thanks,
Amila

[Mark Waite]

You can confirm that a reference repository is being used by looking for the file .git/objects/info/alternates in the repository.  

The file in my sample repository includes the text:

 /home/mwaite/git/bare/jenkins/jenkins.git/objects

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f5630876-867e-4b30-9de5-1d28fa8f8eec%40googlegroups.com.

[Amila Gunathilake]

Hi Mark,

Actually where I can find the file you mention "/home/mwaite/git/bare/jenkins/jenkins.git/objects"  in your repository MarkEWaite/jenkins-bugs ?

Finally I able to get use the GIT Reference repository in my builder machine.   But to get use that I had to replace my old Jenkins Configurations by giving the GIT reference repository path as below. 

 It's really surprised me because, as above screenshot when I include that reference-path configuration in Jenkins GUI only it starting to getting use my Jenkinsfile's  SCM checkout stage code configurations. 


Still I have a small issue as you can see below Jenkins console-output it says "using Shallow fetch with depth 3"  02 times.  (previously it was only one time. So I'm afraid that Jenkins uses both configurations such as Jenkins GUI and Jenkinsfile code configurations here. Is it okay since it may not do any harm to my builds ? 

Thanks again for your quick feedbacks Mr Mark,
Amila

On Monday, February 24, 2020 at 10:17:07 PM UTC-5, Mark Waite wrote:

You can confirm that a reference repository is being used by looking for the file .git/objects/info/alternates in the repository.  

The file in my sample repository includes the text:

 /home/mwaite/git/bare/jenkins/jenkins.git/objects

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f5630876-867e-4b30-9de5-1d28fa8f8eec%40googlegroups.com.

[Amila Gunathilake]

On Tuesday, February 25, 2020 at 3:03:34 AM UTC-5, Amila Gunathilake wrote:

Hi Mark,

Actually where I can find the file you mention "/home/mwaite/git/bare/jenkins/jenkins.git/objects"  in your repository MarkEWaite/jenkins-bugs ?

Finally I able to get use the GIT Reference repository in my builder machine.   But to get use that I had to replace my old Jenkins Configurations by giving the GIT reference repository path as below. 

 

 It's really surprised me because, as above screenshot when I include that reference-path configuration in Jenkins GUI only it starting to getting use my Jenkinsfile's  SCM checkout stage code configurations. 


Still I have a small issue as you can see below Jenkins console-output it says "using Shallow fetch with depth 3"  02 times.  (previously it was only one time. So I'm afraid that Jenkins uses both configurations such as Jenkins GUI and Jenkinsfile code configurations here. Is it okay since it may not do any harm to my builds ? 

 

[Mark Waite]

On Monday, February 24, 2020 at 8:17:07 PM UTC-7, Mark Waite wrote:

You can confirm that a reference repository is being used by looking for the file .git/objects/info/alternates in the repository.  

That sentence is phrased incorrectly.

You can confirm that a reference repository is being used by looking for the file .git/objects/info/alternates in the workspace. (emphasis added)

[Mark Waite]

On Tuesday, February 25, 2020 at 1:06:26 AM UTC-7, Amila Gunathilake wrote:

On Tuesday, February 25, 2020 at 3:03:34 AM UTC-5, Amila Gunathilake wrote:

Hi Mark,

Actually where I can find the file you mention "/home/mwaite/git/bare/jenkins/jenkins.git/objects"  in your repository MarkEWaite/jenkins-bugs ?

Finally I able to get use the GIT Reference repository in my builder machine.   But to get use that I had to replace my old Jenkins Configurations by giving the GIT reference repository path as below. 

 

 It's really surprised me because, as above screenshot when I include that reference-path configuration in Jenkins GUI only it starting to getting use my Jenkinsfile's  SCM checkout stage code configurations. 

That should not surprise you.  If you want to use a reference repository, then you'll need to define the location of that reference repository in each context where a checkout is performed.  Reading the Jenkinsfile from git is a checkout.

 

Still I have a small issue as you can see below Jenkins console-output it says "using Shallow fetch with depth 3"  02 times.  (previously it was only one time. So I'm afraid that Jenkins uses both configurations such as Jenkins GUI and Jenkinsfile code configurations here. Is it okay since it may not do any harm to my builds ? 

 

 

Yes, the git plugin is known to perform two fetch operations for each checkout.  It has been reported as https://issues.jenkins-ci.org/browse/JENKINS-49757 and as https://issues.jenkins-ci.org/browse/JENKINS-56404 .  There is a Google Summer of Code project idea that proposes to fix that by reducing it so that it only calls fetch once.

See https://jenkins.io/projects/gsoc/2020/project-ideas/git-plugin-performance/#fix-known-issues-avoid-redundant-fetch for a discussion of the issue and ideas that have been offered.

Mark Waite