[JIRA] (JENKINS-61009) How to use a reference repository in an organization folder?

57 views
Skip to first unread message

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 7, 2020, 10:05:06 AM2/7/20
to jenkinsc...@googlegroups.com
Mark Waite updated an issue
 
Jenkins / Bug JENKINS-61009
How to use a reference repository in an organization folder?
Change By: Mark Waite
Summary: Jenkins GIT Reference Repository blocker How to use  a reference repository in GitHub Organization Pipeline, where I have multi repositories in one an organization folder . ?
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

kristyk.ashton@gmail.com (JIRA)

unread,
Feb 7, 2020, 10:28:04 AM2/7/20
to jenkinsc...@googlegroups.com

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 8, 2020, 7:28:03 AM2/8/20
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
 
Change By: Mark Waite
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 8, 2020, 5:33:03 PM2/8/20
to jenkinsc...@googlegroups.com

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 8, 2020, 5:33:04 PM2/8/20
to jenkinsc...@googlegroups.com
 

Since there has been no response to my request to confirm the expected options work for this question, I'm closing the question as "Not a defect".

Change By: Mark Waite
Status: Open Fixed but Unreleased
Resolution: Not A Defect

amila.kdam@gmail.com (JIRA)

unread,
Feb 8, 2020, 10:39:03 PM2/8/20
to jenkinsc...@googlegroups.com
Amila Gunathilake reopened an issue
 

Hello Mark Waite , sorry for the late in reply Sir. (just because I was in weekend holidays)

So regarding the question you asked I would like to have the Second option that you mentioned above. 

2.  "Define the reference repository in the organization folder and use a combined repository that includes the content from all the repositories in the organization"

 

But in my GIT-Hub organization I should use the "Townsuite" folder as the reference folder then in-side that "Townsuite" folder there are 15 different Repository folders I have.  Such as WebPortals.git folder, Financial.git folder  etc. 

So my concern is how you going to referenced the "Townsuite" folder for all these 15 different repositories in the Configurations of my GIT-HUB Organization folder _(which is _Townsuite folder_)_ 

my sincere thanks and best wishes for you regarding my question Mr. Marc Waite.

Change By: Amila Gunathilake
Resolution: Not A Defect
Status: Closed Reopened
Assignee: Mark Waite

amila.kdam@gmail.com (JIRA)

unread,
Feb 8, 2020, 10:40:02 PM2/8/20
to jenkinsc...@googlegroups.com
Amila Gunathilake edited a comment on Bug JENKINS-61009
 
Re: How to use a reference repository in an organization folder?
Hello [~markewaite] , sorry for the late in reply Sir. (just because I was in weekend holidays)


So regarding the question you asked I would like to have the Second option that you mentioned above. 

*2.*  "*Define the reference repository in the organization folder and use a [combined repository|https://plugins.jenkins.io/git#combining-repositories] that includes the content from all the repositories in the organization*"

 

But in my GIT-Hub organization I should use the +*"_Townsuite_" folder as the reference folder*+ then in-side that "_Townsuite_" folder there are 15 different Repository folders I have.  Such as WebPortals.git folder, Financial.git folder  etc. 


So my concern is how you going to referenced the "_Townsuite_" folder for all these 15 different repositories in the Configurations of my GIT-HUB Organization folder _(_which is _Townsuite_ folder_)_ 

!image-2020-02-09-09-11-01-619.png!





_my sincere thanks and best wishes for you regarding my question Mr. Marc Waite._

_!image-2020-02-09-09-05-38-996.png!_  

amila.kdam@gmail.com (JIRA)

unread,
Feb 8, 2020, 10:41:03 PM2/8/20
to jenkinsc...@googlegroups.com
Amila Gunathilake edited a comment on Bug JENKINS-61009
Hello [~markewaite] , sorry for the late in reply Sir. (just because I was in weekend holidays)

So regarding the question you asked I would like to have the Second option that you mentioned above. 

*2.*  "*Define the reference repository in the organization folder and use a [combined repository|https://plugins.jenkins.io/git#combining-repositories] that includes the content from all the repositories in the organization*"

 

But in my GIT-Hub organization I should use the +*"_Townsuite_" * folder as the reference folder * + then in-side inside that "_Townsuite_" folder there are 15 different Repository folders I have.  Such as WebPortals.git folder, Financial.git folder  etc. 


So my concern is how you going to referenced the "_Townsuite_" folder for all these 15 different repositories in the Configurations of my GIT-HUB Organization folder _(_which is _Townsuite_ folder_)_ 

!image-2020-02-09-09-11-01-619.png!





_my sincere thanks and best wishes for you regarding my question Mr. Marc Waite._

 

amila.kdam@gmail.com (JIRA)

unread,
Feb 8, 2020, 10:44:02 PM2/8/20
to jenkinsc...@googlegroups.com
Amila Gunathilake edited a comment on Bug JENKINS-61009
Hello [~markewaite] , sorry for the late in reply Sir. (just because I was in weekend holidays) . My defect/issue is still there.

So regarding the question you asked I would like to have the Second option that you mentioned above. 

*2.*  "*Define the reference repository in the organization folder and use a [combined repository|https://plugins.jenkins.io/git#combining-repositories] that includes the content from all the repositories in the organization*"

 

But in my GIT-Hub organization I should use the +*"_Townsuite_"* folder as the reference folder+ then inside that "_Townsuite_" folder there are 15 different Repository folders I have.  Such as WebPortals.git folder, Financial.git folder  etc. 


So my concern is how you going to referenced the "_Townsuite_" folder for all these 15 different repositories in the Configurations of my GIT-HUB Organization folder _(_which is _Townsuite_ folder_)_ 

!image-2020-02-09-09-11-01-619.png!

_my sincere thanks and best wishes for you regarding my question Mr. Marc Waite._

 

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 9, 2020, 4:10:03 AM2/9/20
to jenkinsc...@googlegroups.com

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 9, 2020, 4:10:03 AM2/9/20
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
Change By: Mark Waite
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 9, 2020, 4:26:03 AM2/9/20
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-61009
 
Re: How to use a reference repository in an organization folder?

Amila Gunathilake your reply describes individual repositories under a directory named "TownSuite". If you're using individual repositories under the TownSuite directory, then you must choose option number 1 that I offered, place the definition of the reference repository in each Jenkinsfile for that repository. In that case, the reference repository is not defined in the organization folder at all, since it is defined in each of the individual repositories and is specific for each repository.

I think that is the easiest to manage and the most direct way to provide a reference repository for those repositories that benefit most from it, while allowing you to choose to not use a reference repository for small repositories or infrequently modified repositories.

The second option that I offered requires work from the administrator to create a single git repository on the agent. That single git repository combines the content of all relevant repositories into a single repository. The process for creating that single git repository is described in the documentation that I linked in my earlier description of that option. If there is a single git repository that combines the history of all the git repositories, then that single git repository could be referenced in the definition of the organization folder.

Most people do not want the complexity of managing that single git repository which combines the history of all git repositories. They typically choose to place the definition of the location of the reference repository in the individual repositories as a part of the checkout step in the Jenkinsfile.

amila.kdam@gmail.com (JIRA)

unread,
Feb 9, 2020, 4:54:03 AM2/9/20
to jenkinsc...@googlegroups.com
Amila Gunathilake updated an issue
 
Change By: Amila Gunathilake
Attachment: image-2020-02-09-15-24-32-777.png

amila.kdam@gmail.com (JIRA)

unread,
Feb 9, 2020, 5:07:09 AM2/9/20
to jenkinsc...@googlegroups.com
Amila Gunathilake updated an issue
Change By: Amila Gunathilake
Attachment: image-2020-02-09-15-37-47-089.png

amila.kdam@gmail.com (JIRA)

unread,
Feb 9, 2020, 5:17:05 AM2/9/20
to jenkinsc...@googlegroups.com
Amila Gunathilake commented on Bug JENKINS-61009
 
Re: How to use a reference repository in an organization folder?

Thanks for the quick reply Mr. Mark Waite

Yeah I'm also thought about your 1st option to have a separate checkout step with each reference folders in my Jenkinsfile.  So I already tested that 1st option in my test pipeline then I got an issue because then GIT-Plugin uses both checkout options and then its downloading source codes twice into my agent machine, because its using the default "Repository HTTPS URL" which I configured in Jenkins Configs as well.  You can see that default URL ("Repository HTTPS URL")  configuration in below last screenshot.

      
 Jenkinsfile scm checkout option I added for Webportals.git reference repository.

But also it's using below Jenkins configuration URL which I should configured in Jenkins (because if this url not provided my GIT webhooks are not working) which led Jenkins to automatically downloading the same Webportals.git repository into my Jenkins agent twice.

If I didn't include that "Repository HTTPS URL" in Jenkins Configurations  it will not good for trigger my webhooks I guess because in default Jenkins always searching this URL for connect with the GIT-Hub Repository's  actions (PR) and push triggers.

So Mr Mark my concern is how to got with your above Option 01 (Jenkinsfile checkout stage which I love to go with) in this Situation?  Hope you understand my problem here sir ?

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 9, 2020, 7:58:03 AM2/9/20
to jenkinsc...@googlegroups.com

Amila Gunathilake it appears you are using declarative pipeline. That's a very good choice. Declarative pipeline includes an implicit checkout so that typical pipelines which do not need to customize their checkout step will run without an explicit checkout.

Since you need to control checkout options, you can skip the default checkout by adding the skipDefaultCheckout option in your declarative Pipeline Jenkinsfile:

options {
    skipDefaultCheckout()
}

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 9, 2020, 8:07:03 AM2/9/20
to jenkinsc...@googlegroups.com

You should also remove the deleteDir() if you're concerned enough about performance to use a reference repository and a narrow refspec. The deleteDir removes everything that was already in the directory, including the .git/objects directory with all the repository content.

If you're concerned that your build script doesn't do an adequate job of cleaning the repository, you can either insert a shell/bat step to git clean -xffd or you can add the git extension to Clean after checkout. Both of those do not destroy the previously cloned git repository.

amila.kdam@gmail.com (JIRA)

unread,
Feb 10, 2020, 12:46:05 PM2/10/20
to jenkinsc...@googlegroups.com
Amila Gunathilake updated an issue
Change By: Amila Gunathilake
Attachment: image-2020-02-10-23-16-48-189.png

amila.kdam@gmail.com (JIRA)

unread,
Feb 10, 2020, 1:16:02 PM2/10/20
to jenkinsc...@googlegroups.com
Amila Gunathilake updated an issue
Change By: Amila Gunathilake
Attachment: image-2020-02-10-23-47-02-131.png

amila.kdam@gmail.com (JIRA)

unread,
Feb 10, 2020, 1:24:02 PM2/10/20
to jenkinsc...@googlegroups.com
Amila Gunathilake commented on Bug JENKINS-61009
 
Re: How to use a reference repository in an organization folder?

Mark Waite sir thanks for your help now I'm using above option 01 in Jenkinsfile checkout step code and successfully skipping the Default Checkout from GIT-HUB URL.  Below is my new Jenkinsfile code after changes. 

When I'm running a Build in Jenkins Pipeline for the first time only (rather than the first build any other build's Jenkins Console output doesn't show that keyword "Using reference repository".),  It's printing the "Using reference repository" statement in the Build's Jenkins Console Output (Jenkins log) as below.  
Using reference repository: C:/Program Files (x86)/Jenkins/workspace/GIT_REF/WebPortals.git

 

But also in the Jenkins Console output log its printing as "Fetching upstream changes from remote URL" as well. So here my concern is Mr. Mark if we using the reference repository why Jenkins fetching source code still from GIT Remote repository URL ?
Fetching upstream changes from g...@github.com:AmilaDevops/WebPortals.git
 

Below is the entire Jenkins Log Console Output for that build.

 

Mr. Mark Waite,  So here how can I make sure that Jenkins using my GIT Reference Repository successfully ? 

Is my Jenkinsfile code (checkout: userRemoteConfigs section) for reference repository is okay?  

Thanks & Best Regards,

 

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 10, 2020, 2:00:03 PM2/10/20
to jenkinsc...@googlegroups.com

You've misunderstood the concept of a reference repository. Refer to the command line git documentation where it says:

Using an already existing repository as an alternate will require fewer objects to be copied from the repository being cloned, reducing network and local storage costs. (emphasis added)

A reference repository does not change the source from which the workspace is cloned. A reference repository reduces the network traffic and the local disc use by reusing existing local objects rather than requesting them from the remote git server.

You can confirm that you are using a reference repository by comparing the size of the workspace on disc with the size of a repository cloned without a reference repository. If the workspace on disc is not significantly smaller than the clone created without a reference repository, then there is a risk that the reference repository argument is being ignored or the reference repository does not exist or the reference repository does not contain objects which will assist the repository being cloned.

I'm closing this report as "Not a defect" because it has now become a personal question and answer session.

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 10, 2020, 2:01:03 PM2/10/20
to jenkinsc...@googlegroups.com
Mark Waite closed an issue as Not A Defect
 
Change By: Mark Waite
Status: Open Closed
Resolution: Not A Defect

amila.kdam@gmail.com (JIRA)

unread,
Feb 11, 2020, 5:57:03 PM2/11/20
to jenkinsc...@googlegroups.com
Amila Gunathilake reopened an issue
 

Still my defect is there, because if I don't know whether GIT reference repository using or not this all my implementations in Jenkinsfile Checkout stage will be in vain. 

So could you bare this for couple of days more Mr Mark Waite. Thanks. !!

Change By: Amila Gunathilake
Resolution: Not A Defect
Status: Closed Reopened
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 11, 2020, 7:12:02 PM2/11/20
to jenkinsc...@googlegroups.com
Mark Waite closed an issue as Not A Defect
 

No, this is not a defect.

I've already described the techniques you can use to confirm that a reference repository is having the desired result. Please review the prior comments in this issue for my description of those techniques.

I don't intend to provide further answers on this issue. It is not a defect.

Change By: Mark Waite
Status: Reopened Closed
Resolution: Not A Defect

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 7:58:02 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake updated an issue
Change By: Amila Gunathilake
Attachment: image-2020-02-12-07-57-17-958.png

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 7:59:03 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake updated an issue
Change By: Amila Gunathilake
Attachment: image-2020-02-12-07-58-17-379.png

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 7:59:07 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake commented on Bug JENKINS-61009
 
Re: How to use a reference repository in an organization folder?

I'm sorry  could you tell me how to resolve this error sir ?  We lost our live customer release due to below error.

I'm getting below error when I'm building my Pull Request after all changes you explained above. 

Please see the below my Jenkinsfile code as well I hope above issue causing by the refspec. 

 

 

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 8:00:04 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake edited a comment on Bug JENKINS-61009
I'm sorry  could you tell me how to resolve this error sir ?  We lost our live customer release due to below error.

I'm getting below error when I'm building my Pull Request after all changes you explained above. 
!image-2020-02-12-07- 59-42-137.png!
!image-2020-02-12-07-
58-17-379.png!



Please see the below my Jenkinsfile code as well I hope above issue causing by the refspec. 
!image-2020-02-12-07-57-17-958.png!





 

 

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 8:08:02 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake edited a comment on Bug JENKINS-61009
I'm sorry  could you tell me how to resolve this error sir ?  We lost our live customer release due to below error.

I'm getting below error when I'm building my Pull Request after all changes you explained above. 
!image-2020-02-12-07-59-42-137.png!
!image-2020-02-12-07-58-17-379.png!

Please see the below my Jenkinsfile code as well , I hope above issue causing by the refspec as I underlined in below code
!image-2020-02-12-07-57-17-958.png!

 

 

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 8:15:03 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake reopened an issue
 
Change By: Amila Gunathilake
Resolution: Not A Defect
Status: Closed Reopened

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 8:15:14 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake edited a comment on Bug JENKINS-61009
 
Re: How to use a reference repository in an organization folder?
I'm sorry  this is the final question, could you tell me how to resolve this error sir ?  We lost our live customer release due to below error.


I'm getting below error when I'm building my Pull Request after all changes you explained above. 
!image-2020-02-12-07-59-42-137.png!
!image-2020-02-12-07-58-17-379.png!

Please see the below my Jenkinsfile code as well, I hope above issue causing by the refspec as I underlined in below code. 
!image-2020-02-12-07-57-17-958.png!

 

 

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 10:09:05 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake edited a comment on Bug JENKINS-61009
[~markewaite] yes I compared .git folders as you said and there is a significant change. So I 'm now going to merge my Pull request into my master branch. I'm sorry  this is the final question, could you tell me how to resolve this below error sir ?  We lost our live customer release due to below error.  

I'm getting below error when I'm building my Pull Request after all changes you explained above. 
!image-2020-02-12-07-59-42-137.png!
!image-2020-02-12-07-58-17-379.png!

Please see the below my Jenkinsfile code as well, I hope above issue causing by the refspec as I underlined in below code. 
!image-2020-02-12-07-57-17-958.png!

 

 

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 10:10:03 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake edited a comment on Bug JENKINS-61009
[~markewaite] yes I compared .git folders as you said and there is a significant change difference of folder sizes . So I'm now going to merge my Pull request into my master branch. I'm sorry  this is the final question, could you tell me how to resolve this below error sir ?  


I'm getting below error when I'm building my Pull Request after all changes you explained above. 
!image-2020-02-12-07-59-42-137.png!
!image-2020-02-12-07-58-17-379.png!

Please see the below my Jenkinsfile code as well, I hope above issue causing by the refspec as I underlined in below code. 
!image-2020-02-12-07-57-17-958.png!

 

 

amila.kdam@gmail.com (JIRA)

unread,
Feb 12, 2020, 10:11:03 AM2/12/20
to jenkinsc...@googlegroups.com
Amila Gunathilake edited a comment on Bug JENKINS-61009
Mr. [~markewaite] I'm sorry  this is the final question,  yes I compared .git folders as you said and there is a significant difference of folder sizes. So I'm now going to merge my Pull request into my master branch. I'm sorry  this is the final question, could Could you tell me how to resolve this below error sir ?  


I'm getting below error when I'm building my Pull Request after all changes you explained above. 
!image-2020-02-12-07-59-42-137.png!
!image-2020-02-12-07-58-17-379.png!

Please see the below my Jenkinsfile code as well, I hope above issue causing by the refspec as I underlined in below code. 
!image-2020-02-12-07-57-17-958.png!

 

 

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 12, 2020, 10:22:04 AM2/12/20
to jenkinsc...@googlegroups.com
Mark Waite closed an issue as Not A Defect
Change By: Mark Waite
Status: Reopened Closed
Resolution: Not A Defect

olivier@vernin.me (JIRA)

unread,
Mar 11, 2020, 9:25:16 AM3/11/20
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages