git plugin, use --single-branch option to save disk space?

[David Balazic]

Hi!

With the git plugin, is there a way to clone a repo with the --single-branch option?

We have a freestyle job wit a Git source code, where we specify the branch as: refs/heads/main
And when the build is done, the cloned repo (the .git folder) is very big, about 2.6 GB.
We use a command line git implementation (gitforwindows.org 2.40.1).

Using command line to clone gives this values:

git clone --branch main https://server.example.com/foo/bar "C:\tmp\git test\test_xxx_branch_main"
git clone --single-branch --branch main https://server.example.com/foo/bar "C:\tmp\git test\test_xxx_branch_main_single"

The first creates a 2.6 GB .git folder, while the second a 70 MB sized .git folder.
They both also have about 120 MB of working tree files.

So about 2.4 GB is "wasted" due to unneeded cloned branches.

Is there a way to make the Jenkins git plugin to clone only the required branches?

I realize the plugin uses different commands, but hope the idea is doable.


For reference,
I have Jenkins 2.387.2 on Windows Server 2019
Git plugin 5.0.2
Git implementation: gitforwindows.org 2.40.1
I also tried with JGit (it behaves the same, uses 2.6GB per clone)


Also, the commands used in the build job are:

> git.exe init C:\ProgramData\Jenkins\.jenkins\workspace\TEST build git branch one # timeout=10
> git.exe --version # timeout=10
> git.exe fetch --no-tags --force --progress --depth=1 -- https://server.example.com/foo/bar.git +refs/heads/*:refs/remotes/origin/* # timeout=10

I suspect the asterisk in the last command makes it download more than I need, but I'm no git expert.


Lep pozdrav,
David Balažic





The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Any opinions expressed are mine and do not necessarily represent the opinions of the Company. Emails are susceptible to interference. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is strictly prohibited and may be unlawful. If you have received this message in error, do not open any attachments but please notify the Endava Service Desk on (+44 (0)870 423 0187), and delete this message from your system. The sender accepts no responsibility for information, errors or omissions in this email, or for its use or misuse, or for any act committed or omitted in connection with this communication. If in doubt, please verify the authenticity of the contents with the sender. Please rely on your own virus checkers as no responsibility is taken by the sender for any damage rising out of any bug or virus infection.

Endava plc is a company registered in England under company number 5722669 whose registered office is at 125 Old Broad Street, London, EC2N 1AR, United Kingdom. Endava plc is the Endava group holding company and does not provide any services to clients. Each of Endava plc and its subsidiaries is a separate legal entity and has no liability for another such entity's acts or omissions.

[Mark Waite]

On Friday, May 26, 2023 at 5:46:34 AM UTC-6 David Balazic wrote:

Hi!

With the git plugin, is there a way to clone a repo with the --single-branch option?

We have a freestyle job wit a Git source code, where we specify the branch as: refs/heads/main
And when the build is done, the cloned repo (the .git folder) is very big, about 2.6 GB.
We use a command line git implementation (gitforwindows.org 2.40.1).

Using command line to clone gives this values:

git clone --branch main https://server.example.com/foo/bar "C:\tmp\git test\test_xxx_branch_main"
git clone --single-branch --branch main https://server.example.com/foo/bar "C:\tmp\git test\test_xxx_branch_main_single"

The first creates a 2.6 GB .git folder, while the second a 70 MB sized .git folder.
They both also have about 120 MB of working tree files.

So about 2.4 GB is "wasted" due to unneeded cloned branches.

Is there a way to make the Jenkins git plugin to clone only the required branches?

Yes, you can match the behavior of the --single-branch argument to git clone with the Jenkins git plugin.

Configure a refspec that matches a single branch in the remote configuration section.

Enable "Honor refspec on initial clone"

Those options are described in a git plugin Pipeline example.

If you don't need the history of the repository in your job, you can reduce the size even further by using a shallow clone that only retrieves the most recent commit.

That option is also described in a git plugin PIpeline example.

You can reduce disc use even further if you have a copy of the remote repository already available on the agent.  Then you can use a reference repository.  Reference repositories are generally not helpful when using shallow clone, since the reference repository may not include many of the changes in the shallow clone.

A 10-15 minute video discussing various performance improvement options is also available if you prefer video.

Mark Waite

 

[David Balazic]

Thanks but I already tested those:

 

Refspec: refs/heads/main

Honor refspec on initial clone: enabled

Shallow clone: enabled

Shallow clone depth: 1

 

 

They download a little less : 1.8 GB

 

The credit goes to the shallow clone option. The Honor refspec option does not seem to make any difference. Even the commands seem to be unaffected by it.

Maybe the refspec option is missing from the GUI? The example on the web has it.

 

 

Regards,

David Balažic

Lead engineer                                                                         Endava

 

From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> On Behalf Of Mark Waite
Sent: Friday, 26 May 2023 14:17
To: Jenkins Users <jenkins...@googlegroups.com>
Subject: Re: git plugin, use --single-branch option to save disk space?

 

CAUTION - EXTERNAL EMAIL

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d9b55add-c0ae-4440-ae86-0354d5082fb9n%40googlegroups.com.

[David Balazic]

(sorry for top posting, it's outlook…)

 

For reference, if I do the git commands manually, and replace the fetch with:

 

git.exe fetch -tags --force --progress --  https://server/foo/bar +refs/heads/main:refs/remotes/origin/main

 

then I get the small .git folder. (I replaced the two asterisks with "main")

 

 

Regards,

David Balažic

[Mark Waite]

On Friday, May 26, 2023 at 7:23:39 AM UTC-6 David Balazic wrote:

Thanks but I already tested those:

 

Refspec: refs/heads/main

Honor refspec on initial clone: enabled

Shallow clone: enabled

Shallow clone depth: 1

 

 

They download a little less : 1.8 GB

 

The credit goes to the shallow clone option. The Honor refspec option does not seem to make any difference. Even the commands seem to be unaffected by it.

Maybe the refspec option is missing from the GUI? The example on the web has it.

 

Not missing from the GUI, just hidden by default behind the "Advanced" button on the Repositories configuration of the page.  Click the Advanced button and it will show two more fields, a "Name" field and a "Refspec" field.

 

Mark Waite

[David Balazic]

Thanks!

 

That did the trick.

 

 

David Balažic

Lead engineer                                                                         Endava

 

From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> On Behalf Of Mark Waite
Sent: Friday, 26 May 2023 15:47
To: Jenkins Users <jenkins...@googlegroups.com>
Subject: Re: git plugin, use --single-branch option to save disk space?

 

CAUTION - EXTERNAL EMAIL

On Friday, May 26, 2023 at 7:23:39 AM UTC-6 David Balazic wrote:

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/833aaac1-5ba2-408f-9e02-926bfc9a747fn%40googlegroups.com.