Git LFS and Git for Windows

[Johannes Schindelin]

Hi Taylor (and others who have expressed interest in Git LFS on Windows),

As discussed at the GitMerge, I am looking into Git LFS as part of the Git
for Windows experience. Previously, I was very hesitant (see also this
comment:
https://github.com/git-for-windows/build-extra/pull/110#issuecomment-210480745)
due to its size. Given that there were relatively few requests for Git
LFS, I imagine that only a small fraction of Git for Windows users use Git
LFS, and increasing the installer size by 35% was a bit much.

So I just played with including git-lfs.exe into the Git for Windows
installer, and in my test (simply adding the git-lfs.exe straight from the
AMD64 package at https://github.com/git-lfs/git-lfs/releases/tag/v1.5.5)
it adds 2,200,338 bytes to the installer size.

This is already a ton of an improvement, as the installer increased by
barely more than 6%.

However, you had mentioned that you could squeeze it a little more? I do
not really know how much this means for the installer size, as we use LZMA
to squeeze it as much as possible, and I *think* that debug symbols are
fairly compressible. So let's test this? Could you provide me with a
still-functional but heavily-stripped git-lfs.exe to play with?

Oh, and for the record: I think we should keep git-lfs.exe out of MinGit,
as the entire idea of MinGit was to strip out *as much* as possible
without compromising core Git functionality. If any application needs
MinGit and Git LFS, it is easy enough to coalesce them.

Ciao,
Johannes

[Taylor Blau]

Hi Johannes,

It was a pleasure meeting you at Git Merge in Brussels. I'm on vacation right

now, but I'll be back in the office on Monday (Feb. 13th). Before then, I'll

respond to a few of the points you raised here and send a more detailed response

next week :-).

> However, you had mentioned that you could squeeze it a little more? I do not

> really know how much this means for the installer size, as we use LZMA to

> squeeze it as much as possible, and I *think* that debug symbols are fairly

> compressible. So let's test this? Could you provide me with a still-functional

> but heavily-stripped git-lfs.exe to play with?

I _think_ I can squeeze it a bit more. Since we don't need debugger symbols in a

production release, I think just omitting them will provide a small benefit over

compressing them. I don't have a ton of experience in tweaking these sorts of

`go build` flags, but I'll spend some time on this next week and see how far we

can get it down.

> This is already a ton of an improvement, as the installer increased by

> barely more than 6%.

If not, I think only increasing the installer 6% is a fair trade-off.

> Oh, and for the record: I think we should keep git-lfs.exe out of MinGit,

> as the entire idea of MinGit was to strip out *as much* as possible

> without compromising core Git functionality.

I agree. If a user wants MinGit _and_ LFS, they can just grab a download from

https://git-lfs.github.com.

One other point we already went over in person is that LFS does not currently

have a schedule release cycle. We typically release point releases every 3-5

weeks or so (with the occasional emergency release). If I recall correctly, you

said that this would work fine for the way that Git for Windows is currently

distributed.

I'm going to CC Sebastian Schuberth (@sschuberth) on this thread, since he's

done some recent work on our Windows installer.

We're really excited about getting Git LFS into Git for Windows. Thanks for

getting a thread open about this, and I'm looking forward to diving into this

more next week and getting back to you.

--

Taylor

[Johannes Schindelin]

Hi Taylor,

On Mon, 6 Feb 2017, Taylor Blau wrote:

> It was a pleasure meeting you at Git Merge in Brussels.

The pleasure was mine!

> > However, you had mentioned that you could squeeze it a little more? I
> > do not really know how much this means for the installer size, as we
> > use LZMA to squeeze it as much as possible, and I *think* that debug
> > symbols are fairly compressible. So let's test this? Could you provide
> > me with a still-functional but heavily-stripped git-lfs.exe to play
> > with?
>

> I _think_ I can squeeze it a bit more. Since we don't need debugger
> symbols in a production release, I think just omitting them will provide
> a small benefit over compressing them. I don't have a ton of experience
> in tweaking these sorts of `go build` flags, but I'll spend some time on
> this next week and see how far we can get it down.

Let me know if you need access to a machine where I can set up a script to
test out final installer sizes quickly.

> > This is already a ton of an improvement, as the installer increased by
> > barely more than 6%.
>

> If not, I think only increasing the installer 6% is a fair trade-off.

Agreed. But let's try first.

> One other point we already went over in person is that LFS does not
> currently have a schedule release cycle. We typically release point
> releases every 3-5 weeks or so (with the occasional emergency release).
> If I recall correctly, you said that this would work fine for the way
> that Git for Windows is currently distributed.

Yes, we already have to deal with several mutually incompatible schedule
releases: Git itself, Git Credential Manager, MSYS2's runtime, OpenSSL and
of course OpenSSH. And all the other packages (if something like
Heartbleed raises its ugly head, I will release an "out of band" version
with the affected component updated and fixed).

My personal rule of thumb is: whenever one of the included components is
available at a newer version, I will always try to update our package
repository (Git for Windows' SDK is essentially a friendly fork of MSYS2,
and they use `Pacman` of Arch Linux to manage packages similar to apt-get,
Homebrew and the likes; Git for Windows has its own Pacman repository
where it hosts MSYS2 and MINGW packages). If the update of the component
fixes a serious bug that could affect users negatively, *and* if I am
aware of it, I will release an "out of band" version.

"Out of band" here means that it does not follow upstream Git's release
cycle. When a new Git version comes out, I will always try to drop
everything else (fun, fun) and release a new Git for Windows version with
the same version [*1*]. When I have to release an official version without
any such corresponding upstream Git release, I will append an (2), (3),
(4), etc.

For example, after Git for Windows v2.11.0 came out, there was no upstream
Git release for over six weeks, and since I had some fixes for bugs
affecting users negatively, I released v2.11.0(2) [*2*]. Since that
version introduced a serious bug that was only detected 30 minutes *after*
publishing it, the next day I released v2.11.0(3). After that, upstream
Git v2.11.1 was released, so I rebased Git for Windows' 71 topic branches,
generated all 9 files for release (~350MB), tested it as usual, and
published Git for Windows v2.11.1. In a room full of laptops with only one
wifi. At GitMerge.

> We're really excited about getting Git LFS into Git for Windows. Thanks
> for getting a thread open about this, and I'm looking forward to diving
> into this more next week and getting back to you.

Yes, I think this will improve the user experience. Thank you for your
help!

Ciao,
Johannes

Footnote *1*: There has been one Git version that broke things *only* on
Windows, and we skipped that version: v2.9.1.

Footnote *2*: I should have released that version much sooner. As it was,
v2.11.0 was out for more than a month with known serious issues. In the
future, expect me to be more liberal with out-of-band releases.

[Johannes Schindelin]

Hi Taylor,

On Mon, 6 Feb 2017, Johannes Schindelin wrote:

> On Mon, 6 Feb 2017, Taylor Blau wrote:
>
> > > However, you had mentioned that you could squeeze it a little more?
> > > I do not really know how much this means for the installer size, as
> > > we use LZMA to squeeze it as much as possible, and I *think* that
> > > debug symbols are fairly compressible. So let's test this? Could you
> > > provide me with a still-functional but heavily-stripped git-lfs.exe
> > > to play with?
> >
> > I _think_ I can squeeze it a bit more. Since we don't need debugger
> > symbols in a production release, I think just omitting them will
> > provide a small benefit over compressing them. I don't have a ton of
> > experience in tweaking these sorts of `go build` flags, but I'll spend
> > some time on this next week and see how far we can get it down.
>
> Let me know if you need access to a machine where I can set up a script
> to test out final installer sizes quickly.

To make things easier, I worked on the scripts a bit and updated them to
bundle Git-LFS 1.5.5 as MINGW packages. They were uploaded to our Pacman
repository, meaning that they can be installed in a Git for Windows SDK
using the command `pacman -Sy mingw-w64-x86_64-git-lfs` (replace `x86_64`
by `i686` if you are using a 32-bit setup).

> > > This is already a ton of an improvement, as the installer increased by
> > > barely more than 6%.
> >
> > If not, I think only increasing the installer 6% is a fair trade-off.
>
> Agreed. But let's try first.

Please let me know how it goes ;-)

Ciao,
Johannes

[Taylor Blau]

Hi Johannes,

I'm back from vacation and have a few updates:

> > Let me know if you need access to a machine where I can set up a script
> > to test out final installer sizes quickly.
>
> To make things easier, I worked on the scripts a bit and updated them to
> bundle Git-LFS 1.5.5 as MINGW packages. They were uploaded to our Pacman
> repository, meaning that they can be installed in a Git for Windows SDK
> using the command `pacman -Sy mingw-w64-x86_64-git-lfs` (replace `x86_64`
> by `i686` if you are using a 32-bit setup).

Thank you :-).

> > > If not, I think only increasing the installer 6% is a fair trade-off.
> >
> > Agreed. But let's try first.
>
> Please let me know how it goes ;-)

I just posted https://github.com/git-lfs/git-lfs/pull/1937 which passes the '-s'
and '-w' ldflags to the Go compiler, and was able to reduce the compressed size
of the binaries (on average) 25.6%. On Windows, in particular, the numbers are:

|               | windows-386 | windows-amd64 |
| ------------- | ----------- | ------------- |
| with DWARF    | 3212684     | 3436845       |
| without DWARF | 2348202     | 2548085       |
|               | -26.91%     | -25.86%       |

These changes should be included in the upcoming v2.0.0 release of Git LFS,
which I am happy to notify you of.

Let me know if this is an acceptable size to include in Git for Windows, and if
so, what the next steps going forward are. I'm happy to help however I can :-).

Thanks,
Taylor

[Taylor Blau]

Hi Johannes,

I'm back from vacation and have a few updates:

> > Let me know if you need access to a machine where I can set up a script
> > to test out final installer sizes quickly.
>
> To make things easier, I worked on the scripts a bit and updated them to
> bundle Git-LFS 1.5.5 as MINGW packages. They were uploaded to our Pacman
> repository, meaning that they can be installed in a Git for Windows SDK
> using the command `pacman -Sy mingw-w64-x86_64-git-lfs` (replace `x86_64`
> by `i686` if you are using a 32-bit setup).

Thank you :-).

> > > If not, I think only increasing the installer 6% is a fair trade-off.
> >
> > Agreed. But let's try first.
>
> Please let me know how it goes ;-)