`bundle package --all` with git repositories

[Nathaniel Talbott]

I love the concept of `bundle package --all`, since it takes the awesomeness of `bundle package` and makes it work magically with git sources as well. But: after trying to adopt it on a good sized project, we had to go back to the bad old days of just `bundle package`.

The problem's pretty simple: `--all` caches the gem directory, in a <name>-<ref> format, and causes ridiculous amounts of repository churn every time the dependency updates. Since git-backed gems are often the *most* likely to change - we use them all the time for private shared code that we change fairly regularly, or to track projects closely that change more frequently than they cut releases - the noise got to the point where we couldn't take it anymore and we had to can it.

I'd like to propose - and offer to implement a first pass of - `bundle package --all` actually building and caching a <name>-<ref>.gem file instead of the unpacked gem directory. Will this work or am I missing something that makes it infeasible? Would the bundler team be amenable to such an approach and a patch to implement it? Is there somewhere else I should be asking these questions?

Thanks,

--

Nathaniel

[André Arko]

I believe last time we looked into this, there were a pretty good number of git gems that aren’t buildable directly from their repos. I’m pretty down with the idea, though. Maybe we just end up packing a tarball and using that? It seems like packed git gems shouldn’t actually require git to be used.

As for “repository churn”, do you mean that it causes a lot of file rename operations in git? IMO, that’s actually much better than packed .gem files, because those are completely binary and git can’t share content between them. :(

--
You received this message because you are subscribed to the Google Groups "ruby-bundler" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-bundler...@googlegroups.com.
To post to this group, send email to ruby-b...@googlegroups.com.
Visit this group at http://groups.google.com/group/ruby-bundler.
For more options, visit https://groups.google.com/d/optout.

[Nathaniel Talbott]

RE some gems not being buildable: I was kind of expecting that, but figure we can find some way to pack them even so since we do know they're *usable* as gems.

RE repository churn, space usage is not a concern for us, but the cognitive noise of having a one-line Gemfile.lock change balloon to hundreds of files "changed" is. We found it was obscuring the actual changes that were happening, which had negative effects on our workflow and team dynamics.

So if you're positively disposed to this, I'll start looking at what it will take to make it happen. Thanks for the feedback!

[André Arko]

Tarballs seems good, it would be nice to be able to pack git gems so they can be used on machines without git.

Why don’t you use a subrepository for your vendored gems? Then the “churn” is literally limited to one line in your git diff, and you don’t get repo bloat on top of that.

Thanks for being willing to work on it!

[Nathaniel Talbott]

Not sure what you mean by subrepository: git has so many ways of including other repos, and I can't seem to find one named that. Maybe you mean subtrees? In general we've avoid strategies like that due to painful experiences with submodules in the past, and since using Bundler to handle it is immediately intuitive to Ruby devs and doesn't require any extra git-fu.

Haven't had a chance to hack on this yet, but it's on my short list.

--
Nathaniel

--
You received this message because you are subscribed to a topic in the Google Groups "ruby-bundler" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ruby-bundler/FRDsg8tdJJY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ruby-bundler...@googlegroups.com.

[te...@recombine.com]

Hi Nathaniel,

I love your idea of Bundler trying to build a `gem_name.gem` file instead of storing the gem directory's source files when doing `bundle package --all` with dependencies from git repos.

I definitely agree that this would significantly cut down on 'noise' in the git diffs caused by changes to outside gems.

I was wondering if you had a chance to start hacking on this yet, or if you had any ideas of where in Bundler's source code might be a good starting point for looking to change this, say if I wanted to give it a shot?

Cheers,

Teddy

[Nathaniel Talbott]

So I did start looking at it, and it was non-trivial, and I got sidetracked. It's still on my list, but feel free to jump in if you're keen. I'll be sure to ping here if/when I get a chance to revisit it to make sure I'm not duplicating anyone's effort.

--
Nathaniel

[Teddy Widom]

Sounds good, thanks for the update!