Berksfile locks in repositories

[Matt Ray]

I'm not sure we need to keep Berksfile.lock files in our repositories.
We can be explicit enough in our Berksfiles for dependencies that
results should be consistent and some cookbooks already have the lock
file in their .gitignore anyway. The Berksfile is there for testing
purposes only.

I propose getting consistent on this and removing Berkfile.locks from
the cookbooks. Anyone disagree with this?

Thanks,
Matt Ray
Cloud Integrations Product Lead :: Chef
512.731.2218 :: ma...@getchef.com
mattray :: GitHub :: IRC :: Twitter

[Jay Pipes]

On 12/20/2013 12:13 PM, Matt Ray wrote:
> I'm not sure we need to keep Berksfile.lock files in our repositories.
> We can be explicit enough in our Berksfiles for dependencies that
> results should be consistent and some cookbooks already have the lock
> file in their .gitignore anyway. The Berksfile is there for testing
> purposes only.
>
> I propose getting consistent on this and removing Berkfile.locks from
> the cookbooks. Anyone disagree with this?

Yes, I disagree with this, for the reasons outlined here:

http://stackoverflow.com/questions/4151495/should-gemfile-lock-be-included-in-gitignore

Best,
-jay

[Matt Ray]

There are a lot of threads in that post, but my takeaway is that we
would keep our Berksfile.lock because it signals to other developers
what was known to work at some point? Most of the cookbooks are not
very specific on their dependencies, so the Berksfile.lock files are
quite variable. The openstack-chef-repo Berksfile is the most
proscriptive for dependencies, but even that is somewhat stale.

I'm fine with leaving them, I just want us to be consistent and
understand why they're there.

Thanks,
Matt Ray
Cloud Integrations Product Lead :: Chef
512.731.2218 :: ma...@getchef.com
mattray :: GitHub :: IRC :: Twitter

> --
> You received this message because you are subscribed to the Google Groups
> "opscode-chef-openstack" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opscode-chef-open...@googlegroups.com.
> To post to this group, send email to
> opscode-che...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opscode-chef-openstack/52B5251B.60503%40gmail.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Darren Birkett]

On 21 December 2013 15:38, Matt Ray <ma...@getchef.com> wrote:
> The openstack-chef-repo Berksfile is the most
> proscriptive for dependencies, but even that is somewhat stale.

How would we propose that this gets updated? Should it be hooked in
to the gating in some way?

It seems to me that the likely reason for it being there at all, at
least at the moment, is as an artifact of running a berks
install/upload, so the version in the repo just happens to contain
commit references for all the cookbooks as they were at the time this
repo was created.

Darren

[Andy McCrae]

Could we not remove the ref: from all the Berksfile.lock in the parent repo - that way the ref will be updated to the latest "head" of the specific repo's. It seems that we're locking in the specific commit reference, which will then become stale. This seems a bit counter-intuitive especially for repositories that we control (e.g. the cookbook-openstack repositories), for external repositories that we have no control over, I can understand locking the reference. 

Andy

[Hugh Saunders]

I am in favor of remove all commit SHAs that relate to stackforge repos from Berksfile.lock. If a stackforge cookbook repo has a bad commit, that should be fixed in the stackforge cookbook repo, not by pointing to an old commit in Berksfile.lock. 

Gating on integration tests would help to guarantee that the heads of the cookbook repos are good, but thats another thread.

--

Hugh Saunders

[Craig Tracey]

I think there is some confusion around Berkshelf here. When we call out our Berkshelf dependencies we are not specifying them by their sha...just their version. The Berksfike.lock is an artifact of a berks run and is used almost as a cache for Berkshelf...so that it doesnt have to re-verify, through introspection, that it has the right version of the cookbook.

We should not be hand-modifying the Berksfile.lock. Having the lock file speeds up build time and provides the consumer of a cookbook a "known good" reference. They are obviously free to update at will.

--

You received this message because you are subscribed to the Google Groups "opscode-chef-openstack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opscode-chef-open...@googlegroups.com.
To post to this group, send email to opscode-che...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/CAAD%2BY3WOuqoqkfHu%3DXoiCnT-5F74eh-7%2B3N1K_9606WaduWJKw%40mail.gmail.com.

[Andy McCrae]

On Friday, 17 January 2014 13:42:54 UTC, Craig Tracey wrote:

I think there is some confusion around Berkshelf here. When we call out our Berkshelf dependencies we are not specifying them by their sha...just their version. The Berksfike.lock is an artifact of a berks run and is used almost as a cache for Berkshelf...so that it doesnt have to re-verify, through introspection, that it has the right version of the cookbook.

We should not be hand-modifying the Berksfile.lock. Having the lock file speeds up build time and provides the consumer of a cookbook a "known good" reference. They are obviously free to update at will.

I haven't seen this happening though. I made a commit earlier today which failed gating because of the "ref:" line in the Berksfile.lock, which pointed to a much older commit - I had to manually edit and remove the "ref:" line, the tests then succeeded as expected. If that is the purpose of the Berksfile.lock file, then should it be removed from all parent repositories so that it is generated on a "per use" basis?

[Hugh Saunders]

On 17 January 2014 13:42, Craig Tracey <craig...@gmail.com> wrote:

We should not be hand-modifying the Berksfile.lock. Having the lock file speeds up build time and provides the consumer of a cookbook a "known good" reference. They are obviously free to update at will.

Problem is that in some of the repos (openstack-chef-repo for example) the Berksfile.lock is not a known good set of cookbooks, its an old set of cookbooks that is not updated regularly (last update ~2 months ago). 

As the Berksfile.lock is not well maintained, perhaps it should be removed as originally suggested?

I'm not too sure what the best solution is, but having to remove Berksfile.lock after each checkout seems like an odd workflow.

--

Hugh Saunders

[Craig Tracey]

So what would have happened in your case is that the CI berkshelf process would have used all of the cached cookbooks it knew about via the Berksfike.lock (with the exception of your ref removal) and then generated a new sha locally for the now-missing cookbook sha. Once it had that it would run the standard gates.

The problem with that is that now the Berksfile.lock on the CI slave is different than what is in git, and the "known good" reference is no longer valid for the next consumer.

This is very likely to be an issue during these periods of heavy churn between the cookbooks... such add where we are now. The correct way to pull in the correct shas is with a 'berks update'. Perhaps one way we could help alleviate this pain is with a script that sets up some precommit hooks... one of which being checking the berks status.

>
> --
> You received this message because you are subscribed to the Google Groups "opscode-chef-openstack" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to opscode-chef-open...@googlegroups.com.
> To post to this group, send email to opscode-che...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/5b2cbc0a-eb9b-4222-b6f1-08c0ae34c60d%40googlegroups.com.

[Craig Tracey]

FWIW I am not making commentary on the usefulness/maintainability of Berksfile.lock just stating that removing the shas invalidates its intended purpose.

--

You received this message because you are subscribed to the Google Groups "opscode-chef-openstack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opscode-chef-open...@googlegroups.com.
To post to this group, send email to opscode-che...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/CAAD%2BY3VzqD_7g3h%2BqXhLdcfXO4PZR7hd0eMPiF0ND6Jh%3DOv8eQ%40mail.gmail.com.

[Andy McCrae]

On Friday, 17 January 2014 14:18:21 UTC, Craig Tracey wrote:

On Jan 17, 2014 8:50 AM, "Andy McCrae" <andy....@gmail.com> wrote:
>
>
>
> On Friday, 17 January 2014 13:42:54 UTC, Craig Tracey wrote:
>>
>> I think there is some confusion around Berkshelf here. When we call out our Berkshelf dependencies we are not specifying them by their sha...just their version. The Berksfike.lock is an artifact of a berks run and is used almost as a cache for Berkshelf...so that it doesnt have to re-verify, through introspection, that it has the right version of the cookbook.
>>
>> We should not be hand-modifying the Berksfile.lock. Having the lock file speeds up build time and provides the consumer of a cookbook a "known good" reference. They are obviously free to update at will.
>
> I haven't seen this happening though. I made a commit earlier today which failed gating because of the "ref:" line in the Berksfile.lock, which pointed to a much older commit - I had to manually edit and remove the "ref:" line, the tests then succeeded as expected. If that is the purpose of the Berksfile.lock file, then should it be removed from all parent repositories so that it is generated on a "per use" basis?

So what would have happened in your case is that the CI berkshelf process would have used all of the cached cookbooks it knew about via the Berksfike.lock (with the exception of your ref removal) and then generated a new sha locally for the now-missing cookbook sha. Once it had that it would run the standard gates.

The problem with that is that now the Berksfile.lock on the CI slave is different than what is in git, and the "known good" reference is no longer valid for the next consumer.

This is very likely to be an issue during these periods of heavy churn between the cookbooks... such add where we are now. The correct way to pull in the correct shas is with a 'berks update'. Perhaps one way we could help alleviate this pain is with a script that sets up some precommit hooks... one of which being checking the berks status.

I actually changed the sha rather than removing the ref: line, but I know what you mean - I'm still not entirely sure I see the benefits of the .lock file - surely all the cookbook-openstack-* should be in a known good state in the repositories - I'm not sure I see the need for the "ref: sha" to be set for these cookbooks, and since this file isn't updated on the majority of commits (and would have to be updated in a lot of other repositories berksfile.lock file!) I'm really not seeing the benefit.

The CI slave bases it off the git repository, and will pull the version down again, so if the ref: line was not present at all, it would just generate a ref: sha on every gate test - in the same way as I would if I were to pull the repository and run berks install. 

If we had a way to automatically update the berksfile.lock, for each repository on every commit, it would be fine, but I think at the moment the current action list for making changes is (as a crude generalisation):

<change code>

rm Berksfile.lock

berks install

Since removing berksfile.lock to get things to work is the first step this is clearly not a suitable solution, and updating/committing a changed version of berksfile.lock on every commit seems a poor solution (since you would also need to update the other cookbooks berksfile.lock).

[Hugh Saunders]

Lets vote! 

https://review.openstack.org/#/c/67478/

--

Hugh Saunders

[Darren Birkett]

I tend to agree with Matt's original suggestion to remove them, for all the reasons discussed.  I understand the point that Jay was making, wrt the fact that it in theory represents a known good state.

At the moment though, until we get some proper CI testing/gating going on, with post-commit actions to update a potentially good lockfile, I'm in favour of removing.

- Darren

[Chris Laco]

Just wanted to chime in with a few thoughts, mostly comparing this process to Gemfile/Gemfile.lock.

First, I think removing the Berksfile.lock from master in all repos is the right thing to do. It has no meaning as master is where developers work.

With that said, I think the stable branches _should_ contain a Berksfile.lock which represents a known good set of versions that have been tested and released to work together from the perspective of someone downloading, berks installing, kitchen-testing that cookbook as well as the parent repo loading all of those cookbooks.

To me, this is just how Gemfile/lock works in practice usually:  http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/

Developers ignore and should exclude .lock files from the repo. But when released, the .lock files should be included, and ensure that the consumer get exactly the same versions that were tested and has a much better chance of success.

Now, in an ideal world, berks would be honoring the entirety of the Berkshelf.lock file, but it doesn't; specifically with dependencies. It just ignores the lockfile contents and does what it wants.

IIRC, this is getting fixed in 3.x. As a workaround, berks _will_ honor versions if they're also specified in Berkshelf itself.

But, we also should have real version numbers/tags in stable release to use in depends statements.

---

From: opscode-che...@googlegroups.com [opscode-che...@googlegroups.com] on behalf of Darren Birkett [darren....@gmail.com]
Sent: Friday, January 17, 2014 8:40 AM
To: Hugh Saunders
Cc: opscode-che...@googlegroups.com
Subject: Re: [chef-openstack] Berksfile locks in repositories

--
You received this message because you are subscribed to the Google Groups "opscode-chef-openstack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opscode-chef-open...@googlegroups.com.
To post to this group, send email to opscode-che...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/CAHX4EnO5aedcVzSSmQJvOVdmVm_Bhgo_9WtChAgA8fo8i8L91w%40mail.gmail.com.

[Darren Birkett]

+1 to lockfiles in stable releases.  Seems like this is where they are actually meant for, and not master/dev branches.

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/41B7689F0255FC48AF2637D2B3B089BE0198FB76%40ORD1EXD04.RACKSPACE.CORP.

[Craig Tracey]

++

I was formulating essentially the same response.

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/41B7689F0255FC48AF2637D2B3B089BE0198FB76%40ORD1EXD04.RACKSPACE.CORP.

[Jay Pipes]

On Fri, 2014-01-17 at 16:48 +0000, Chris Laco wrote:
> Just wanted to chime in with a few thoughts, mostly comparing this
> process to Gemfile/Gemfile.lock.

> First, I think removing the Berksfile.lock from master in all repos is
> the right thing to do. It has no meaning as master is where developers
> work.

> With that said, I think the stable branches _should_ contain a
> Berksfile.lock which represents a known good set of versions that have
> been tested and released to work together from the perspective of
> someone downloading, berks installing, kitchen-testing that cookbook
> as well as the parent repo loading all of those cookbooks.

> To me, this is just how Gemfile/lock works in practice usually:
> http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/
> Developers ignore and should exclude .lock files from the repo. But
> when released, the .lock files should be included, and ensure that the
> consumer get exactly the same versions that were tested and has a much
> better chance of success.

The above makes a ton of sense actually. Well said, Chris. I would
support the above strategy.

Best,
-jay

[Matt Ray]

This sounds like a good plan, we just need to update documentation to
capture the decision. I'll add that as a comment to the reviews that
remove the Berksfile.lock from master to update the README.md for it.

Thanks,
Matt Ray
Cloud Integrations Product Lead :: Chef
512.731.2218 :: ma...@getchef.com
mattray :: GitHub :: IRC :: Twitter

> --
> You received this message because you are subscribed to the Google Groups "opscode-chef-openstack" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to opscode-chef-open...@googlegroups.com.
> To post to this group, send email to opscode-che...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/1389983417.7857.19.camel%40serialcoder.

[John Dewey]

The reason Berksfile.lock was originally included in the repos was due to Berskshelf’s dependency resolver.  We would run into issues where tests would pass locally but not remotely on the CI server.  Berkshelf was not loading the proper cookbooks as specified by the Berksfile.

At the time Berkshelf 2.x was released, and as a work-a-round Jamie suggested committing the Berksfile.lock, which corrected the dependency resolution.  

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/41B7689F0255FC48AF2637D2B3B089BE0198FB76%40ORD1EXD04.RACKSPACE.CORP.