Can't clone from github anymore...

[Doug Davis]

A few days ago I started to see an issue trying to clone one of our repos and I'm a bit confused by the errors I'm seeing.  Keep in mind that I could be doing something wrong, but given our repo/process hasn't changed I'm starting to think there's something on the CF github side of things that isn't quite right.   Perhaps someone on the list could help.

We have a repo that has cf-release as a submodule - right now its pointing to cf-release at v150
Normally, we do a recursive git clone and it pulls in cf-release and all submodules just fine
However, now when we do the clone we see this:

Initialized empty Git repository in /home/user/t/ibm-release/releases/cf-release/src/hm9000/src/github.com/coreos/go-raft/.git/ remote: Counting objects: 1884, done. remote: Compressing objects: 100% (693/693), done. remote: Total 1884 (delta 1191), reused 1884 (delta 1191) Receiving objects: 100% (1884/1884), 543.68 KiB | 305 KiB/s, done. Resolving deltas: 100% (1191/1191), done. fatal: reference is not a tree: 9fed6b9e77a96f1ab3a4c02bc8dbed72fbac9eb0 Unable to checkout '9fed6b9e77a96f1ab3a4c02bc8dbed72fbac9eb0' in submodule path 'src/github.com/coreos/go-raft' Failed to recurse into submodule path 'src/hm9000' Failed to recurse into submodule path 'releases/cf-release'

it looks like its failing trying to clone "go-raft" at the "9fed6b..." level.  Some other things I'm seeing:
- I think this reference between HM and raft has been removed in a newer version (post v150):
   https://github.com/cloudfoundry/hm-workspace/commit/560aab3196a9122a62eb84907ef3b8c634833513
   but given I should be on a fixed/old commit level of cf-release, I'm not sure why this would impact me.  
- I think go-raft has been renamed since:
    https://github.com/coreos/go-raft
   gets routed to:
    https://github.com/coreos/raft
- And in particular,  https://github.com/coreos/go-raft/tree/9fed6b9e77a96f1ab3a4c02bc8dbed72fbac9eb0   gets routed to the "raft" URL just fine

Anyone else seeing this?  And in particular, anyone understand what's going on? and how to fix it?  I still feel like a newbie to git, but I would think that being on an old/static commit level would shield me from future changes like this.  Although, a git repo rename could play havoc with things I guess.

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

[James Bayer]

doug,

i heard something about an issue with one of the commits earlier in the week. i'll ask about this at standup today and we should have an answer in a few hours.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Thank you,

James Bayer

[Christopher Ferris]

Thanks, James!

[James Bayer]

here's an update from aram, the runtime anchor...we still need more information.

I believe this is a known issue caused by the maintainer of coreos/go-raft doing a force-push to their repo which has rewritten history.

I think the decision was to leave our repos as-is rather than re-write our history as well.

I believe Onsi discovered this and he may have more details.

[James Bayer]

alex suraci says this should be fixed on master, which means when we cut v155 that it's fixed. v155 is blocked on finding a cloud controller bug that is pretty impactful to cloud controller stability.

i'm not sure what that means for earlier releases yet.

[Doug Davis]

yea, not sure what this means for those who are pre-155 - like us.  Is there a way to re-write history again so those missing links are valid?

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

James Bayer ---01/30/2014 11:20:09 AM---alex suraci says this should be fixed on master, which means when we cut v155 that it's fixed. v155

[James Bayer]

matthew kocher is going to send a message with more detail.

a bit more to the story is that the final release blobs should have previously released final releases. however, you will have to use those as-is and not be able to easily modify those for changes because coreos/raft has moved a git submodule pointer that used to resolve ok. this is a limitation of having git dependencies with submodules to a git repo that someone else controls.

[Matthew Kocher]

Final releases don't rely on the working tree being up to date, you can still `bosh create release releases/cf-xyz.yml` even though you can't check out the submodule to the right hash.

That submodule was only used in development, our fix for this was just to remove it from the repo.  If you need to make changes on an older branch of cf-release you should be able to remove the submodule as well.

--

Matthew Kocher

Director of Engineering, Cloud Foundry

On Thu, Jan 30, 2014 at 9:50 AM, James Bayer <jba...@gopivotal.com> wrote:

matthew kocher is going to send a message with more detail.

a bit more to the story is that the final release blobs should have previously released final releases. however, you will have to use those as-is and not be able to easily modify those for changes because coreos/raft has moved a git submodule pointer that used to resolve ok. this is a limitation of having git dependencies with submodules to a git repo that someone else controls.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Christopher Ferris]

"this is a limitation of having git dependencies with submodules to a git repo that someone else controls."

indeed - there's a lesson in there somewhere

Chris

[Dr Nic Williams]

If only someone, at some time, across the history of Computer Science had invented some sort of versioned packaging concept for software libraries and then sufficiently documented and demonstrated the idea across many programming languages such that the golang core team could have been sufficiently educated about it :)

[Doug Davis]

But I can't clone the cf-release repo in order to do a bosh create release.

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

Matthew Kocher ---01/30/2014 01:01:30 PM---Final releases don't rely on the working tree being up to date, you can still `bosh create release r

From: Matthew Kocher <mko...@pivotallabs.com>
To: vcap...@cloudfoundry.org,
Date: 01/30/2014 01:01 PM
Subject: Re: [vcap-dev] Can't clone from github anymore...

---

[Dr Nic Williams]

Doug, I was able to clone cf-release. Are you still having issues?

--

Dr Nic Williams

Stark & Wayne LLC - consultancy for Cloud Foundry users

http://drnicwilliams.com

http://starkandwayne.com

cell +1 (415) 860-2185

twitter @drnic

[Doug Davis]

Yes, sort of. I can't recursively checkout my parent repo (which has cf-release as a submodule).  
What I don't understand about git is that a checkout <myrepo> + recursive submodule update fails when I do it on 'master'.  But it works just fine with I do it on a branch of my repo.  Its very confusing to me.
I guess as long as I can modify my stuff to only do a submodule update at one level instead of recursive I'll live - for now.

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

Dr Nic Williams ---01/30/2014 07:03:38 PM---Doug, I was able to clone cf-release. Are you still having issues? On Thu, Jan 30, 2014 at 1:50 PM,

From: Dr Nic Williams <drnicw...@gmail.com>
To: "vcap...@cloudfoundry.org" <vcap...@cloudfoundry.org>,

[Doug Davis]

To reproduce it:
git clone https://github.com/cloudfoundry/cf-release
cf cf-release
git checkout v150
git submodule update --recursive --init

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

Doug Davis---01/30/2014 11:04:22 PM---Yes, sort of. I can't recursively checkout my parent repo (which has cf-release as a submodule).

[Dr Nic Williams]

Doug, if any of the 3rd party repos are screwed, you may need to manually fix the .gitmodules, .git/, etc.

I am unsure of what can be done to fix all old tags.

Perhaps if you can create a script to fix any old tagged version of cf-release then other people would be v appreciative!

On Sat, Feb 1, 2014 at 2:09 PM, Doug Davis <d...@us.ibm.com> wrote:

To reproduce it:
git clone https://github.com/cloudfoundry/cf-release
cf cf-release
git checkout v150
git submodule update --recursive --init


thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

<graycol.gif>

Doug Davis---01/30/2014 11:04:22 PM---Yes, sort of. I can't recursively checkout my parent repo (which has cf-release as a submodule).

From: Doug Davis/Raleigh/IBM@IBMUS
To: vcap...@cloudfoundry.org,
Date: 01/30/2014 11:04 PM
Subject: Re: [vcap-dev] Can't clone from github anymore...

---

Yes, sort of. I can't recursively checkout my parent repo (which has cf-release as a submodule).  
What I don't understand about git is that a checkout <myrepo> + recursive submodule update fails when I do it on 'master'.  But it works just fine with I do it on a branch of my repo.  Its very confusing to me.
I guess as long as I can modify my stuff to only do a submodule update at one level instead of recursive I'll live - for now.

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

<graycol.gif>

Dr Nic Williams ---01/30/2014 07:03:38 PM---Doug, I was able to clone cf-release. Are you still having issues? On Thu, Jan 30, 2014 at 1:50 PM,

From: Dr Nic Williams <drnicw...@gmail.com>
To: "vcap...@cloudfoundry.org" <vcap...@cloudfoundry.org>,
Date: 01/30/2014 07:03 PM
Subject: Re: [vcap-dev] Can't clone from github anymore...

---

Doug, I was able to clone cf-release. Are you still having issues?


On Thu, Jan 30, 2014 at 1:50 PM, Doug Davis <d...@us.ibm.com> wrote:

But I can't clone the cf-release repo in order to do a bosh create release. 



thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

<graycol.gif>

[Doug Davis]

While I'm sure its possible, its not trivial when you're starting two levels (or more) above the problem submodule.  It means having to do one level at a time until you reach it.  Blech!  :-)

Are there plans in place to avoid this type of situation in the future?

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

"Dr Nic Williams" ---02/01/2014 05:12:53 PM---Doug, if any of the 3rd party repos are screwed, you may need to manually fix the .gitmodules, .git/

[James Bayer]

none that have been put into motion yet because it involves moving away from submodules to 3rd party repos and that isn't trivial.

my understanding of the situation is that you can still upload the final releases to a bosh director because the blobs are referenced in the release. however, this doesn't cover the use case of making modifications to an older final release before using it sadly.

Thank you,

James Bayer

[Doug Davis]

Yep - which is exactly our situation.   :-(

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

James Bayer ---02/02/2014 02:56:27 AM---none that have been put into motion yet because it involves moving away from submodules to 3rd party