Can't clone from github anymore...

175 views
Skip to first unread message

Doug Davis

unread,
Jan 30, 2014, 10:22:02 AM1/30/14
to vcap...@cloudfoundry.org

A few days ago I started to see an issue trying to clone one of our repos and I'm a bit confused by the errors I'm seeing.  Keep in mind that I could be doing something wrong, but given our repo/process hasn't changed I'm starting to think there's something on the CF github side of things that isn't quite right.   Perhaps someone on the list could help.

We have a repo that has cf-release as a submodule - right now its pointing to cf-release at v150
Normally, we do a recursive git clone and it pulls in cf-release and all submodules just fine
However, now when we do the clone we see this:

    Initialized empty Git repository in /home/user/t/ibm-release/releases/cf-release/src/hm9000/src/github.com/coreos/go-raft/.git/
    remote: Counting objects: 1884, done.
    remote: Compressing objects: 100% (693/693), done.
    remote: Total 1884 (delta 1191), reused 1884 (delta 1191)
    Receiving objects: 100% (1884/1884), 543.68 KiB | 305 KiB/s, done.
    Resolving deltas: 100% (1191/1191), done.
    fatal: reference is not a tree: 9fed6b9e77a96f1ab3a4c02bc8dbed72fbac9eb0
    Unable to checkout '9fed6b9e77a96f1ab3a4c02bc8dbed72fbac9eb0' in submodule path 'src/github.com/coreos/go-raft'
    Failed to recurse into submodule path 'src/hm9000'
    Failed to recurse into submodule path 'releases/cf-release'

it looks like its failing trying to clone "go-raft" at the "9fed6b..." level.  Some other things I'm seeing:
- I think this reference between HM and raft has been removed in a newer version (post v150):
   https://github.com/cloudfoundry/hm-workspace/commit/560aab3196a9122a62eb84907ef3b8c634833513
   but given I should be on a fixed/old commit level of cf-release, I'm not sure why this would impact me.  
- I think go-raft has been renamed since:
    https://github.com/coreos/go-raft
   gets routed to:
    https://github.com/coreos/raft
- And in particular,  https://github.com/coreos/go-raft/tree/9fed6b9e77a96f1ab3a4c02bc8dbed72fbac9eb0   gets routed to the "raft" URL just fine

Anyone else seeing this?  And in particular, anyone understand what's going on? and how to fix it?  I still feel like a newbie to git, but I would think that being on an old/static commit level would shield me from future changes like this.  Although, a git repo rename could play havoc with things I guess.

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

James Bayer

unread,
Jan 30, 2014, 10:46:02 AM1/30/14
to vcap...@cloudfoundry.org
doug,

i heard something about an issue with one of the commits earlier in the week. i'll ask about this at standup today and we should have an answer in a few hours.


To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.



--
Thank you,

James Bayer

Christopher Ferris

unread,
Jan 30, 2014, 11:14:11 AM1/30/14
to vcap...@cloudfoundry.org
Thanks, James!

James Bayer

unread,
Jan 30, 2014, 11:18:29 AM1/30/14
to vcap...@cloudfoundry.org
here's an update from aram, the runtime anchor...we still need more information.

I believe this is a known issue caused by the maintainer of coreos/go-raft doing a force-push to their repo which has rewritten history.

I think the decision was to leave our repos as-is rather than re-write our history as well.

I believe Onsi discovered this and he may have more details.

James Bayer

unread,
Jan 30, 2014, 11:20:01 AM1/30/14
to vcap...@cloudfoundry.org
alex suraci says this should be fixed on master, which means when we cut v155 that it's fixed. v155 is blocked on finding a cloud controller bug that is pretty impactful to cloud controller stability.

i'm not sure what that means for earlier releases yet.

Doug Davis

unread,
Jan 30, 2014, 11:32:17 AM1/30/14
to vcap...@cloudfoundry.org

yea, not sure what this means for those who are pre-155 - like us.  Is there a way to re-write history again so those missing links are valid?



thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.



Inactive hide details for James Bayer ---01/30/2014 11:20:09 AM---alex suraci says this should be fixed on master, which means James Bayer ---01/30/2014 11:20:09 AM---alex suraci says this should be fixed on master, which means when we cut v155 that it's fixed. v155

James Bayer

unread,
Jan 30, 2014, 12:50:52 PM1/30/14
to vcap...@cloudfoundry.org
matthew kocher is going to send a message with more detail.

a bit more to the story is that the final release blobs should have previously released final releases. however, you will have to use those as-is and not be able to easily modify those for changes because coreos/raft has moved a git submodule pointer that used to resolve ok. this is a limitation of having git dependencies with submodules to a git repo that someone else controls.

Matthew Kocher

unread,
Jan 30, 2014, 1:00:59 PM1/30/14
to vcap...@cloudfoundry.org
Final releases don't rely on the working tree being up to date, you can still `bosh create release releases/cf-xyz.yml` even though you can't check out the submodule to the right hash.

That submodule was only used in development, our fix for this was just to remove it from the repo.  If you need to make changes on an older branch of cf-release you should be able to remove the submodule as well.

--
Matthew Kocher
Director of Engineering, Cloud Foundry


On Thu, Jan 30, 2014 at 9:50 AM, James Bayer <jba...@gopivotal.com> wrote:
matthew kocher is going to send a message with more detail.

a bit more to the story is that the final release blobs should have previously released final releases. however, you will have to use those as-is and not be able to easily modify those for changes because coreos/raft has moved a git submodule pointer that used to resolve ok. this is a limitation of having git dependencies with submodules to a git repo that someone else controls.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

Christopher Ferris

unread,
Jan 30, 2014, 2:11:42 PM1/30/14
to vcap...@cloudfoundry.org
"this is a limitation of having git dependencies with submodules to a git repo that someone else controls."

indeed - there's a lesson in there somewhere

Chris

Dr Nic Williams

unread,
Jan 30, 2014, 2:16:50 PM1/30/14
to vcap...@cloudfoundry.org
If only someone, at some time, across the history of Computer Science had invented some sort of versioned packaging concept for software libraries and then sufficiently documented and demonstrated the idea across many programming languages such that the golang core team could have been sufficiently educated about it :)

Doug Davis

unread,
Jan 30, 2014, 4:50:12 PM1/30/14
to vcap...@cloudfoundry.org

But I can't clone the cf-release repo in order to do a bosh create release.



thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.



Inactive hide details for Matthew Kocher ---01/30/2014 01:01:30 PM---Final releases don't rely on the working tree being up to Matthew Kocher ---01/30/2014 01:01:30 PM---Final releases don't rely on the working tree being up to date, you can still `bosh create release r



From: Matthew Kocher <mko...@pivotallabs.com>
To: vcap...@cloudfoundry.org,
Date: 01/30/2014 01:01 PM
Subject: Re: [vcap-dev] Can't clone from github anymore...





Dr Nic Williams

unread,
Jan 30, 2014, 7:03:07 PM1/30/14
to vcap...@cloudfoundry.org
Doug, I was able to clone cf-release. Are you still having issues?
--
Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
twitter @drnic
graycol.gif

Doug Davis

unread,
Jan 30, 2014, 11:04:05 PM1/30/14
to vcap...@cloudfoundry.org

Yes, sort of. I can't recursively checkout my parent repo (which has cf-release as a submodule).  
What I don't understand about git is that a checkout <myrepo> + recursive submodule update fails when I do it on 'master'.  But it works just fine with I do it on a branch of my repo.  Its very confusing to me.
I guess as long as I can modify my stuff to only do a submodule update at one level instead of recursive I'll live - for now.


thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.


Inactive hide details for Dr Nic Williams ---01/30/2014 07:03:38 PM---Doug, I was able to clone cf-release. Are you still havinDr Nic Williams ---01/30/2014 07:03:38 PM---Doug, I was able to clone cf-release. Are you still having issues? On Thu, Jan 30, 2014 at 1:50 PM,

Doug Davis

unread,
Feb 1, 2014, 5:06:22 PM2/1/14
to vcap...@cloudfoundry.org

To reproduce it:
git clone https://github.com/cloudfoundry/cf-release
cf cf-release
git checkout v150
git submodule update --recursive --init




thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.


Inactive hide details for Doug Davis---01/30/2014 11:04:22 PM---Yes, sort of. I can't recursively checkout my parent repo (whicDoug Davis---01/30/2014 11:04:22 PM---Yes, sort of. I can't recursively checkout my parent repo (which has cf-release as a submodule).

Dr Nic Williams

unread,
Feb 1, 2014, 5:12:44 PM2/1/14
to vcap...@cloudfoundry.org, vcap...@cloudfoundry.org
Doug, if any of the 3rd party repos are screwed, you may need to manually fix the .gitmodules, .git/, etc.

I am unsure of what can be done to fix all old tags.

Perhaps if you can create a script to fix any old tagged version of cf-release then other people would be v appreciative!




On Sat, Feb 1, 2014 at 2:09 PM, Doug Davis <d...@us.ibm.com> wrote:

To reproduce it:
git clone https://github.com/cloudfoundry/cf-release
cf cf-release
git checkout v150
git submodule update --recursive --init


thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.


<graycol.gif>
Doug Davis---01/30/2014 11:04:22 PM---Yes, sort of. I can't recursively checkout my parent repo (which has cf-release as a submodule).

From: Doug Davis/Raleigh/IBM@IBMUS
To: vcap...@cloudfoundry.org,
Date: 01/30/2014 11:04 PM
Subject: Re: [vcap-dev] Can't clone from github anymore...




Yes, sort of. I can't recursively checkout my parent repo (which has cf-release as a submodule).  
What I don't understand about git is that a checkout <myrepo> + recursive submodule update fails when I do it on 'master'.  But it works just fine with I do it on a branch of my repo.  Its very confusing to me.
I guess as long as I can modify my stuff to only do a submodule update at one level instead of recursive I'll live - for now.

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.


<graycol.gif>
Dr Nic Williams ---01/30/2014 07:03:38 PM---Doug, I was able to clone cf-release. Are you still having issues? On Thu, Jan 30, 2014 at 1:50 PM,

From:
Dr Nic Williams <drnicw...@gmail.com>
To:
"vcap...@cloudfoundry.org" <vcap...@cloudfoundry.org>,
Date:
01/30/2014 07:03 PM
Subject:
Re: [vcap-dev] Can't clone from github anymore...




Doug, I was able to clone cf-release. Are you still having issues?


On Thu, Jan 30, 2014 at 1:50 PM, Doug Davis <
d...@us.ibm.com> wrote:
    But I can't clone the cf-release repo in order to do a bosh create release. 



    thanks
    -Doug
    ________________________________________________________
    STSM |  Standards Architect  |  IBM Software Group

    (919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
    The more I'm around some people, the more I like my dog.



    <graycol.gif>

Doug Davis

unread,
Feb 1, 2014, 10:39:49 PM2/1/14
to vcap...@cloudfoundry.org

While I'm sure its possible, its not trivial when you're starting two levels (or more) above the problem submodule.  It means having to do one level at a time until you reach it.  Blech!  :-)

Are there plans in place to avoid this type of situation in the future?



thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.



Inactive hide details for "Dr Nic Williams" ---02/01/2014 05:12:53 PM---Doug, if any of the 3rd party repos are screwed, you ma"Dr Nic Williams" ---02/01/2014 05:12:53 PM---Doug, if any of the 3rd party repos are screwed, you may need to manually fix the .gitmodules, .git/

James Bayer

unread,
Feb 2, 2014, 2:56:14 AM2/2/14
to vcap...@cloudfoundry.org
none that have been put into motion yet because it involves moving away from submodules to 3rd party repos and that isn't trivial.

my understanding of the situation is that you can still upload the final releases to a bosh director because the blobs are referenced in the release. however, this doesn't cover the use case of making modifications to an older final release before using it sadly.
Thank you,

James Bayer
graycol.gif

Doug Davis

unread,
Feb 2, 2014, 8:49:16 AM2/2/14
to vcap...@cloudfoundry.org

Yep - which is exactly our situation.   :-(



thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.



Inactive hide details for James Bayer ---02/02/2014 02:56:27 AM---none that have been put into motion yet because it involves mJames Bayer ---02/02/2014 02:56:27 AM---none that have been put into motion yet because it involves moving away from submodules to 3rd party

Reply all
Reply to author
Forward
0 new messages