broken submodule Git, again

[Bob Carpenter]

I am sooo frustrated. Every time I try to play with submodules, I
wind up breaking my build and blocking myself from doing any further
work. Waaaah!!!


Step 0. everything working fine locally, not up to date with origin


Step 1. Update cmdstan -- no problem:

~/cmdstan(develop)$ git pull

....


Step 2. Try to update cmdstan/stan --- FAIL!

~/cmdstan/stan(feature/issue-1057-generalize-indexing)$ git pull origin develop
From https://github.com/stan-dev/stan
* branch develop -> FETCH_HEAD
warning: Failed to merge submodule lib/stan_math (commits don't follow merge-base)
Auto-merging lib/stan_math
CONFLICT (submodule): Merge conflict in lib/stan_math
Automatic merge failed; fix conflicts and then commit the result.

~/cmdstan/stan(feature/issue-1057-generalize-indexing|MERGING)$ git status
# On branch feature/issue-1057-generalize-indexing
# You have unmerged paths.
# (fix conflicts and run "git commit")
#
# Changes to be committed:
#
# modified: README.md
# modified: src/docs/bibtex/all.bib
# modified: src/docs/stan-reference/acknowledgements.tex
# modified: src/docs/stan-reference/distributions.tex
# modified: src/doxygen/stanlogo-main.png
#
# Unmerged paths:
# (use "git add <file>..." to mark resolution)
#
# both modified: lib/stan_math
#
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: src/stan/model/indexing/lvalue.hpp
# modified: src/stan/model/indexing/rvalue.hpp
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# src/stan/model/indexing/deep_copy.hpp
# src/stan/model/indexing/rvalue_at.hpp
# src/stan/model/indexing/rvalue_index_size.hpp
# src/test/unit/model/indexing/deep_copy_test.cpp

At this point, trying to update fails miserably:

~/cmdstan/stan(feature/issue-1057-generalize-indexing|MERGING)$ make math-update
git submodule init
git submodule update --recursive
Skipping unmerged submodule lib/stan_math


Help. I'm again blocked from doing any work anywhere.

- Bob

[Daniel Lee]

That's bad. Would you be ok nuking the math library? If so:

rm -rf lib/stan_math
git checkout -- lib
make math-revert

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Michael Betancourt]

make-math update doesn’t merge — you have to do that with
a commit. Why isn’t stan_math up to date with develop?

[Bob Carpenter]

No luck.

~/cmdstan/stan(feature/issue-1057-generalize-indexing|MERGING)$ rm -rf lib/stan_math/

~/cmdstan/stan(feature/issue-1057-generalize-indexing|MERGING)$ git checkout -- lib
error: path 'lib/stan_math' is unmerged

~/cmdstan/stan(feature/issue-1057-generalize-indexing|MERGING)$ make math-revert
git submodule update --init --recursive
Skipping unmerged submodule lib/stan_math

- Bob

[Bob Carpenter]

If I knew the answers to any of these questions, I probably
wouldn't get myself into this fix every time I try to update.
I'm just fundamentally confused as to why it keeps breaking.

I need some step-by-step instructions that will work to update
submodules without borking my whole repo.

Why when I update stan does it break the math sub-repo rather
than updating it? Why can't I get rid of it now and update?
No clue.

- Bob

[Michael Betancourt]

There’s some missing info here:

From what repository are you editing files and trying to update?
For example, are you checking out a fresh copy of cmdstan,
cd’ing not stan/lib/stan_math, and then editing the math library?

[Bob Carpenter]

Part of the problem is that I don't know how to talk about what I
did. I'm happy to monkey-type things in that you ask me to.

As far as I know, there were no changes in stan/lib/stan_math.
The way I'm trying to work, I check out cmdstan, then whatever (can't
recall) to check out stan within it, and so on down to lib/stan_math.
Then I checkout a new branch in cmdstan/stan and work on that.
Again, ***without touching cmdstan or lib/stan_math***.

But every time I try to update, all hell breaks
loose and I get stuck. Probably because I just do git pull
reflexively rather than using the make targets.

- Bob

[Bob Carpenter]

I'm still completely blocked.

Now that I nuked lib/stan_math, is there any way to force it to put
the right one back? Is it then OK to merge so I don't get the annoying
MERGING thing on my command line? Or can I just ignore it? I'm really
confused as you can see.

The other problem is I'm afraid to touch any of this stuff I don't
understand for fear of messing up the whole repo commit chain. I'd commit
to stan-dev/stan and just back everything else out, but it won't let me
until I fix the merge.

- Bob

[Michael Betancourt]

Here’s my best guess of what happened

1) You checked out a fresh copy of cmdstan. This includes tagged versions
of each of the submodules

cmdstan (tag1)
cmdstan/stan (tag1)
cmdstan/stan/lib/stan_math (tag1)

2) You then move down into stan and make edits to files

cmdstan/stan (tag2) <— files modified
cmdstan/stan/lib/stan_math (tag1) <— no changes!

3) In the meantime updates to the math library are merged into stan develop,

cmdstan/stan (develop) <— updated from tag1 to point to the new updated math library
cmdstan/stan/lib/stan_math (develop) <— modified from tag1

Now when you try to merge stan develop into your local branch, git thinks

- Any differences in stan between tag2 and develop? Yes, the stan math submodule is modified.
Treat submodule differences as conflicts and stop the automatically merge.

At which point you have to manually update the math library,

cd lib/stan_math
git pull —ff
cd ../..
git pull —ff

In other word, fast forward merges (git pull -ff) aren’t recursive so if there are
conflicts in the submodules then you’ll have to fix them yourself.

I think.

[Michael Betancourt]

Try this:

1. Reset lib/stan_math to what it was before you nuked it,

// assuming that you’re in stan
> git checkout HEAD — lib/stan_math

2. See if you still have the merge conflict between the version of stan_math
that develop points to and the version that you point to,

> git status

3. If so, then update stan_math,

> cd lib/stan_math
> git pull -ff
// should be no merge conflicts

4. Now go back and see if there’s still a conflict,

> cd ../..
> git status
// should be no merge conflicts

5. Resolve the original merge conflict

> git commit

6. Have a drink.

[Daniel Lee]

I want you unblocked asap, but might have to wait until we're sitting face to face tomorrow.

One thing I noticed was a trailing slash when you removed the math lib. That doesn't remove the whole directory. (I don't know if I suggested a trailing slash, but there shouldn't be one.)

I'll be in tomorrow probably before you get in. Should we make shell scripts / make targets to do this management?

[Bob Carpenter]

You're spot on with (1) and (2). Then I tried to just
do a "git pull origin". I know "ff" means "fast forward",
but I have no idea what that means in git terms --- people
keep trying to explain it and it just won't stick.

I really have no sense of direction. All this forward
and back and relative stuff really really really gets me
lost.

And now I can't do anything because I followed Daniel's advice
to remove stan/lib/stan_math and now I can't figure out how to
get it back. Nothing I try works because of the merge conflicts.

And I don't understand why if I don't touch the library, it
doesn't get updated. Isn't that the whole point?

What's the downside to not having them be submodules? All
it's giving me is grief. Is the upside that you're supposed
to mark from the higher level repo where you're at in the lower
level one? I just can't get that to work properly.

- Bob

[Daniel Lee]

Bob, I see the problem you're seeing. I'm trying to figure out what's going on your branch... something really weird is happening and I don't know what it is yet.

In the meantime, let's get you back before trying to merge develop. Do this to get out of the merge hell that you're in:

> git merge --abort

I'll follow up with instructions on what to do.

Daniel

P.S. I didn't realize the trailing slash doesn't matter. I've confirmed: it doesn't matter.

[Daniel Lee]

Bob, can you push your branch wherever it is? I'll work through the merge hell. I don't really know how this is happening, but I see it too. I can figure this out, I think.

Digging into it:
- Part of the problem is that the lib/stan_math hash that you have on your branch points to nothing on the GitHub repo. If you look at the branch on GitHub (https://github.com/stan-dev/stan/tree/feature/issue-1057-generalize-indexing/lib) you can see "stan_math @ 699d1e1".

- I figured out one way to get around it:

  1. move the math submodule to the current version of develop. From cmdstan: cd stan/lib/stan_math; git checkout develop; git pull --ff

  2. merge develop: git merge develop ("git pull origin develop" causes all hell to break loose)

Daniel

[Bob Carpenter]

Thanks! That was like a magic wand. The update was totally
non-critical --- what I'm doing shouldn't touch anything else,
so no rush to get it up to date. I just update out of habit
when my side's stable because I'd rather catch conflicts earlier
than later.

- Bob

[Daniel Lee]

Did it work?!?!?

[Bob Carpenter]

Luckily, it shouldn't be merge hell in the usual sense --- I
think it's all just pointers. I haven't actually changed anything
in the Math lib.

Now that I'm back to an OK state, I think I only need to know how
I am supposed to update the submodule.

- Bob

[Daniel Lee]

(I interpreted your message as one of success, but just wanted to make sure you were actually in a state where you can move forward.)

I don't know the difference between "git pull origin develop" and "git fetch; git merge develop" -- I'm on stackoverflow now reading, but it looks like it should be the same thing. I'm obviously missing something.

[Bob Carpenter]

Oh, and I pushed where I'm at on my branch, even though I know it'll
break the build, but now I'm even more confused on stan_math:

~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$

It says it's on the same branch as stan? But I never created a
branch with that name in stan_math!

At the higher levels, everything's clean:

~/cmdstan/stan(feature/issue-1057-generalize-indexing)$ git status
# On branch feature/issue-1057-generalize-indexing
nothing to commit, working directory clean


And then cmdstan is where I expect, because I've added new commits
to stan under it.

~/cmdstan(develop)$ git status
# On branch develop
# Your branch is ahead of 'origin/develop' by 2 commits.
# (use "git push" to publish your local commits)

#
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#

# modified: stan (new commits)
#
no changes added to commit (use "git add" and/or "git commit -a")



- Bob

[Bob Carpenter]

Yeah, I can work now. No idea how to get my math back to where
it should be, but I don't need a newer version until I want to
merge everything.

I thought "git pull origin develop" just pulled the most recent
version of develop from the origin and merged it. Running "git merge develop"
after running the above never does anything.

Thanks a ton for helping out. I'm really OK to work now
and won't try to touch the math lib.

- Bob

[Daniel Lee]

On Thu, Oct 22, 2015 at 9:36 PM, Bob Carpenter <ca...@alias-i.com> wrote:

Oh, and I pushed where I'm at on my branch, even though I know it'll
break the build, but now I'm even more confused on stan_math:

~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$

It says it's on the same branch as stan?  But I never created a
branch with that name in stan_math!

Ah... that's what your submodule is pointing at. It doesn't exist on the GitHub repo, so that's what's causing issues. My guess is that you have a local branch that you named that, maybe you accidentally created the branch on math instead of stan?

Anyway, I know how to get you out of this bind. From ~/cmdstan/stan/lib/stan_math:

$ git checkout develop

$ git pull --ff

That's going to get you up to date with develop.

$ cd ../../

$ git add lib/stan_math

$ git commit -m "Updating Math library to develop."

That'll add the commit for the submodule.

$ git push

That *should* get you to where you need to be.

[Bob Carpenter]

My love for submodules is not growing.

Still totally messed up. Watch below. Checking out
develop or generalize-indexing from either stan or
stan_math changes the other. WTF? Is my terminal just
confused?


~/cmdstan/stan(develop)$ cd lib/stan_math/

~/cmdstan/stan/lib/stan_math(develop)$ cd ../..

~/cmdstan/stan(develop)$ git checkout feature/issue-1057-generalize-indexing
Switched to branch 'feature/issue-1057-generalize-indexing'

~/cmdstan/stan(feature/issue-1057-generalize-indexing)$ git status
# On branch feature/issue-1057-generalize-indexing
nothing to commit, working directory clean

~/cmdstan/stan(feature/issue-1057-generalize-indexing)$ cd lib/stan_math/

~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$

~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$ git checkout develop
Switched to branch 'develop'
Your branch is ahead of 'origin/develop' by 4 commits.

(use "git push" to publish your local commits)

~/cmdstan/stan/lib/stan_math(develop)$ cd ../../

~/cmdstan/stan(develop)$

- Bob

[Daniel Lee]

I wonder if you have some merge bubbles in there. Since you're lib/stan_math is already nuked, do this to just get yourself up to the latest develop state:

cd ~/cmdstan/stan/lib/stan_math
git reset --hard origin/develop

After that last line, you should see something like:
HEAD is now at 4cb436e Merge pull request #181 from stan-dev/feature/issue-180-clean_up_ode

And getting the status now should be ok:
$ git status
On branch develop
Your branch is up-to-date with 'origin/develop'.

nothing to commit, working directory clean

I don't know how to diff history well enough to tell where those 4 commits are coming from.

[Bob Carpenter]

No luck. My indexing branch is stubborn about
wanting to take stan_math along with it. Is it possible
this is just a mess in the terminal package displaying?
Is there another way to check which submodule I have? I
thought you showed me a (hidden?) file somewhere.


~/cmdstan/stan/lib/stan_math(develop)$ git reset --hard origin/develop
HEAD is now at 2d57e0e Merge pull request #1578 from stan-dev/feature/issue-1576-better-doc-for-Wiener-distribution

~/cmdstan/stan/lib/stan_math(develop)$

~/cmdstan/stan/lib/stan_math(develop)$ cd ../../

~/cmdstan/stan(develop)$ git checkout feature/issue-1057-generalize-indexing
Switched to branch 'feature/issue-1057-generalize-indexing'

~/cmdstan/stan(feature/issue-1057-generalize-indexing)$ cd lib/stan_math/

~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$


At this point, I think it'd be safe just to completely check out
everything from scratch. It seems totally amateurish, but that's
me when it comes to Git.

- Bob

[Daniel Lee]

I thought that would have reset it to the upstream origin/develop, but clearly I was wrong. It looks like your branch is still behind.

stackoverflow (http://stackoverflow.com/questions/1628088/reset-local-repository-branch-to-be-just-like-remote-repository-head) is telling me I should have told you this:

cd ~/cmdstan/stan/lib/stanmath

git fetch origin

git reset --hard origin/develop

And the hidden file:

$ cd ~/cmdstan/stan

$ cat .gitmodules
[submodule "lib/stan_math"]
    path = lib/stan_math
    url = https://github.com/stan-dev/math.git
$ git submodule status

(the .gitmodules file doesn't really say much)

[Bob Carpenter]

Thanks a lot. Your help is the only thing keeping me from
jumping up and down on my computer (not that it's my computer's
fault).

But alas, no luck:

~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$ git fetch origin

~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$ git reset --hard origin/develop

HEAD is now at 2d57e0e Merge pull request #1578 from stan-dev/feature/issue-1576-better-doc-for-Wiener-distribution

~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$


So then I thought I'd try this:

~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$ git checkout develop
Switched to branch 'develop'

~/cmdstan/stan/lib/stan_math(develop)$ git fetch origin

~/cmdstan/stan/lib/stan_math(develop)$ git reset --hard origin/develop
HEAD is now at 2d57e0e Merge pull request #1578 from stan-dev/feature/issue-1576-better-doc-for-Wiener-distribution

~/cmdstan/stan/lib/stan_math(develop)$

So far, so good. But then boom, it pulls along stan:

~/cmdstan/stan/lib/stan_math(develop)$ cd ../../

~/cmdstan/stan(develop)$


~/cmdstan/stan(develop)$ cat .gitmodules

[submodule "lib/stan_math"]
path = lib/stan_math
url = https://github.com/stan-dev/math.git

Yup, doesn't say much. But wasn't there a file that had the
commit hash or whatever it's called for the submodule? I'd like to
test if it's just the terminal messing me up.

And now I'm starting to really freak out, because it looks like
my issue no longer has any of the indexing code in it.

ls src/stan/model/

~/cmdstan/stan(feature/issue-1057-generalize-indexing)$ ls src/stan/model/
model_header.hpp prob_grad.hpp util.hpp
~/cmdstan/stan(feature/issue-1057-generalize-indexing)$



I had just changed a whole bunch of it, as in at least thirty or forty
hours of work, before trying to push.

I'm going to sit in the corner and cry now.

- Bob

[Bob Carpenter]

OK, it looks like my last commit and push was taken by Git,
because when I look at the feature/issue-1057-generalize-indexing
on the GitHub web page, everything's there. When I look
at it after this

> ~/cmdstan/stan/lib/stan_math(feature/issue-1057-generalize-indexing)$ git fetch origin

nothing's there.

- Bob

[Daniel Lee]

ok... then I think you need to do a submodule update

> cd ~/cmdstan/stan

> git submodule update --init --recursive

[Bob Carpenter]

Hey, that put me back in confusing number land.

No idea why all the work on my branch I can see on the GitHub
page isn't checked out into my version.

Sooo, I thought I'd do just a simple pull, and wham bam, back
to merge conflict.

~/cmdstan/stan(feature/issue-1057-generalize-indexing)$ git submodule update --init --recursive
Submodule path 'lib/stan_math': checked out '4cb436e76820535fb1ea3be3868212be6b28f6e2'

~/cmdstan/stan(feature/issue-1057-generalize-indexing)$ cd lib/stan_math/

~/cmdstan/stan/lib/stan_math((4cb436e...))$ cd ..

~/cmdstan/stan/lib(feature/issue-1057-generalize-indexing)$ cd ..

~/cmdstan/stan(feature/issue-1057-generalize-indexing)$ git pull

warning: Failed to merge submodule lib/stan_math (commits don't follow merge-base)
Auto-merging lib/stan_math
CONFLICT (submodule): Merge conflict in lib/stan_math
Automatic merge failed; fix conflicts and then commit the result.

~/cmdstan/stan(feature/issue-1057-generalize-indexing|MERGING)$


~/cmdstan/stan(feature/issue-1057-generalize-indexing|MERGING)$ git status
# On branch feature/issue-1057-generalize-indexing

# Your branch and 'origin/feature/issue-1057-generalize-indexing' have diverged,
# and have 19 and 44 different commits each, respectively.
# (use "git pull" to merge the remote branch into yours)
#

# You have unmerged paths.
# (fix conflicts and run "git commit")
#
# Changes to be committed:
#

# modified: src/docs/stan-reference/language.tex
# modified: src/docs/stan-reference/programming.tex
# modified: src/stan/lang/ast.hpp
# modified: src/stan/lang/ast_def.cpp
# modified: src/stan/lang/generator.hpp
# modified: src/stan/lang/grammars/expression_grammar.hpp
# modified: src/stan/lang/grammars/expression_grammar_def.hpp
# new file: src/stan/lang/grammars/indexes_grammar.hpp
# new file: src/stan/lang/grammars/indexes_grammar_def.hpp
# new file: src/stan/lang/grammars/indexes_grammar_inst.cpp
# modified: src/stan/lang/grammars/statement_grammar.hpp
# modified: src/stan/lang/grammars/statement_grammar_def.hpp
# modified: src/stan/lang/grammars/term_grammar.hpp
# modified: src/stan/lang/grammars/term_grammar_def.hpp
# modified: src/stan/lang/grammars/var_decls_grammar_def.hpp
# new file: src/stan/meta/matrix.hpp
# new file: src/stan/model/indexing.hpp
# new file: src/stan/model/indexing/deep_copy.hpp
# new file: src/stan/model/indexing/index.hpp
# new file: src/stan/model/indexing/index_list.hpp
# new file: src/stan/model/indexing/lvalue.hpp
# new file: src/stan/model/indexing/rvalue.hpp
# new file: src/stan/model/indexing/rvalue_at.hpp
# new file: src/stan/model/indexing/rvalue_index_size.hpp
# new file: src/stan/model/indexing/rvalue_return.hpp
# modified: src/stan/model/model_header.hpp
# modified: src/test/unit/lang/ast_test.cpp
# modified: src/test/unit/lang/generator_test.cpp
# new file: src/test/unit/meta/matrix/indexed_type_test.cpp
# new file: src/test/unit/meta/util.hpp
# new file: src/test/unit/model/indexing/deep_copy_test.cpp
# new file: src/test/unit/model/indexing/index_list_test.cpp
# new file: src/test/unit/model/indexing/index_test.cpp
# new file: src/test/unit/model/indexing/lvalue_test.cpp
# new file: src/test/unit/model/indexing/rvalue_return_test.cpp
# new file: src/test/unit/model/indexing/rvalue_test.cpp

#
# Unmerged paths:
# (use "git add <file>..." to mark resolution)
#
# both modified: lib/stan_math
#

So there's all my work sitting there like it's a new file. WTF? Now
I'm really too nervous to touch it.

There's still a whole lot about this whole process I'm not getting.
How'd that give me merge conflicts?

- Bob

[Daniel Lee]

no clue. I'm baffled. Let's talk in person tomorrow. I'm having trouble staying awake now.

Daniel

[Michael Betancourt]

Whenever people get up ping me and we can figure this out in a Hangout

using screen sharing.  Should be straightforward.

[Bob Carpenter]

I just blew away the whole repo in the broken state
and cloned a new one. No problem now.

Thanks for all the help. I really appreciate it.
I think I'm still a couple steps away from really
understanding what's going on with submodules (despite
Daniel explaining it to me a few times now, each time in
a way that seems totally clear as he's talking but seems
to elude me later).

- Bob