Documentation on what to do when Pkg.publish fails

[Douglas Bates]

Once again I am facing

julia> Pkg.publish()

INFO: Validating METADATA

INFO: Pushing RCall permanent tags: v0.1.1

INFO: Submitting METADATA changes

INFO: Forking JuliaLang/METADATA.jl to dmbates

Enter host password for user 'dmbates':

ERROR: key not found: "token"

 in getindex at dict.jl:617

 in token at pkg/github.jl:53

 in req at pkg/github.jl:61

 in POST at pkg/github.jl:74

 in fork at pkg/github.jl:87

 in pull_request at pkg/entry.jl:292

 in publish at pkg/entry.jl:359

 in anonymous at pkg/dir.jl:28

 in cd at ./file.jl:20

 in __cd#228__ at ./pkg/dir.jl:28

 in publish at pkg.jl:57

This what happens to me the majority of the times when I try to publish an update.  I don't understand the circumstances under which this happens and how, or even if, it can be avoided.

If I check the status of my local METADATA repository I see

bates@thin40:~/.julia/v0.3/METADATA$ git status

On branch metadata-v2

Your branch is ahead of 'origin/metadata-v2' by 18 commits.

  (use "git push" to publish your local commits)

nothing to commit, working directory clean

bates@thin40:~/.julia/v0.3/METADATA$ git pull

Already up-to-date.

As I understand it I don't want to push the changes to my github fork because of those 18 commits.  Any pull request that I generate from my fork will have 17 superfluous commits. 

84a074c * Tag RCall v0.1.1

c1719a1 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

483e380 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

ece31d5 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

6db3d8d * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

2e40156 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

f72bc8e * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

cdfceb2 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

48f91c5 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

397a530 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

cd91640 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

c8d76e6 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

c698b4d * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

23b927a * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

ead1dc7 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

8eefce8 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

0536c0b * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

233cca4 * Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

Again, I don't understand how, or even if, this can be avoided.  I trust that this can be solved with 'git rebase' but I have a 99.999% chance of making things even worse for myself if I try to do that.

Is there any documentation on what is happening when forking JuliaLang/METADATA.jl to dmbates goes wrong?  Is it related to my having a 'token = ...' in the github section of ~/.gitconfig?  If so, should I remove that?

My usual recourse is to remove my ~/.julia directory and start over from scratch, which makes releasing a new version of a package much more awkward than it should be.  I have reached the point when I don't want to develop Julia packages simply because the process of releasing a new version is so unbelievably difficult.

I find it ironic that the purpose of git is to preserve history of changes and I seem always to end up in situations where the only way out is to start over from scratch and wipe out all the history.

[Douglas Bates]

P.S. If someone in the JuliaStats group could publish version 0.1.1 of RCall, I would appreciate it.  I doubt I will be in a position to do so in the foreseeable future.

[Andreas Noack]

Done in 

https://github.com/JuliaLang/METADATA.jl/commit/9430eb6427a4dfade0ee2db6dbc7967084a59abf

[Douglas Bates]

On Friday, March 13, 2015 at 10:45:49 AM UTC-5, Andreas Noack wrote:

Done in 

https://github.com/JuliaLang/METADATA.jl/commit/9430eb6427a4dfade0ee2db6dbc7967084a59abf

Thank you.

[Andreas Noack]

I also saw the "...token..." thing recently and I could fix it by following the advice in the blue box in 

http://julia.readthedocs.org/en/latest/manual/packages/#publishing-your-package

i.e. by deleting package manager entry in the settings at guthub.com.

[Douglas Bates]

On Friday, March 13, 2015 at 11:37:37 AM UTC-5, Andreas Noack wrote:

I also saw the "...token..." thing recently and I could fix it by following the advice in the blue box in 

http://julia.readthedocs.org/en/latest/manual/packages/#publishing-your-package

i.e. by deleting package manager entry in the settings at guthub.com.

Thanks, Andreas.  I remembered seeing that advice but I thought, mistakenly, that it would apply only to package management in Julia v"0.4-".

Does this have any effect on ending up with a commit labelled

Merge branch 'metadata-v2' of git://github.com/JuliaLang/METADATA.jl into metadata-v2

every time that I run 'Pkg.update()'.  I had, perhaps naively, thought that Pkg.update() would pull from git://github.com/JuliaLang/METADATA.jl and, in my experience, a pull does not cause a separate commit of a merge.  Is it related to having a fork of METADATA.jl under my github account so there are three repositories that somehow need to be reconciled?  I suppose I could delete the forked copy of METADATA.jl on my github account after every Pkg.publish() but that seems to be unnecessarily complicated.

Is the fact that I use more than one computer for development what is tripping me up?  That seems to be implied by the advice that you directed me to.  Is using more than one computer considered so unusual that there have to be special procedures to use regarding Julia packages if you do?

[Tony Kelman]

A pull does cause a merge commit, unless it's a fast-forward update, meaning your existing history exactly matched some past state of the upstream. You get merge commits if you have some local committed but un-pushed changes, then do Pkg.update() before publishing them.

I personally think it would be better if Pkg.publish() would create a new branch first, then commit to it. It's analogous to the advice we give newcomers with PR's to base Julia, it's not good to make commits straight to your fork's master, since keeping things updated as upstream changes becomes more complicated and requires either rebasing or merge commits.

[andy hayden]

Pull = fetch + merge. If there are additional commits on your branch then merge will add a merge commit.

Perhaps it's better to rebase rather than merge in Pkg.update... For most people should this be a fast-forward?

There's a "TODO handle merge conflicts" perhaps it should raise if it can't fast-forward...

I like to use git fetch + git merge --ff-only when updating master (or git pull --rebase when updating branches).

[Douglas Bates]

The problem for me is that I can't determine under what circumstances I will get these "merge" commits and I don't find out until it is too late. I just suddenly learn that my METADATA repository is thoroughly screwed up.

As far as I know all I have done is run Pkg.update().  I have no idea why Pkg.update() would result in a merge commit if I am working from a clean package directory.  I also don't know how to determine if it has created a merge commit.  I could do everything by hand, regularly checking the git status of the METADATA from outside Julia but if that is necessary then why have the Pkg module at all.

This is the source of my frustration  I am willing do things in a roundabout way (like systematically removing the token from github account if there is a possibility that I have used more than one computer) but if I don't know that cause of these issues it is hard to work around them.

I do not have a way of releasing a new version of a package that I can feel confident will work, which makes me think that I am not cut out to be a Julia developer.

[Tony Kelman]

That's not good. Your local METADATA isn't exactly screwed up, it just looks messy and will have a bunch of extraneous history in it that isn't relevant to the PR you'll eventually file to tag or register whatever package you're working on.

I just filed a PR at https://github.com/JuliaLang/julia/pull/10529 that might mitigate this by preferring rebase instead of merge in Pkg.update.

The circumstances under which you get merge commits are the following:

1. You do Pkg.tag or Pkg.register locally, which makes new commits to your local copy of METADATA.

2. You do Pkg.update a few times, which merges in any new versions of other packages from upstream METADATA

If you do Pkg.update after Pkg.tag or Pkg.register but before Pkg.publish and the corresponding new PR getting merged into upstream METADATA, then your local copy of METADATA will gradually accumulate new merge commits, one for each time you get new information from Pkg.update.

[Keith]

So it looks like pull/10529 has been merged.  How does that change the picture?

[Tony Kelman]

For those brave people who use 0.4 nightlies or source builds (which also happens to have a much faster Pkg.publish, but also many other breaking changes), when you do Pkg.update you should no longer get merge commits, instead any local commits you may have in your personal copy of METADATA should get automatically rebased on top of the latest from upstream each time you do Pkg.update. We'll see how well this works in practice.

[Tim Holy]

+1