Commits and refs affected by "git push -f"

[Kohsuke Kawaguchi]

As I suggested in the e-mail thread, I've written a little program that
looks at the GitHub events API to figure out the push activities from
Luca, and assembled a list of refs and affected commits. The result is
in [1].

The exact code is at [2], and I invite others to check my sanity. The
basic idea is to look at the events time line, and find the problematic
push.

The meaning of the output is:

repository name, before, after, ref

"before" is the commit that's lost.

I'm going to compare this with the CSV file from GitHub now to see if
there's a pattern of discrepancy here.

[1] https://gist.github.com/kohsuke/7438914
[2]
https://github.com/jenkinsci/backend-git-pushf-finder/tree/eeb462c47dbaa45f2170ccada487eed33da81193
--
Kohsuke Kawaguchi http://kohsuke.org/

[Luca Milanesio]

I will cross-check with the list provided by GitHub.
(was doing a similar exercise on the suspect 54 repos anyway)

Luca.

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Luca Milanesio]

Cross-checking the two lists I have noticed:

a) The list from GitHub has lots of apparently incorrect post-push SHA-1 according to Kohsuke's list (88 repos)

b) With regards to the pre-push SHA-1, the two lists are almost identical with the following 4 exceptions:

git-client-plugin => pre-push and post-push inverted between the two lists
humbug-plugin => pre-push and post-push (GitHub) corresponds to the post-push (Kohsuke)
promoted-builds-plugin => pre-push and post-push (GitHub) corresponds to the post-push (Kohsuke)
scoring-load-balancer-plugin => pre-push and post-push (GitHub) corresponds to the post-push (Kohsuke)

c) There are repositories where the push was actually on other branches

git-plugin,refs/heads/refactoring
json-lib,refs/heads/rebase-2.4
perforce-plugin,refs/heads/multiple-scm-instatination-fix
pitmutation-plugin,refs/heads/generic-metrics
prqa-plugin,refs/heads/1.2.1
skype-im-plugin,refs/heads/v2dev
stashnotifier-plugin,refs/heads/develop
subversion-plugin,refs/heads/refactoring
transifex-plugin,refs/heads/gh-pages

d) The repos with not enough history or depth

None of those repositories were in the list of the 186 ones mentioned: can't find any of them on my logs.
This means that in the event time window captured by GitHub there wasn't any activity related to my account or any activity at all.

--- * ---

My suggestions:

a) I guess this is due of some initial 'git push' made for recovering the situation at the time that GitHub took the snapshot (the post-push are mostly incorrect, not the pre-push)

b) I will make further analysis on the 4 exceptions

c) Those branches are definitely to be restored, using the pre-push SHA1 from Kohsuke's list

d) Possibly cross-check with the plugins maintainers, but they should be OK and not require fixing.

--- * ---

Any other thoughts / feedback ?

Luca.

[Kohsuke Kawaguchi]

On 11/12/2013 02:34 PM, Luca Milanesio wrote:
> I will cross-check with the list provided by GitHub.
> (was doing a similar exercise on the suspect 54 repos anyway)

Yes, please cross check to validate my sanity.

I've compared this list with the CSV file from GitHub that you posted
earlier. The program I used is in the same GitHub repository.

https://gist.github.com/kohsuke/7440208

This is not the best formatted text, but the way you read this is that
given a line like this:

> GitHub:220610b->64db20c Kohsuke:220610b->c5bcbfd build-pipeline-plugin

According to GitHub's CSV, Luca has overwritten 220610b by 64db20c, but
according to my program that looks at the event API, Luca has
overwritten 220610b by c5bcbfd. This is the master branch in the
'build-pipeline-plugin'.

So one naturally wonders where the discrepancy comes from, and when I
look at the events output, I think it becomes clearer:

- open [1] in the browser (I captured this as of this writing in [2])
- This records Luca's push in line 350 (event ID: 1883629947)
- Head was 220610b and the push moved it to c5bcbfd, which is the
value my program reported.
- And if you scroll up, you see jenkinsadmin had pushed after luca
did, moving the ref from c5bcbfd to 64db20c, which is the value
GitHub CSV reported. This is probably Nicolas pushing changes from
jenkins.ci.cloudbees.com workspaces, which explains the login name
"jenkinsadmin"

My guess is that Nathan took the "force pushed to" SHA1 by looking at
the commit the master branch was pointing to, when he did his thing.
That is to say, if somebody has pushed changes after Luca did but before
Nathan looked at, that'd create a discrepancy in our two data.

In any case, the tip of the branch after a forced push is less
interesting. What's important is what the tip was before Luca's push.

So my program pays special attention when two data reports different
commit IDs as to what the repository was pointing to before Luca's push.
[2] prints out "^^^ discrepancy on the original commit ^^^ " when that
is the case.

Fortunately, there's only four of them. This includes
scoring-load-balancer-plugin reported by ikedam, which gives me a bit of
confidence in the approach. I'm going to look at these four more carefully.


This exercise also revealed another data loss not caught earlier by
GitHub's CSV, which are pushes to branches other than the 'master'
branch. The data from the events API correctly report them, and you can
see them all in [2] as well.


Since Git does not allow retrieval of commits by their SHA1, I'm going
to try if Git would let me tag a commit that I don't have locally. At
some low enough level this should work, but I don't know if the command
line git would take it. If that doesn't work, I think we need to
assemble the list of SHA1 and ref names so that we can ask someone from
GitHub to create refs for them.


In the mean time, if you notice any data loss that's not in the
'recovery' branch, please let us know so that we can continue to
validate and verify our salvage efforts.


[1] https://api.github.com/repos/jenkinsci/build-pipeline-plugin/events
[2] https://gist.github.com/kohsuke/7440304

--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Jenkins Enterprise, our professional version of Jenkins

[Kohsuke Kawaguchi]

On 11/12/2013 03:13 PM, Luca Milanesio wrote:
> d) The repos with not enough history or depth
>
> None of those repositories were in the list of the 186 ones mentioned: can't find any of them on my logs.
> This means that in the event time window captured by GitHub there wasn't any activity related to my account or any activity at all.

Maybe you have already explained this elsewhere, but how did you come up
with the list of 186 repositories in the first place? Is that what you
had on your Gerrit instance when it did "git push -f"?


The repositories listed under "[WARNING] events don't go back far
enough" are repositories that do report some events but none before the
Nov 9th. Thus there's a suspicion that some of those event history was
already lost, hence the warning. There are only three, and I think all
is accouned for:

- awseb-deployment-plugin (created yesterday)
- backend-git-pushf-finder (created today)
- websphere-deployer-plugin (created today)


Repositories listed under "[INFO ] no events in" are more numerous. I
agree with you that most likely there has been no activities for these
repositories in quite some time. For now I think we'd have to assume
that no commits were lost on them.

(If we have more cycles we could check the last commit in these repos to
really verify that none of them contain any new commits)

> --- * ---
>
> My suggestions:
>
> a) I guess this is due of some initial 'git push' made for recovering

> the situation at the timethat GitHub took the snapshot (the post-push

> are mostly incorrect, not the pre-push)

Agreed.

> b) I will make further analysis on the 4 exceptions
>
> c) Those branches are definitely to be restored, using the pre-push SHA1 from Kohsuke's list
>
> d) Possibly cross-check with the plugins maintainers, but they should be OK and not require fixing.

Yes, let's list up the repos of (b)+(c) above and find maintainers.

> --- * ---
>
> Any other thoughts / feedback ?
>
> Luca.
>
>
> On 12 Nov 2013, at 22:34, Luca Milanesio <luca.mi...@gmail.com> wrote:
>
>> I will cross-check with the list provided by GitHub.
>> (was doing a similar exercise on the suspect 54 repos anyway)
>>
>> Luca.
>>
>> On 12 Nov 2013, at 21:27, Kohsuke Kawaguchi <k...@kohsuke.org> wrote:
>>
>>>
>>> As I suggested in the e-mail thread, I've written a little program that looks at the GitHub events API to figure out the push activities from Luca, and assembled a list of refs and affected commits. The result is in [1].
>>>
>>> The exact code is at [2], and I invite others to check my sanity. The basic idea is to look at the events time line, and find the problematic push.
>>>
>>> The meaning of the output is:
>>>
>>> repository name, before, after, ref
>>>
>>> "before" is the commit that's lost.
>>>
>>> I'm going to compare this with the CSV file from GitHub now to see if there's a pattern of discrepancy here.
>>>
>>> [1] https://gist.github.com/kohsuke/7438914
>>> [2] https://github.com/jenkinsci/backend-git-pushf-finder/tree/eeb462c47dbaa45f2170ccada487eed33da81193
>>> --
>>> Kohsuke Kawaguchi http://kohsuke.org/
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>

[lucamilanesio]

Hi Kohsuke,

I agree with your conclusions ... that was my impression as well when I was analysing the 54 repos with pre = post: possibly Nathan thought that no other pushes were made after my forced push on Sunday.

One more detail about the 4 discrepancies on the pre-push SHA1:

comparing the pre-push SHA1 / post-push SHA1 and their respective commit dates, I would say that *your* pre-push SHA1 seems correct: possibly then a manualy "copy&paste" mistake by Nathan.

With regards to the current situation of those 4 repos:

git-client-plugin => OK

humbug-plugin => *OLD* (pointing the post-push SHA1)

promoted-builds-plugin => OK

scoring-load-balancer-plugin =>  *OLD* (pointing the post-push SHA1)

Last point: how to recover old SHA1 that were not labelled by Nathan ?

You can actually create a branch from the GitHub GUI: I will check if you can specify the branch point with a SHA1 so we can do it ourselves as well.

P.S. Let's not forget the repos with the other non-master branches to recover :-)

Luca.

[lucamilanesio]

Good news: I managed to create a branch based on a generic SHA1 directly from the GitHub GUI.

This allows us to set labels to the pre-push SHA1 for fetching and recovering them.

@Kohsuke: do you want me to create those labels on the mentioned projects ?

humbug-plugin => *OLD* (pointing the post-push SHA1)

scoring-load-balancer-plugin =>  *OLD* (pointing the post-push SHA1)

Luca.

[Kohsuke Kawaguchi]

Can you come over to the IRC channel so that we can talk in real time?

See http://jenkins-ci.org/content/chat for instructions.

On 11/12/2013 03:43 PM, lucamilanesio wrote:
> Hi Kohsuke,
> I agree with your conclusions ... that was my impression as well when I
> was analysing the 54 repos with pre = post: possibly Nathan thought that
> no other pushes were made after my forced push on Sunday.
>
> One more detail about the 4 discrepancies on the pre-push SHA1:
> comparing the pre-push SHA1 / post-push SHA1 and their respective commit
> dates, I would say that *your* pre-push SHA1 seems correct: possibly
> then a manualy "copy&paste" mistake by Nathan.
>
> With regards to the current situation of those 4 repos:
> git-client-plugin => OK
> humbug-plugin => *OLD* (pointing the post-push SHA1)
> promoted-builds-plugin => OK
> scoring-load-balancer-plugin => *OLD* (pointing the post-push SHA1)

I verified that git-client-plugin and promoted-builds-plugin are good.

> Last point: how to recover old SHA1 that were not labelled by Nathan ?
> You can actually create a branch from the GitHub GUI: I will check if
> you can specify the branch point with a SHA1 so we can do it ourselves
> as well.

If you have Nathan's e-mail address, I think it's better to ask Nathan
again.

I just experimented locally by creating two repositories trying to push
SHA1 into a ref when I don't have it locally,

git push origin bc48f6d072a267c6d8feaaea0a555411676b4a27:refs/tags/foo

... but this didn't work.

> P.S. Let's not forget the repos with the other non-master branches to
> recover :-)

Yes. Will come back to this.

> jenkins.ci.cloudbees.com <http://jenkins.ci.cloudbees.com>

> <https://groups.google.com/groups/opt_out>.

[Luca Milanesio]

Hi Kohsuke,
see below my feedback.

On 12 Nov 2013, at 23:38, Kohsuke Kawaguchi <kkawa...@cloudbees.com> wrote:

> On 11/12/2013 03:13 PM, Luca Milanesio wrote:
>> d) The repos with not enough history or depth
>>
>> None of those repositories were in the list of the 186 ones mentioned: can't find any of them on my logs.
>> This means that in the event time window captured by GitHub there wasn't any activity related to my account or any activity at all.
>
> Maybe you have already explained this elsewhere, but how did you come up with the list of 186 repositories in the first place? Is that what you had on your Gerrit instance when it did "git push -f"?

It was not a manual command execution of "git push -f" but an uncontrolled (from my side) behaviour of the Gerrit replication plugin.
The "force" option is (unlucky choice IMHO) implicit and optional: if you don't say anything (as I did) assumes force by default ;-(

As I do have the logs (on my side) of the catastrophe happened on Sun 10th, I extracted the list of repositories from there.

[Kohsuke Kawaguchi]

Luca and I took the conversation to IRC. Based on the list I created we
are reasonably confident that we recovered all the lost commits at this
point.


The following two repositories has the correct 'recovery' branch
pointing to the commit just prior to "git push -f". The original list
from GitHub incorrectly reported that these repositories do not have any
lost commits:

- humbug-plugin
- scoring-load-balancer-plugin

The following two repositories are in the same category, except all the
commits have already been resurrected by the time we got to it. So we
didn't do anything:

- promoted-builds-plugin
- git-client-plugin

GitHub only restored ref updates that have affected the master branch of
all the repositories. My program identified that more branches in some
repositories have been affected. We've manually inspected them and fixed
them all up.

> aeacaf8->61dcb7f git-plugin branch:refs/heads/refactoring
> 1ac9a85->bd241e6 json-lib branch:refs/heads/rebase-2.4
> 67323c6->fc3c580 perforce-plugin branch:refs/heads/multiple-scm-instatination-fix
> 70e0ba9->5a253ab pitmutation-plugin branch:refs/heads/generic-metrics
> fe15a76->2f5e05e prqa-plugin branch:refs/heads/1.2.1
> 7e81d63->4dad46b skype-im-plugin branch:refs/heads/v2dev
> b65b0d9->28fc04f stashnotifier-plugin branch:refs/heads/develop
> aae02ab->0512752 subversion-plugin branch:refs/heads/refactoring
> 327c4e4->6463c81 transifex-plugin branch:refs/heads/gh-pages

The way you read the list above is that, for example, git-plugin had its
'refactoring' branch unexpectedly moved from aeacaf8 to 61dcb7f, so we
restored it by fast-forwarding the 'refactoring' branch to aeacaf8,
which is where it should be.

For each such non-master ref updates, we also created
'recovery-$BRANCHNAME' to designate the point of the commit just prior
to "git push -f". So in case of git-plugin, there's
'recovery-refactoring' that points to aeacaf8.

In all the 9 cases, update to the branch (like 'refactoring') to the
correct commit was done in fast-forwarding. This means there's no data
loss between my attempt and possible attempts by respective plugin
developers.


And finally, for all the 186 repositories whose master branch was
affected, I've fast-forwareded 'master' to 'recovery' as I proposed
yesterday. That is, if the master branch (who lost some commits) didn't
receive any commits since the "git push -f", this fix will resurrect the
master branch as it was just prior to "git push -f".

Most repositories fit this category, including those who have already
been restored, where recovery and master already point to the same
commit. The entire execution record of this is in
https://gist.github.com/kohsuke/7442162.

The following repositories did not take the fast-forwarding:

> error: failed to push some refs to 'g...@github.com:jenkinsci/conditional-buildstep-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/copyartifact-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/datical-db-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/disk-usage-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/email-ext-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/extra-columns-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/git-client-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/ironmq-notifier-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/jobConfigHistory-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/matrix-auth-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/maven-info-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/mesos-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/promoted-builds-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/puppet-jenkins.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/scoring-load-balancer-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/timestamper-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/transifex-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/translation-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/view-job-filters-plugin.git'
> error: failed to push some refs to 'g...@github.com:jenkinsci/xvnc-plugin.git'

This can mean one of two things:

1) Commits have already been recovered and more commits were added,
in which case the repository is in good shape.

2) recovery and master have diverged, in which case a merge is likely
needed.

I'm going to look at them to classify them in those two categories.

Kohsuke Kawaguchi http://kohsuke.org/

[Kohsuke Kawaguchi]

I went through this list, by running "git rev-list
origin/master..origin/recovery" to find out what commits their master
branches are missing.

The following 3 repositories belong to the class #2 above, where master
and recovery have actually diverged. See the list below. I manually
inspected them all, and I'm going to just run "git merge recovery" from
master.

In all the other repositories, the lost commits have been recovered.
That is, their master branch contains all the commits in the recovery
branch.


puppet-jenkins
b0b5f5e84f466bcd925a89e81d91d9923b439094
6041785c38b7eb36c13c57a759fd97d36c24d25f
6f0e4d5e2c1955a3732fd066cfa8faf063792ced
7eb07a95eaa1f9b61ec86b1f5ef0256e4b7e96a5
fa4deb90478b70fc5e408e908b1ad104c29f7e8d
833197a370bb7f9bf2787f508051154574e9f1b4
ffae859f551e32af85cebe72e424939fc327a01e
f433fa5bf82b26db52a35063a0a71f58448d8504
8709d40c0f898987b5903cfa57a8382c995188e0
c130234b5fa139e5fa3aef888e721d47bb0461e6
f8b3006c7af9140239b65985a1a6c76a0e85c851
7a66f3f098e1665360270e49d3312e54bfcd4bf1
ca78023cfb3c922486d340a3f687162902e19b37
c47ba351261e855f49a9e6dbf9d40479cbd03e38
66da22cced7cb821762d0e0cc73abc8407e557a9
3a93b53b4b646b787829efa46e4995f3f60a3ec8
ff57fb1e6f554390b053c176d35375caab20bb1d
c6da8d753a9a2461beef912df5b9bd05b20392f6
1b8e186c5cb57391323512a4c720e34fc96f03c0
bf398f28f8210475b65b427c3288d837be63d7e2
8d375f76257ae0db415bc7bc7587ee76280fb44b
bef0b486906b5ef94b38fd407da73b1ed3616124
a237522ff50ea092078cf71142863399a02cf652
412e9c3bc1189e58f03e23c3ab3d486021d4a122
133279e3aa7a056e2209e5db1abd3c0325e1717c
97af684a4dc0556ef5cedf61569e29674b77f9b2
1e769502395941eee43c950c64ae87d41a7d565f
6550deeb9c62932c70202cf58036541b982603bb
6ecb4023c48d335882fad07e0bb97ce5c7e0dcc1
2c9916fdec0deee8dc4b91d5852b229b924ac118
87a8e884caf2b34dc6ab7fcc31e07d2c0515a860
15db701af46050470c25b30f67ece47a1421361a
50665743a048fe04ef03bdeb29406c25f3dd8c1b
3ebde028402cddb1710c6c5e0a080a4bdcc92b28
0e31849cdfac835474013872d408077120d0c674
23d3f181825d3b205965e4343bb14298512a2aff
2d9ff026a0dc882fc97c4ce25290a9ca13836dd3
734e6a1da7203f9c1e43252f047b72a5a9da28a6
956a1be8c1fc8403052bba22b907655dfa02e849
df63e8ef9ce66e3feb47a32768c55171a7cc9046
449dadeb27bc8aca7e903ca7547eacf87e957b26
c0690ebee1d76f4ca74a062219fe0f9d8043e9dc
dc5aa24ffcbe2d19aabc8aad33036259c653ae4b
c8c101896b2fc67b18fbee56a10b39e0346fcfc0
9ed8855e7bf99f10efb9f806074441a9601ae09a
351f48169ff7a10df09a7342f79f887fc8e4b90a
6480820b7feda96a0bd2c3456800366bb2cb1f9b
87853699842a6bc9983b75fd9c2aca5f01fd0863
723680c252c56d844aa579c04731f73fbc42d5be
6fd5cd93b73d6a90cf7ac877f82eb15c66b46ac5
47d3feed4866a2bb7e8516035e3abfd82b3792e8
1eda89e6d724aa0243cffe68aefa0cfc08e8f57e
c817968d466d521f01531bad2ad87fbd5b248a1a
6d3de01b2d2f8159b11917a8b0c8b7c54b0918fc
b2f14add1097be9e2e71c22fb76b31497e1cbed0
994f7b2a418ac3308cd8919b13fe86a25c925a7b
14d0f02fd36cc50a425ec0779674246fa051bd6c
0ab3128ba894674d3b42fab5d3d769053f04b06d
efd29b0a2b242e9526f2078d2093a89a7f68e4fc
91c8510cb0532507bc1197a231cc83e653cc938d
f0bcd8d62fd2686c39f6b39e0cba48387e1b0b12
ecc9216c0b9fb20e395a45779be83804012cc43b
4680a9fdf9013fe824470be6939e1ab9640515e0
b388d1935d533b04be00ecd5d4a5e7d2d48218a0
74f578b1d5bc262fa0c8a1f0f9edfd63dd1a8cff
13c0def605176e483b3a2960a06ae1709d621b10
c950ff3a0f47bae2603a8f45b784992efb8b0919
314e0de5d86701e7c3d1c8a2b3aefd32bc658878
0828b469d1a8df8e4789d3070d8e2e5bdc4ea34c
fc9bda9f8430051f1630c607894076ab287e93e7
69e6286f5eded5d77cbd5e05e25222747c8c4647
90bb47943bb283bfe6520dd0ad811160cf078bc3
79b2c353a7930fc00cb845dc26d06a1f8782935c
c65e3f2864f90d5aa27aec8f6c9d7fb980a13b3a
dca6ebf28e8e89fe1d5a4359c9c7317816135417
a999701d71998119e3c34b85ea83e223145f666d
7447d7fa7c7d581b9eb0939f042158433a952323
view-job-filters-plugin
6f27fa9fb9bb5c1a7905a538907185f519d16f8f
c89f171d7e7215c8a62ba2497fb4caa555eb42be
xvnc-plugin
e891d20a7d7453a718abe60a56436c383f9bb503

[Kohsuke Kawaguchi]

I merged the three repositories and pushed new master.

> puppet-jenkins
> view-job-filters-plugin
> xvnc-plugin

This concludes the data recovery. All the commits have been restored,
all the branches have been updated, and there's no action needed by anyone.

That said, it'd be great if people can test my claim by trying to find
any missing commits, to boost our confidence level in the recovery
process (or find missing commits while we can still save them.)

I'll work with Luca to write a summary on the incident and the data
recovery.

[Luca Milanesio]

Hi Kohsuke,
that's really good news !

I would like once again to publicly apologise with all the Jenkins Developers Community for the inconvenience caused by my forced push on the 10th of November.
The problem was caused by the unfortunate combination of fatal mistakes on my side: not the tool's (Git, Gerrit or GitHub) fault ... just a very bad day of mistakes from me :-(

A big thank to all the people that cooperated and supported the recover the correct point on 186 repositories (!) including Kohsuke and Nathan.
I am keen to help in the post-mortem report and analysis in order to help the community to avoid that similar situations would happen in the future.

Thanks again for your understanding.

Luca.

[FredG]

Hi,

As of late last night (GMT+1) the missing commits for the extra-columns-plugin were not recovered.
The plugin was on Lucas "impacted plug-ins" list and I found a recovery branch for it,
but I had to push the commits again myself (worked without a problem).

Just wondering if there are other plugins that are not up-to-date yet.

Regards,

Fred

[Jesse Glick]

On Tue, Nov 12, 2013 at 8:51 PM, Kohsuke Kawaguchi <k...@kohsuke.org> wrote:
>> error: failed to push some refs to 'g...@github.com:jenkinsci/matrix-auth-plugin.git'

This was a weird case. I think it is fine now. When splitting the
plugin, I had to do two emergency micro releases after 1.0, which I
did in anonymous branches that got “jumping fish”-merged back into
master, though I might not have pushed the final merge of 1.0.2 from
my local clone. And then on Monday morning (before seeing the notices
about the GitHub problem) I did a few more commits and a Maven
release, which I guess pushed the master HEAD back up, and also made
your recovery script fail.

[Kohsuke Kawaguchi]

On 11/13/2013 03:26 AM, FredG wrote:
> Hi,
>
> As of late last night (GMT+1) the missing commits for the
> extra-columns-plugin were not recovered.
> The plugin was on Lucas "impacted plug-ins" list and I found a recovery
> branch for it,
> but I had to push the commits again myself (worked without a problem).

Hmm. I wonder if you did the push slightly before I did mine.

According to the events time line, at 2013-11-13T00:14:01Z you pushed to
master and moved from cbaaef99711c8d0a8072021ad2648d16043055f0 (which is
what luca had set) to e4644c5ede5cca6b365aa48bb659c6170e4638e5 (which is
one commit ahead of the recovery point.)

There are a few more pushes from you afterward, and the last one moves
the master to 86169ef8a62ddd587dea871d64d3fa2dd7241daa at
2013-11-13T00:28:45Z, which translates to about 5:30pm my time, which is
just before my recovery->master fast-forward.

So I still stand by the assumption that all commits were recovered as of
6:13pm Nov 12th PST.

> Just wondering if there are other plugins that are not up-to-date yet.
>
> Regards,
>
> Fred
>
> On Wednesday, November 13, 2013 3:13:47 AM UTC+1, Kohsuke Kawaguchi wrote:
>
>
> I merged the three repositories and pushed new master.
>
> > puppet-jenkins
> > view-job-filters-plugin
> > xvnc-plugin
>
> This concludes the data recovery. All the commits have been restored,
> all the branches have been updated, and there's no action needed by
> anyone.
>
> That said, it'd be great if people can test my claim by trying to find
> any missing commits, to boost our confidence level in the recovery
> process (or find missing commits while we can still save them.)
>
> I'll work with Luca to write a summary on the incident and the data
> recovery.
>
>
> --
> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
> Try Jenkins Enterprise, our professional version of Jenkins
>

> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-de...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[FredG]

OK, I must have missed something.
I was under the impression that the major part of the recovery was already done
and just a few special branches were left, before I pushed my changes.
That's why I assumed that the plug-in was missing.

Sorry for the noise.

[Ikedam]

Hello.

> > error: failed to push some refs to 'g...@github.com:jenkinsci/copyartifact-plugin.git'

I found something strange had happened to copyartifact-plugin.
It had been already recovered to a commit of an abondoned pull request,
https://github.com/jenkinsci/copyartifact-plugin/pull/24 (46540cc88dee2ecc56ede5c389b48e4f89d1c539).
This seemed cause the above push failure.

I have already recovered it by adding a reverting commit.
Same things may have happened to other failed plugins.

What I know from https://api.github.com/repos/jenkinsci/copyartifact-plugin/events are:

* 2013/10/11 16:55:13 push from jglick 5153c27 -> dc98ffe (this was the commit that should be recovered to).
* 2013/11/03 07:44:39 pull request from ikedam 46540cc
* 2013/11/10 01:58:32 push from lucamilanesio dc98ffe -> f468296
* 2013/11/11 14:07:44 push from jenkinsadmin f468296 -> 46540cc (strange!)

Both "recovery" branch and Kohsuke's list (https://gist.github.com/kohsuke/7438914) points to
the correct commit dc98ffe.
I have no idea what made jenkinsadmin to push that commit
(The recovery should be done with id "kohsuke").

ikedam

[Kohsuke Kawaguchi]

I think I know what happened.

The push from "jenkinsadmin" was done by ndeloof, who had a direct
access to workspaces used by http://jenkins.ci.cloudbees.com/. My
understanding is that he had taken the current workspace of the
respective plugin projects, and then run git push from there.

Now, if the project had last built a pull request at the time of this,
then the master on these workspaces would contain the pull request that
was built. So the push would end up integrating these pull requests.

Looks like I'll have to check the push from jenkinsadmin to see if this
has affected other plugins. Oy.

> <https://gist.github.com/kohsuke/7442162>.

> <http://jenkins.ci.cloudbees.com <http://jenkins.ci.cloudbees.com>>

> For more options, visit https://groups.google.com/groups/opt_out.