strange pushes on GitHub

79400 views
Skip to first unread message

domi

unread,
Nov 10, 2013, 10:47:04 AM11/10/13
to Jenkins Developers
Hi everyone and specially Luca,

yesterday something strange has happen to many Jenkins repositories on GitHub (more then 50 Repos)…
Luca Milanesio seems have pushed to many many repositories without really changing anything - at least I have not seen anything changed.
If you look at the activity stream/feed at https://github.com/organizations/jenkinsci and go back to yesterday, you’ll find many pushes to many different repositories.
Maybe I’m wrong and someone has an easy explanation for this?
But for me this looks like someone did not really now how Git/GitHub works and accidentally pushed to every single repo - maybe someone else could also have a look and see if there is something broken now?

regards Domi

Marcelo

unread,
Nov 10, 2013, 10:56:15 AM11/10/13
to jenkin...@googlegroups.com
Regards

Marcelo Rebasti



--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Vincent Latombe

unread,
Nov 10, 2013, 11:21:00 AM11/10/13
to Jenkins Dev
It seems like a lot of repositories are back to previous revisions. git-client-plugin's master now points to 1.4.3-SNAPSHOT whereas it should be pointing to 1.4.7-SNAPSHOT...

Vincent


2013/11/10 Marcelo <mreb...@gmail.com>

ogondza

unread,
Nov 10, 2013, 11:25:25 AM11/10/13
to jenkin...@googlegroups.com
Definitely related, selenium-tests repo lost 17+ commits after such push.

domi

unread,
Nov 10, 2013, 11:51:26 AM11/10/13
to Jenkins Developers, Lu...@milanesio.org
As I don’t know if Luca is following this list too, I’m adding Lucas mail on the to list… (got the email address from his GitHub account)
Please Luca can you explain what you have done to all this repositories?
regards Domi


On 10.11.2013, at 17:25, ogondza <ogo...@gmail.com> wrote:

Definitely related, selenium-tests repo lost 17+ commits after such push.

Arnaud Héritier

unread,
Nov 10, 2013, 1:24:32 PM11/10/13
to jenkin...@googlegroups.com, Jenkins Developers, lu...@milanesio.org
I confirm we lost many commits. damned github that doesn't allow to disable the history rewrite at least on master branch. 
Last commit on Xcode repo is referencing one done by Jerome Lacoste 5 montages ago while KK worked on it recently. 
I don't know how we can restore everything easily. We need to find the recent hash and relative history in a clone we the push of yesterday was retrieved. 


Sent from Mailbox for iPhone

Arnaud Héritier

unread,
Nov 10, 2013, 1:45:03 PM11/10/13
to jenkin...@googlegroups.com, Jenkins Developers, lu...@milanesio.org
S/montages/monthes
S/we/with

Stupid dictionary 


Sent from Mailbox for iPhone


Luca Milanesio

unread,
Nov 10, 2013, 1:55:08 PM11/10/13
to jenkin...@googlegroups.com, Arnaud Héritier
Hi all,
I have triggered an involuntary "forced push" last night on the list of Jenkins-CI plugins indicated below in this e-mail.

My apology 

I did not realise that I actually had forced push permissions and I do apologise for the inconvenience caused.
The operations pushed back the all the branches to around 1 month. The history is not lost and is still on the GitHub server but on detached branches.

The solution

I can raise a request to GitHub to provide the "reflog" of those repositories and restore the branches to the point before my forced push.
Alternatively the owners of those repositories can still perform a "forced push" to restore the correct position of the branches.
(if you would like to do so, please write to the mailing list so that we do not overlap the recovery operations)

The full list

See below the full list of repositories impacted:

antexec-plugin.git
artifactory-plugin.git
associated-files-plugin.git
audit2db-plugin.git
audit-trail-plugin.git
backend-pull-request-greeter.git
beaker-builder-plugin.git
branch-api-plugin.git
build-flow-plugin.git
buildgraph-view.git
build-pipeline-plugin.git
build-timeout-plugin.git
buildtriggerbadge-plugin.git
bytecode-compatibility-transformer.git
ci-game-plugin.git
clearcase-plugin.git
clearcase-ucm-plugin.git
cloudbees-folder-plugin.git
cloudbees-plugin-gateway.git
cloudtest-plugin.git
clover-plugin.git
cobertura-plugin.git
collabnet-plugin.git
collapsing-console-sections-plugin.git
compact-columns-plugin.git
compress-artifacts-plugin.git
conditional-buildstep-plugin.git
config-file-provider-plugin.git
configurationslicing-plugin.git
copyartifact-plugin.git
copy-project-link-plugin.git
copy-to-slave-plugin.git
cppcheck-plugin.git
credentials-plugin.git
crowd2-plugin.git
crowd-plugin.git
customtools-plugin.git
cvsclient.git
cvs-plugin.git
dashboard-view-plugin.git
datical-db-plugin.git
dependency-check-plugin.git
deploy-plugin.git
disable-failed-job-plugin.git
disk-usage-plugin.git
doclinks-plugin.git
dry-plugin.git
dynamic-axis-plugin.git
ec2-plugin.git
elastic-axis-plugin.git
email-ext-plugin.git
envinject-lib.git
envinject-plugin.git
extended-choice-parameter-plugin.git
extra-columns-plugin.git
extras-executable-war.git
extreme-feedback-plugin.git
gearman-plugin.git
gerrit-trigger-plugin.git
gitbucket-plugin.git
git-chooser-alternative-plugin.git
git-client-plugin.git
git-plugin.git
global-build-stats-plugin.git
global-variable-string-parameter-plugin.git
gradle-jpi-plugin.git
grails-plugin.git
greenballs-plugin.git
groovy-postbuild-plugin.git
heavy-job-plugin.git
hockeyapp-plugin.git
http-request-plugin.git
humbug-plugin.git
instant-messaging-plugin.git
integrity-plugin.git
ironmq-notifier-plugin.git
ivytrigger-plugin.git
jacoco-plugin.git
jclouds-plugin.git
jira-plugin.git
jobConfigHistory-plugin.git
job-dsl-plugin.git
job-import-plugin.git
job-poll-action-plugin.git
jquery-plugin.git
jquery-ui-plugin.git
json-lib.git
kiuwan-plugin.git
label-verifier-plugin.git
ldap-plugin.git
leiningen-plugin.git
lib-annotation-indexer.git
lib-task-reactor.git
lib-windows-remote-command.git
literate-cli.git
logfilesizechecker-plugin.git
m2release-plugin.git
m2-repo-reaper-plugin.git
mailer-plugin.git
matrix-auth-plugin.git
maven-hpi-plugin.git
maven-info-plugin.git
mercurial-plugin.git
mesos-plugin.git
metadata-plugin.git
mock-security-realm-plugin.git
msbuild-plugin.git
naginator-plugin.git
nerrvana-plugin.git
nested-view-plugin.git
next-build-number-plugin.git
next-executions-plugin.git
parameterized-trigger-plugin.git
perforce-plugin.git
performance-plugin.git
persona-plugin.git
pitmutation-plugin.git
plain-credentials-plugin.git
plugin-compat-tester.git
postbuildscript-plugin.git
promoted-builds-plugin.git
prqa-plugin.git
publish-over-cifs-plugin.git
puppet-jenkins.git
radiatorview-plugin.git
rapiddeploy-plugin.git
release-plugin.git
repo-plugin.git
rich-text-publisher-plugin.git
robot-plugin.git
run-condition-plugin.git
rvm-plugin.git
scm2job-plugin.git
scm-api-plugin.git
scoring-load-balancer-plugin.git
script-scm-plugin.git
selenium-axis-plugin.git
selenium-builder-plugin.git
selenium-tests.git
skype-im-plugin.git
skytap-cloud-plugin.git
smartfrog-plugin.git
sms-plugin.git
sounds-plugin.git
ssh-agent-plugin.git
ssh-credentials-plugin.git
sshd-module.git
ssh-slaves-plugin.git
starteam-plugin.git
stashnotifier-plugin.git
subversion-plugin.git
suppress-stack-trace-plugin.git
swarm-plugin.git
synergy_scm-plugin.git
tap-plugin.git
teamconcert-plugin.git
testlink-plugin.git
tfs-plugin.git
thin-backup-plugin.git
throttle-concurrent-builds-plugin.git
tikal-multijob-plugin.git
timestamper-plugin.git
token-macro-plugin.git
transifex-plugin.git
translation-plugin.git
trilead-ssh2.git
unity3d-plugin.git
veracode-scanner-plugin.git
view-job-filters-plugin.git
virtualbox-plugin.git
vsphere-cloud-plugin.git
vstestrunner-plugin.git
walldisplay-plugin.git
warnings-plugin.git
weblogic-deployer-plugin.git
winstone.git
wix-plugin.git
ws-cleanup-plugin.git
xcode-plugin.git
xstream.git
xtrigger-lib.git
xunit-plugin.git
xvfb-plugin.git
xvnc-plugin.git

The prevention

Can we prevent this to happen again ?
I personally do not work on any of those repositories but still have "forced push" permissions ... and so many other people have.
I don't see the value of having such power of "potential disruption" associated to my account :-( ... can we remove the forced push by default and enable on a case-by-case basis ?

--- * ---

If you would like to propose an alternative approach to resolve the problem, feel free to follow-up !

... and again ... accept my sincere apologies :-(

Luca.

ogondza

unread,
Nov 10, 2013, 2:05:39 PM11/10/13
to jenkin...@googlegroups.com, Arnaud Héritier
I have already asked Github support to provide hashes of former master positions and referenced this thread. Will report ASAP.

Luca Milanesio

unread,
Nov 10, 2013, 2:10:24 PM11/10/13
to jenkin...@googlegroups.com, Arnaud Héritier, sup...@github.com
*UPDATE*
I have just send the request to GitHub support on-line, looping them as well in this e-mail.

@GitHub Support: please consider this request as a matter of urgency, thank you.

Luca.

Arnaud Héritier

unread,
Nov 10, 2013, 2:30:24 PM11/10/13
to ogondza, jenkin...@googlegroups.com
Thx guys for these details. We are all humans and subject of doing errors. 
I hope we'll have some help from github but I'm not sure they may spend many time to help us for an error not coming from their side. 
In any case it is recommended to NOT PULL current master branches from these repositories, your local copy may be more up-to-date. If you have a recent hash you can at least try to push them in github under a branch named "master-fix" the time we find the best solution to repare all master repositories. 
If you commited on these repositories since yesterday verify that you didn't started your work from the wrong old hash. 

Idea : in our CI server logs we should be able to find the lastest hash of each repo in the related plugin build. After that we have to find how to restore them. It's easy to do it when you have this commit locally but AFAIK it's not if it is only in the remote (you don't get orphelin commits when you clone a repo)

Arnaud


Sent from Mailbox for iPhone


Luca Milanesio

unread,
Nov 10, 2013, 2:41:40 PM11/10/13
to jenkin...@googlegroups.com, Arnaud Héritier, sup...@github.com
*UPDATE* after a cross-check I need to add 2 extra repositories to the list:
accurev-plugin
analysis-core-plugin

The total number of impacted repositories should then be: 186

@ALL: is there anyone with direct contact of some GitHub-er ? It would be best to resolve the problem *before* the start of the working week on Monday.

Thanks to everyone that contributed to resolve the problem !

Luca.

Luca Milanesio

unread,
Nov 10, 2013, 2:47:50 PM11/10/13
to jenkin...@googlegroups.com, ogondza
Hi Arnaud,
thanks for the understanding :-) I am used to warn other people from doing any "forced push" ... and Murphy's law wanted this time happening to me :-(

On 10 Nov 2013, at 19:30, Arnaud Héritier <aher...@gmail.com> wrote:

> Thx guys for these details. We are all humans and subject of doing errors.
> I hope we'll have some help from github but I'm not sure they may spend many time to help us for an error not coming from their side.

Technically yes, but for the "customer relationship" and because it is an easy recovery (git reflog + git branch), they could make some effort to help us out.
JenkinsCI is possibly the largest customer they have worldwide ;-)

> In any case it is recommended to NOT PULL current master branches from these repositories, your local copy may be more up-to-date. If you have a recent hash you can at least try to push them in github under a branch named "master-fix" the time we find the best solution to repare all master repositories.
> If you commited on these repositories since yesterday verify that you didn't started your work from the wrong old hash.
>
> Idea : in our CI server logs we should be able to find the lastest hash of each repo in the related plugin build. After that we have to find how to restore them. It's easy to do it when you have this commit locally but AFAIK it's not if it is only in the remote (you don't get orphelin commits when you clone a repo)

Yes, that's what I thought as well ... but definitely the "git reflog" on GitHub server would be more reliable.
The CI workspace (assuming that Git clones are not shallow copies) may have as well those commits as well, but again GitHub would be safer and better IMHO.

Luca

Marcus Bauer

unread,
Nov 10, 2013, 2:52:39 PM11/10/13
to jenkin...@googlegroups.com
Hi,

I don't think GitHub.com has any possibilities for disabling force pushes, this seems to be exclusive to GH Enterprise only.

The JaCoCo repo where I initially noticed this was restored by Dominik Stadler (centic9) few minutes ago.

Marcus

Mirko Friedenhagen

unread,
Nov 10, 2013, 3:00:30 PM11/10/13
to jenkin...@googlegroups.com
Hello,

I saw these strange pushes as well, jobConfigHistory-plugin was
failing because master was reset to a commit which was two months old.
I see that https://github.com/jenkinsci/maven-hpi-plugin/tree/master
is pointing to an old commit as well, the new version 1.106 is much
newer than master.

Regards
Mirko
Regards Mirko
--
http://illegalstateexception.blogspot.com/
https://github.com/mfriedenhagen/
https://bitbucket.org/mfriedenhagen/

Luca Milanesio

unread,
Nov 10, 2013, 4:40:28 PM11/10/13
to jenkin...@googlegroups.com
That's really pitty :-( ... force push are dangerous, especially if you don't have control over the Git Server.

Typically recovering a force push is straightforward:
1. git reflog > look at the SHA-1 before the forced push
2. git branch -f <name> <sha-1>

But if you don't have control over the Git repo on the Server, that you need to prevent force push to happen: unless they want everyone to buy GH Enterprise !

I still hope GitHub will do 1. and 2. for us :-)

Luca.

Dominik Stadler

unread,
Nov 10, 2013, 4:59:42 PM11/10/13
to jenkin...@googlegroups.com
Hi,

I didn't know that this was a bigger problem and thus just pushed the previous commits again, in case of the jacoco-plugin repo it was easy as I just committed changes yesterday and thus could easily push the latest changes together with the missing commits.

The really bad thing in my view is that as a user of the repo you do not see at all that it happened. Git and Github is all about history of changes, but in this case your repo looks like a month ago with no visible trace on github that something newer was there at some time... Quite dangerous....

Dominik.

Slide

unread,
Nov 10, 2013, 5:06:36 PM11/10/13
to Jenkins Dev, Arnaud Héritier
I did a force push to email-ext and it seems to be fine now.

Thanks,

slide

Arnaud Héritier

unread,
Nov 10, 2013, 5:11:34 PM11/10/13
to Slide, Jenkins Dev
In theory, to restore the master to the latest hash (and others branches - As I can see in Github organisation history) no --force will be required. You won't change the history, you just re-add missing commits on top of the old one.
--
-----
Arnaud Héritier
Mail/GTalk: aheritier AT gmail DOT com
Twitter/Skype : aheritier

Arnaud Héritier

unread,
Nov 10, 2013, 5:16:55 PM11/10/13
to jenkin...@googlegroups.com
I hope that Github will help us. That may save us a lot of time. Perhaps giving us a copy of all repositories could be enough and we may dig ourselves in reflogs to restore branches.
Yes that's really annoying that they don't allow us to block force push on some branches. Because of that when I migrated my company from subversion to Github I setup a daily external clone/backup process of all repositories in a case where we had such issue and we weren't easily able to identify a recent clone amonst ours developers (which is a children game in a company compared in an OSS like Jenkins)

Christopher Orr

unread,
Nov 10, 2013, 5:42:53 PM11/10/13
to jenkin...@googlegroups.com
With 678 members in the jenkinsci organisation, we're probably
relatively exceptional; I can't imagine GitHub have much incentive to
implement the type of fine-grained force-push permissions we would need.

Plus I think even with the number of members we have, git accidents are
very rare, i.e. it's likely easier to fix mistakes in git, than to
maintain a big set of permissions for new and existing users/repos.

BTW, out of curiosity, how did you manage to force push such a large
number of repositories at the same time? :)

Regards,
Chris


On 11/10/2013 10:40 PM, Luca Milanesio wrote:
> That's really pitty :-( ... force push are dangerous, especially if you
> don't have control over the Git Server.
>
> Typically recovering a force push is straightforward:
> 1. git reflog > look at the SHA-1 before the forced push
> 2. git branch -f <name> <sha-1>
>
> But if you don't have control over the Git repo on the Server, that you
> need to prevent force push to happen: unless they want everyone to buy
> GH Enterprise !
>
> I still hope GitHub will do 1. and 2. for us :-)
>
> Luca.
>
> On 10 Nov 2013, at 19:52, Marcus Bauer <mab...@gmail.com
> <mailto:mab...@gmail.com>> wrote:
>
>> Hi,
>>
>> I don't think GitHub.com <http://GitHub.com> has any possibilities for
>> disabling force pushes, this seems to be exclusive to GH Enterprise
>> <https://enterprise.github.com/help/articles/disable-force-pushes> only.
>>
>> The JaCoCo repo where I initially noticed this was restored by Dominik
>> Stadler (centic9) few minutes ago.
>>
>> Marcus
>>
>> Am Sonntag, 10. November 2013 19:55:08 UTC+1 schrieb lucamilanesio:
>>
>> Hi all,
>> I have triggered an involuntary "forced push" last night on the
>> list of Jenkins-CI plugins indicated below in this e-mail.
>>
>> *_My apology _*
>>
>> I did not realise that I actually had forced push permissions and
>> I do apologise for the inconvenience caused.
>> The operations pushed back the all the branches to around 1 month.
>> The history is not lost and is still on the GitHub server but on
>> detached branches.
>>
>> *_The solution_*
>> *_
>> _*
>> I can raise a request to GitHub to provide the "reflog" of those
>> repositories and restore the branches to the point before my
>> forced push.
>> /_Alternatively the owners of those repositories can still perform
>> a "forced push" to restore the correct position of the branches._/
>> (if you would like to do so, *_please write to the mailing list so
>> that we do not overlap the recovery operations_*)
>>
>> *_The full list_*
>> *_The prevention_*

Luca Milanesio

unread,
Nov 10, 2013, 6:24:09 PM11/10/13
to jenkin...@googlegroups.com
GitHub model is based on pull requests and not on shared Git repositories: that's why protections against forced push is typically not needed (typically repo is pushed by only its maintainers and not by contributors).
The way we use GitHub with Jenkins is not very typical for  them :-)

On 10 Nov 2013, at 22:42, Christopher Orr <ch...@orr.me.uk> wrote:

With 678 members in the jenkinsci organisation, we're probably relatively exceptional; I can't imagine GitHub have much incentive to implement the type of fine-grained force-push permissions we would need.

Plus I think even with the number of members we have, git accidents are very rare, i.e. it's likely easier to fix mistakes in git, than to maintain a big set of permissions for new and existing users/repos.

Yes they are rare ! ... but when they happen you need to sort them out :-(

BTW, out of curiosity, how did you manage to force push such a large number of repositories at the same time? :)

Problems with the Gerrit replication plugin ... unfortunate combination of three events (Murphy's law ?):
- working on a directory where I had the Jenkins plugins repos cloned (but outdated to around 1 month ago)
- having a Gerrit replication plugin pointing to that directory (I did not realised was using *forced push* that until later today)
- having the permissions of making *forced push* on repos that I did not work on (I did not realised that until today)

I have removed those local repos on my machine ... so can't do it again, but we need to sort out the mess in the meantime :-(
I am submitting now a patch to Gerrit replication plugin to *forbid* forced push by configuration: so at least I can block them from my side.

Possibly a bit late ... closing the gates when the sheep just escaped ;-(

Luca.

Christopher Orr

unread,
Nov 10, 2013, 6:42:46 PM11/10/13
to jenkin...@googlegroups.com
On 11/11/2013 12:24 AM, Luca Milanesio wrote:
>> BTW, out of curiosity, how did you manage to force push such a large
>> number of repositories at the same time? :)
>
> Problems with the Gerrit replication plugin ... unfortunate combination
> of three events (Murphy's law ?):
> - working on a directory where I had the Jenkins plugins repos cloned
> (but outdated to around 1 month ago)
> - having a Gerrit replication plugin pointing to that directory (I did
> not realised was using *forced push* that until later today)
> - having the permissions of making *forced push* on repos that I did not
> work on (I did not realised that until today)
>
> I have removed those local repos on my machine ... so can't do it again,
> but we need to sort out the mess in the meantime :-(
> I am submitting now a patch to Gerrit replication plugin to **forbid*
> forced push by configuration*: so at least I can block them from my side.

Yeah, that's unlucky.

One thing I sometimes do for repos where I have commit access, but don't
want to push, is to clone the repository via HTTP. That way, any
accidental attempts at pushing won't use my ssh key, and so GitHub will
prompt me for my password.

Regards,
Chris

Luca Milanesio

unread,
Nov 11, 2013, 3:11:17 AM11/11/13
to jenkin...@googlegroups.com
Does anyone have news from GitHub ?
They haven't responded at all to my customer support request :-(

Luca.

Vojtech Juranek

unread,
Nov 11, 2013, 3:15:03 AM11/11/13
to jenkin...@googlegroups.com
On Sunday 10 November 2013 21:40:28 Luca Milanesio wrote:
> That's really pitty :-( ... force push are dangerous, especially if you
> don't have control over the Git Server.

I wonder if we can use our all.git [1] somehow (in the worst case scenario
that github doesn't help us). When it try to clone it, it fails with remote
error and when looking into web UI the changes are already synchronized with
github. But IMHO still worth to investigate, orphan commits could still be
there

[1] http://git.jenkins-ci.org/?p=all.git;a=summary
signature.asc

Emanuele Zattin

unread,
Nov 11, 2013, 3:21:31 AM11/11/13
to jenkin...@googlegroups.com
Maybe somebody from the bay area could try driving to the Github office on Monday morning. Scott Chacon would probably be good bet.

Emanuele Zattin
---------------------------------------------------
-I don't have to know an answer. I don't feel frightened by not knowing things; by being lost in a mysterious universe without any purpose — which is the way it really is, as far as I can tell, possibly. It doesn't frighten me.- Richard Feynman

nicolas de loof

unread,
Nov 11, 2013, 4:05:30 AM11/11/13
to jenkin...@googlegroups.com, Arnaud Héritier
git fetch + git push restored git-client and git plugins history


2013/11/10 Luca Milanesio <luca.mi...@gmail.com>

Stephen Connolly

unread,
Nov 11, 2013, 4:09:04 AM11/11/13
to jenkin...@googlegroups.com, Arnaud Héritier
I had pushed releases of credentials, ssh-credentials, ssh-agent and ec2 on Friday, so I just re-pushed them. I don't think anyone made changes to scm-api or branch-api since my last push, so I pushed those from my local also

Mads Nielsen

unread,
Nov 11, 2013, 5:06:43 AM11/11/13
to jenkin...@googlegroups.com, Arnaud Héritier
What is the recommended way to get a full restore, in the case where you do not happen to have a local clone of the repository which was cloned before the accident?

Mads Nielsen
Consultant
Praqma A/S

Tel: +45 50 98 18 09
Mail: m...@praqma.net
web: www.praqma.net

nicolas de loof

unread,
Nov 11, 2013, 5:25:17 AM11/11/13
to jenkin...@googlegroups.com, Arnaud Héritier
in worst case, we have a clone on jenkins.ci.cloudbees.com we can use to push back to github.
just need some scripting to use it.

would you like me to look into this option ?


2013/11/11 Mads Nielsen <m...@praqma.net>

Arnaud Héritier

unread,
Nov 11, 2013, 5:36:17 AM11/11/13
to nicolas de loof, jenkin...@googlegroups.com
Hi Nicolas. 

  For me it would be the safer issue. You could wait for tomorrow (it's a day off today in France) but we just need to take care to not loose these repositories. 

Sent from Mailbox for iPhone


nicolas de loof

unread,
Nov 11, 2013, 5:38:50 AM11/11/13
to Arnaud Héritier, jenkin...@googlegroups.com
I can look into this while traveling to devoxx tomorrow


2013/11/11 Arnaud Héritier <aher...@gmail.com>

Luca Milanesio

unread,
Nov 11, 2013, 5:58:04 AM11/11/13
to jenkin...@googlegroups.com, nicolas de loof, Arnaud Héritier
Yes, I think that would be a sensible approach.
As some of the people started already to push their up-to-date repo, it is better a push from the CI node rather than a "reset" from the GitHub guys at this stage.

Luca.

Hakan Tandoğan

unread,
Nov 11, 2013, 7:16:43 AM11/11/13
to jenkin...@googlegroups.com, Arnaud Héritier
I pushed transifex-plugin.git from my local tree, all fine for this plugin now.

Regards,
Hakan

Mike Caspar

unread,
Nov 11, 2013, 7:22:58 AM11/11/13
to jenkin...@googlegroups.com
HI there,
I just tried to connect to check out the plugin I maintain (ironmq-notifier).
I have lost access to the repo completely.
I suspect that since I'm new, my rights were reverted due to this somehow.
Could someone add me back into the plugin ironmq-notifier.
Thanks

userid:mikecaspar

Mike Caspar

unread,
Nov 11, 2013, 7:24:59 AM11/11/13
to jenkin...@googlegroups.com
Actually, looks like it was just some weird caching problem.
Back into the repo now.

nicolas de loof

unread,
Nov 11, 2013, 9:09:29 AM11/11/13
to jenkin...@googlegroups.com, Arnaud Héritier
I've pushed plugin's master branches from jenkins.ci.cloudbees.com

some of then were rejected, like 

g...@github.com:jenkinsci/warnings-plugin.git

 ! [rejected]        master -> master (non-fast-forward)


Anyway I guess most plugin repo should be restored now.


2013/11/11 Hakan Tandoğan <hakan.t...@gmail.com>

Mads Nielsen

unread,
Nov 11, 2013, 9:16:32 AM11/11/13
to jenkin...@googlegroups.com, Arnaud Héritier
I can confirm that the ones i was interested in are OK: 


Thanks nicolas!

Vivekanand S V

unread,
Nov 11, 2013, 9:20:06 AM11/11/13
to jenkin...@googlegroups.com
This one https://github.com/jenkinsci/copy-to-slave-plugin recovered only some commits.

nicolas de loof

unread,
Nov 11, 2013, 9:26:14 AM11/11/13
to jenkin...@googlegroups.com
master branch is up-to-date with the one on jenkins.ci.cloudbees.com

do you have a local clone you could push from ?


2013/11/11 Vivekanand S V <svvive...@gmail.com>

oliver gondža

unread,
Nov 11, 2013, 9:28:18 AM11/11/13
to jenkin...@googlegroups.com, nicolas de loof
Thanks nicolas,

Could you list the repos that rejected the push and point us to the backup
repos so we can fix the rest manually? For instance
next-build-number-plugin is missing 6 commits compared to my local copy. I
assume this is not the only one.

--
oliver

Ulli Hafner

unread,
Nov 11, 2013, 9:33:52 AM11/11/13
to jenkin...@googlegroups.com
I already pushed my local commits again, so the history of the warnings plug-in is up to date again.
signature.asc

nicolas de loof

unread,
Nov 11, 2013, 9:37:36 AM11/11/13
to oliver gondža, jenkin...@googlegroups.com

Here are the repo that fail to git push as "[rejected]        master -> master (non-fast-forward)"


branch-api-plugin.git

credentials-plugin.git

deployit-plugin.git

ec2-cloud-axis-plugin.git

ec2-plugin.git

email-ext-plugin.git

flexible-publish-plugin.git

gerrit-trigger-plugin.git

git-client-plugin.git

git-plugin.git

ironmq-notifier-plugin.git

jacoco-plugin.git

jobConfigHistory-plugin.git

literate-plugin.git

maven-info-plugin.git

maven-plugin.git

perforce-plugin.git

rebuild-plugin.git

sounds-plugin.git

ssh-slaves-plugin.git

subversion-plugin.git

synergy_scm-plugin.git

testlink-plugin.git

tfs-plugin.git

transifex-plugin.git

warnings-plugin.git












2013/11/11 oliver gondža <ogo...@gmail.com>

Slide

unread,
Nov 11, 2013, 9:40:35 AM11/11/13
to Jenkins Dev, oliver gondža
I already pushed the correct stuff to email-ext-plugin.git, so that one is done.

Regards,

slide


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Iñaki Albors

unread,
Nov 11, 2013, 10:09:37 AM11/11/13
to jenkinsci-dev
next-executions-plugin restored

Vivekanand S V

unread,
Nov 11, 2013, 10:16:44 AM11/11/13
to jenkin...@googlegroups.com
@nicolas, I have a local clone which I last used to push a release. What do I need to do ?

Stefan Brausch

unread,
Nov 11, 2013, 10:32:33 AM11/11/13
to jenkin...@googlegroups.com
logfilesizechecker and scm2job restored from local copy  (wasn’t complete)

nicolas de loof

unread,
Nov 11, 2013, 10:56:33 AM11/11/13
to jenkin...@googlegroups.com
just git push origin master

the unintentional force push just reverted history some weeks ago, so this will re-add the missing commits


2013/11/11 Vivekanand S V <svvive...@gmail.com>
@nicolas, I have a local clone which I last used to push a release. What do I need to do ?

Vivekanand S V

unread,
Nov 11, 2013, 11:09:12 AM11/11/13
to jenkin...@googlegroups.com
@nicolas, its done, copy-to-slave is upto date now.


Thanks a lot.

Vojtech Juranek

unread,
Nov 11, 2013, 2:04:49 PM11/11/13
to jenkin...@googlegroups.com
Hi,
just FIY, if your plugin is not on this list, this doesn't mean the commits
were successfully restored - e.g. one of my plugins (beaker-builder) is not on
the list and the last commit is 3 months old [1], while actual last commit is
one week old [2], so please double check your plugins
Vojta

[1] https://github.com/jenkinsci/beaker-builder-plugin/commits/master
[2] https://github.com/vjuranek/beaker-builder-plugin/commits/master
> 2013/11/11 oliver gondža <ogo...@gmail.com>
signature.asc

nicolas de loof

unread,
Nov 11, 2013, 2:17:30 PM11/11/13
to jenkin...@googlegroups.com
yes indeed, this is just the last state on CI job, no guarantee it hasn't been updated in the meantime based on github forced push


2013/11/11 Vojtech Juranek <vjur...@redhat.com>

Mirko Friedenhagen

unread,
Nov 11, 2013, 3:00:53 PM11/11/13
to jenkin...@googlegroups.com
I pushed my local copy of jobConfigHistory-plugin without force and
all commits are current now.
Regards Mirko
--
http://illegalstateexception.blogspot.com/
https://github.com/mfriedenhagen/
https://bitbucket.org/mfriedenhagen/

Rob Petti

unread,
Nov 11, 2013, 3:09:24 PM11/11/13
to jenkin...@googlegroups.com, Arnaud Héritier
perforce-plugin.git has been restored from my local copy.

It may be beneficial to look into implementing finer grained access control for individual repos. For example, I thought that only myself, Oleg, and the Jenkins project owners had write access to the perforce plugin. If everyone has access to everything, then this sort of thing could happen again.

On Sunday, 10 November 2013 11:55:08 UTC-7, lucamilanesio wrote:
Hi all,
I have triggered an involuntary "forced push" last night on the list of Jenkins-CI plugins indicated below in this e-mail.

My apology 

I did not realise that I actually had forced push permissions and I do apologise for the inconvenience caused.
The operations pushed back the all the branches to around 1 month. The history is not lost and is still on the GitHub server but on detached branches.

The solution

I can raise a request to GitHub to provide the "reflog" of those repositories and restore the branches to the point before my forced push.
Alternatively the owners of those repositories can still perform a "forced push" to restore the correct position of the branches.
(if you would like to do so, please write to the mailing list so that we do not overlap the recovery operations)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Kohsuke Kawaguchi

unread,
Nov 11, 2013, 3:19:43 PM11/11/13
to jenkin...@googlegroups.com
On 11/10/2013 10:55 AM, Luca Milanesio wrote:
> Hi all,
> I have triggered an involuntary "forced push" last night on the list of
> Jenkins-CI plugins indicated below in this e-mail.
>
> *_My apology _*
>
> I did not realise that I actually had forced push permissions and I do
> apologise for the inconvenience caused.
> The operations pushed back the all the branches to around 1 month. The
> history is not lost and is still on the GitHub server but on detached
> branches.
>
> *_The solution_*
> *_
> _*
> I can raise a request to GitHub to provide the "reflog" of those
> repositories and restore the branches to the point before my forced push.
> /_Alternatively the owners of those repositories can still perform a
> "forced push" to restore the correct position of the branches._/
> (if you would like to do so, *_please write to the mailing list so that
> we do not overlap the recovery operations_*)

If they can just create refs for all the tips of the affected branches
before a forced push in a distinct names, then we can write a script
that compares that with the current tip of the branches (which may or
may not be fixed by someone manually already) and do the right thing.



>
> *_The full list_*
> *_The prevention_*
>
> Can we prevent this to happen again ?
> I personally do not work on any of those repositories but still have
> "forced push" permissions ... and so many other people have.
> I don't see the value of having such power of "potential disruption"
> associated to my account :-( ... can we remove the forced push by
> default and enable on a case-by-case basis ?
>
> --- * ---
>
> If you would like to propose an alternative approach to resolve the
> problem, feel free to follow-up !
>
> ... and again ... accept my sincere apologies :-(
>
> Luca.
>
> On 10 Nov 2013, at 18:24, Arnaud H�ritier <aher...@gmail.com
> <mailto:aher...@gmail.com>> wrote:
>
>> I confirm we lost many commits. damned github that doesn't allow to
>> disable the history rewrite at least on master branch.
>> Last commit on Xcode repo is referencing one done by Jerome Lacoste 5
>> montages ago while KK worked on it recently.
>> I don't know how we can restore everything easily. We need to find the
>> recent hash and relative history in a clone we the push of yesterday
>> was retrieved.
>>
>> �
>> Sent from Mailbox <https://www.dropbox.com/mailbox> for iPhone
>>
>>
>> On Sun, Nov 10, 2013 at 5:51 PM, domi <do...@fortysix.ch
>> <mailto:do...@fortysix.ch>> wrote:
>>
>> As I don�t know if Luca is following this list too, I�m adding
>> Lucas mail on the to list� (got the email address from his GitHub
>> account)
>> Please Luca can you explain what you have done to all this
>> repositories?
>> regards Domi
>>
>>
>> On 10.11.2013, at 17:25, ogondza <ogo...@gmail.com
>> <mailto:ogo...@gmail.com>> wrote:
>>
>>> Definitely related, selenium-tests repo lost 17+ commits after
>>> such push.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the
>>> Google Groups "Jenkins Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to jenkinsci-de...@googlegroups.com
>>> <mailto:jenkinsci-de...@googlegroups.com>.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Jenkins Developers" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to jenkinsci-de...@googlegroups.com
>> <mailto:jenkinsci-de...@googlegroups.com>.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-de...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.


--
Kohsuke Kawaguchi http://kohsuke.org/

Kohsuke Kawaguchi

unread,
Nov 11, 2013, 3:23:24 PM11/11/13
to jenkin...@googlegroups.com

Yes, I was thinking about the same. The last run of this is Nov 9,
11:24pm in EST.

I really hope this is before the incident. But I'll find out soon.

On 11/11/2013 12:15 AM, Vojtech Juranek wrote:
> On Sunday 10 November 2013 21:40:28 Luca Milanesio wrote:
>> That's really pitty :-( ... force push are dangerous, especially if you
>> don't have control over the Git Server.
>
> I wonder if we can use our all.git [1] somehow (in the worst case scenario
> that github doesn't help us). When it try to clone it, it fails with remote
> error and when looking into web UI the changes are already synchronized with
> github. But IMHO still worth to investigate, orphan commits could still be
> there
>
> [1] http://git.jenkins-ci.org/?p=all.git;a=summary
>


--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Jenkins Enterprise, our professional version of Jenkins

Andrew Bayer

unread,
Nov 11, 2013, 3:25:57 PM11/11/13
to jenkin...@googlegroups.com
fwiw, gradle-jpi-plugin, throttle-concurrent-builds-plugin, jclouds-plugin and associated-files-plugin are all up to date now.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

Kohsuke Kawaguchi

unread,
Nov 11, 2013, 4:22:29 PM11/11/13
to jenkin...@googlegroups.com, oliver gondža

You should push 'master' in these repos to a separate ref so that we can
compare.
> 2013/11/11 oliver gondža <ogo...@gmail.com <mailto:ogo...@gmail.com>>
>
> Thanks nicolas,
>
> Could you list the repos that rejected the push and point us to the
> backup repos so we can fix the rest manually? For instance
> next-build-number-plugin is missing 6 commits compared to my local
> copy. I assume this is not the only one.
>
> --
> oliver
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-de...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.


Kohsuke Kawaguchi

unread,
Nov 11, 2013, 5:08:19 PM11/11/13
to jenkin...@googlegroups.com
On 11/11/2013 12:23 PM, Kohsuke Kawaguchi wrote:
>
> Yes, I was thinking about the same. The last run of this is Nov 9,
> 11:24pm in EST.
>
> I really hope this is before the incident. But I'll find out soon.

Unfortunately, it appears that the last sync process has run after
luca's "push -f".

I'll hold on to this repo just in case, but resurrecting lost commits
from this appears hopeless.

Luca Milanesio

unread,
Nov 11, 2013, 5:22:55 PM11/11/13
to jenkin...@googlegroups.com
Good news from GitHub: they have extracted the full list of SHA-1 before the forced push !
Many thanks to Nathan Witmer :-)

See below the full CSV with the SHA-1.
He created as well a branch named 'recovery' that points to the candidate point for restoring the master branch.

Hope this will help to sort out the remaining repos.

Luca.

> Hi Luca.
>
> Oh man, that sinking feeling!
>
> But, no worries: I've gone through each of the repos listed above and done the following:
>
> * retrieved the SHA for the previous `master` before you force-pushed
> * created a branch called `recovery` pointing to each former master
>
> In some cases, these are the same.
>
> I can go further and reset the master refs to their former shas if you'd like, or you can recover these yourself. To do so, in each repo:
>
> $ git fetch
> $ git checkout master
> $ git reset --hard origin/recovery
> $ git push --force origin master
>
> I've attached a csv containing the shas (former master and forced master) for each branch, for your reference.
>
> Good luck!
>
> Nathan.
jenkinsci_restore.csv

Kohsuke Kawaguchi

unread,
Nov 11, 2013, 5:27:14 PM11/11/13
to jenkin...@googlegroups.com
We at the #jenkins channel in IRC were just wondering what the 'recovery' branch is.

I think it makes sense to bring back 'master' to 'recovery' where it is fast-forward, but let's walk bit slowly here...


2013/11/11 Luca Milanesio <luca.mi...@gmail.com>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.




--
Kohsuke Kawaguchi

Kohsuke Kawaguchi

unread,
Nov 11, 2013, 5:46:48 PM11/11/13