strange pushes on GitHub

[domi]

Hi everyone and specially Luca,

yesterday something strange has happen to many Jenkins repositories on GitHub (more then 50 Repos)…
Luca Milanesio seems have pushed to many many repositories without really changing anything - at least I have not seen anything changed.
If you look at the activity stream/feed at https://github.com/organizations/jenkinsci and go back to yesterday, you’ll find many pushes to many different repositories.
Maybe I’m wrong and someone has an easy explanation for this?
But for me this looks like someone did not really now how Git/GitHub works and accidentally pushed to every single repo - maybe someone else could also have a look and see if there is something broken now?

regards Domi

[Marcelo]

Domi, maybe it's related to this mail http://jenkins-ci.361315.n4.nabble.com/Jacoco-plugin-repo-reset-tt4680030.html.

Regards

Marcelo Rebasti

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Vincent Latombe]

It seems like a lot of repositories are back to previous revisions. git-client-plugin's master now points to 1.4.3-SNAPSHOT whereas it should be pointing to 1.4.7-SNAPSHOT...

Vincent

2013/11/10 Marcelo <mreb...@gmail.com>

[ogondza]

Definitely related, selenium-tests repo lost 17+ commits after such push.

[domi]

As I don’t know if Luca is following this list too, I’m adding Lucas mail on the to list… (got the email address from his GitHub account)

Please Luca can you explain what you have done to all this repositories?

regards Domi

On 10.11.2013, at 17:25, ogondza <ogo...@gmail.com> wrote:

Definitely related, selenium-tests repo lost 17+ commits after such push.

[Arnaud Héritier]

I confirm we lost many commits. damned github that doesn't allow to disable the history rewrite at least on master branch. 

Last commit on Xcode repo is referencing one done by Jerome Lacoste 5 montages ago while KK worked on it recently. 

I don't know how we can restore everything easily. We need to find the recent hash and relative history in a clone we the push of yesterday was retrieved. 

—
Sent from Mailbox for iPhone

[Arnaud Héritier]

S/montages/monthes

S/we/with

Stupid dictionary 

—
Sent from Mailbox for iPhone

[Luca Milanesio]

Hi all,

I have triggered an involuntary "forced push" last night on the list of Jenkins-CI plugins indicated below in this e-mail.

My apology 

I did not realise that I actually had forced push permissions and I do apologise for the inconvenience caused.

The operations pushed back the all the branches to around 1 month. The history is not lost and is still on the GitHub server but on detached branches.

The solution

I can raise a request to GitHub to provide the "reflog" of those repositories and restore the branches to the point before my forced push.

Alternatively the owners of those repositories can still perform a "forced push" to restore the correct position of the branches.

(if you would like to do so, please write to the mailing list so that we do not overlap the recovery operations)

The full list

See below the full list of repositories impacted:

antexec-plugin.git

artifactory-plugin.git

associated-files-plugin.git

audit2db-plugin.git

audit-trail-plugin.git

backend-pull-request-greeter.git

beaker-builder-plugin.git

branch-api-plugin.git

build-flow-plugin.git

buildgraph-view.git

build-pipeline-plugin.git

build-timeout-plugin.git

buildtriggerbadge-plugin.git

bytecode-compatibility-transformer.git

ci-game-plugin.git

clearcase-plugin.git

clearcase-ucm-plugin.git

cloudbees-folder-plugin.git

cloudbees-plugin-gateway.git

cloudtest-plugin.git

clover-plugin.git

cobertura-plugin.git

collabnet-plugin.git

collapsing-console-sections-plugin.git

compact-columns-plugin.git

compress-artifacts-plugin.git

conditional-buildstep-plugin.git

config-file-provider-plugin.git

configurationslicing-plugin.git

copyartifact-plugin.git

copy-project-link-plugin.git

copy-to-slave-plugin.git

cppcheck-plugin.git

credentials-plugin.git

crowd2-plugin.git

crowd-plugin.git

customtools-plugin.git

cvsclient.git

cvs-plugin.git

dashboard-view-plugin.git

datical-db-plugin.git

dependency-check-plugin.git

deploy-plugin.git

disable-failed-job-plugin.git

disk-usage-plugin.git

doclinks-plugin.git

dry-plugin.git

dynamic-axis-plugin.git

ec2-plugin.git

elastic-axis-plugin.git

email-ext-plugin.git

envinject-lib.git

envinject-plugin.git

extended-choice-parameter-plugin.git

extra-columns-plugin.git

extras-executable-war.git

extreme-feedback-plugin.git

gearman-plugin.git

gerrit-trigger-plugin.git

gitbucket-plugin.git

git-chooser-alternative-plugin.git

git-client-plugin.git

git-plugin.git

global-build-stats-plugin.git

global-variable-string-parameter-plugin.git

gradle-jpi-plugin.git

grails-plugin.git

greenballs-plugin.git

groovy-postbuild-plugin.git

heavy-job-plugin.git

hockeyapp-plugin.git

http-request-plugin.git

humbug-plugin.git

instant-messaging-plugin.git

integrity-plugin.git

ironmq-notifier-plugin.git

ivytrigger-plugin.git

jacoco-plugin.git

jclouds-plugin.git

jira-plugin.git

jobConfigHistory-plugin.git

job-dsl-plugin.git

job-import-plugin.git

job-poll-action-plugin.git

jquery-plugin.git

jquery-ui-plugin.git

json-lib.git

kiuwan-plugin.git

label-verifier-plugin.git

ldap-plugin.git

leiningen-plugin.git

lib-annotation-indexer.git

lib-task-reactor.git

lib-windows-remote-command.git

literate-cli.git

logfilesizechecker-plugin.git

m2release-plugin.git

m2-repo-reaper-plugin.git

mailer-plugin.git

matrix-auth-plugin.git

maven-hpi-plugin.git

maven-info-plugin.git

mercurial-plugin.git

mesos-plugin.git

metadata-plugin.git

mock-security-realm-plugin.git

msbuild-plugin.git

naginator-plugin.git

nerrvana-plugin.git

nested-view-plugin.git

next-build-number-plugin.git

next-executions-plugin.git

parameterized-trigger-plugin.git

perforce-plugin.git

performance-plugin.git

persona-plugin.git

pitmutation-plugin.git

plain-credentials-plugin.git

plugin-compat-tester.git

postbuildscript-plugin.git

promoted-builds-plugin.git

prqa-plugin.git

publish-over-cifs-plugin.git

puppet-jenkins.git

radiatorview-plugin.git

rapiddeploy-plugin.git

release-plugin.git

repo-plugin.git

rich-text-publisher-plugin.git

robot-plugin.git

run-condition-plugin.git

rvm-plugin.git

scm2job-plugin.git

scm-api-plugin.git

scoring-load-balancer-plugin.git

script-scm-plugin.git

selenium-axis-plugin.git

selenium-builder-plugin.git

selenium-tests.git

skype-im-plugin.git

skytap-cloud-plugin.git

smartfrog-plugin.git

sms-plugin.git

sounds-plugin.git

ssh-agent-plugin.git

ssh-credentials-plugin.git

sshd-module.git

ssh-slaves-plugin.git

starteam-plugin.git

stashnotifier-plugin.git

subversion-plugin.git

suppress-stack-trace-plugin.git

swarm-plugin.git

synergy_scm-plugin.git

tap-plugin.git

teamconcert-plugin.git

testlink-plugin.git

tfs-plugin.git

thin-backup-plugin.git

throttle-concurrent-builds-plugin.git

tikal-multijob-plugin.git

timestamper-plugin.git

token-macro-plugin.git

transifex-plugin.git

translation-plugin.git

trilead-ssh2.git

unity3d-plugin.git

veracode-scanner-plugin.git

view-job-filters-plugin.git

virtualbox-plugin.git

vsphere-cloud-plugin.git

vstestrunner-plugin.git

walldisplay-plugin.git

warnings-plugin.git

weblogic-deployer-plugin.git

winstone.git

wix-plugin.git

ws-cleanup-plugin.git

xcode-plugin.git

xstream.git

xtrigger-lib.git

xunit-plugin.git

xvfb-plugin.git

xvnc-plugin.git

The prevention

Can we prevent this to happen again ?

I personally do not work on any of those repositories but still have "forced push" permissions ... and so many other people have.

I don't see the value of having such power of "potential disruption" associated to my account :-( ... can we remove the forced push by default and enable on a case-by-case basis ?

--- * ---

If you would like to propose an alternative approach to resolve the problem, feel free to follow-up !

... and again ... accept my sincere apologies :-(

Luca.

[ogondza]

I have already asked Github support to provide hashes of former master positions and referenced this thread. Will report ASAP.

[Luca Milanesio]

*UPDATE*

I have just send the request to GitHub support on-line, looping them as well in this e-mail.

@GitHub Support: please consider this request as a matter of urgency, thank you.

Luca.

[Arnaud Héritier]

Thx guys for these details. We are all humans and subject of doing errors. 

I hope we'll have some help from github but I'm not sure they may spend many time to help us for an error not coming from their side. 

In any case it is recommended to NOT PULL current master branches from these repositories, your local copy may be more up-to-date. If you have a recent hash you can at least try to push them in github under a branch named "master-fix" the time we find the best solution to repare all master repositories. 

If you commited on these repositories since yesterday verify that you didn't started your work from the wrong old hash. 

Idea : in our CI server logs we should be able to find the lastest hash of each repo in the related plugin build. After that we have to find how to restore them. It's easy to do it when you have this commit locally but AFAIK it's not if it is only in the remote (you don't get orphelin commits when you clone a repo)

Arnaud

—
Sent from Mailbox for iPhone

[Luca Milanesio]

*UPDATE* after a cross-check I need to add 2 extra repositories to the list:

accurev-plugin
analysis-core-plugin

The total number of impacted repositories should then be: 186

@ALL: is there anyone with direct contact of some GitHub-er ? It would be best to resolve the problem *before* the start of the working week on Monday.

Thanks to everyone that contributed to resolve the problem !

Luca.

[Luca Milanesio]

Hi Arnaud,
thanks for the understanding :-) I am used to warn other people from doing any "forced push" ... and Murphy's law wanted this time happening to me :-(

On 10 Nov 2013, at 19:30, Arnaud Héritier <aher...@gmail.com> wrote:

> Thx guys for these details. We are all humans and subject of doing errors.
> I hope we'll have some help from github but I'm not sure they may spend many time to help us for an error not coming from their side.

Technically yes, but for the "customer relationship" and because it is an easy recovery (git reflog + git branch), they could make some effort to help us out.
JenkinsCI is possibly the largest customer they have worldwide ;-)

> In any case it is recommended to NOT PULL current master branches from these repositories, your local copy may be more up-to-date. If you have a recent hash you can at least try to push them in github under a branch named "master-fix" the time we find the best solution to repare all master repositories.
> If you commited on these repositories since yesterday verify that you didn't started your work from the wrong old hash.
>
> Idea : in our CI server logs we should be able to find the lastest hash of each repo in the related plugin build. After that we have to find how to restore them. It's easy to do it when you have this commit locally but AFAIK it's not if it is only in the remote (you don't get orphelin commits when you clone a repo)

Yes, that's what I thought as well ... but definitely the "git reflog" on GitHub server would be more reliable.
The CI workspace (assuming that Git clones are not shallow copies) may have as well those commits as well, but again GitHub would be safer and better IMHO.

Luca

[Marcus Bauer]

Hi,

I don't think GitHub.com has any possibilities for disabling force pushes, this seems to be exclusive to GH Enterprise only.

The JaCoCo repo where I initially noticed this was restored by Dominik Stadler (centic9) few minutes ago.

Marcus

[Mirko Friedenhagen]

Hello,

I saw these strange pushes as well, jobConfigHistory-plugin was
failing because master was reset to a commit which was two months old.
I see that https://github.com/jenkinsci/maven-hpi-plugin/tree/master
is pointing to an old commit as well, the new version 1.106 is much
newer than master.

Regards
Mirko
Regards Mirko
--
http://illegalstateexception.blogspot.com/
https://github.com/mfriedenhagen/
https://bitbucket.org/mfriedenhagen/

[Luca Milanesio]

That's really pitty :-( ... force push are dangerous, especially if you don't have control over the Git Server.

Typically recovering a force push is straightforward:

1. git reflog > look at the SHA-1 before the forced push

2. git branch -f <name> <sha-1>

But if you don't have control over the Git repo on the Server, that you need to prevent force push to happen: unless they want everyone to buy GH Enterprise !

I still hope GitHub will do 1. and 2. for us :-)

Luca.

[Dominik Stadler]

Hi,

I didn't know that this was a bigger problem and thus just pushed the previous commits again, in case of the jacoco-plugin repo it was easy as I just committed changes yesterday and thus could easily push the latest changes together with the missing commits.

The really bad thing in my view is that as a user of the repo you do not see at all that it happened. Git and Github is all about history of changes, but in this case your repo looks like a month ago with no visible trace on github that something newer was there at some time... Quite dangerous....

Dominik.

[Slide]

I did a force push to email-ext and it seems to be fine now.

Thanks,

slide

Website: http://earl-of-code.com

[Arnaud Héritier]

In theory, to restore the master to the latest hash (and others branches - As I can see in Github organisation history) no --force will be required. You won't change the history, you just re-add missing commits on top of the old one.

--

-----

Arnaud Héritier

http://aheritier.net

Mail/GTalk: aheritier AT gmail DOT com

Twitter/Skype : aheritier

[Arnaud Héritier]

I hope that Github will help us. That may save us a lot of time. Perhaps giving us a copy of all repositories could be enough and we may dig ourselves in reflogs to restore branches.

Yes that's really annoying that they don't allow us to block force push on some branches. Because of that when I migrated my company from subversion to Github I setup a daily external clone/backup process of all repositories in a case where we had such issue and we weren't easily able to identify a recent clone amonst ours developers (which is a children game in a company compared in an OSS like Jenkins)

[Christopher Orr]

With 678 members in the jenkinsci organisation, we're probably
relatively exceptional; I can't imagine GitHub have much incentive to
implement the type of fine-grained force-push permissions we would need.

Plus I think even with the number of members we have, git accidents are
very rare, i.e. it's likely easier to fix mistakes in git, than to
maintain a big set of permissions for new and existing users/repos.

BTW, out of curiosity, how did you manage to force push such a large
number of repositories at the same time? :)

Regards,
Chris

On 11/10/2013 10:40 PM, Luca Milanesio wrote:
> That's really pitty :-( ... force push are dangerous, especially if you
> don't have control over the Git Server.
>
> Typically recovering a force push is straightforward:
> 1. git reflog > look at the SHA-1 before the forced push
> 2. git branch -f <name> <sha-1>
>
> But if you don't have control over the Git repo on the Server, that you
> need to prevent force push to happen: unless they want everyone to buy
> GH Enterprise !
>
> I still hope GitHub will do 1. and 2. for us :-)
>
> Luca.
>
> On 10 Nov 2013, at 19:52, Marcus Bauer <mab...@gmail.com

> <mailto:mab...@gmail.com>> wrote:
>
>> Hi,
>>
>> I don't think GitHub.com <http://GitHub.com> has any possibilities for

>> disabling force pushes, this seems to be exclusive to GH Enterprise

>> <https://enterprise.github.com/help/articles/disable-force-pushes> only.

>>
>> The JaCoCo repo where I initially noticed this was restored by Dominik
>> Stadler (centic9) few minutes ago.
>>
>> Marcus
>>
>> Am Sonntag, 10. November 2013 19:55:08 UTC+1 schrieb lucamilanesio:
>>
>> Hi all,
>> I have triggered an involuntary "forced push" last night on the
>> list of Jenkins-CI plugins indicated below in this e-mail.
>>

>> *_My apology _*

>>
>> I did not realise that I actually had forced push permissions and
>> I do apologise for the inconvenience caused.
>> The operations pushed back the all the branches to around 1 month.
>> The history is not lost and is still on the GitHub server but on
>> detached branches.
>>

>> *_The solution_*
>> *_
>> _*

>> I can raise a request to GitHub to provide the "reflog" of those
>> repositories and restore the branches to the point before my
>> forced push.

>> /_Alternatively the owners of those repositories can still perform
>> a "forced push" to restore the correct position of the branches._/
>> (if you would like to do so, *_please write to the mailing list so
>> that we do not overlap the recovery operations_*)
>>
>> *_The full list_*

>> *_The prevention_*

[Luca Milanesio]

GitHub model is based on pull requests and not on shared Git repositories: that's why protections against forced push is typically not needed (typically repo is pushed by only its maintainers and not by contributors).

The way we use GitHub with Jenkins is not very typical for  them :-)

On 10 Nov 2013, at 22:42, Christopher Orr <ch...@orr.me.uk> wrote:

With 678 members in the jenkinsci organisation, we're probably relatively exceptional; I can't imagine GitHub have much incentive to implement the type of fine-grained force-push permissions we would need.

Plus I think even with the number of members we have, git accidents are very rare, i.e. it's likely easier to fix mistakes in git, than to maintain a big set of permissions for new and existing users/repos.

Yes they are rare ! ... but when they happen you need to sort them out :-(

BTW, out of curiosity, how did you manage to force push such a large number of repositories at the same time? :)

Problems with the Gerrit replication plugin ... unfortunate combination of three events (Murphy's law ?):

- working on a directory where I had the Jenkins plugins repos cloned (but outdated to around 1 month ago)

- having a Gerrit replication plugin pointing to that directory (I did not realised was using *forced push* that until later today)

- having the permissions of making *forced push* on repos that I did not work on (I did not realised that until today)

I have removed those local repos on my machine ... so can't do it again, but we need to sort out the mess in the meantime :-(

I am submitting now a patch to Gerrit replication plugin to *forbid* forced push by configuration: so at least I can block them from my side.

Possibly a bit late ... closing the gates when the sheep just escaped ;-(

Luca.

[Christopher Orr]

On 11/11/2013 12:24 AM, Luca Milanesio wrote:
>> BTW, out of curiosity, how did you manage to force push such a large
>> number of repositories at the same time? :)
>
> Problems with the Gerrit replication plugin ... unfortunate combination
> of three events (Murphy's law ?):
> - working on a directory where I had the Jenkins plugins repos cloned
> (but outdated to around 1 month ago)
> - having a Gerrit replication plugin pointing to that directory (I did
> not realised was using *forced push* that until later today)
> - having the permissions of making *forced push* on repos that I did not
> work on (I did not realised that until today)
>
> I have removed those local repos on my machine ... so can't do it again,
> but we need to sort out the mess in the meantime :-(

> I am submitting now a patch to Gerrit replication plugin to **forbid*
> forced push by configuration*: so at least I can block them from my side.

Yeah, that's unlucky.

One thing I sometimes do for repos where I have commit access, but don't
want to push, is to clone the repository via HTTP. That way, any
accidental attempts at pushing won't use my ssh key, and so GitHub will
prompt me for my password.

Regards,
Chris

[Luca Milanesio]

Does anyone have news from GitHub ?
They haven't responded at all to my customer support request :-(

Luca.

[Vojtech Juranek]

On Sunday 10 November 2013 21:40:28 Luca Milanesio wrote:
> That's really pitty :-( ... force push are dangerous, especially if you
> don't have control over the Git Server.

I wonder if we can use our all.git [1] somehow (in the worst case scenario
that github doesn't help us). When it try to clone it, it fails with remote
error and when looking into web UI the changes are already synchronized with
github. But IMHO still worth to investigate, orphan commits could still be
there

[1] http://git.jenkins-ci.org/?p=all.git;a=summary

[Emanuele Zattin]

Maybe somebody from the bay area could try driving to the Github office on Monday morning. Scott Chacon would probably be good bet.

Emanuele Zattin
---------------------------------------------------
-I don't have to know an answer. I don't feel frightened by not knowing things; by being lost in a mysterious universe without any purpose — which is the way it really is, as far as I can tell, possibly. It doesn't frighten me.- Richard Feynman

[nicolas de loof]

git fetch + git push restored git-client and git plugins history

2013/11/10 Luca Milanesio <luca.mi...@gmail.com>

[Stephen Connolly]

I had pushed releases of credentials, ssh-credentials, ssh-agent and ec2 on Friday, so I just re-pushed them. I don't think anyone made changes to scm-api or branch-api since my last push, so I pushed those from my local also

[Mads Nielsen]

What is the recommended way to get a full restore, in the case where you do not happen to have a local clone of the repository which was cloned before the accident?

Mads Nielsen Consultant Praqma A/S Tel: +45 50 98 18 09 Mail: m...@praqma.net web: www.praqma.net

[nicolas de loof]

in worst case, we have a clone on jenkins.ci.cloudbees.com we can use to push back to github.

just need some scripting to use it.

would you like me to look into this option ?

2013/11/11 Mads Nielsen <m...@praqma.net>

[Arnaud Héritier]

Hi Nicolas. 

  For me it would be the safer issue. You could wait for tomorrow (it's a day off today in France) but we just need to take care to not loose these repositories. 

—
Sent from Mailbox for iPhone

[nicolas de loof]

I can look into this while traveling to devoxx tomorrow

2013/11/11 Arnaud Héritier <aher...@gmail.com>

[Luca Milanesio]

Yes, I think that would be a sensible approach.

As some of the people started already to push their up-to-date repo, it is better a push from the CI node rather than a "reset" from the GitHub guys at this stage.

Luca.

[Hakan Tandoğan]

I pushed transifex-plugin.git from my local tree, all fine for this plugin now.

Regards,

Hakan