Thu Mar 28 12:59:34.304 [mongosMain] MongoS version 2.4.1 starting: pid=9218 port=30000 64-bit host=
host15.somedomain.com (--help for usage)
Thu Mar 28 12:59:34.304 [mongosMain] git version: 1560959e9ce11a693be8b4d0d160d633eee75110
Thu Mar 28 12:59:34.304 [mongosMain] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
ue }
ms)
Thu Mar 28 12:59:45.807 [mongosMain] waited 11s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 12:59:56.846 [mongosMain] waited 22s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:00:07.890 [mongosMain] waited 33s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:00:18.931 [mongosMain] waited 44s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:00:29.970 [mongosMain] waited 55s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:00:41.008 [mongosMain] waited 66s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:00:52.060 [mongosMain] waited 77s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:01:03.101 [mongosMain] waited 88s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:01:14.141 [mongosMain] waited 99s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:01:25.207 [mongosMain] waited 110s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:01:36.246 [mongosMain] waited 121s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:01:47.293 [mongosMain] waited 132s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:01:58.333 [mongosMain] waited 143s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:02:09.372 [mongosMain] waited 154s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:02:20.465 [mongosMain] waited 165s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:02:31.505 [mongosMain] waited 176s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:02:42.544 [mongosMain] waited 187s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:02:53.583 [mongosMain] waited 198s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:03:04.621 [mongosMain] waited 209s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:03:15.658 [mongosMain] waited 220s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:03:26.698 [mongosMain] waited 231s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:03:37.738 [mongosMain] waited 242s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:03:48.776 [mongosMain] waited 254s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:03:59.814 [mongosMain] waited 265s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:04:10.854 [mongosMain] waited 276s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:04:21.893 [mongosMain] waited 287s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:04:32.931 [mongosMain] waited 298s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:04:43.969 [mongosMain] waited 309s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:04:55.009 [mongosMain] waited 320s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:05:06.049 [mongosMain] waited 331s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:05:17.087 [mongosMain] waited 342s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:05:28.125 [mongosMain] waited 353s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:05:39.164 [mongosMain] waited 364s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:05:50.206 [mongosMain] waited 375s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:06:01.244 [mongosMain] waited 386s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:06:12.282 [mongosMain] waited 397s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:06:23.321 [mongosMain] waited 408s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:06:34.360 [mongosMain] waited 419s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:06:45.401 [mongosMain] waited 430s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:06:56.440 [mongosMain] waited 441s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:07:07.480 [mongosMain] waited 452s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:07:18.520 [mongosMain] waited 463s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:07:29.560 [mongosMain] waited 474s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:07:40.600 [mongosMain] waited 485s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:07:51.639 [mongosMain] waited 496s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:08:02.677 [mongosMain] waited 507s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:08:13.716 [mongosMain] waited 518s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:08:24.755 [mongosMain] waited 529s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:08:35.794 [mongosMain] waited 541s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:08:46.834 [mongosMain] waited 552s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:08:57.872 [mongosMain] waited 563s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:09:08.912 [mongosMain] waited 574s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:09:19.956 [mongosMain] waited 585s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:09:30.996 [mongosMain] waited 596s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:09:42.037 [mongosMain] waited 607s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:09:53.076 [mongosMain] waited 618s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:10:04.117 [mongosMain] waited 629s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:10:15.157 [mongosMain] waited 640s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:10:26.197 [mongosMain] waited 651s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:10:37.237 [mongosMain] waited 662s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:10:48.275 [mongosMain] waited 673s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:10:59.315 [mongosMain] waited 684s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:11:10.355 [mongosMain] waited 695s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:11:21.395 [mongosMain] waited 706s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:11:32.434 [mongosMain] waited 717s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:11:43.473 [mongosMain] waited 728s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:11:54.514 [mongosMain] waited 739s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:12:05.554 [mongosMain] waited 750s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:12:16.592 [mongosMain] waited 761s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:12:27.631 [mongosMain] waited 772s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:12:38.671 [mongosMain] waited 783s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:12:49.711 [mongosMain] waited 794s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:13:00.753 [mongosMain] waited 805s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:13:11.791 [mongosMain] waited 817s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:13:22.831 [mongosMain] waited 828s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:13:33.869 [mongosMain] waited 839s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:13:44.910 [mongosMain] waited 850s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:13:55.949 [mongosMain] waited 861s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:14:06.991 [mongosMain] waited 872s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:14:18.030 [mongosMain] waited 883s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:14:29.070 [mongosMain] waited 894s for distributed lock configUpgrade for upgrading config database to new format v4
Thu Mar 28 13:14:35.092 [mongosMain] forcing lock 'configUpgrade/host15.somedomain.com:30000:1364468909:1804289383' because elapsed time 900327 > takeover time 900000
Thu Mar 28 13:14:35.529 [mongosMain] lock 'configUpgrade/host15.somedomain.com:30000:1364468909:1804289383' successfully forced
Thu Mar 28 13:14:37.092 [mongosMain] distributed lock 'configUpgrade/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 5154342bf6e04e7181edb6da
Thu Mar 28 13:14:37.095 [mongosMain] starting upgrade of config server from v3 to v4
Thu Mar 28 13:14:37.095 [mongosMain] starting next upgrade step from v3 to v4
Thu Mar 28 13:14:37.096 [mongosMain] about to log new metadata event: { _id: "host15.somedomain.com-2013-03-28T12:14:37-5154342df6e04e7181edb6db", server: "
host15.somedomain.com", clientAddr: "N/A", time: new Date(1364472877096), what: "starting upgrade of config database", ns: "config.version", details: { from: 3, to: 4 } }
Thu Mar 28 13:14:39.265 [mongosMain] forcing upgrade locks of previous failed upgrade with id 51542836a70d009c157f195b
Thu Mar 28 13:14:39.742 [mongosMain] stale mongos detected 345437 minutes ago, network location is
host10.somedomain.com:30000, not checking version
Thu Mar 28 13:14:39.742 [mongosMain] stale mongos detected 335023 minutes ago, network location is
host12.somedomain.com:30000, not checking version
Thu Mar 28 13:14:39.742 [mongosMain] stale mongos detected 277985 minutes ago, network location is
host16.somedomain.com:30000, not checking version
Thu Mar 28 13:14:39.742 [mongosMain] stale mongos detected 159343 minutes ago, network location is host9:30000, not checking version
Thu Mar 28 13:14:39.742 [mongosMain] stale mongos detected 257 minutes ago, network location is
host15.somedomain.com:30000, not checking version
Thu Mar 28 13:14:39.742 [mongosMain] stale mongos detected 258 minutes ago, network location is
host9.somedomain.com:30000, not checking version
Thu Mar 28 13:14:39.742 [mongosMain] stale mongos detected 278 minutes ago, network location is
host2.somedomain.com:30000, not checking version
Thu Mar 28 13:14:40.183 [mongosMain] acquiring locks for 14 sharded collections...
Thu Mar 28 13:14:40.959 [mongosMain] distributed lock 'backend.cash_transactions/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543430f6e04e7181edb6e8
Thu Mar 28 13:14:41.768 [mongosMain] distributed lock 'backend.clickables/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543430f6e04e7181edb6e9
Thu Mar 28 13:14:42.751 [mongosMain] distributed lock 'backend.f2011_friendly_matches/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543431f6e04e7181edb6ea
Thu Mar 28 13:14:43.560 [mongosMain] distributed lock 'backend.f2011_gko_tournaments/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543432f6e04e7181edb6eb
Thu Mar 28 13:14:44.386 [mongosMain] distributed lock 'backend.f2011_players/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543433f6e04e7181edb6ec
Thu Mar 28 13:14:45.210 [mongosMain] distributed lock 'backend.f2011_playerteams/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543434f6e04e7181edb6ed
Thu Mar 28 13:14:46.021 [mongosMain] distributed lock 'backend.f2011_rbcl_matches/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543435f6e04e7181edb6ee
Thu Mar 28 13:14:47.000 [mongosMain] distributed lock 'backend.f2011_rbcl_playerteams/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543436f6e04e7181edb6ef
Thu Mar 28 13:14:47.881 [mongosMain] distributed lock 'backend.f2011_rbcl_standings_history/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543437f6e04e7181edb6f0
Thu Mar 28 13:14:48.658 [mongosMain] distributed lock 'backend.notifications/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543437f6e04e7181edb6f1
Thu Mar 28 13:14:48.658 [mongosMain] acquired 10 locks out of 14 for config upgrade
Thu Mar 28 13:14:49.326 [mongosMain] distributed lock 'backend.purchases/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543438f6e04e7181edb6f2
Thu Mar 28 13:14:50.237 [mongosMain] distributed lock 'backend.rewards/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 51543439f6e04e7181edb6f3
Thu Mar 28 13:14:51.146 [mongosMain] distributed lock 'backend.user_game_summaries/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 5154343af6e04e7181edb6f4
Thu Mar 28 13:14:51.993 [mongosMain] distributed lock 'backend.users/host15.somedomain.com:30000:1364471974:1804289383' acquired, ts : 5154343bf6e04e7181edb6f5
Thu Mar 28 13:14:51.993 [mongosMain] copying collection and chunk metadata to working and backup collections...