10:22:06,364 INFO [razor.web.log] (http-/0.0.0.0:8080-2) 127.0.0.1 - - [19/Mar/2015 10:22:06] "GET /api " 200 4962 0.0670
10:22:06,806 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) CREATE TABLE IF NOT EXISTS "schema_info" ("version" integer DEFAULT 0 NOT NULL)
10:22:06,814 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) SELECT * FROM "schema_info" LIMIT 1
10:22:06,820 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.000000s) SELECT 1 AS "one" FROM "schema_info" LIMIT 1
10:22:06,828 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.000000s) SELECT count(*) AS "count" FROM "schema_info" LIMIT 1
10:22:06,836 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) SELECT "version" FROM "schema_info" LIMIT 1
10:22:06,843 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) SELECT "version" FROM "schema_info" LIMIT 1
10:22:06,854 INFO [razor.web.log] (http-/0.0.0.0:8080-2) 127.0.0.1 - - [19/Mar/2015 10:22:06] "GET /api/commands/reboot-node " 200 2308 0.0540
10:22:06,889 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.000000s) CREATE TABLE IF NOT EXISTS "schema_info" ("version" integer DEFAULT 0 NOT NULL)
10:22:06,897 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) SELECT * FROM "schema_info" LIMIT 1
10:22:06,906 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.000000s) SELECT 1 AS "one" FROM "schema_info" LIMIT 1
10:22:06,913 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.000000s) SELECT count(*) AS "count" FROM "schema_info" LIMIT 1
10:22:06,921 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) SELECT "version" FROM "schema_info" LIMIT 1
10:22:06,928 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.000000s) SELECT "version" FROM "schema_info" LIMIT 1
10:22:06,938 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.000000s) SELECT * FROM "nodes" WHERE ("name" = 'node15') LIMIT 1
10:22:06,951 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) SELECT * FROM "nodes" WHERE ("name" = 'node15') LIMIT 1
10:22:06,975 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) BEGIN
10:22:06,983 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) INSERT INTO "commands" ("command", "submitted_by", "submitted_at", "status", "finished_at", "params") VALUES ('reboot-node', NULL, '2015-03-19 10:22:06.949000-0400', 'finished', '2015-03-19 10:22:06.973000-0400', '{"name":"node15"}') RETURNING *
10:22:06,999 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) COMMIT
10:22:07,007 INFO [razor.web.log] (http-/0.0.0.0:8080-2) 127.0.0.1 - - [19/Mar/2015 10:22:07] "POST /api/commands/reboot-node " 202 98 0.1220
10:22:19,883 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) CREATE TABLE IF NOT EXISTS "schema_info" ("version" integer DEFAULT 0 NOT NULL)
10:22:19,895 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) SELECT * FROM "schema_info" LIMIT 1
10:22:19,905 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) SELECT 1 AS "one" FROM "schema_info" LIMIT 1
10:22:19,912 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) SELECT count(*) AS "count" FROM "schema_info" LIMIT 1
10:22:19,919 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.000000s) SELECT "version" FROM "schema_info" LIMIT 1
10:22:19,926 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.000000s) SELECT "version" FROM "schema_info" LIMIT 1
10:22:19,936 INFO [razor.web.log] (http-/0.0.0.0:8080-2) 10.108.81.78 - - [19/Mar/2015 10:22:19] "GET /svc/mk/extension.zip " 404 40 0.0600
10:22:20,139 INFO [razor.sequel] (http-/0.0.0.0:8080-2) (0.001000s) CREATE TABLE IF NOT EXISTS "schema_info" ("version" integer DEFAULT 0 NOT NULL)
I don't see anything else related to the nodes we are trying to reboot. We have some other systems that are undergoing maintenance and I see razor is repeatedly attempting to get the power state for 4 of them (unsuccessfully)
10:37:49,285 INFO [razor.messaging.sequel] (Thread-3 (HornetQ-client-global-threads-1414145431)) retry message ID:9f353111-ccf3-11e4-9a14-bb950502efae after 1.02: executing ["ipmitool", "-I", "lanplus", "-H", "192.168.2.33", "-U", "user", "-f", "/tmp/ipmitool-password20150319-52745-11tcjfo", "power", "status"] failed: #<Process::WaitThread:0x53022940>
Error: Unable to establish IPMI v2 / RMCP+ session
Error: Unable to establish IPMI v2 / RMCP+ session
Error: Unable to establish IPMI v2 / RMCP+ session
Unable to get Chassis Power Status
Maybe this is clogging up the queue? I'm not sure how to manage the queue.