Servers stuck in initializing state again / reboot results in Authentication failure messages

[Srinivasan Subramanian]

Again the launch of farm resulted in Servers being stuck in “initializing” state.  Scalarizr showed no errors.  I then rebooted the server via Scalr console and the following messages popped up in the instance scalarizr.log

 

2012-03-10 22:09:52,231 - WARNING - scalarizr.messaging.p2p.producer - Message 'RebootStart' not delivered (message_id: b7b6ddfa-86bf-4546-9d71-c3ee5e646a57)

2012-03-10 22:09:52,232 - ERROR - scalarizr.messaging.p2p.producer - Cannot authenticate on message server. Signature doesn't match

2012-03-10 22:09:52,310 - WARNING - scalarizr.messaging.p2p.producer - Message 'RebootFinish' not delivered (message_id: d8da609d-9adb-4bd2-b240-52c816f352b7)

2012-03-10 22:09:52,310 - ERROR - scalarizr.messaging.p2p.producer - Cannot authenticate on message server. Signature doesn't match

 

Obviously some thing is failing.  I am just unable to get this to run consistently.  All ssl_access and ssl_error logs on scalr do not indicate any issues.  cron is running happily.

 

Any tips/help on getting this resolved will be a blessing.

 

Regards

Srini

[Srinivasan Subramanian]

Tried with a brand new farm.  Same issue, where it is stuck in “initializing” stage.

 

1. cron jobs are running

2. instance 8013 is reachable from scalr server

3. scalr internal messaging for this instance is empty!

 

Obviously something is wrong.  I just terminated the instance to check if scalr would recognize and handle it, the following was seen in the scalarizr log:

 

2012-03-10 22:36:28,116 - INFO - scalarizr.scripts.halt - Starting halt script    .

2012-03-10 22:36:29,123 - WARNING - scalarizr.messaging.p2p.producer - Message    HostDown' not delivered (message_id: 327637f3-1362-4f82-83e6-32598cd31442)

2012-03-10 22:36:29,123 - ERROR - scalarizr.messaging.p2p.producer - Cannot authenticate on message server. Signature doesn't match

2012-03-10 22:36:58,579 - INFO - scalarizr - [pid: 1461] Stopping scalarizr 0.7.188

2012-03-10 22:36:58,580 - INFO - scalarizr - [pid: 1562] SNMP process terminated

 

Can someone help please?

Srini

[Srini]

This was an issue with my server, nothing to do with Scalr .. somehow after some time it was losing connectivity (guessing) .. I moved it to a new server and it seems OK so far.

Srini