Scalr 4.5 upgrade

[Mike McCarthy]

Hi,

I've recently upgraded Scalr from 4.4 to 4.5.  I've created a role that has scalarizr-ec2 v2.4.1 running on it and have been launching a test farm with this role in it.   If I leave Scalr alone after launching the farm it sits in Initializing state until it times out.  In my Scalr message sender log I can see these sorts of entries for the server that is launching:

[05/Mar/2014 11:14:05][0.0.30][msg_sender][2167][139995350418896] ERROR Delivery failed, message:'eb052699-9395-4dc3-b779-d34d4887617a' error:'<type 'exceptions.Exception'> server:e775d281-59ad-4abf-9509-5fea59471a67 error:can't determine IP line: 274'

[05/Mar/2014 11:31:33][0.0.30][msg_sender][5950][140567929726416] ERROR Delivery failed, message:'5219eb8c-f2ed-443d-aa0e-7db9616d9331' error:'<type 'exceptions.Exception'> server:e775d281-59ad-4abf-9509-5fea59471a67 error:can't determine IP line: 274'

If I go into the Scalr Internal Messaging for the server while it is launching I find it sat on the "out/HostInit" message or "out/HostUp" message having failed after three attempts.   If for each of these messages I choose to resend them from the actions menu I end up with a complete launch of the server.    Is there something I could have missed in the upgrade that would cause these messages to be sent automatically?   As far as I'm aware I've got the required security groups in place on the EC2 side as this worked when it was on 4.4.  I've got cron jobs and the required daemons set up as detailed in the wiki page for upgrading from 4.4 to 4.5.  Is there anything else I should be running or something I've missed?

Thanks,

Mike

[Mike McCarthy]

Just updating on this.  I've added a little more output to the msg_sender.py script that is generating the error messages above.   

In the current version on git I've changed this:

lines 179-180

if not ip:
            raise Exception("server:%s error:can't determine IP" % server['server_id'])

To this:

if not ip:
            raise Exception("server:%s error:can't determine IP l %s r %s X" % (server['server_id'],server['local_ip'],server['remote_ip'])

I'm now seeing this in the logs:

[05/Mar/2014 17:10:17][0.0.30][msg_sender][20124][139675050296784] ERROR Delivery failed, message:'9e8348d5-309a-4bbd-8abe-069e991f56a9' error:'<type 'exceptions.Exception'> server:066add24-80e7-4ed2-adbb-d54b7ab0434f error:can't determine IP l 10.116.144.91 r  X line: 274'

So it looks like an IP is at least being pulled but not used.   As a test I've added a line prior to the "if not ip:" forcing ip = server['local_ip'] as that appears to have an IP.   This has stopped the messages in the logs above but the messages still don't appear to be getting through and as before I'm still able to manually push through the messages from the Scalr UI and end up with a running farm but if left alone the farm just times out still. 

[Mike McCarthy]

A further update on this.   I've done some deeper debugging and found that the messages were being sent on port 80.   I've added "port=8013" after the line where I forced the ip variable to use server['local_ip'] and now I have Scalr functioning automatically and farms launch without any user intervention.  

All I'd really like to know now is if these changes I've made are sound or if there's something else I've missed elsewhere that would make it so that these script changes aren't needed.   To help anyone seeing the same issue I've seen this is the change to the code that I have made to msg_sender.py:

179         if not ip:

180                 if server['local_ip']:

181                         port = 8013

182                         ip = server['local_ip']

183                 else:

184                         raise Exception("server:%s error:can't determine IP l %s r %s X" % (server['server_id'],server['local_ip'],server['remote_ip']))