Pending state for servers in the farm.

[Dale-Kurt Murray]

In Core Settings, the Event Handler URL and Server IP Address of the scalr server is set, however server instances are terminated and re-launched on AWS EC2 after a few minutes when the Farm is launched. Scalr does however report the status of the active servers as pending.

There also seems to be an issue with creating Rackspace Roles, the temporary servers are created and are being provisioned but the process takes much longer that usual.

[Dale-Kurt Murray]

After reviewing the logs while attempting to build a role I found that the server had issues with SSH during the build and provisioning of a role

http://cl.ly/3g2t3u1f0q3G2K021G1i

Any thoughts on why this would happen?

[Nick Toursky]

Are you able to connect to this server from the host where Scalr is running.

Is SSH2 PHP extension installed?

--
You received this message because you are subscribed to the Google Groups "scalr-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/scalr-discuss/-/UQkfhbsLxZwJ.

To post to this group, send email to scalr-...@googlegroups.com.
To unsubscribe from this group, send email to scalr-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scalr-discuss?hl=en.

[Igor Savchenko]

Rackspace reporting that server status is ERROR. This server status
returned by Rackspace API. Most likely server failed to provision. You
can ask RS support for more information.

Regards,
Igor

[Dale-Kurt Murray]

The SSH2 PHP extension is installed and works fine

[Dale-Kurt Murray]

@DicsyDel

You may be correct, I'm having similar problem with the farm when I configure a App and Database server, and launch the farm. One of the server keeps terminating and goes to a pending, then terminates again. This happens consistently, however one of the two server is spun up with no issues.

[Dale-Kurt Murray]

The log show the following message "Server '83d24b0d-b0e0-4df7-aac8-b9c65762e9d0' did not send 'hostInit' event in 2400 seconds after launch (Try increasing timeouts in role settings). Considering it broken. Terminating instance." Is this a common occurrence?

[Igor Savchenko]

This error means that server (scalarizr - scalr agent on server)
didn't send a message that server has been initialized. And this means
that OS didn't boot properly. This is not a scalr issue. Again, I
would advise you to ask RS support, why servers didn't become ACTIVE
and shows ERROR state instead.

Regards,
Igor

> --
> You received this message because you are subscribed to the Google Groups
> "scalr-discuss" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/scalr-discuss/-/xOb4cioB9n0J.

[Dale-Kurt Murray]

It would seem AWS is suffering from a similar issue as well.

[Igor Savchenko]

If you have the same issue with AWS then:
1. Make sure that at AWS server is in running state and console output
shows that system boot normally.
2. If everything is okay with 1, it means that messaging between
scalr<->scalarizr doesn't work. If you click to [options] for this
server in scalr and then 'Internal scalr messging' and it would be
messages in delivering state - it means that you have issues with
cronjobs. if you won't see any messages - means that scalarizr cannot
reach scalr and deliver message - you will need to check
/var/log/scalarizr.log on instance for more details.

Regards,
Igor

[Dale-Kurt Murray]

When the instances are launched, it never gets to a running state, it usually is terminated before then (http://cl.ly/3y252N3p1u3z0I3l3e0O).

The event log shows "Message: Server '8fe0d2e5-cebb-4d70-97d1-1f920c060adb' did not send 'hostInit' event in 4800 seconds after launch (Try increasing timeouts in role settings). Considering it broken. Terminating instance."

[Dale-Kurt Murray]

On AWS this is what's happening - http://cl.ly/2o3x1i0v0q0u2s1r1x15

[Igor Savchenko]

messaging between
scalr<->scalarizr doesn't work. If you click to [options] for this
server in scalr and then 'Internal scalr messging' and it would be
messages in delivering state - it means that you have issues with
cronjobs. if you won't see any messages - means that scalarizr cannot
reach scalr and deliver message - you will need to check
/var/log/scalarizr.log on instance for more details.

Regards,
Igor

On 13 February 2012 10:56, Dale-Kurt Murray <dalekur...@gmail.com> wrote:
> On AWS this is what's happening - http://cl.ly/2o3x1i0v0q0u2s1r1x15
>

> --
> You received this message because you are subscribed to the Google Groups
> "scalr-discuss" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/scalr-discuss/-/oXzPgCv2BDoJ.

[Dale-Kurt Murray]

Thanks Igor,

So, I checked the Internal Messages for the server and that was blank, which mean Scalrizr canon reach my scalr server and delivery messages. I will check the scalarizr log and let you know what's happening there shortly.

[Dale-Kurt Murray]

I have another problem, I don't have the PEM file to access the server. I'm going to launch the farm using Rackspace and use the java ssh console.

[Dale-Kurt Murray]

On Rackspace I got the /var/log/scalarizr.log file

cat scalarizr.log
2012-02-13 01:38:14,217 - INFO - scalarizr - [pid: 10396] Starting scalarizr 0.7.176
2012-02-13 01:38:14,700 - INFO - scalarizr - [pid: 10405] Starting scalarizr 0.7.176
2012-02-13 01:38:14,702 - INFO - scalarizr.config - State: importing
2012-02-13 01:38:15,254 - INFO - scalarizr.messaging.p2p.consumer - Building message consumer server on 0.0.0.0:8013
2012-02-13 01:38:15,261 - INFO - scalarizr.handlers.lifecycle - Server will be imported into Scalr
2012-02-13 01:38:15,310 - INFO - scalarizr.snmp.agent - [pid: 10451] Starting SNMP server on 0.0.0.0:8014
2012-02-13 01:40:02,859 - INFO - scalarizr.config - State: rebundling
2012-02-13 01:40:04,361 - INFO - scalarizr.handlers.rebundle - Lookup server 108.166.104.46 on CloudServers
2012-02-13 01:40:07,747 - INFO - scalarizr.handlers.rebundle - Creating server image. server id: 20586432, image name: 'rackspace-mysqllvm64-ubuntu-10-04-20120213014002'
2012-02-13 01:40:11,156 - INFO - scalarizr.handlers.rebundle - Checking that image 18729497 is completed
2012-02-13 19:53:45,947 - INFO - scalarizr - [pid: 928] Starting scalarizr 0.7.177
2012-02-13 19:54:16,212 - INFO - scalarizr - Server was started after rebundle. Performing some cleanups
2012-02-13 19:54:16,213 - INFO - scalarizr.config - State: bootstrapping
2012-02-13 19:54:16,810 - INFO - scalarizr.handlers - Stopping mysql. (Configuring)
2012-02-13 19:54:18,100 - INFO - scalarizr.messaging.p2p.consumer - Building message consumer server on 0.0.0.0:8013
2012-02-13 19:54:18,104 - INFO - scalarizr.handlers.lifecycle - Starting initialization
2012-02-13 19:54:18,415 - INFO - scalarizr.snmp.agent - [pid: 1042] Starting SNMP server on 0.0.0.0:8014
2012-02-13 19:54:19,802 - INFO - scalarizr.config - State: initializing

[itdept]

This is the same exact ISSUE i am having.

[Dale-Kurt Murray]

@itdept Are you experiencing this on both AWS and Rackspace? What version of Scalr are you using?

[Dale-Kurt Murray]

Is anyone having similar issues?

[Dale-Kurt Murray]

Igor,

I double checked my configuration for Scalr and found that I was missing the SNMPStatsPoller cronjob.

Now my cronjob looks like this:

*/2 * * * * /usr/bin/php -q /var/scalr/app/cron-ng/cron.php --Poller
* * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --Scheduler
*/10 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --MySQLMaintenance
* * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --DNSManagerPoll
17 5 * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --RotateLogs
*/2 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --EBSManager
*/20 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --RolesQueue
*/5 * * * * /usr/bin/php -q /var/scalr/app/cron-ng/cron.php --DbMsrMaintenance
*/2 * * * * /usr/bin/php -q /var/scalr/app/cron-ng/cron.php --Scaling
*/5 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --DBQueueEvent
*/2 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --SzrMessaging
*/4 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --RDSMaintenance
*/2 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --BundleTasksManager
* * * * * /usr/bin/php -q /var/scalr/app/cron-ng/cron.php --ScalarizrMessaging
* * * * * /usr/bin/php -q /var/scalr/app/cron-ng/cron.php --MessagingQueue
*/2 * * * * /usr/bin/php -q /var/scalr/app/cron-ng/cron.php --DeployManager
* * * * * root /usr/bin/php -q /var/scalr/app/cron-ng/cron.php --SNMPStatsPoller

However, that has not resolve the problem I'm having. EC2 instances launched in the Farm are reported as pending, however they seem to be up and running. As a result I'm not able to login to through SSH because the PEM file is not accessible via Scalr as a result of it being in pending state.

I'm not sure what's going on with this, but this is the only project I have which has this issue :\

[itdept]

Hi,
Yes on both. I am using ver 2.5.

The servers are completed, but scalr lists them as pending or incompleted.

[Dale-Kurt Murray]

At the moment I'm even having issues building roles on Rackspace

Bundle Task - http://cl.ly/3w0t3w101H0z0v2X082Y
Log - http://cl.ly/2s1r3S2B122k343n0m3t

Now I'm a little frustrated

[Sebastian Stadil]

Knowing Rackspace's API, I'm guessing that the logs show some weird responses from the API call. Take a look at that and contact Rackspace.

--
You received this message because you are subscribed to the Google Groups "scalr-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/scalr-discuss/-/JwoT1KXsXZsJ.

[Dale-Kurt Murray]

I'm not seeing any logging information under the API log view in scalr, but I suspect that is for API calls to the Scalr API itself not with third-party APIs.

This seems to be an issue that is consistently happening when attempting to launch instances either on AWS or Rackspace. I re-traced my steps in the setup of the environment and scalr and found that I had not had the SNMPStatPoller in the cronjob, which lead me to believe that may have been the problem. After adding it to the cron, the results were the same.

I think this can be a simple issue and may just be human error as this is the only project I have currently which I have ever had this kind of issue. I had completed an upgrade to 2.5 last week on another project and it works just fine.

Sadly enough another user has reported having similar issues with his setup as well when launching instances on AWS and Rackspace.

[itdept]

This again, is the same exact issue we are having. The servers spam ACTIVE.

[itdept]

Also, the servers scalarizer log sits waiting on port 8014:

tail scalarizr.log
2012-02-15 20:51:48,008 - INFO - scalarizr - [pid: 2630] Starting scalarizr 0.7.178
2012-02-15 20:51:49,099 - INFO - scalarizr - [pid: 2639] Starting scalarizr 0.7.178
2012-02-15 20:51:49,100 - INFO - scalarizr.config - State: importing
2012-02-15 20:51:50,031 - INFO - scalarizr.messaging.p2p.consumer - Building message consumer server on 0.0.0.0:8013
2012-02-15 20:51:50,034 - INFO - scalarizr.handlers.lifecycle - Server will be imported into Scalr
2012-02-15 20:51:50,192 - INFO - scalarizr.snmp.agent - [pid: 2678] Starting SNMP server on 0.0.0.0:8014

[Srini]

Hi Dale

Did you find a fix for this? I am having the exact same error. Base
roles are spun up but the App role consistently is stuck in the
Pending state.

Regards
Srini

On Feb 16, 12:41 am, Dale-Kurt Murray <dalekurt.mur...@gmail.com>
wrote:

> I'm not seeing any logging information under the API log view in scalr, but
> I suspect that is for API calls to the Scalr API itself not with
> third-party APIs.
>
> This seems to be an issue that is consistently happening when attempting to
> launch instances either on AWS or Rackspace. I re-traced my steps in the
> setup of the environment and scalr and found that I had not had the
> SNMPStatPoller in the cronjob, which lead me to believe that may have been
> the problem. After adding it to the cron, the results were the same.
>
> I think this can be a simple issue and may just be human error as this is
> the only project I have currently which I have ever had this kind of issue.
> I had completed an upgrade to 2.5 last week on another project and it works
> just fine.
>
> Sadly enough another user has reported having similar issues with his setup
> as well when launching instances on AWS and Rackspace.
>

> On Wed, Feb 15, 2012 at 2:32 PM, Sebastian Stadil <sebast...@scalr.com>wrote:
>
>
>
>
>
>
>
> > Knowing Rackspace's API, I'm guessing that the logs show some weird
> > responses from the API call. Take a look at that and contact Rackspace.
>
> > On Wed, Feb 15, 2012 at 11:30 AM, Dale-Kurt Murray <
> > dalekurt.mur...@gmail.com> wrote:
>
> >> At the moment I'm even having issues building roles on Rackspace
>
> >> Bundle Task -http://cl.ly/3w0t3w101H0z0v2X082Y

> >> Log -http://cl.ly/2s1r3S2B122k343n0m3t

[Dale-Kurt Murray]

At the moment I am using an earlier version of Scalr, however be sure that in Core Settings, the Event Handler URL and Server IP Address of the scalr server is set

[Srinivasan S]

That is set correctly. Anyways I ended up restarting the scalr server and things were fine again. Strange

Srini

Sent on my BlackBerry® from Vodafone

---

From: Dale-Kurt Murray <dalekur...@gmail.com>

Sender: scalr-...@googlegroups.com

Date: Wed, 29 Feb 2012 08:42:34 -0500

To: <scalr-...@googlegroups.com>

ReplyTo: scalr-...@googlegroups.com

Subject: Re: Pending state for servers in the farm.

[Dale-Kurt Murray]

How much memory do you have on your scalr instance?

[Srinivasan Subramanian]

Hi Dale

 

I don’t have access to the server yet .. will check and let you know.

 

What is the recommendation?  I haven’t found any so far on the wiki.

 

Regards

Srini

> >> mailto:scalr-discuss%2Bunsu...@googlegroups.com.

> >> For more options, visit this group at
> >>http://groups.google.com/group/scalr-discuss?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "scalr-discuss" group.
> > To post to this group, send email to scalr-...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > mailto:scalr-discuss%2Bunsu...@googlegroups.com.

> > For more options, visit this group at
> >http://groups.google.com/group/scalr-discuss?hl=en.

--
You received this message because you are subscribed to the Google Groups "scalr-discuss" group.
To post to this group, send email to scalr-...@googlegroups.com.

To unsubscribe from this group, send email to mailto:scalr-discuss%2Bunsu...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/scalr-discuss?hl=en.

--
You received this message because you are subscribed to the Google Groups "scalr-discuss" group.
To post to this group, send email to scalr-...@googlegroups.com.

To unsubscribe from this group, send email to mailto:scalr-discuss%2Bunsu...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/scalr-discuss?hl=en.

--
You received this message because you are subscribed to the Google Groups "scalr-discuss" group.
To post to this group, send email to scalr-...@googlegroups.com.

To unsubscribe from this group, send email to mailto:scalr-discuss%2Bunsu...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/scalr-discuss?hl=en.

[Dale-Kurt Murray]

At the moment I am running an earlier version of Scalr on one instance and the latest on another. The results I have had varied on both instances.