Account Suspended

[Paul Peeler]

Our account was suspended. I wasn't aware that it had been enabled again, and I began to get emails. Unfortunately the message is somewhat generic. Does anyone know what this generally means? How do you go about testing?

Message: 

SpotCloud has detected a problem with your provider API and connect retrieve the list of running instances. Please correct this problem as soon as possible to avoid instance termination.

[Adam Boduch]

Hi,

I responded to your email you sent to us.

Basically, it looks like SpotCloud was having trouble connecting to your
provider API.

This is a useful tool to help providers diagnose API connectivity
issues:

https://spotcloud.appspot.com/provider/logs

Please let me know if you get everything sorted out.

Adam

[Paul Peeler]

It doesn't really tell me much other than that it can't get /vm/list at random intervals but most of the time it can. The box hasn't been down, and nothing has changed as far as configuration on our end. In the same minute it can get /ptemplate/list and not /vm/list.

[Adam Boduch]

This sounds like there is something wrong with your provider setup. Can
you take a look at your logs and let me know if there is anything
interesting in the /vm/list requests?

[Paul Peeler]

Example logs below with my username modified. There is nothing in nginx error logs pertaining to /vm/list or the two IPs that seem to consistently monitor our server.

74.125.44.87 - - [17/May/2011:07:50:25 -0400] "GET /rest/hosting/vm/list?ecp_auth_digest=Zqs9srRpq1vpMR%2B%2FyOmn7mYlRYo%3D&ecp_username=myuser HTTP/1.1" 200 45 "-" "AppEngine-Google; (+http://code.google.com/appengine; appid: spotcloud)" "-"

64.233.172.18 - - [17/May/2011:07:55:19 -0400] "GET /rest/hosting/vm/list?ecp_auth_digest=Zqs9srRpq1vpMR%2B%2FyOmn7mYlRYo%3D&ecp_username=myuser HTTP/1.1" 200 45 "-" "AppEngine-Google; (+http://code.google.com/appengine; appid: spotcloud)" "-"

[Paul Peeler]

Anything on this? Can we test?

On Tuesday, May 17, 2011 8:04:52 AM UTC-4, Paul Peeler wrote:
> Example logs below with my username modified. There is nothing in nginx error logs pertaining to /vm/list or the two IPs that seem to consistently monitor our server.</div>
>
> </div>
> 74.125.44.87 - - [17/May/2011:07:50:25 -0400] &quot;GET /rest/hosting/vm/list?ecp_<WBR>auth_digest=Zqs9srRpq1vpMR%2B%<WBR>2FyOmn7mYlRYo%3D&amp;ecp_username=<WBR>myuser HTTP/1.1&quot; 200 45 &quot;-&quot; &quot;AppEngine-Google; (+<a href="http://code.google.com/appengine" target="_blank">http://code.google.com/<WBR>appengine</a>; appid: spotcloud)&quot; &quot;-&quot;</div>
> 64.233.172.18 - - [17/May/2011:07:55:19 -0400] &quot;GET /rest/hosting/vm/list?ecp_<WBR>auth_digest=Zqs9srRpq1vpMR%2B%<WBR>2FyOmn7mYlRYo%3D&amp;ecp_username=<WBR>myuser HTTP/1.1&quot; 200 45 &quot;-&quot; &quot;AppEngine-Google; (+<a href="http://code.google.com/appengine" target="_blank">http://code.google.com/<WBR>appengine</a>; appid: spotcloud)&quot; &quot;-&quot;</div>
>
> </div>

[Paul Peeler]

We are continuing to get error messages. What other steps can we take to resolve this?

[Adam Boduch]

Hi,

It looks like the logs below are showing successful API calls. We'll
need to figure out what is failing. Can you email me your full log
files so I can further diagnose?

Thanks,

Adam

[Adam Boduch]

I'm seeing several "Could not contact the server" messages in your
provider API logs. This only seems to happen on the "/vm/list" API.

I would suggest trying to reproduce this issue yourself. Everything is
configured correctly on SpotCloud. You're getting notifications from
SpotCloud because it cannot retrieve the list of instances. This is a
requirement.

Regards,

Adam

[Paul Peeler]

Where do I send it?

[Paul Peeler]

In the logs at https://spotcloud.appspot.com/provider/logs it shows that we have been checked up to 3:56PM on May 19th. I assume that the time is off on the polling server...

[Adam Boduch]

ad...@enomaly.com

On Thu, 2011-05-19 at 08:45 -0700, Paul Peeler wrote:
> Where do I send it?

[Adam Boduch]

The log times are displayed in UTC.

[Paul Peeler]

Please confirm you received my email. 

[Paul Peeler]

Any updates here? I'm getting nothing back from anyone via email or groups...

[Derek Anderson]

I think the group is having mail delivery issues. I just received this now.

I'll follow up with Adam tomorrow to make sure he received a copy of this.

[Adam Boduch]

Hi,

I re-enabled your account yesterday. Please let me know if you
continue experiencing problems with your account.

Adam

[Paul Peeler]

Problems persist. I continue to get warnings and errors about insufficient resources.

[Adam Boduch]

Anything interesting turn up in your ECP logs?

[Paul Peeler]

Always the same problem. I18N_INSUFFICIENT_RESOURCES is reported in the transaction log. Any issue reported to me refers to /vm/list.

[Paul Peeler]

Additionally, agent logs still say the same thing.

tail /opt/enomalism2/logs/agent.log 

IOError: [Errno 2] No such file or directory: u'/opt/enomalism2/repo/e228a8f3-3a10-11e0-bd47-00259013616c.xvm2'

2011-05-26 11:26:30-0400 [XmlStream,client] [WARNING - e2_pdaemon.agent.xmpp.XMPPAgent] Sending stanza on agent_info of 

2011-06-02 10:45:45-0400 [-] [ERROR - e2_pdaemon.agent.agent.HostAgent] Provision failure, reason: [Errno 2] No such file or directory: u'/opt/enomalism2/repo/e228a8f3-3a10-11e0-bd47-00259013616c.xvm2'

2011-06-02 10:45:45-0400 [-] [ERROR - e2_pdaemon.agent.commands.ProvisionCommand] Provision failed: [Errno 2] No such file or directory: u'/opt/enomalism2/repo/e228a8f3-3a10-11e0-bd47-00259013616c.xvm2'

Traceback (most recent call last):

 File "twisted/internet/defer.py", line 601, in _deferGenerator

 File "build/bdist.linux-x86_64/egg/e2_pdaemon/agent/agent.py", line 527, in provision

IOError: [Errno 2] No such file or directory: u'/opt/enomalism2/repo/e228a8f3-3a10-11e0-bd47-00259013616c.xvm2'

[Adam Boduch]

Hi,

This appears to be a permission issue. Please verify that your
spotcloud user has the appropriate permissions.

Regards,

Adam

[Paul Peeler]

Attached are screenshots. I configured permissions based on the install docs. If there is something incorrect could you please point it out.

[Adam Boduch]

Hi,

Hmmm. Permissions look fine to me. I just enabled your account and
sent out another test instance. Let me know if you get the same error
in your agent logs.

Regards,

Adam

[Paul Peeler]

Below is the entry into the agent logs on both agents that are currently running. My manager still reports I18N_INSUFFICIENT_RESOURCES as well.

2011-06-08 14:50:03-0400 [-] [ERROR - e2_pdaemon.agent.agent.HostAgent] Provision failure, reason: [Errno 2] No such file or directory: u'/opt/enomalism2/repo/e228a8f3-3a10-11e0-bd47-00259013616c.xvm2'

2011-06-08 14:50:03-0400 [-] [ERROR - e2_pdaemon.agent.commands.ProvisionCommand] Provision failed: [Errno 2] No such file or directory: u'/opt/enomalism2/repo/e228a8f3-3a10-11e0-bd47-00259013616c.xvm2'

Traceback (most recent call last):

 File "twisted/internet/defer.py", line 601, in _deferGenerator

 File "build/bdist.linux-x86_64/egg/e2_pdaemon/agent/agent.py", line 527, in provision

IOError: [Errno 2] No such file or directory: u'/opt/enomalism2/repo/e228a8f3-3a10-11e0-bd47-00259013616c.xvm2'

2011-06-08 14:50:03-0400 [-] [ERROR - e2_pdaemon.agent.agent.HostAgent] Provision failure, reason: [Errno 2] No such file or directory: u'/opt/enomalism2/repo/e228a8f3-3a10-11e0-bd47-00259013616c.xvm2'

2011-06-08 14:50:03-0400 [-] [ERROR - e2_pdaemon.agent.commands.ProvisionCommand] Provision failed: [Errno 2] No such file or directory: u'/opt/enomalism2/repo/e228a8f3-3a10-11e0-bd47-00259013616c.xvm2'

[Paul Peeler]

To be thorough, /opt/enomalism2/repo/ does exist, the file it is looking for does not exist on the agent nodes. It does however exist on the manager node.

[Adam Boduch]

Hi,

Well that is good news, seems like the manager is actually downloading
and installing it. Let me ask around here and see if I can find some
insight.

Regards,

Adam

[Adam Boduch]

Hi,

I have a couple things for you to check.

Make sure this folder exists on your agent:

ls -alrt /var/lib/xen/images/repo

If so, verify your storage target settings and check your filesystem
mounts. The details of how to setup the repo are in the ECP
documentation.

Regards,

Adam

On Wed, 2011-06-08 at 11:55 -0700, Paul Peeler wrote:

[Paul Peeler]

I don't see anything in the installation docs about setting up repos. I don't see any mention of the words xen or repo in SpotCloudInstallationGuide.pdf so please let me know if I have the wrong docs. I was directed via email to mount my local storage to /var/lib/xen/images, hence the lost and found directory in the listing below.

As it turns out, the repo directory doesn't exist. I have created it on both agents. Please point me to the docs that will inform me of the correct permissions, or mention here what they should be.

ls -alrt /var/lib/xen/images/

total 24

drwx------ 2 root root 16384 May  9 12:07 lost+found

drwxr-xr-x 3 root root  4096 May 12 11:21 ..

drwxr-xr-x 3 root root  4096 Jun  8 14:50 .

ls -alrt /var/lib/xen/

total 16

drwxr-xr-x 21 root root 4096 May 12 11:21 ..

drwxr-xr-x  3 root root 4096 May 12 11:21 .

drwxr-xr-x  3 root root 4096 Jun  8 14:50 images

[Paul Peeler]

Account suspended again...

[Adam Boduch]

Hi,

We know why this is happening now. And it has to do with your ECP
installation. I'll take a look around to find you some repo setup
instructions.

Regards,

Adam

On Wed, 2011-06-08 at 19:52 -0700, Paul Peeler wrote:
> Account suspended again...

[Paul Peeler]

Awesome, thanks.

[Paul Peeler]

Anything else on this? I would like to get this running before the weekend if possible.

[Adam Boduch]

Hi,

You should find the ECP documentation on /opt/enomalism2/docs. The
admin guide should give you everything you need to properly setup your
repo.

Regards,

Adam

[Paul Peeler]

Looks like storagetarget was in the old form. That and a change to repodir have been made. Can we attempt another test?

[Adam Boduch]

Hi,

Just created an instance.

Regards,

Adam