Cloudservers-uk delete errors

[Chris Strand]

Hey,

Since Sunday evening I have getting Exceptions on what appears to be a somewhat random basis when trying to shut down nodes running on the Rackspace UK public cloud.

jclouds log message:

23:37:23.869 [i/o thread 1]    WARN  o.j.h.h.BackoffLimitedRetryHandler - Cannot retry after server error, command has exceeded retry limit 5: [method=CloudServersAsyncClient.deleteServer, request=DELETE https://lon.servers.api.rackspacecloud.com/v1.0/10001547/servers/10112625 HTTP/1.1]

The exception and message:

org.jclouds.http.HttpResponseException: command: DELETE https://lon.servers.api.rackspacecloud.com/v1.0/10001547/servers/10112625 HTTP/1.1 failed with response: HTTP/1.1 501 Not Implemented; content: [This method may not be used.]

Full stack trace uploaded to paste bin: http://pastebin.com/H0rZ0eVx

I am pretty sure that this must be a problem with Rackspace's API for a number of reasons, but I thought I would check here to see if anyone has experienced the same thing or can think of anything that I could be doing wrong on my side? As you can imagine this is playing havoc with my automation so any help is greatly appreciated!

Chris

[Andrei Savu]

I had the same issue. I think we need to contact Rackspace support about this.

Chris

--
You received this message because you are subscribed to the Google Groups "jclouds" group.
To post to this group, send email to jcl...@googlegroups.com.
To unsubscribe from this group, send email to jclouds+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jclouds?hl=en.

[Chris Strand]

I have been in contact with them over the last couple of days. They admitted they had API issues over the weekend (but didn't report it to their status blog/twitter, tut tut!) and claimed everything should be fixed. They also said that they have been unable to reproduce this issue, so thanks for letting me know I'm not the only one!

I guess then this will just serve as a heads up for anyone else using Rackspace at the moment, I'll repost here if anything useful comes out of their support.

Chris

[peter veentjer]

We are running into the same problems since 24 hours when we create
new instances in the London datacenter through the API, I filed a
ticket and they responded:

------------------------------------------------

Hi Peter,

We've been seeing an odd build issue today that's been affecting
builds intermittently. In most cases building a new server does go
through.

In the meantime I've deleted the following servers for you as they
aren't usable,

.....

Sorry about this, but our operations people are looking into it. In
the meantime let me forward this ticket on to our account managers for
credit review.

....

------------------------------------------------

So something is failing on their side.

On Feb 23, 9:52 am, Chris Strand <clastr...@gmail.com> wrote:
> I have been in contact with them over the last couple of days. They
> admitted they had API issues over the weekend (but didn't report it to
> their status blog/twitter, tut tut!) and claimed everything should be
> fixed. They also said that they have been unable to reproduce this issue,
> so thanks for letting me know I'm not the only one!
>
> I guess then this will just serve as a heads up for anyone else using
> Rackspace at the moment, I'll repost here if anything useful comes out of
> their support.
>
> Chris
>

> On 23 February 2012 04:21, Andrei Savu <savu.and...@gmail.com> wrote:
>
>
>
>
>
>
>
> > I had the same issue. I think we need to contact Rackspace support about
> > this.
>

> > On Thu, Feb 23, 2012 at 12:08 AM, Chris Strand <clastr...@gmail.com>wrote:
>
> >> Hey,
>
> >> Since Sunday evening I have getting Exceptions on what appears to be a
> >> somewhat random basis when trying to shut down nodes running on the
> >> Rackspace UK public cloud.
>
> >> jclouds log message:
> >> 23:37:23.869 [i/o thread 1]    WARN  o.j.h.h.BackoffLimitedRetryHandler -
> >> Cannot retry after server error, command has exceeded retry limit 5:
> >> [method=CloudServersAsyncClient.deleteServer, request=DELETE

> >>https://lon.servers.api.rackspacecloud.com/v1.0/10001547/servers/1011...]

>
> >> The exception and message:
> >> org.jclouds.http.HttpResponseException: command: DELETE

> >>https://lon.servers.api.rackspacecloud.com/v1.0/10001547/servers/1011...failed with response: HTTP/1.1 501 Not Implemented; content: [This

[Adrian Cole]

Thanks for the update, Peter.

This happens occasionally with Rackspace, and brings up a point as to what to do about it?  For example, it is possible we could disable a context who throws continuous service errors and return an explicit exception type like SNAFUException, or ServiceOutageException which applications could catch instead of hunting for 500s. 

There is a similar need for apis who have extensions that are not enabled, or fail for whatever reason.  It could be best to disable certain failing commands for a while, which could allow the caller to queue until the situations resolve.

Thoughts?
-A

[Chris Strand]

I have been wanting to have something in my codebase such that if the error rate for a certain provider becomes too high then I would automatically swap over to an alternate one. I haven't thought through a possible implementation for that yet but having more explicit exceptions could certainly help.

RE: Rackspace problems I've never had this exact API shutdown problem before but in the past I have regularly experienced servers booting up into error state (and I then have to get support to shut them down manually).

A minor suggestion for now, from my logs I can see that the logging level for five failed attempts in a row is just WARN, shouldn't this be bumped up to ERROR?

Chris

PS: Rackspace have also confirmed to me that they are able to reproduce the 501 API error and are working on a solution.

[Adrian Cole]

Good point, Chris. We could bump the logging level up on 5th error;
can you throw an issue in for this?

Besides nodes in rackspace coming up in error state, I've noticed the
service returning 503 errors. I think that per-provider, or maybe
even per api w/in a provider, we'd want to have some signature of
"enough's enough" so maybe we should get more feedback about what
those lines should be? Possibly we may want to take an api out
manually, too...

thx.
-A

[Chris Strand]

I have created an issue at  http://code.google.com/p/jclouds/issues/detail?id=848 for the logging levels.

There are a whole host of things that could be considered unacceptable when interacting with a provider, here are a few (in no particular order) that annoy me:

• servers take way above the usual time to boot up
• any 500+ HTTP errors with the API
• servers going into some kind of error state on boot up

I'm sure there are many more but these are ones I have personally encountered. As for a way to handle this neatly I'm afraid I have no ideas so far!

Chris