Service Deletion Stuck in In Progress

[Matt Cholick]

I have a space with several services stuck in the deleting state. In the json that comes back, the state looks like:

"last_operation": {
  "type": "delete",
  "state": "in progress",
  "description": "",
  "updated_at": "2015-04-27T20:30:30Z"
}

This results in "cf services" returning several entries in the "delete in progress" state. Re-issuing the delete command fails with a 400, saying there's another operation in progress. The install itself seems healthy, as I've been able to create and delete several services since observing this behavior.

Is there a way to the delete to try again or otherwise unstick the job form the API? What's the best way of fixing the state?

-Matt Cholick

[dsa...@pivotal.io]

Hi Matt,

We've got a couple questions that will help us debug this:

1. What kind of deployment is this? Is it a private deployment? If so, do can you tell us which version of cf-release is deployed? Are you the operator/admin of this CF deployment? 

2. What service is this an instance of? Do you have access to the broker? Is it attempting to implement asynchronous service instance operations?

3. Finally, if you run `cf curl /v2/events?q=actee:<SERVICE INSTANCE GUID>`, you should be able to learn more about recent operations on the instance. Can you please show us the list of events, with any sensitive data redacted?

Thanks!

David && Raina

CF Services API Team

[Matt Cholick]

David && Raina,

This is a private deployment & I'm one of the operators. We're running cf-release 200.

This is happening on multiple services. Two of them are:

• https://github.com/pivotal-cf/cf-redis-release
• https://github.com/cloudfoundry/cf-mysql-release
  

I'm unsure if these implement the async service operations or not. I can dig if you folks don't know, but I figure they're pivotal service implementations so you likely do. I do have complete access to the brokers.

Here's the call you asked for, some some data tweaked or stripped. It's pretty much what I'd expect: creation then the deletion that hasn't gone through.

{
   "total_results": 2,
   "total_pages": 1,
   "prev_url": null,
   "next_url": null,
   "resources": [
      {
         "metadata": {
            "guid": "a42cf808-2bcf-4a97-87de-c9af12eabc16",
            "url": "/v2/events/a42cf808-2bcf-4a97-87de-c9af12eabc16",
            "created_at": "2015-04-15T16:00:44Z",
            "updated_at": null
         },
         "entity": {
            "type": "audit.service_instance.create",
            "actor": "REDACTED-GUID",
            "actor_type": "user",
            "actor_name": "de...@unknown.org",
            "actee": "REDACTED-GUID",
            "actee_type": "service_instance",
            "actee_name": "cache-01",
            "timestamp": "2015-04-15T16:00:44Z",
            "metadata": {
               "request": {
                  "name": "cache-01",
                  "service_plan_guid": "8e1b9237-773c-473a-8b9e-8863fa28ab0d",
                  "space_guid": "REDACTED-GUID"
               }
            },
            "space_guid": "REDACTED-GUID",
            "organization_guid": "REDACTED-GUID"
         }
      },
      {
         "metadata": {
            "guid": "845e00cd-c3bb-445f-81ea-d04bda50963f",
            "url": "/v2/events/845e00cd-c3bb-445f-81ea-d04bda50963f",
            "created_at": "2015-04-27T20:30:37Z",
            "updated_at": null
         },
         "entity": {
            "type": "audit.service_instance.delete",
            "actor": "REDACTED-GUID",
            "actor_type": "user",
            "actor_name": "de...@unknown.org",
            "actee": "REDACTED-GUID",
            "actee_type": "service_instance",
            "actee_name": "cache-01",
            "timestamp": "2015-04-27T20:30:37Z",
            "metadata": {
               "request": null
            },
            "space_guid": "REDACTED-GUID",
            "organization_guid": "REDACTED-GUID"
         }
      }
   ]
}

Thanks for taking a look.

-Matt

[Denilson Nastacio]

I would like to see the answer for one of the original questions from Matt:

" Is there a way to the delete to try again or otherwise unstick the job from the API? What's the best way of fixing the state? "

[Mike Youngstrom]

We had a similar issue that appears to be the same.  Our problem only affected old services created prior to our deploying 20x.  When we researched this issue weeks ago we found that pivotal and found and fixed it in a later release but I cannot seem to find the tracker issue ATM.  An upgrade to latest CF version may fix you.  An upgrade wasn't an option for us so we fixed our issue by modifying the CCDB.  The fix for us was to create service_instance_operations rows for all of the services that didn't have one.  We did so executing the following queries against our CCDB:

//This query will give you an idea of how many service instances your system has.

select count(*) from service_instances;

//This query will show you how many service_instance_operations you have.  The difference of the two will tell you about how many service_instance_operations rows you need to insert.

select count(*) from service_instance_operations;

//This query will create a service_instance_operations entry for the next service_instance that doesn't have a service_instance_operations.  You must execute this query for until every service_instance has a service_instance_operation.  Once it stops updating rows you can stop.

insert into service_instance_operations (guid, updated_at, service_instance_id, type, state, description) select UUID(), NOW(), id, 'update', 'succeeded', '' from service_instances where id not in (select service_instance_id from service_instance_operations) limit 1;

//This resets the state of all the services stuck in "in progress"

update service_instance_operations set state='succeeded' where state='in progress';

//This gives all the operations a description other than null (not sure if this is necessary but it is what we did.

update service_instance_operations set description='' where description is null;

After doing all that our services were unstuck from being "in progress" and none of our services had this issue anymore.

Hope it helps.

Mike

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/a2371381-d2b3-4762-a406-65ece9fc9127%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Matt Cholick]

Mike,

That's great; thanks for the detailed series of SQL statement. That will help quite a bit.

-Matt