Problem with case list reports and case APIs

[Cory Zue]

Hi all,

We're having a problem with one of the pieces of infrastructure used to generate the case list reports (including case reassignment), and apis. We are looking into it urgently and will let you know as soon as things are back up and running.

Cory

[Cory Zue]

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

[Sarah Bird]

Hi Cory,

I'm now getting a 500 error?

Bird

--
You received this message because you are subscribed to the Google Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
skype: birdsarah
email: sb...@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

[Sarah Bird]

Sorry, to clarify, that's when I'm trying to get cases through the API.

CommcareHQ is saying "No data available to display. Please try changing your filters."

Thanks,

Bird

[Cory Zue]

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking into the root cause again.

Cory

[Emeka2015]

Just so you also know Cory, I generated a case list before your email. With 16 cases, 3 of which were closed, but now regenerating that same user cases now has 14 cases with 2 closed users. Meaning 2 cases not appearing.

-Emeka Chukwu-

Sent from my iPhone

[Daniel Myung]

Hi Users,

Since the resurfaced problem a few minutes ago, we've again restarted our search service system which is slowly coming back online - when it's in a partial state - it may return data that it has available. 

OK, the services are fully back online - can you test again?

If there is missing data, there is a chance that newly registered cases that came in during the outage were not picked up.

Dan

[Daniel Myung]

To clarify:

The case api and search service is fully back online, we are now running tests to ensure our search index is up to date with the database to ensure that the two are in sync.

Dan

[Sarah Bird]

500 again?

[Daniel Myung]

Yes, sorry - it's gone offline again - am working on getting it back up.

[Sarah Bird]

No worries. Difficult day :( Thanks for all your hard work.

[Daniel Myung]

Dear Users,

We have completed a rebalance and reallocation of resources on our search+api machines and restored its stability, but we'll be continuing to keep a close eye on things.  We have resumed synchronization for the past hour with the database, and it is back in sync, and the search+api service is responding well. So we think the issues have been resolved and will be OK. The issue stemmed from our case API/search index being crowded on just 1 of our 2 servers causing it to not respond in a timely manner. I have spread the case index over the 2 machines and it is behaving significantly better now.

We're also going to be focusing on longer term fixes to address the underlying problem - and will keep you posted should anything require attention again.

Thanks for the quick feedback and alerts when issues did come up, and apologies again for the inconvenience.

Dan

[Sarah Bird]

Looking good from here. Thanks!