Writing a “circuit breaker” for use in Django

[Peter Coles]

I’d like to write a circuit breaker wrapper to use with services that I call from inside a Django app, and I’d like to get some pointers on the following questions:

1. What’s the most global, persisting, shared-across-requests type of data I can read & write to memory?
   
2. However I can do this, does this require using `threading.Lock()`? (looking around `dispatch.Signal` I see some use of it)
   
3. If this data is shared per-process, what’s a good way to see how many processes it would be (for uwsgi is that just the # of worker processes)?

The term “circuit breaker” is referring to service clients that cut themselves off from making future requests to a service after the service has failed to connect or read/write after a certain error threshold. They can employ retries to eventually right themselves. If a service continues to fail, even despite strict timeouts, it can significantly slow down a site or even cause the webserver to run out of threads and it will effectively reach a DoS scenario. The clients are per-webserver (or maybe in this case, per-process) and all independently can open or close their “circuits” based on what they’re experiencing.

To effectively track the effectiveness of service calls without having a performance impact on the webserver, it really needs to store this info in memory—which is the source of my original questions—and I assume this means I’m stuck with only info per-process? I’m OK with the info disappearing after server restarts, but might there be some gotchas around other configurations, like the `--harakiri` option for uwsgi? As stated above, I’m also a little unclear on when using threading locks would be required vs not, given that python uses the GIL—but maybe certain configurations do allow for non-thread-safe things to happen?

I found 2 barebones approaches on github, but the correctness/effectiveness of each was quite unclear:

• https://github.com/cuker/django-patchboard/blob/master/patchboard/circuitbreaker.py
  
• https://github.com/globocom/memcached_memoize/blob/master/memcached_memoize/decorators/circuit_breaker.py
  

Any help would be appreciated—whatever I build for this would definitely be open-sourced to the community!

[Bill Freeman]

In Django, requests should not wait, since the threads are relatively heavyweight in python, and need to be a limited resource.  The alternative is a large number of processes, which is also expensive.

So you must design a scheme in which the client polls for a result, meaning that you cannot expect it to be handled by the same process, let alone thread.  So the success of a response (and its data) must be stored in a resource shared across processes and threads.  Most obviously, this is you database, whether by and ORM Model, or (if small enough) in session data.  But it is also possible to use a secondary, in RAM store, such as redis, or maybe memcached (or an ad hoc separate process, by why reinvent the wheel?).

An alternative is to have these requests handled by something that does well with Websockets, such as tornado.  (Django may have work toward Websocket support, but I'm not aware of anything satisfying.)

Or, in the non-browser case of the client being able to provide a response endpoint, a Celery task could retry, and wen successful could POST to that endpoint.

Doing Websockets or long poll type interactions isn't what Django is designed to do well.  Some machinations are appropriate if most of what you have to do fits Django, but if all you need to do is drive a screw, don't waste time filing a blade onto the face of a hammer.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/35f791f2-d857-44de-a9e7-e52e346728b9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Coles]

Thank you for your input. I appreciate you trying to steer my away from potentially terrible design, however, I am still interested in someone addressing my main questions. Some level of service calls is inevitable in any interesting web application—even a call to an in-memory cache, like Redis or Memcached, counts as a service call that could fail (a truly resilient web-site would not fall over if the cache disappeared from DNS for 10 minutes). In fact one of the examples I linked to was a circuit breaker for memcached.

I could leverage something like the local memory cache to store this very light-weight information for a service client (which maybe would off-set that disclaimer in the docs about not using it in production) or as seen in the other examples, try something like storing values on a “global” class object or instance. I was curious if anyone who has worked with stuff like this in Django might have some pointers. Maybe Django Developers would be a better forum?

My original questions:

1. What’s the most global, persisting, shared-across-requests type of data I can read & write to memory?
   
2. However I can do this, does this require using `threading.Lock()`? (looking around `dispatch.Signal` I see some use of it)
3. If this data is shared per-process, what’s a good way to see how many processes it would be (for uwsgi is that just the # of worker processes)?

[François Schiettecatte]

Hi

Specifically addressing your questions:

- I would look at using shared memory to store data that you can share across processes, plenty of resources on the web that talk about this.

- You might want to use a small file to store data and fcntl.flock() to control access so you are not reading/writing at the same time.

I run Django via wsgi which spawns multiple processes so there is nothing within Django per-se that you can use, you have to go outside, i.e. shared memory, database or file.

François

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1935fd88-1c8d-4cb3-aa67-ef3aa4b36e75%40googlegroups.com.

[Avraham Serour]

> 1. What’s the most global, persisting, shared-across-requests type of data I can read & write to memory?

Don't complicate your life, just store data in the database

> 2. However I can do this, does this require using `threading.Lock()`? (looking around `dispatch.Signal` I see some use of it)

If you use the ORM you won't need any of this, you can have many processes even on different machines

> 3. If this data is shared per-process, what’s a good way to see how many processes it would be (for uwsgi is that just the # of worker processes)?

Of course you can, the question is why? Each request should be isolated and it shouldn't make a difference on how many workers you have

As per design, maybe you want an asynchronous processing, meaning the web request would just register that a request to calculate something was made, send the request to the jobs queue and return saying the request was received please come back later

you can use celery for this, there are others but it is fairly popular

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1935fd88-1c8d-4cb3-aa67-ef3aa4b36e75%40googlegroups.com.

[Bill Freeman]

If you have your heart set on a circuit breaker pattern (patterns are overrated, see http://www.paulgraham.com/icad.html), consider implementing it in the client.

If you need a long poll solution, note that you can serve some urls from uwsgi/Django and others from tornado or twisted.

If you are willing to actually poll, the result of a backend service failing should be a 500 or maybe 400 or 300, series return code -- not having the Django request wait.  One that your client will interpret as try later: you can include a suggested wait time in the response.

You can keep it all in Django, but you are unlikely to find examples, since most folks would have done it one of the other ways, so you are breaking new ground on your own.  (I'm hoping to bee shown to be wrong by someone who has implemented such a long poll solution, and who is free to talk about it.)

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFWa6tKfoCQYHeF-hC1ZTDPbNOTCUX%3D-GnE3AfnDtyeMQ2nuNw%40mail.gmail.com.

[Peter Coles]

Thank you for the pointed response, François. I was thinking about accessing some shared memory vs just doing something simpler and just being per-process. The prior sounds more thorough, but the nature of what I’d like to use it for is already OK with separate clients across machines, so maybe across processes isn’t too bad either.

Bill & Avaraham, here’s an example from the netflix blog of what I’m talking about: http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html — it’s not a new idea and it’s an industry standard approach to building resilient, high-capacity websites and services (especially with any SOA or even micro-service architecture).

I’ll go my own path from here :)