Sharding with Django

[Brett Elliott]

How can I 'shard' with Django? What I mean by sharding is accessing
multiple databases to combine data from multiple databases. This would
require access to the database connection API so that the view has
access to multiple database connections. Perhaps I could add code to a
filter to set up the connections, then somehow I can access those
connections from the view.


Thanks for any assistance,

Brett

[Malcolm Tredinnick]

At the moment, this isn't supported. There was some multi-db support
done at one point and I thought somebody was going to be hosting that
externally whilst they worked on it, but nothing seems to have been
announced. However, that's not really sharding in any case: each field
(possibly each model) would all be stored in a single database, just
using multiple databases for storing different fields/models. Sharding
usually refers to splitting the data set so that one group of rows are
in one database and another are somewhere else (i.e. the same field has
data in different locations). Sharding is much harding to make work in a
truly generic sense that multiple database support.

Regards,
Malcolm

--
Honk if you love peace and quiet.
http://www.pointy-stick.com/blog/

[Brett Elliott]

On Nov 15, 4:39 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:

Yes that is a better description of sharding. I do use sharding in
another project and I find it essential to scaling, but I had to do a
bit of one-off Java trickery to get it to work correctly. Sharding is
very had to get right in the general sense. One project that I know
of, Hibernate Shards for Java( http://www.hibernate.org/414.html,
thanks Google), seems to do a good job at it although I haven't used
it.

Assuming you have a person stored in one database and the person's
relatives stored in another database, I imagine an API that would
allow something like:

person = Person.shards.get(pk=123)

relatives = person.getRelatives()

then the shards manager would do:

def getRelatives(person):
database_id = person.database_id
dbinst = database.get(database_id)
relatives = Relative.shardmanager.get(person=person)
return relatives

I can start looking at the Django code to see if it can be made to do
something similar.

[crybaby]

python orm sqlalchemy allows sharding. I would like to me sqlalchemy
integrated into django. I am python noob, so I can't go around
hacking the framework. I like django, but the whole framework is not
decoupleable.