One database per client

[Diego Woitasen]

Hi,

 I'm developing a SaaS application. I want to store the collections of every client in different databases. I was reading and search in Google and haven't found a practical way to do it.

 If there a way to switch the database at runtime? Globally. My application has two parts, one runs one process per client and the other one is a Django front-end. In the latest, I should find a way to switch the DB at runtime when the user is logged. In the first case is easy.

Regards,

  Diego

[Diego Woitasen]

Hints? Context managers looks too complex, I need something easier :)

I tried calling register_connection() to modify the default connection but it doesn't work :(

[Diego Woitasen]

I forgot to mention that I'm using Flask with the restful extension.

[dorian i]

How does the process know for what client it is working?

[Diego Woitasen]

Session information related with the current request.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "MongoEngine Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/mongoengine-users/69dzr3stPb4/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to
> mongoengine-us...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Diego Woitasen

[Ross Lawley]

As of 0.8.0 there is a switch database context manager.  That might work for you.

Ross

You received this message because you are subscribed to the Google Groups "MongoEngine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongoengine-us...@googlegroups.com.

[Diego Woitasen]

Yes, but it requires a switch per class. I want to do it globallly.

[Ross Lawley]

Perhaps a custom context manager for the request to setup the connections - or monkey patching the connection code.

Ross

[Diego Woitasen]

Yes, I had a look at the context manager and could do it in the
distpatch_request() function of the Resource object of Flask-restful.

Anyway, it would be great to have a generic solution, If there is a
good idea, I have no problem to create a pull request :)

Regards,
Diego

[Ross Lawley]

Happy to take some code! - might be one for mongoengine-extras as it is quite niche.

Ross

[Diego Woitasen]

Let me know what do you think about this and I'll write a more generic
solution to be added to MongoEngine extras. May be a new XDocument
class can be used these special cases.

https://gist.github.com/diegows/5654284

Regards,
Diego

[dorian i]

What do you think of the comments in this issue Diego ? (https://github.com/MongoEngine/mongoengine/issues/222)

On Thursday, April 25, 2013 8:00:08 PM UTC+2, Diego Woitasen wrote:

[Diego Woitasen]

Interesting. I'll took some ideas from there to write the generic implementation. The code that I pasted in gist is working right in my app. When I finish some details there I'll post new code.

--

[dorian i]

https://gist.github.com/ddorian/5659485 (has many flaws but it is a start?)

On Thursday, April 25, 2013 8:00:08 PM UTC+2, Diego Woitasen wrote:

[dorian i]

Why are you using one db per customer?

On Thursday, April 25, 2013 8:00:08 PM UTC+2, Diego Woitasen wrote:

[Diego Woitasen]

Because there are some collections that are really huge, specially for
some clients.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "MongoEngine Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/mongoengine-users/69dzr3stPb4/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to
> mongoengine-us...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
Diego Woitasen

[dorian i]

But why not 100 collections in one db, to lower the overhead of .ns files and to create less .db files?

[Diego Woitasen]

Well, I haven't considered that. If you have link about that kind of details of Mongo I could consider a change in a future release ;-)

[dorian i]

Basically you have a (default) limit of 22,000 namespaces for each database. A collection/index is a namespace. Depending on your schema:

Suppose you have 10 collections per client and each collection has 5 indexes. That's 50 namespaces. Leave a little room for new namespaces (changes and features in the future), say another 50.
So 1 client needs 100 namespaces. That means ~200 clients in one db(22,000/100~=200).

Then there are several ways to specify in which db the clients collection will reside.

I also suppose that there is a collection with every client record(not client's data). You can create a field in this document called "db" and another called "collection_number"(a SequenceField will be easier).

When a client document is created you can check each db that exists for the number of collections that each have.

Or you can create some db and when a client is created choose a random db. Over time check if some db's are reaching the maximum number of namespaces and remove them from you random.choices(list).

If a client has collections[pages, users, log]. For this client you create collections[pages24, users24, log24]. (24 in this case is the client.collection_number)

And then on every request:

client = Client.objects.get(pk=pk)

client_data = Page.objects(database=client.db, collections=client.collection_number)

[Ross Lawley]

Hi all,

Interesting conversation - I'm happy to for more feature requests on github (pull requests are better ;)) - either for core mongoengine or extras :D

Ross

You received this message because you are subscribed to the Google Groups "MongoEngine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongoengine-us...@googlegroups.com.

[Diego Woitasen]

I think the pull requests should go for pymongo :)

[dorian i]

This is easly achived in plain pymongo !?

The hard part is changing/patching the internals of mongoengine to be more flexible and overridable.

[Diego Woitasen]

I've been reading a lot of code the last days...

I think it should be done in Pymongo, because MongoEngine doesn't have
access to the low level details of the cursors.

Right now the Pymongo engine works in this way (this is not the real
code, just a simplified version):

if len(self.batch) == 0:
self.batch = self.get_another_batch_from_server() #this operation is blocking.

return self.some_data_from_batch()

I think it should work in this way:

if self.batch_requested:
result = self.get_batch_from_server() # non-blocking
if len(result) > 0:
self.batch_requested = False

if len(self.batch) < (self.batch_size/2):
self.request_another_batch_from_server() #non blocking function
self.batch_requested = True

return self.some_data_from_batch()

I'll have some time next week to work on this, comments are welcome in
the meantime :)

Regards,
Diego

[dorian i]

Both your last replies are for another thread Diego. The "Understanding cursors" on group/mongodb one !

[Diego Woitasen]

ups, I didn't read the title :)

[Diego Woitasen]

Going back to this topic, after running the application for a couple of weeks using one DB per client I think it was a good decision. MongoDB locks the database while it's reading/writing it. So if I have huge and busy collections, putting them in a separate DB makes the locks independent per each client.

Just a comment that could be helpful for somebody else.

On Thursday, April 25, 2013 3:00:08 PM UTC-3, Diego Woitasen wrote:

[dorian i]

But if you have a big number of clients or many small clients the overhead of .db/.ns files becomes larger.

--

You received this message because you are subscribed to a topic in the Google Groups "MongoEngine Users" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongoengine-users/69dzr3stPb4/unsubscribe.

[Diego Woitasen]

Can you explain that overhead? Because at first, I don't see any problem to have more files.

[dorian i]

See: http://docs.mongodb.org/manual/reference/configuration-options/#smallfiles

[Diego Woitasen]

Ok, but it's not my case. I'm not using that option and I have big
databases (specially, one big collections) and the db quantity in the
future is uncertain :)

If the number is large, I won't care about that because I'll have
enough budget to change anything :)

Regards,
Diego

Diego Woitasen