Web-framework+db with the widest scalability?

[Alec Taylor]

Disclosure: I have posted this question on stackoverflow and comp.lang.python.

I am building a project requiring high performance and scalability, entailing:

• Role-based authentication with API-key licensing to access data of specific users
• API exposed with REST (XML, JSON), XMLRPC, JSONRPC and SOAP
• "Easily" configurable getters and setters to create APIs accessing the same data but with input/output in different schemas

A conservative estimate of the number of tables—often whose queries require joins—is: 20.

Which database type—e.g.: NoSQL or DBMS—key-value data store or object-relational database—e.g.: Redis or PostgreSQL—and web-framework—e.g. Django, Web2Py, Tornado, Pylons or Flask—would you recommend?

Thanks for all suggestions

[Vlad K.]

There's no right answer to that.

I suppose the choice of DB depends on your data structure. Do you need simple key-value storage, data store, or normalized database and relations? Can you live with denormalized data? Do you need complex queries for which SQL is most suitable? Do you need map/reduce functionality? Do you need ACID compliance or can live with missing, unsynced or corrupt data in case of a failure?

You're mentioning tables and joins so methinks that alone limits you to a RDBMS, or at least makes everything else a huge pain. Or maybe your app does not even need joins?

A highly biased answer from my experience would be: PostgreSQL. I've tried MongoDB, for example, used it in a project for a year and found that there is nothing in MongoDB that I cannot do with Postgres, which I'm familiar with and been using for many years. I can also use and/or tune Postgres (and the OS) to drop ACID compliance and behave just like MongoDB. But that means that I can revert it back to fully ACID compliant with just a few changes in configuration, if I regret dropping ACID (no pun intended), instead of having to switch entire DB backend and all the models. I can make it multi-master statement-level replicated with no integrity checks for maximum performance and maximum scalability and hope there's no failure, or if failure doesn't mean big deal.

I can rapidly prototype and develop the app with an excellent ORM such as SQLAlchemy and later switch to highly optimized queries or even move parts of data business logic to DB via stored procedures / functions, if I wanted to. I can always put memcached in between the app and the db and make it as consistent as I want. Or even add a persistent storage NoSQL layer to help out where and if needed.

If you haven't seen it, I suggest this interesting, albeit a bit flamey vid:  http://youtu.be/b2F-DItXtZs


DB abstraction? Perhaps, but my opinion is why don't use the maximum a (R)DBMS can offer, which often means forfeiting portability, unless you need it.

As for Python frameworks, it also depends what you need. Full blown stacks, thin apps or my personal favorite Pyramid which sits somewhere in between those two and I can bend it to my will any way I want and not be limited with assumptions and inadequate (for my needs) design decisions, often inherent in a full blown stack.

Sounds to me like you're dealing with a rather big project or at least something that you expect to become a rather big project at which point even minute changes to the system may turn out to be huge pain. So with that I suggest you to try out several DBMS solutions and see which suits you better and make sure you're familiar with the benefits and drawbacks of each, especially with a NoSQL system in which sometimes drawbacks are not as obvious until you hit particular situations and they become obvious.

.oO V Oo.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/pylons-discuss/-/eRHQsprDYQgJ.
To post to this group, send email to pylons-...@googlegroups.com.
To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.