Re: Web2Py in cluster and DATABASES folder

[Niphlod]

ehm... deploy. production. cluster.
You say that "some .table files are created" hence I think you're using the same database uri both for production and development (if any). If .table files are created, it's safe to assume that the underlying code changes too, right ?
If so, a simply copy/rsync from a "development" to all the "cluster" machines suffices, without you ending in misery. Why all the pain if you have to sync the code accross cluster if .table files can travel the same way as your codebase ?

Il giorno giovedì 23 agosto 2012 17:14:51 UTC+2, Mike ha scritto:

We are planning on deploying our Web2Py application under Apache-WSGI on a cluster of servers all accessing the same multiple databases. Since the new databases will be created on the fly, as needed, we cannot predict beforehand how many we will end up using and, consequently, how many files will end up being created in DATABASES folder. As a result, I am trying to create sub-folders under DATABASES and set MIGRATE parameter on .define_table() call for a given database to <sub-folder name>/<table name>.table. Such approach allows me to better control the overall number of files in a single folder as well as make the overall picture more manageable. There is one problem with this approach and one question for the experts:

1. A problem - we are using Auth to authenticate users and so need to perform auth.define_tables(). By passing to it migrate="<sub-folder name>/", I almost made it create all the auth_ table files in that sub-folder, except that there is a typo in TOOLS.PY, line 1420 in my source code for version 1.99.7, where instead of expected
   
migrate=self.__get_migrate(settings.table_cas_name, migrate),
                           
it goes
   
migrate=self.__get_migrate(settings.table_event_name, migrate),

I've fixed it in my copy and would like to request that the fix should be made in the official release code base as well.

2. A question - should DATABASES folder be shared among the Apache-Web2Py machines or should each maintain its own copy? More specifically, is there any locking mechanism on those .table files built specifically for the purpose of sharing between threads/machines? I intend to use migrate_enabled=False when creating an instance of DAL if I know that the tables have already been created, but to make sure that multiple machines won't try to create the new database/new sub-folder at the same time, I will probably have to build some locks outside DAL. Is this even the right way of thinking about this problem?

Sorry for the long-winded question. Great framework, BTW. :)

[Massimo Di Pierro]

If they share the database, they should also share the folder. Anyway, mind that file IO is a major bottleneck and accessing a shared filesystem is every worse.

Setting migrate=False prevents the file system access completely.

Ideally should have a way to set migrate=True only when the new organization joins and migrate=True when it joined already. You should be able to perform this check without looking at the filesystem.

On Thursday, 23 August 2012 12:15:49 UTC-5, Mike wrote:

Perhaps I haven't made myself clear enough...

Forget about development environment - think cluster deployment scenario only.

I can't create databases in advance - in fact, the server code is written to create a new schema (using MySQL parlance) every time a new group of customers (an "organization") is joining the site. As such, any of the servers in the cluster at any point in time could be faced with the task of creating a new schema, a new sub-folder inside DATABASES and initializing the schema with appropriate tables and sub-folder - with appropriate .table files. My only issue is whether or not DATABASES folder should be shared among the machines in the cluster (like SESSIONS, for instance) or if each of them should maintain its own (less preferable, for various reasons). And that bug will have to be fixed, of course.