Couple of questions about improving performance in models

[Lisandro]

Straight to the point:

I'm using lazy_tables=True and migrate=False

In my model, I have several Method Fields.

Some of them use a function that is declared inside the same model, like this:

db.auth_user.give_me_data = Field.Method(lambda row: _give_me_data(row.auth_user))

def _give_me_data(user):
    # calculation and obtention of data
    # return data

So first question: should I move declaration of _give_me_data function to a module? 

Or is it the same because I would have to import it anyway in the model?

Just to remind, I'm using lazy_tables=False, so maybe the code isn't really executed until the table is called. I'm not sure about this.

Usage of Storage class objects

In my models, I create some Storage objects, because it's really useful for setting and retrieving data from those objects. I use them like this:

from gluon.tools import Storage

CONFIG = Storage()
CONFIG.variable1 = 'value-variable-1'
CONFIG.variable2 = 'value-variable-2'

I don't put much data inside them, just config variables.

However, my question is: should I use other data type to storage those variables? Or it won't make any difference?

Thats all for now. Any comment will be appreciated. Thanks in advance!

[Anthony]

On Thursday, March 17, 2016 at 1:08:49 PM UTC-4, Lisandro wrote:

Straight to the point:

I'm using lazy_tables=True and migrate=False

In my model, I have several Method Fields.

Some of them use a function that is declared inside the same model, like this:

db.auth_user.give_me_data = Field.Method(lambda row: _give_me_data(row.auth_user))

def _give_me_data(user):
    # calculation and obtention of data
    # return data

So first question: should I move declaration of _give_me_data function to a module? 

Or is it the same because I would have to import it anyway in the model?

Just to remind, I'm using lazy_tables=False, so maybe the code isn't really executed until the table is called. I'm not sure about this.

Note, as soon as you do db.auth_user, the table is no longer lazy. Maybe not a big deal for this one table.

Probably doesn't matter much where you define the function, though I suppose there could be a slight performance benefit to putting it in a module (where it will only be defined once instead of on every request).
 

Usage of Storage class objects

In my models, I create some Storage objects, because it's really useful for setting and retrieving data from those objects. I use them like this:

from gluon.tools import Storage

CONFIG = Storage()
CONFIG.variable1 = 'value-variable-1'
CONFIG.variable2 = 'value-variable-2'

I don't put much data inside them, just config variables.

However, my question is: should I use other data type to storage those variables? Or it won't make any difference?

Again, probably not a big deal, but you could move that to a module, or maybe use AppConfig (which caches the configuration data after the initial setup).

Anthony

[Lisandro]

Thank you very much for the quick answer.

Your first comment made me think about the "lazy_tables" with the definition of virtual method fields.

If you say that doing db.tablename the table is no longer lazy, then it wouldn't make sense to use lazy_tables in a particular case where almost each table has a virtual method field, am I right?

I mean, for example, if this is my model:

db.define_table('first_table', Field('field1'), Field('field2'))

db.first_table.field3 = Field.Method(lambda row: 'value for field3')


db.define_table('second_table', Field('field1'), Field('field2'))

db.second_table.field3 = Field.Method(lambda row: 'value for field3')



...then it would make no sense to use lazy_tables=True, because anyway I'm calling both tables in the model, right?

If this is the case, is there any alternative to this?

I'm thinking to define a function in a module, the function would take the row as an argument and return the value. 

However, it wouldn't be possible to call the function in the "row.method" way, but if it improves performance, I will do it that way.

I would appreciate any comment or advise on this, if any.

Thanks again!

[Anthony]

On Thursday, March 17, 2016 at 1:53:43 PM UTC-4, Lisandro wrote:

Thank you very much for the quick answer.

Your first comment made me think about the "lazy_tables" with the definition of virtual method fields.

If you say that doing db.tablename the table is no longer lazy, then it wouldn't make sense to use lazy_tables in a particular case where almost each table has a virtual method field, am I right?

I mean, for example, if this is my model:

db.define_table('first_table', Field('field1'), Field('field2'))

db.first_table.field3 = Field.Method(lambda row: 'value for field3')


This won't help with db.auth_user, which is defined automatically (though you can define it manually yourself if preferred), but you can simply do:

db.define_table('first_table',
                Field('field1'),

                Field('field2'),
                Field.Method('Field3', lambda row: 'value for field3'))

Note, the first argument of Field.Method() and Field.Virtual() should always be the field name anyway, even if you are defining them like this:

db.mytable.myfield = Field.Method('myfield', ...)

There are particular cases where leaving the name out can lead to problems.

Anthony

[Lisandro]

Oh I see! So I can define the virtual/method fields that way, and then it does make sense to keep using lazy_tables. Completely missed that point.

Thank you very much! 

As always, web2py's community rocks. 

[Lisandro]

I will bother with one last related question: what about callback triggers?

I'm defining them like this:

db.define_table('mytable', Field('field1'), Field('field2'))  # defines the table
from myglobals import myfunction  # imports global function
db.mytable._after_update.append(lambda s, r: myfunction(s, r))  # defines after update callback trigger

In this case, I guess lazy_tables=True wouldn't make sense, because I'm calling db.mytable so it triggers the definition of the table.

Is there a way of specifying callback triggers directly when calling define_table?

[Anthony]

See http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#on_define.

[Lisandro]

Anthony, thank you very much for your help.

I've successfully changed the definition of the callback trigger, like this:

db.define_table('mytable',
  Field('myfield'),
  on_define=lambda table: table._after_update.append(lambda s, r: myfunction(s, r))
)

I'm tempted to ask one more thing that I'm wondering about:

In my model, I  have several tables (lets say about 8 or 9) that are used to store information about list of options. 

These tables only have a few records each, and they will never grow.

For example, one table could be "days" (I don't have this table specifically, it's for the example).

So, let's say the table "days" has a "name" field and a "plural_name" field.

The table is populated with data only once, at application startup, and remains the same forever: "days" table will only have 7 records, one for each day of the week.

Now, the part that I'm worried about is this: inside the same model, I store each record in a variable, like this:

db.define_table('days', Field('name'), Field('plural_name'))

MONDAY = db.days[1]
TUESDAY = db.days[2]
WEDNESDAY = db.days[3]
THURSDAY = db.days[4]
FRIDAY = db.days[5]
SATURDAY = db.days[6]
SUNDAY = db.days[7]

I use that to be able to reference each day from the app code, but I've realised that in this way, the "days" table won't be lazy, because it's called to get the record.

So my question is: what would be the correct approach for storing that kind of records in variables, without compromising lazy tables?

I've thouth of an answer myself, but I'm not sure. 

One possible approach would be to define Storage objects instead of pointing to the records. 

Because I already know the data of the days table, I can replace the previous code with this:

from gluon.tools import Storage

MONDAY = Storage()
MONDAY.id = 1
MONDAY.name = 'monday'
MONDAY.plural_name = 'mondays'

Would it be better in terms of performance?

Of course, I wouldn't have methods available at the row level, but I don't use them in this case.  

[Dave S]

On Friday, March 18, 2016 at 10:19:40 AM UTC-7, Lisandro wrote:

Anthony, thank you very much for your help.

 

[...]

I'm tempted to ask one more thing that I'm wondering about:

In my model, I  have several tables (lets say about 8 or 9) that are used to store information about list of options. 

These tables only have a few records each, and they will never grow.

For example, one table could be "days" (I don't have this table specifically, it's for the example).

So, let's say the table "days" has a "name" field and a "plural_name" field.

The table is populated with data only once, at application startup, and remains the same forever: "days" table will only have 7 records, one for each day of the week.

Now, the part that I'm worried about is this: inside the same model, I store each record in a variable, like this:

db.define_table('days', Field('name'), Field('plural_name'))

MONDAY = db.days[1]
TUESDAY = db.days[2]
WEDNESDAY = db.days[3]
THURSDAY = db.days[4]
FRIDAY = db.days[5]
SATURDAY = db.days[6]
SUNDAY = db.days[7]

I use that to be able to reference each day from the app code, but I've realised that in this way, the "days" table won't be lazy, because it's called to get the record.

So my question is: what would be the correct approach for storing that kind of records in variables, without compromising lazy tables?

For tables this small, does lazy make a difference?  You're only calling the Field constructor twice in this example.

You may have a good reason for having this sort of thing as a table, but if you're just wanting a global list, putting an actual list in the globals seems reasonable.

/dps

[Anthony]

On Friday, March 18, 2016 at 1:19:40 PM UTC-4, Lisandro wrote:

Now, the part that I'm worried about is this: inside the same model, I store each record in a variable, like this:

db.define_table('days', Field('name'), Field('plural_name'))

MONDAY = db.days[1]
TUESDAY = db.days[2]
WEDNESDAY = db.days[3]
THURSDAY = db.days[4]
FRIDAY = db.days[5]
SATURDAY = db.days[6]
SUNDAY = db.days[7]

I use that to be able to reference each day from the app code, but I've realised that in this way, the "days" table won't be lazy, because it's called to get the record.

Forget about the laziness of the table here -- that's trivial compared to the time it will take to run seven separate queries on every request. Instead, you should retrieve all the rows in a single query and cache the result for a really long time.
 

I've thouth of an answer myself, but I'm not sure. 

One possible approach would be to define Storage objects instead of pointing to the records. 

Because I already know the data of the days table, I can replace the previous code with this:

from gluon.tools import Storage

MONDAY = Storage()
MONDAY.id = 1
MONDAY.name = 'monday'
MONDAY.plural_name = 'mondays'

Would it be better in terms of performance?

Of course, I wouldn't have methods available at the row level, but I don't use them in this case.  

Yes, something like this would be even better if you really don't have any need for database functionality here. You could even put the data in a module and import.

Anthony