Suggestion - a pickled DAL field

[Omri Har-Shemesh]

Hi web2pyers,

very often, I have a field in the table which has to hold a complicated

value (most often numpy arrays). The way I implement it is that I use

"text" as the type of field, and then simply pickle.dumps(my_value) when

I save the field, and then pickle.loads(value_from_db) to access the field

again. 

My suggestion is simple - create field that automatically pickles the values

it gets and unpickles them on extraction. Is this already implemented? 

Do you have other suggestions on how to implement this so that I won't need

to pickle every time I access the table?

Best wishes,

Omri

[Anthony]

You could use a SQLCustomType field: http://web2py.com/books/default/chapter/29/6#Custom-Field-types-(experimental).

Anthony

[Derek]

Make it a computed field?

[Anthony]

On Tuesday, July 24, 2012 5:58:29 PM UTC-4, Derek wrote:

Make it a computed field?

I'm not sure that would be helpful in this case. A computed field computes its value automatically based on other fields in the record, but in this case, he needs to pickle an object, which is not one of the other fields. Also, it needs to be unpickled when queried, which a computed field wouldn't handle. 

Anthony

[Omri Har-Shemesh]

Thanks!

I will try it out :)

[Toby Shepard]

I haven't used one, but I remember coming across it in the manual.
Don't you just pass a lambda when you create the field. It would
seem that you could do anything you want to the value without consulting
other fields.

[Anthony]

>     Make it a computed field?
>
>
> I'm not sure that would be helpful in this case. A computed field
> computes its value automatically based on other fields in the record,
> but in this case, he needs to pickle an object, which is not one of the
> other fields. Also, it needs to be unpickled when queried, which a
> computed field wouldn't handle.
>

I haven't used one, but I remember coming across it in the manual.
Don't you just pass a lambda when you create the field.  It would
seem that you could do anything you want to the value without consulting
other fields.

Are you talking about a computed field? The compute function only runs if you don't pass a value to the field -- it calculates a value for the field based on the values of other fields in the record. Also, it doesn't do anything to the value upon query, so wouldn't help with the unpickling in this case.

Anthony

[howesc]

For what it is worth i have used custom fields for this type of thing with great success!

[Omri]

So I tried it out, and it works great, except for one thing - when I try to use record.as_dict() to get

the structure which I can return to a JSON-RPC call which I normally use to communicate with my

client, it simply does not return the field. 

I tried to print it out and I see very clearly that as long as it is a Row object there is no problem, but

as soon as I run .as_dict() it disappears.

My definitions are as follows:

from gluon.dal import SQLCustomType

pickled = SQLCustomType(

    type = 'text',

    native = 'text',

    encoder = (lambda x: pickle.dumps(x)),

    decoder = (lambda x: pickle.loads(x))

)

then I use

db.define_table("my_table",

    Field("pickled_field", type=pickled)

)

but when I fetch the record it does not go to as_dict() (actually I use as_list() but both don't work).

Any suggestions? Is there a way to teach as_dict to interpret this field type?

Best wishes,

Omri

--
 
 
 

[Omri]

I use the pickled field to store a dictionary. Since dictionary

is not defined as a serializable type for the as_dict function,

it is not returned when calling to as_dict. 

Adding it (dal.py line 6183) solved my problem, but I would

like to keep this change when I update my web2py next. Is

there a different way (without changing the dal) to achieve this?

Could you add it to the dal.py? Is it without risks?

Thanks,

Omri

[Derek]

You could monkey patch it in 0.py in models, perhaps.

[Omri]

Hi Derek,

thanks! the as_dict method has already been fixed in trunk to handle

my use-case.

best wishes,

Omri

--
 
 
 

[Derek]

aww, I love to see a cute monkey patch.