Uploading data as blob -- hitting a brick wall

[Joe Barnhart]

I'm hitting a brick wall when trying to convert my table (under PostgreSQL) to use "blob" storage.  The problem seems to be that web2py doesn't properly recognize and initialize the blob field.  For example, when I tried this:

            db.define_table("fileobject",

                Field("upload_data","blob"),

                Field("upload","upload",uploadfield='upload_data'),

                ...

The code fails when uploading the blob.  In the DAL, in the code for Field.store():

        self_uploadfield = self.uploadfield

        if isinstance(self_uploadfield,Field):

            blob_uploadfield_name = self_uploadfield.uploadfield

            keys={self_uploadfield.name: newfilename,

                  blob_uploadfield_name: file.read()}

            self_uploadfield.table.insert(**keys)

            ...

 

In this case, the test to see if self_uploadfield is a Field FAILS -- because the uploadfield still has the string "upload_data", not the corresponding Field.

Next, I thought I'd try the old-school way, I put the Field definition as the argument to uploadfield:

            db.define_table("fileobject",

                Field("upload","upload",uploadfield=Field("upload_data","blob")),

                ...

Again, this FAILS, but for a different reason -- the "table" attribute of the "upload_data" Field never got initialized to "fileobject"!

I just can't find a way around this.  I'm using web2py version 2.5.1-stable under OSX (running from source).

-- Joe B.

[Joe Barnhart]

I had an idea to work around the problem...

Define the blob field in the table and then set the "uploadfield" parameter (of the "upload" field) AFTER table creation!  Perfect (I thought), I'll have the contents of the uploadfield as a Field (not a string) and I'll have the "table" attribute of the blob field set to the proper table. 

            db.define_table("fileobject",

                Field("upload_data","blob"),

                Field("upload","upload",uploadfield="upload_data"),

                ...)
            db.fileobject.upload.uploadfield=db.fileobject.upload_data

It shoulda worked...  But it didn't.

The problem was that the line:

blob_uploadfield_name = self_uploadfield.uploadfield

Now resolves to "True" -- because the blob field has its upload field parameter set to "true" by default.

Fine, I'll change the DAL and set it to the name of the blob field.

blob_uploadfield_name = self_uploadfield.name

Hooray!  I finally loaded data into my blob field!  Only... it loaded TWO ROWS in the table.  The first row holds only the binary data (and a length field I computed) while the next row holds all the metadata!

5;"";;"2013-07-29 18:27:00";505;"";"";"";"ZnJvbS...KQ=="
6;".py";1;"2013-07-29 18:27:14";;"form";"fileobject.upload.97409537b5b00657.706f72747363616e2e7079.py";"portscan.py";""

Looking at the code, I can see the handling for "upload" to a blob field results in an immediate "insert" statement, and that generates a row in the table without waiting for any other fields.

self_uploadfield.table.insert(**keys)

I would stake my meager reputation as a Python code reader that the "upload" to blob feature of web2py has been majorly borked, perhaps in a recent update.  It doesn't work anything like described in the PDF or online book.  It appears almost like it expects a separate upload TABLE, and it inserts the data into that other table while processing the insert for the "master" table.

If anyone is using 2.5.1 to store data in a database as blobs, and is not using GAE, how'd you do it??!?

-- Joe B.

[Niphlod]

answered in the other thread. This is just another case where packing a simple app and attaching to the thread showing exactly what your code does would have saved (you and me) lots of headaches :-P

[Joe Barnhart]

Hi Niphlod --

Thank you for pointing me in the right direction.  Although my application is too complicated to simplify (or more likely I'm not skilled enough to do it quickly) I was able to create a simple "form" example like you tested and put breakpoints in WingIDE to find out what is actually going on.

I discovered the place in SQLFORM.accepts() where the dirty work is done and what is needed to replace it.  Here's the code for those who follow this thread later... (if anyone is crazy enough!)

newfilename = field.store(source_file, original_filename,

                         field.uploadfolder)

fields[fieldname] = newfilename

if isinstance(field.uploadfield, str):

    fields[field.uploadfield] = source_file.read()

The var "field" in this case has been set to the table.upload field, so the variable "uploadfield" is tested to see if it is a string and the file is read into the blob data field if true.

I'll be able to test my application using DB-stored data now.  I'm not sure if its advantages will outweigh the drawbacks, but at least I can give it a proper try now.  Thanks again!

-- Joe B.