RuntimeError: Unable to handle upload

[Bill Thayer]

I am attempting to write a function that will eventually go in to a migration script to upload legacy data files. As you can see I use the DAL and define the wiki_media table. The function below looks just like the example in the book.However when I run it from the command line I get an error that the _attempt_upload(fields) method cannot handle the upload. See below.

Doe anyone know of a way to insert files into the database with a script? I am using an upload field instead of upload directory. Also using a recent trunk version of web2py.

import os, sys

from gluon import DAL, Field

from easygui import fileopenbox, diropenbox

db = DAL('oracle://...',

         auto_import=True)

db.define_table('wiki_media',

                Field('wiki_page', 'reference wiki_page'),

                Field('title', required=True),

                Field('filename', 'upload', required=True),

                Field("filedata", "blob"),

                migrate=False)

db.wiki_media.filename.uploadfield='filedata'

file_id = None

def upload_file():

    """demonstrate uploading a file programmatically,

    """

    stream=None

    file_path=fileopenbox(msg='Pick a file', title='Choose file', default=None)

    if file_path:

        print 'file_path= '.join([file_path])

        stream = open(file_path, 'rb')

    else: return None

    file_id=db.wiki_media.insert(filename=db.wiki_media.filename.store(stream,file_path),

                       filedata=stream.read())

    stream.close()

    return file_id

Error:

Microsoft Windows [Version 6.1.7600]

Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\bthayer>cd %WEB2PY_HOME%

C:\web2py-1045bab06391\web2py-1045bab06391>cd appl*/TAMOTO

C:\web2py-1045bab06391\web2py-1045bab06391\applications\TAMOTO>cd private

C:\web2py-1045bab06391\web2py-1045bab06391\applications\TAMOTO\private>python

Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32

Type "help", "copyright", "credits" or "license" for more information.

>>> import test_tamoto_rpc as trpc

>>> trpc.upload_file()

C:\Users\bthayer\Documents\car\carfiles.txt

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "test_tamoto_rpc.py", line 33, in upload_file

    filedata=stream.read())

  File "C:\web2py-1045bab06391\web2py-1045bab06391\gluon\dal.py", line 8276, in insert

    self._attempt_upload(fields)

  File "C:\web2py-1045bab06391\web2py-1045bab06391\gluon\dal.py", line 8258, in _attempt_upload

    raise RuntimeError("Unable to handle upload")

RuntimeError: Unable to handle upload

>>>

Thanks for your time.

-Bill

[Alan Etkin]

I am attempting to write a function that will eventually go in to a migration script to upload legacy data files. As you can see I

"Unable to handle load" is too generic for this case. I'd better temporarily let dal not to catch the error so there's a more precise description of it. Perhaps there are encoding errors when writing the stream output?. BTW, Is that the recommended way of adding upload fields data programatically (calling the .store method)?

[Bill Thayer]

Hi Alan,

Yes the book has the example as written because I am uploading directly to the database. Otherwise I get an error that store requires a value for uploadfolder.

Sorry but I've been looking for how t temporarily not let the dal catch the error. I tried running my app with the -S option but it's the same result.

Regards,

Bill

[Alan Etkin]

I did not explain well what I meant:

File "C:\web2py-1045bab06391\web2py-1045bab06391\gluon\dal.py", line 8258, in _attempt_upload

    raise RuntimeError("Unable to handle upload")

It seems that dal is returning a generic message so my idea was to take out any exception handling in the dal.py module that does the file writing. Well, actually there is not such exception handling clause. It's just that the method called requires that:

a) if the object passed has a file attribute, it is supposed to have a filename attribute
b) if the object has a .read method, then it looks for a name attribute

Other cases raise the OperationalError output you are getting.

Perhaps this can help debugging the code to create the files.

[Alan Etkin]

Replacing the store call with this should avoid the error:



file_id=db.wiki_media.insert(filename=db.wiki_media.filename.store(stream, filename=<a file name>, path=file_path)

[Bill Thayer]

Hi Alan,

Just now getting back to this days later...

Your code also threw the same error so before I un-commented the if statements in dal.py I added some print statements and can see the list of fields, then the name of the filename field as 'filename' followed by wiki_media.filename.9b724ad17775d60e.63617266696c65732e747874.txt

as the value being tested for a 'file' attribute and a 'filename' attribute before calling the store function...(again?) This is confusing so I guess I'll do as you suggest and take out those lines of code.

>>> import test_tamoto_rpc as trpc

>>> trpc.upload_file()

C:\Users\bthayer\Documents\car\carfiles.txt

wiki_media.id

wiki_media.filedata

wiki_media.cell_id

wiki_media.lib_id

wiki_media.layout_id

wiki_media.wiki_page

wiki_media.title

wiki_media.meas_id

wiki_media.filename

filename

wiki_media.filename.9b724ad17775d60e.63617266696c65732e747874.txt

Traceback (most recent call last):

  File "<console>", line 1, in <module>

  File "C:\web2py-1045bab06391\web2py-1045bab06391\applications\TAMOTO\private\test_tamoto_rpc.py", line 38, in upload_file

    filedata=stream.read())

  File "C:\web2py-1045bab06391\web2py-1045bab06391\gluon\dal.py", line 8280, in insert

    self._attempt_upload(fields)

  File "C:\web2py-1045bab06391\web2py-1045bab06391\gluon\dal.py", line 8262, in _attempt_upload

    raise RuntimeError("Unable to handle upload")

RuntimeError: Unable to handle upload

>>>

[Bill Thayer]

So I just replaced the first if statement with 'if True:' so that the code would execute. Looks like a unicode object does not have an attribute 'file' 

(InteractiveConsole)

>>> import test_tamoto_rpc as trpc

>>> trpc.upload_file()

C:\Users\bthayer\Documents\car\carfiles.txt

wiki_media.id

wiki_media.filedata

wiki_media.cell_id

wiki_media.lib_id

wiki_media.layout_id

wiki_media.wiki_page

wiki_media.title

wiki_media.meas_id

wiki_media.filename

filename

wiki_media.filename.a8857e792725cc9c.63617266696c65732e747874.txt

Traceback (most recent call last):

  File "<console>", line 1, in <module>

  File "C:\web2py-1045bab06391\web2py-1045bab06391\applications\TAMOTO\private\test_tamoto_rpc.py", line 38, in upload_file

    filedata=stream.read())

  File "C:\web2py-1045bab06391\web2py-1045bab06391\gluon\dal.py", line 8281, in insert

    self._attempt_upload(fields)

  File "C:\web2py-1045bab06391\web2py-1045bab06391\gluon\dal.py", line 8258, in _attempt_upload

    new_name = field.store(value.file,filename=value.filename)

AttributeError: 'unicode' object has no attribute 'file'

>>>

[Alan Etkin]

Your code also threw the same error so before I un-commented the if statements in dal.py I added some print statements

Did you try the store command as I posted it (passing pathname and filename) without changing dal.py? What is the error output?

[Bill Thayer]

Yes. That is what I meant. I even chased down the call in the code to see where you got it. 

Running db.executesql(...) might be a workaround but in order to do that I need to change the filename field type to 'string' then make the insert to get the id then build an UPDATE command.

Here is what worked just now:

# -*- coding: cp1252 -*-

import os, sys

from gluon import DAL, Field

from gluon.tools import Auth

from gluon.validators import IS_SLUG

from easygui import fileopenbox, diropenbox

db = DAL('oracle://<user>/<pwd>@<server>:1521',

         auto_import=True)

file_id = None

def upload_file():

    """demonstrate uploading a file to tomoto programatically.
       this will only work for files accessible from the local machine.

    """

    stream=None

    file_path=fileopenbox(msg='Pick a file', title='Picking files', default=None)

    if file_path:

        print 'file_path= ' + file_path

        stream = open(file_path, 'rb')

    else: return None

    

    blobdoc = stream.read()

    path,filename = os.path.split(file_path)

    #workaround to possible surprise bug in web2py?

    db.wiki_media.filename.type='string'

    # need an id to send to Oracle

    file_id=db.wiki_media.insert(filename=filename)

    db.commit()

    if file_id:

        print 'file_id=' + str(file_id)

    db.executesql('UPDATE WIKI_MEDIA SET FILEDATA =:blobdoc WHERE ID =:file_id',

          placeholders={'file_id':file_id, 'blobdoc':blobdoc})

    db.commit()

    stream.close()

    return file_id

Wish there was something a bit more elegant and reusable. Looked at the custom_store attribute but could quite wrap my head around how to get the id. 

[Alan Etkin]

Yes. That is what I meant. I even chased down the call in the code to see where you got it.

Tried this in a console and it works, I could not reproduce the error.

# myfile is an open file object
file_id=db.mytable.insert(myupload=db.mytable.myupload.store(myfile,
        filename="myfilename.txt", path="/home/spametki/"),
    myuploadfield=myfile.read())

Make sure the filename passed to store is an str object (not unicode) because otherwise it may raise a RuntimeError. If the error persists, it may be a bug.
 

[Bill Thayer]

Hi Alan. Seeing your code must have helped because I did get it to run. 

The following shows a demo script with functions to upload a file and download a file utilizing a table containing a blob field. It worked twice on a pdf for me. Note that I had to change the filename type from 'upload' to 'string' in order to bypass an error message in the dal._attempt_upload(fields).

# -*- coding: cp1252 -*-

import os, sys

from gluon import DAL, Field, fileutils

from easygui import fileopenbox, diropenbox

db = DAL('oracle://...',

         auto_import=True)  ## Still playing with this one to try and avoid re-defining table?

db.define_table('wiki_media',

                Field('filename', 'upload', uploadfield='filedata'),

                Field('title', 'string'),

                Field('filedata', 'blob'),

                migrate=False, fake_migrate=True)

def upload_file():

    """demonstrate uploading a file to database programatically,

    """

    stream=None

    file_path=fileopenbox(msg='Pick a file', title='Picking files', default=None)

    if file_path:

        print 'file_path= ' + file_path

        stream = open(file_path, 'rb')

    else:

        print 'file_path undefined'

        return None

    

    path,filename = os.path.split(file_path)

    #workaround to possible suprise bug in web2py?

    db.wiki_media.filename.type='string'

    file_id=db.wiki_media.insert(filename=db.wiki_media.filename.store(stream, filename=filename, path=path), filedata=stream.read())

    db.commit()

    stream.close()

    if file_id:

        print 'file_id=' + str(file_id)

    return file_id

def download_file(file_id, table='wiki_media'):

    """ In this demo will download the file with the given file_id from the

        given table. Table must have a filename field of type upload and a

        filedata field of type blob

    """

    table = db[table]

    if file_id:

        print 'file_id=' + str(file_id)

    else:

        print 'file_id is empty'

        return None

    

    directory=diropenbox(title='Select destination directory')

    

    record = table(table.id==file_id)

    (filename, stream) = table.filename.retrieve(record.filename)

    pathname = os.path.join(directory,filename)

    import shutil           #In fileutils.py the developer mentioned he suspects a bug in shutil.copyfileobj
                            #but I didn't know how to get the length of a cStringIO object to 
                            #use fileutils.copystream()

    shutil.copyfileobj(stream,open(pathname,'wb'))

    stream.close()

    return

Hope it helps someone.

-Bill