Understanding the download function

[Andy W]

I have a multi-tenant application, where users can upload files. I save these in a different directory for each client (tenant), so I can keep tabs on the overall disk space and number of files uploaded by each.

This works fine when the table is defined in a model file - from the view I can download existing files and upload new ones:

Model:

from gluon import *
import os
client_id=3 #hard coded here for illustration
           
db.define_table('attachment_model',
    Field('attached_file', 'upload', label="Upload new file",
          uploadfolder=os.path.join(request.folder, 'uploads', str(client_id), 'attachments'),

          requires=IS_NOT_EMPTY(), autodelete=True),
    Field('filename', type='string', length=150, writable=False),
    migrate=True)

Controller:

def model_based():
    db.attachment_model.filename.readable=False
    form_attachment=SQLFORM(db.attachment_model,
                 autodelete=True, labels=None, deletable=True,
                 fields=['attached_file'], submit_button='Attach file')
    if hasattr(request.vars.attached_file, "filename"):
        #save original file name
        form_attachment.vars.filename=request.vars.attached_file.filename
    if form_attachment.process().accepted:
        response.flash = 'attachment saved'
    elif form_attachment.errors:
        response.flash = 'form has errors'
    # list existing attachments
    rows=db(db.attachment_model.id>0).select()
    response.view = 'attachment.html'
    return dict(form_attachment=form_attachment,
                rows=rows)

View:

{{extend 'layout.html'}}
<h1>
    Attached files
</h1>
    <table class="table-striped">
        <thead>
            <tr>
                <td>id</td>
                <td>filename</td>
            </tr>
        </thead>
        <tbody>
            {{for r in rows:}}
                <tr>
                    <td>{{=r.id}}</td>
                    <td>{{=A(r.filename, _href=URL('download', args=(str(client_id) + '/attachments/' + r['attached_file'])))}}</td>
                </tr>
            {{pass}}
        </tbody>
    </table>
   
<h2>Add new attachment</h2>
<div class="form-inline well">
    {{=form_attachment}}
</div>

My issue is when I re-write the above so the model definition is moved to a module.

Module mod_attachment.py:

from gluon import *
import os

class Attachment_module(object):
    def __init__(self, db):
        self.db = db

    def define_tables(self):
        db = self.db
        client_id=3
        if not 'attachment_module' in db.tables:
            db.define_table('attachment_module',
                Field('attached_file', 'upload', label="Upload new file",
                      uploadfolder=os.path.join(current.request.folder, 'uploads', str(client_id), 'attachments'),
                      requires=IS_NOT_EMPTY(), autodelete=True),
                Field('filename', type='string', length=150, writable=False),
                Field('request_tenant', type='integer', default=client_id,
                      readable=False, writable=False),
                migrate=True)

Revised controller:

def module_based():
    from mod_attachment import Attachment_module
    attachment_module = Attachment_module(db)
    attachment_module.define_tables()
    db.attachment_module.filename.readable=False
    form_attachment=SQLFORM(db.attachment_module,
                 autodelete=True, labels=None, deletable=True,
                 fields=['attached_file'], submit_button='Attach file')
    if hasattr(request.vars.attached_file, "filename"):
        #save original file name
        form_attachment.vars.filename=request.vars.attached_file.filename
    if form_attachment.process().accepted:
        response.flash = 'attachment saved'
    elif form_attachment.errors:
        response.flash = 'form has errors'
    # list existing attachments
    rows=db(db.attachment_module.id>0).select()
    response.view = 'attachment.html'
    return dict(form_attachment=form_attachment,
                rows=rows)

With this approach, I can still list the uploaded files and add additional ones (using the same view as before), but the download function no longer works.

So my question is why not? Do I have to modify the standard download function, and how?

Hope these are not daft questions - any pointers would be appreciated.

Andy

[Anthony]

Assuming you are talking about the standard download() function in the default.py controller of the scaffolding application, note that it uses response.download() to retrieve the file, and as per the documentation, it looks at only the last URL arg to obtain the filename (from which it extracts the database table name and field name in order to get the uploadfolder and retrieve the file). So, in your link URL:

URL('download', args=(str(client_id) + '/attachments/' + r['attached_file'])

The str(client_id) and "attachements" parts of the URL are completely ignored and therefore unnecessary.

Of course, because response.download() works by getting the uploadfolder from the upload field definition, it requires that the relevant database model actually be defined. In your case, though, you are only defining the db.attachment_module table when inside the module_based() function, so when in the download() function, that model doesn't exist, and response.download() therefore cannot find the relevant upload folder.

If you don't want to have to ensure that the model has been defined when the download() function is called, you can continue using links like the ones you have generated, but in that case you won't be able to use response.download() and should instead use response.stream(). The former is designed to work specifically with upload fields in DAL models (and therefore requires access to the models), whereas the latter is a more general method for streaming any file back to the browser (though it requires that you know the full path to the file on the filesystem).

Anthony

[Andy W]

Many thanks for the clear explanation!

Andy

[Rudy]

Hi Andy / Anthony,

May i ask how you have addressed the multi-tenant requirement separating all those uploaded files from different tenant to keep track of their storage usage.

I want to use the client id as part of the uploadfolder, it worked as a test below when I defined the uploadfolder in db.py with hardcoded client_id. In reality, the client_id varies (is there a way to specify the table company's id in the uploadfolder in db.py?), I thought of specifying the uploadfolder info in the action where I would do my upload (edit_company()), I could upload it but not download. 

Is it the same issue as Anthony mentioned (for test purpose, i used the same edit_company()'s SQLFORM to upload and download? i thought it would execute the line db.company.logo.uploadfolder=os.path.join(request.folder,'uploads', str(company_id)) before it executes  form=SQLFORM(db.company, record=record, deletable=True, showid=True, upload=URL('download')).process(), obviously i missed out some fundamental concept (no i didn't understand why Andy couldn't specify the table definition in the module as SQLFORM and view.html would be executed after calling attachment_module.define_tables()). Any insight will be much appreciated.

in db.py

client_id=3 #hard coded here for illustration

db.define_table('company',

Field('logo', 'upload', label='Company Logo',  uploadfolder=os.path.join(request.folder, 'uploads', client_id), uploadseparate=True, autodelete=True))

in default.py

def edit_company():

    company_id = request.args(0)

    record=db.company(company_id)

    db.company.logo.uploadfolder=os.path.join(request.folder,'uploads', str(company_id))

    db.company.chop.uploadfolder=os.path.join(request.folder,'uploads', str(company_id))

    form=SQLFORM(db.company, record=record, deletable=True, showid=True, upload=URL('download')

                 ).process()

    if form.accepted:

        session.flash='Info: you have updated your company details!!'

        redirect(URL('manage_company'))

    elif form.errors:

        response.flash='Error: inside edit_company, form raised error!!'

    else:

        pass

    return locals()

[Andy W]

Hi Rudy

Have you tried re-defining the download function to include the path to the
location you have used for storing the uploaded files? For example, in my
case I ended up with the following in my controller:

def download():
    import os
     file_path = os.path.join(request.folder,'uploads', str(auth.user.client), request.args(0))
     return response.stream(open(file_path), attachment=False, chunk_size=4096)

Obviously you need to edit file_path in the above to match your situation.

Andy

[Rudy]

Hi Andy,

It works as you suggested below, thanks so much.

Anthony, i understand what you meant of  "In your case, though, you are only defining the db.attachment_module table when inside the module_based() function, so when in the download() function, that model doesn't exist, and response.download() therefore cannot find the relevant upload folder.", thanks for sharing your insight.

Rudy

[Anthony]

On Monday, October 9, 2017 at 1:42:59 AM UTC-4, Andy W wrote:

Hi Rudy

Have you tried re-defining the download function to include the path to the
location you have used for storing the uploaded files? For example, in my
case I ended up with the following in my controller:

def download():
    import os
     file_path = os.path.join(request.folder,'uploads', str(auth.user.client), request.args(0))
     return response.stream(open(file_path), attachment=False, chunk_size=4096)

This shouldn't be necessary. As long as you have the client id available in the request (e.g., in auth.user.client), you can simply use it to specify the uploadfolder when defining the model. In db.by:

db.define_table('company',
    Field('logo', 'upload', label='Company Logo',

          uploadfolder=os.path.join(request.folder, 'uploads', auth.user.client if auth.user else ''),
          uploadseparate=True, autodelete=True)),
    ...)

Anthony