Inconsistent file upload behaviour

[Francis Windram]

Hi All,

I'm having a very very weird problem with some upload processing I'm performing in web2py.

Firstly for context here is the code I am using:

Controller:

def validate_test():
    form = SQLFORM.factory(
        Field('csvfile', 'upload'), table_name="dataset_upload")
    if form.validate():
        logger.info("form validating")
        candidate_file = request.vars.csvfile.file.read()
        logger.info("{}".format(candidate_file))
    elif form.process().accepted:
        response.flash = 'form accepted'
    elif form.errors:
        response.flash = 'form has errors'
    return dict(form=form)

And the View:

{{extend 'layout.html'}}

<div style="background: linear-gradient(240deg, lightgrey, whitesmoke)" class="jumbotron">
<br><br>
{{=form}}
</div>

This is essentially as simple as it can be. The user should upload a csv file, and I wish to run a suite of validators on that csv file before returning to the user whether the validation was passed or where it failed and on what.

Now this is the first step, just checking that the form csv is readable by the server. And here is where I hit the problem:

Sometimes I get out (as expected) the contents of the csv file from logger.info("{}".format(candidate_file))

Most of the time I just get out a blank string, even with the same input file!




I feel like I'm going mad, doing the same thing twice and getting different results. It smells to me like caching or race conditions but I just have no idea what really is going on or why this might be happening.




Do you have any ideas?




Thanks!

P.S. also more generally is this a good way to perform validation on a csv file without any database IO? I want users to be able to check whether their file fits the canonical format prior to actually attempting to commit it to the database?
It would be nice to be able to use the Web2Py validators to do this validation, but I don't really want to start creating temporary staging tables to perform this validation prior to upload, especially as I have a multi-table structure which the data will be going in to, but it comes in as a flat csv.


[Francis Windram]

Just to add to this, I've done some further testing and can confirm a few more bits about the inconsistency.

Firstly the behaviour is not browser-specific, occurring in both firefox and chrome.

Secondly it seems to fail always when running web2py normally. When running in the debugger in pycharm it sometimes works, and sometimes fails in precisely the same manner. I am somewhat at my wits end.

My only thoughts are either a caching issue or a race condition, but I can't see any immediate reason for either of these. Even when starting with a fresh session we seem to see this behaviour...

[Leonel Câmara]

You probably just need to rewind the file because request.vars already loaded it.

request.vars.csvfile.file.seek(0)

candidate_file = request.vars.csvfile.file.read()





Instead you can just do:

candidate_file = request.vars.csvfile.value




Which uses the value of the file already got by request