Cornice / Colander design for file upload

[Thierry Florac]

Hi,

I'm starting to use Cornice, Colander and Swagger and I'm trying to create a REST service which should allow users to upload file(s)...

Until now I'm using "multipart/form-data" encoding and it seems OK, but :

 - is it actually a good practice to handle file uploads in a REST API using this encoding? Should I prefer something like JSON with base64 string encoding (including for use cases with large data files)??

 - how should I handle schema verification with Colander for file upload fields (what I actually receive using form-data encoding is a CGI FieldStorage object) ?

Best regards for any advise,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

[Michael Merickel]

To me it just kind of depends on what level of atomicity you need in your API endpoint. If you can accept the binary data without any other parameters then that's great, just do that. If you need it alongside other input then multipart is great. Some people also marshal that stuff in x-foo headers but I think there's very few cases where that's compelling. BASE64'd binary data into JSON is the least desirable but it really depends on the situation - I'm definitely guilty of using it in certain scenarios (if you just really really really want a JSON body cuz that's what's easiest on the client-side). 

It's worth noting that Deform has its own mechanism for storing the file upload on the server such that you can connect it up to the form data from a separate request. This is handy to avoid re-transferring the data when parts of the form fail to validate.

The CGI FieldStorage object is basically a wrapper around the content, providing a file-like interface for messing with it as webob may have done you a solid and stored the data to disk instead of blowing out your memory. You can always just read the content in from there. You'll also want to pull the content-type from that object. After that validation is up to you.

Finally, if the request got all the way to your WSGI app then it's almost guaranteed that it was already fully parsed/received from the client by your WSGI server or any middleware/API calls so it's (probably) too late at that point to really validate and reject based on the size. You'd want to do that upstream prior to the body being read if it's really large data you're worried about receiving.

- Michael

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWBv8Ufg-ezeHPqeRXjwuF2f-8iSE8VRy7k3_n9iRp-LqA%40mail.gmail.com.

[Thierry Florac]

Hi Michael,

I finally made my REST API service to accept both multipart/form-data and JSON base64 encoded data...

Implementation is probably not "perfect" (I found that Colander is probably lacking a few documentation about creating and registering custom types and converters), but at least it works!  :)

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/314E86A9-A36F-439F-AD01-3F581ED87B92%40gmail.com.

[Steve Piercy]

On 2/9/21 12:56 AM, Thierry Florac wrote:

> I found that Colander is probably lacking a few documentation about creating and registering custom types and converters

I'd like to improve the docs. Would you please open an issue with what you think is missing from this section?

https://docs.pylonsproject.org/projects/colander/en/latest/extending.html

I'd be happy to collaborate to make them more helpful.

--steve

[Thierry Florac]

Hi Steve,

My message is probably wrong!

As far as I can remember, the miss was not in Colander documentation, but in Cornice Swagger extension documentation about types converters registration; an issue is already open to add this to it's documentation...

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/aa88b42d-33a0-b3ad-cd6e-739ad81d3296%40gmail.com.