Issues migrating dynamic CSV to Python3

[Jonathan Vanasco]

I discovered an issue with our code when generating dynamic CSVs under Python3.  I'm hoping someone can point me in the right direction.  I couldn't find the appropriate migration/changelog information.  This works fine under Python2.

The generic way we create/serve the CSV and check it in tests are below.

The two problems:

1. Under Python3, an exception is thrown under waitress, because of attempted string+byte concatenation. (https://github.com/Pylons/waitress/blob/master/src/waitress/task.py#L316)

    >    towrite += data + b"\r\n"

    > TypeError: can only concatenate str (not "bytes") to str

2. I picked this up in a unit test; i can't seem to access any sort of body/text/content off the response.

    from pyramid.request import Request

    from pyramid import testing

    self.config = config = testing.setUp()

    app = self.config.make_wsgi_app()

    req = Request.blank(csv_link)

    req.remote_addr = "127.0.0.1"

    resp = req.get_response(app)

    self.assertEqual(resp.status_code, 200)

    self.assertTrue(resp.text.startswith("User CPU time,"))

---

 def view_csv(request):

    csvfile = StringIO()

    csvwriter = csv.writer(csvfile)

    for row in ((1, "a"), (2, "b")):

        csvwriter.writerow(row)

    csvfile.seek(0)

    as_csv = Response(content_type="text/csv", body_file=csvfile, status=200)

    as_csv.headers["Content-Disposition"] = str("attachment; filename=example.csv")

    return as_csv

---

[Bert JW Regeer]

Your body_file is not bytes, but str. You need to make sure that what you pass to body_file returns bytes.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/7d3c278f-8d20-4456-b05c-33c8d2cb6a67n%40googlegroups.com.

[Jonathan Vanasco]

Thanks so much, Bert. That should have been obvious!

As an interim solution, I'm just wrapping the inputs:

    if six.PY3:

        csvfile = BytesIO(csvfile.read().encode("utf-8"))

        csvfile.seek(0)

[Theron Luhn]

Since you’re just dumping the entire file into BytesIO anyways, wouldn’t it be easier to do `body=csvfile.read().encode(“utf8”)` and skip BytesIO?

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/e73aae83-0f7e-414c-a9c3-c20957793a0dn%40googlegroups.com.

[Jonathan Vanasco]

Actually,  just using `body=csvfile.read()` works on Python2 & Python3 !

[Jonathan Vanasco]

Thanks, everyone!

Now I've got both these debugtoolbar plugins updated PyPi and running on Github Actions with expanded test-suites:

* https://github.com/jvanasco/pyramid_debugtoolbar_api_performance
* https://github.com/jvanasco/pyramid_debugtoolbar_api_sqlalchemy

[Mike Orr]

On Sun, Oct 18, 2020 at 1:22 PM Bert JW Regeer <xist...@0x58.com> wrote:
>
> Your body_file is not bytes, but str. You need to make sure that what you pass to body_file returns bytes.

I thought that but 'writer = csv.writer(response.body_file)' works for
me, and my row items are 'str'. In Python 2 I had to convert unicode
to boytes or I'd get an error, but Python 3 accepts 'str' values
directly and the resulting file download is correct.

[Bert JW Regeer]

That is because response.body_file is https://github.com/Pylons/webob/blob/99890b763dbd441ca1e5f14f8418efa2a8173ab0/src/webob/response.py#L1445 which calls https://github.com/Pylons/webob/blob/99890b763dbd441ca1e5f14f8418efa2a8173ab0/src/webob/response.py#L671 on write.

It does the conversion for you...

That's not the same as setting body_file directly on the response, which does NOT go through that same mechanism and instead assumes that the file is a file in binary mode.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3DurwVOtKjYweOyzG7YLA%2BQNb_hVG5zrrA2cjRoG%2BQmtB8Q%40mail.gmail.com.