Cythonizing Tornado - anyone done it recently?

[Adrian]

I last looked into this years ago but decided it wasn't worth pursuing. 

But now Tornado is fully integrated with asyncio and should compile, and has type hints, and Cython supports those type hints... it might be a decent win.

Has anyone done it and has results/methods to share?

I also expect that at some point, I'm going to go down the rabbit hole of trying to compile and Cythonize templates, and figure out how to call them with the right namespace.

[Ben Darnell]

Looks like it's been six years since I last looked at this. Before we had type hints, you could get about a 25% performance improvement by compiling all of Tornado (with small patches) with cython. I'm not sure that type hints will help much, though - there just aren't that many performance-critical places in Tornado where you could give a sufficiently precise type hint. My notes from my previous explorations are in https://groups.google.com/g/python-tornado/c/Kele72rCUaE/m/TUgyCJifNb4J

-Ben

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/7f3f0ef7-0c98-4577-84db-68670ce3bc01n%40googlegroups.com.

[Adrian]

We've only got two errors:

[1/1] Cythonizing /home/tornado/log.pyx

Error compiling Cython file:

------------------------------------------------------------

...

def _safe_unicode(s: Any) -> str:

    try:

        return _unicode(s)

    except UnicodeDecodeError:

        return repr(s)

                  ^

------------------------------------------------------------

log.pyx:78:19: str objects do not support coercion to unicode, use a unicode string literal instead (u'')

and:

[1/1] Cythonizing /home/tornado/template.pyx

Error compiling Cython file:

------------------------------------------------------------

...

    def each_child(self) -> Iterable["_Node"]:

        return (self.body,)

    def generate(self, writer: "_CodeWriter") -> None:

        block = writer.named_blocks[self.name]

        with writer.include(block.template, self.line):

                   ^

------------------------------------------------------------

template.pyx:560:20: Expected an identifier

[Adrian]

Solved the first one with: return str(repr(s)). Solved the second one by just renaming the writer.include method. Now I just have to rejig all my imports and I can run some benchmarks.

[Rajdeep Rath]

Thank you for sharing. Does that mean tornado applications can be cythonized properly ?

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/955f7b16-9370-4bad-954d-aec5a0f1be18n%40googlegroups.com.

[Ben Darnell]

It's been possible to cythonize tornado *applications* for at least several years now (although I admit I haven't tried it recently). Adrian is talking about cythonizing Tornado itself.

On Tue, 16 Feb 2021 at 5:27 AM, Adrian <aflit...@gmail.com> wrote:

Solved the first one with: return str(repr(s)). Solved the second one by just renaming the writer.include method. Now I just have to rejig all my imports and I can run some benchmarks.

FWIW, with the setup.py changes from https://github.com/bdarnell/tornado/compare/master...cython#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7 I was able to do some testing without changing all my imports.

-Ben

 

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CAP869y8U78GBdRXdp9CqABjPVjyjqyGVRpgAuNsG7tfJEmUzbw%40mail.gmail.com.

[Rajdeep Rath]

Ah I see. Understood 

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CAFkYKJ6PmRKn5otbL8Vd16y66OxTBTcop5TOcDkYq0xHxWbPpg%40mail.gmail.com.