Re: [tornado] Why no Windows support?

[Ben Darnell]

That thread is three years old; the pull request was eventually merged.  Tornado is currently usable on windows for development purposes, but you probably shouldn't run a production site on it (on windows we are limited to 64 simultaneous connections).  

-Ben

On Tue, Jan 29, 2013 at 2:30 AM, Ralph Möritz <ralm...@gmail.com> wrote:

I found this thread where someone forked the official GitHub Tornado repo & added support for Windows. He issued a pull request but was ignored - why is this? Are the developers of Tornado not interested in adding Windows support?

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ralph Möritz]

Yes, I found this out for myself after installing from PyPI and seeing that it works. Needless to say I felt like such an idiot for having posted this! In my defence though, the README & website really should be updated to say that Windows *is* supported.

[aliane abdelouahab]

i think why the document is not update is:
the problem with windows is the lack of epoll() or kqeue(), tornado
will run using select() which is old.
so the 10k problem is not solved on windows (...?)

On 30 jan, 10:49, Ralph Möritz <ralmor...@gmail.com> wrote:
> Yes, I found this out for myself after installing from PyPI and seeing that
> it works. Needless to say I felt like such an idiot for having posted this!
> In my defence though, the README & website really should be updated to say
> that Windows *is* supported.
>
>
>
>
>
>
>
> On Tuesday, January 29, 2013 5:36:24 PM UTC+2, Ben Darnell wrote:
>
> > That thread is three years old; the pull request was eventually merged.
> >  Tornado is currently usable on windows for development purposes, but you
> > probably shouldn't run a production site on it (on windows we are limited
> > to 64 simultaneous connections).
>
> > -Ben
>

> > On Tue, Jan 29, 2013 at 2:30 AM, Ralph Möritz <ralm...@gmail.com<javascript:>
> > > wrote:
>
> >> I found this thread<https://groups.google.com/forum/?fromgroups=#!topic/python-tornado/Z1...>where someone forked the official GitHub Tornado repo & added support for

> >> Windows. He issued a pull request but was ignored - why is this? Are the
> >> developers of Tornado not interested in adding Windows support?
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Tornado Web Server" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to python-tornad...@googlegroups.com <javascript:>.
> >> For more options, visithttps://groups.google.com/groups/opt_out.

[Andrew Grigorev]

Just my 50 cents...

There is a big difference between "yes, it works somehow on Windows" and "Windows is supported". May be a little note about "Windows could be used as a development environment for tornado applications" would be more reasonable?

There is nothing about Windows in tornado docs, except some records in changelog http://www.tornadoweb.org/documentation/search.html?q=Windows, which actually looks like "Windows is supported". So I'm confused about is it supported or not.

30.01.2013 13:49, Ralph Möritz пишет:

-- 
Andrew

[aliane abdelouahab]

i actually do this, i try my code under windows, because i've a laptop
and linux is not good friend with my battery life.... so yes it works,
maybe as the best as i can get, i've everything, maybe my application
is a basic, but i use several libraries and i dont find any problem!
i'm using tornado latest stable, because the post-1 dont work with
Mongotor, i use Mongodb with it and the application runs good, so hope
the Windows support will be added in the documentation.

On 30 jan, 11:24, Andrew Grigorev <and...@ei-grad.ru> wrote:
> Just my 50 cents...
>
> There is a big difference between "yes, it works somehow on Windows" and
> "Windows is supported". May be a little note about "Windows could be
> used as a development environment for tornado applications" would be
> more reasonable?
>
> There is nothing about Windows in tornado docs, except some records in

> changeloghttp://www.tornadoweb.org/documentation/search.html?q=Windows,

> which actually looks like "Windows is supported". So I'm confused about
> is it supported or not.
>

> 30.01.2013 13:49, Ralph M�ritz ?????:

>
>
>
>
>
>
>
>
>
> > Yes, I found this out for myself after installing from PyPI and seeing
> > that it works. Needless to say I felt like such an idiot for having
> > posted this! In my defence though, the README & website really should
> > be updated to say that Windows *is* supported.
>
> > On Tuesday, January 29, 2013 5:36:24 PM UTC+2, Ben Darnell wrote:
>
> >     That thread is three years old; the pull request was eventually
> >     merged.  Tornado is currently usable on windows for development
> >     purposes, but you probably shouldn't run a production site on it
> >     (on windows we are limited to 64 simultaneous connections).
>
> >     -Ben
>

> >     On Tue, Jan 29, 2013 at 2:30 AM, Ralph M�ritz <ralm...@gmail.com

> >     <javascript:>> wrote:
>
> >         I found this thread

> >         <https://groups.google.com/forum/?fromgroups=#%21topic/python-tornado/...>

> >         where someone forked the official GitHub Tornado repo & added
> >         support for Windows. He issued a pull request but was ignored
> >         - why is this? Are the developers of Tornado not interested in
> >         adding Windows support? --
> >         You received this message because you are subscribed to the
> >         Google Groups "Tornado Web Server" group.
> >         To unsubscribe from this group and stop receiving emails from
> >         it, send an email to python-tornad...@googlegroups.com

> >         <javascript:>.

> >         For more options, visit
> >        https://groups.google.com/groups/opt_out

> >         <https://groups.google.com/groups/opt_out>.

>
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Tornado Web Server" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to python-tornad...@googlegroups.com.
> > For more options, visithttps://groups.google.com/groups/opt_out.
>
> --

> Andrew

[Ben Darnell]

On Wed, Jan 30, 2013 at 5:24 AM, Andrew Grigorev <and...@ei-grad.ru> wrote:

Just my 50 cents...

There is a big difference between "yes, it works somehow on Windows" and "Windows is supported". May be a little note about "Windows could be used as a development environment for tornado applications" would be more reasonable?

Yes, exactly.  Here's what I said in a previous thread on this topic:

""">> Tornado works on windows now, but doesn't scale very well - the limit

>> on the number of file descriptors for select() is lower than for unix
>> (although I think it's become somewhat reasonable in modern versions),
>> and performance would presumably be better with a windows-specific
>> implementation.  More importantly, this would be a sign that someone
>> has looked into a windows implementation more deeply than just seeing
>> that the tests pass and nothing seems obviously broken, which is a key
>> part of the distinction between "working" and "supported".
>
>
> What would the criteria be for Windows gaining "supported" status,
> specifically?

The main thing is simply someone who wants to support it. :)  If you'd
like to take on that responsibility, I think we can easily move
Windows to the "supported for development but not for production"
column (which, to be honest, is what Mac OS should be downgraded to
given the poor networking performance I've seen in Lion).  To remove
the "not for production" qualification, I think we'd need need a
native IOLoop and positive reports from someone who's taken the leap
and run in production on windows despite the lack of "official"
support."""

Since then, I've gotten a better understanding of what it would take to support IOCP (which is the windows analogue to epoll/kqueue), and the changes that are needed go deeper than just a new IOLoop implementation.  The whole interface of the IOLoop and IOStream would have to change.  I think for this reason windows is unlikely to rise above its second-class citizenship in Tornado.

-Ben

[Riccardo Magliocchetti]

Hello,

Is this still the case? Possibly something in tornado 4 made supporting IOCP easier?

thanks

--
Riccardo Magliocchetti

[A. Jesse Jiryu Davis]

This is still the case, Riccardo.

--

You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[aliane abdelouahab]

you can check here :D

https://github.com/tornadoweb/tornado/blob/master/tornado/ioloop.py#L236

[Riccardo Magliocchetti]

> you can check here :D
> https://github.com/tornadoweb/tornado/blob/master/tornado/ioloop.py#L236

Thanks for the hint. My question was more about the "whole interface of the
IOLoop and IOStream would have to change" part. Maybe it was already been worked
out in tornado 4.

--
Riccardo Magliocchetti
@rmistaken

http://menodizero.it

[A. Jesse Jiryu Davis]

It hasn't, and from what I've heard Ben say on the topic, first-class support for Windows based on IOCP is not in Tornado's future. asyncio does support Windows, and I believe you could run Tornado with the asyncio event loop in Python 3.4+ on Windows with good results, but I don't have personal experience with that setup.

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ben Darnell]

On Sat, Feb 20, 2016 at 9:10 AM, A. Jesse Jiryu Davis <je...@emptysquare.net> wrote:

It hasn't, and from what I've heard Ben say on the topic, first-class support for Windows based on IOCP is not in Tornado's future. asyncio does support Windows, and I believe you could run Tornado with the asyncio event loop in Python 3.4+ on Windows with good results, but I don't have personal experience with that setup.

Unfortunately running Tornado on top of asyncio doesn't help with windows performance. The differences between the windows and posix models cannot be hidden behind the Tornado IOLoop interface; supporting windows would require changes to higher level interfaces (IOStream) as well. (and these changes are unlikely to ever happen)

In asyncio, there are two windows event loops (https://docs.python.org/3/library/asyncio-eventloops.html#windows): SelectorEventLoop and ProactorEventLoop. ProactorEventLoop is the high-performance IOCP one, but it doesn't support the add_reader() and add_writer() methods that Tornado's asyncio bridge relies on, so you have to use SelectorEventLoop instead. SelectorEventLoop has exactly the same limitations as Tornado's native IOLoop on windows.

-Ben

[Riccardo Magliocchetti]

Hello Ben,

Il 21/02/2016 21:12, Ben Darnell ha scritto:
> On Sat, Feb 20, 2016 at 9:10 AM, A. Jesse Jiryu Davis <je...@emptysquare.net
> <mailto:je...@emptysquare.net>> wrote:
>
> It hasn't, and from what I've heard Ben say on the topic, first-class
> support for Windows based on IOCP is not in Tornado's future. asyncio

> /does/ support Windows, and I believe you could run Tornado with the asyncio

> event loop in Python 3.4+ on Windows with good results, but I don't have
> personal experience with that setup.
>
>
> Unfortunately running Tornado on top of asyncio doesn't help with windows
> performance. The differences between the windows and posix models cannot be
> hidden behind the Tornado IOLoop interface; supporting windows would require
> changes to higher level interfaces (IOStream) as well. (and these changes are
> unlikely to ever happen)
>
> In asyncio, there are two windows event loops
> (https://docs.python.org/3/library/asyncio-eventloops.html#windows):
> SelectorEventLoop and ProactorEventLoop. ProactorEventLoop is the
> high-performance IOCP one, but it doesn't support the add_reader() and
> add_writer() methods that Tornado's asyncio bridge relies on, so you have to use
> SelectorEventLoop instead. SelectorEventLoop has exactly the same limitations as
> Tornado's native IOLoop on windows.

Thanks for the explanation. Is this the case for twisted IOLoop too? May i add
some of these information to the documentation?

thanks

[Ben Darnell]

Yes, the same is true for TwistedIOLoop: Twisted works well on Windows, but TwistedIOLoop does not make Tornado work well on Windows. (same for libuv and https://github.com/saghul/tornaduv). There is no way to get windows support by plugging something in at or below the IOLoop; the changes must be further-reaching. Feel free to submit an update to the docs.

 

thanks

--
Riccardo Magliocchetti
@rmistaken

http://menodizero.it

[Walter Gamba]

Hi Ben,

we are thinking wether it is feasable to add support for Windows IOCP in Tornado, since we really need it in a project of ours.

We are aware we would need change the IOStream class as well as the IOLoop subclass, but while reading your comments we got worried.

We were wondering why you state that "(and these changes are unlikely to ever happen)" regarding IOStream? Do you think this is really too complicated (i.e. we have overlooked something vital) or simply no one in the Tornado community is interested in this? 

Thanks, 

Walter

[Ben Darnell]

On Tue, Feb 23, 2016 at 1:29 PM, Walter Gamba <walter...@gmail.com> wrote:

Hi Ben,

we are thinking wether it is feasable to add support for Windows IOCP in Tornado, since we really need it in a project of ours.

We are aware we would need change the IOStream class as well as the IOLoop subclass, but while reading your comments we got worried.

We were wondering why you state that "(and these changes are unlikely to ever happen)" regarding IOStream? Do you think this is really too complicated (i.e. we have overlooked something vital) or simply no one in the Tornado community is interested in this? 

You probably already know this, but here's some background in the hopes that this post can become the definitive answer on Windows support in Tornado. (The previous sentence virtually guarantees that I will make some embarrassing typo here)

The issue is that in order to support both IOCP and select-style event loops, you need a fundamentally different interface (I'll use asyncio as my IOCP-friendly example throughout this discussion, but the same things are mostly true of Twisted under slightly different names). Tornado's event loop has three versatile methods: add_handler, update_handler, and remove_handler (asyncio's equivalent is a set of four methods: add_reader, remove_reader, add_writer, remove_writer). From these methods we can create servers (tornado.netutil.add_accept_handler), clients (IOStream and SSLIOStream), and other things like PipeIOStream. Tornado itself doesn't include any code related to UDP, but this could be added by third parties using the existing IOLoop methods. 

IOCP, however, cannot support this style in which one method can act on many kinds of file descriptors. Instead, asyncio has a variety of methods directly on the event loop: create_connection(), create_unix_connection(), create_server(), connect_{read,write}_pipe(), etc. Asyncio has UDP support in the create_datagram_endpoint() method, which is important because it would not be possible for a third party to add this from the outside. 

Therefore, supporting IOCP means deprecating the core methods of the Tornado IOLoop and introducing a suite of new methods in their place. It may be possible to hide much of this change from the user (e.g. by switching between two implementations in `IOStream.__new__`), but it's still a significant increase in API surface area and complexity. It would introduce a division in the ecosystem between applications that use the new IOCP-friendly interfaces and those that use the old style (for example, CurlAsyncHTTPClient and the Momoko PostgreSQL driver rely on the IOLoop {add,update,remove}_handler methods and I'm not sure if it's even possible to adapt them to an IOCP-friendly form). 

So it's a major change, but we've made major changes before, like the introduction of coroutines. This is where the more personal and controversial arguments begin. I don't personally value Windows support by enough to do much work on this myself. Even if someone else did the work, I would hesitate to merge it because I *like* the simplicity and abstraction of Tornado's current interface. Reasonable people can differ on whether file descriptors are a good abstraction (hi Glyph!), but I think the idea that "everything's a file (descriptor)" has been very useful. I wasn't around when the project that became Tornado got started, but I think that the complexity of Twisted's Reactor interface was a big part of why the Friendfeed team decided to start from scratch (even though I have come to respect Twisted's decisions more as I learn about the reasons behind them). In short, even if Tornado loses some Windows users to Twisted or asyncio (or even a fork of Tornado!), I'm not sure that's worth the costs of introducing better Windows support.

-Ben

[Walter Gamba]

Hi Ben, 

in fact I didn't know many of the things you wrote. Thanks.

My request is basely customer-driven. If it were for me I'd stick with Tornado and Linux and be happy, but out there there are still people using Windows, so.. 

I will review your detailed explanation with my friends who are more Windows-savvy, but I think you made the point very clear. 

Thanks again for your time and kindness in being very plain and detailed.

I think we will have to move our code to Twisted, and I am already fearing every moment of it.. 

Walter

[Riccardo Scarsi]

Hi Ben,

I cannot fully understand why you talk about major changes in the IOLoop API, probably there are details in the current implementation that I do not consider.

IOCP (https://msdn.microsoft.com/it-it/library/windows/desktop/aa365198(v=vs.85).aspx) works with many kinds of file descriptors (handles in the Windows terminology):  for example, files, mail slots, pipes and socket.

From the read/write API point of view the same function (ie WriteFile https://msdn.microsoft.com/en-us/library/windows/desktop/aa365747(v=vs.85).aspx ) works in the same way on any supported handles ( files, mail slots, pipes and socket). 

The main difference is that you will get a IOCP notification only if you call the WriteFile or ReadFile with a specific parameters (LPOVERLAPPED) set, but this could be done in a specific IOStream implementation.

So I think that the main difference is about the low level read/write API and not in the IOLoop interface.

Do you think am I missing some important point?

Thanks

Riccardo

[Ben Darnell]

On Thu, Feb 25, 2016 at 6:56 AM, Riccardo Scarsi <sca...@gmail.com> wrote:

Hi Ben,

I cannot fully understand why you talk about major changes in the IOLoop API, probably there are details in the current implementation that I do not consider.

IOCP (https://msdn.microsoft.com/it-it/library/windows/desktop/aa365198(v=vs.85).aspx) works with many kinds of file descriptors (handles in the Windows terminology):  for example, files, mail slots, pipes and socket.

From the read/write API point of view the same function (ie WriteFile https://msdn.microsoft.com/en-us/library/windows/desktop/aa365747(v=vs.85).aspx ) works in the same way on any supported handles ( files, mail slots, pipes and socket). 

The main difference is that you will get a IOCP notification only if you call the WriteFile or ReadFile with a specific parameters (LPOVERLAPPED) set, but this could be done in a specific IOStream implementation.

So I think that the main difference is about the low level read/write API and not in the IOLoop interface.

Do you think am I missing some important point?

That's a good question; my knowledge of IOCP is mostly second-hand, so it is possible that there is a better way to do it (but all the examples I'm aware of use an interface similar to the ones I described).

One important difference is that IOCP is a *completion-based* interface while select-style event loops are *readiness-based*. That means that select() tells you when you can try your read or write again, but IOCP actually does the read for you and gives you the data (Frankly, the IOCP approach makes a lot more sense, and if I were designing an OS from scratch that's probably the way I'd go). So you can't use IOCP with the same readiness-based methods that we have now, but maybe it's possible to use just a few completion-based methods instead of the broad interface described in my previous message. (Given a readiness-based event loop it's straightforward to provide a completion-based interface, but you can't go the other way around). However, I'm not sure how much that actually saves: a read-ready callback may be followed by accept() (for listening sockets), recv (for regular sockets), or read (for pipes), so you'd still need almost as many methods as before. 

-Ben