[websocket] websocket.write_message when data is available at fd connected to psuedo terminal

[james walker]

Hi All,

Python version: 3.8

Tornado version: 6.1/latest

I have a fd connected to a pseudo terminal, I am using select.select(fd, []. []) to read from this. Is there a way to use asyncio/tornado to call write_message once data is available?

rlist, wlist, xlist = select.select([self.application.settings["pty_master_fd"]], [], [], timeout_sec)

Basically when there is a value in rlist I need to read and send the data to websocket.

I have seen the other thread and some stackoverflow answers , all of them suggest to keep a list of connected ws clients and loop over them to write_message. This seems a bit hackish to me.

Is there a way I can use something aysncio/tornado to run the read_emit_pty_otuput function in background and whenever data is available write that to websocket ?

Thanks,

PS: Please forgive me if this has already been answered many times.
I am very new to tornado and asyncio.

[Rajdeep Rath]

Not sure how others would do it but for me I would loop over each client and for each put the message into the message queue of the client. Then from there each client handler writes it to the client socket. It uses await to read write with the queue. Not sure if there would be another way?

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/7b5a61b5-73f4-4333-85bc-2c125597e148n%40googlegroups.com.

[james walker]

Currently the method read_emit_pty_otuput is a blocking method,  I need to call it from open and put in background, for that I was thinking of using IOLoop.spawn_callback. As far as I could understand for adding a callback the read_emit_pty should be a coroutine.

Why will each client need a message queue ?  Can I not directly write_message to the client socket ?

[Pierce Lopez]

As you noticed, the blocking is a problem. Waiting for fd readiness is one of the jobs of the ioloop, so you shouldn't have to do that yourself with select(). I'd try to use a PipeIOStream to read from the fd, but I'm not sure how well it works with a pty.

https://www.tornadoweb.org/en/stable/iostream.html#tornado.iostream.PipeIOStream

If not using the ioloop to wait for the fd readiness, then you'll need to spawn a thread to do it, and then use the add_callback() method of the main ioloop to transfer execution back to the main ioloop's thread before calling any other tornado functions/methods (e.g. write_message()).

https://www.tornadoweb.org/en/stable/ioloop.html#tornado.ioloop.IOLoop.add_callback

- Pierce

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/59949a25-ee7d-4b70-a046-4c5b13d23bd6n%40googlegroups.com.

[james walker]

Thanks for your response pierce,

I can not find example usage of PipeIOStream in the docs, can you please clarify these two things for me ?

1) Where should I instantiate the PipeIOStream ? 

   

2) What method should be used to read from it?  read_from_fd ? or read_bytes ? 

App setup:

open() method:

After print("3--") the server seems to have hanged. It does not respond to ws.send("a") from frontend.

initialize_pseduo_terminal

3) Unless the read_pty_output_and_emit returns something the open() method will not close right ? Instead I want to run the read_pty_output_and_emit forever in background until on_close is called, and as soon as there is data fd it should send websocket event.

PS: If the code is not clear, this is the backend for in browser python repl. Browser sends the input through websocket, input is passed to the pseudoterminal which passes it to the python3 subprocess.

When python3 subprocess has some output to give, it writes on the "pty_master_fd". Whenever we have something to read on this "pty_master_fd" I need to send websocket message to browser.

Thanks your help. 

[Pierce Lopez]

In your open() method, you don't want to pause that indefinitely with await self.read_pty_output_and_emit(), you instead want to do something like IOLoop.current().spawn_callback(self.read_pty_output_and_emit) to let that run after open() has returned.

You can enable websocket "ping" and timeout in the tornado.web.Application settings. But that's not why things aren't working right at the start.

Forking the process, and using a pty, put you in a world of complexities that I have not mixed with python async ioloops. However, if you can do without the pty, you can use tornado.process.Subprocess and it can create PipeIOStreams for you:

https://www.tornadoweb.org/en/stable/process.html#tornado.process.Subprocess

- Pierce

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/1392a978-09a9-4f1c-91f5-fe38dfdbe646n%40googlegroups.com.

[james walker]

Thanks, 

Yes I figured out the same that I have to use the spawn_callback. Now at least server is responding correctly. 

The callback never seems to run though, when I receive a message I write this to the fd.

Yes I agree that this mixes too many things, but I have seen in many answers here and on stack overflow that @Ben Darnell has suggested that we can make pty work with tornado.

I can't understand when the callback is called, the documentation suggests that spawn_callback "Calls the given callback on the next IOLoop iteration." 

Q1) Does this mean whenever cycles are free callback will run ? If not then when exactly the callback runs

And https://www.tornadoweb.org/en/stable/guide/coroutines.html#how-to-call-a-coroutine, at the end of this there is a section that say Finally if IOLoop is not yet running then you need to do IOLoop.current.run_sync()

I am using this IOLoop.current().start() to start the loop in main. 
Q2) So long as the server is running my IOLoop wil always be running right ?

Q3) So now the only question remains is how to trigger the callback on some custom event ?

Thank you for taking the time to answer my queries. :)

[Pierce Lopez]

Yes the IOLoop should still be running, and the async method callback should run less than a millisecond after spawn_callback().

I suggest building up familiarity with the basics of async functions and event loops, by starting with a demo application that does not use pty or fork (or threads). Using things that are designed for asyncio, like all the conveniences offered by tornado itself (http client, queues, etc) is reasonably straightforward. Integrating with blocking/forking/threading code is possible but fairly advanced, and if you get it slightly wrong the failures will be mysterious.

- Pierce

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/8fbaf34a-d8db-4e6b-b252-5a743fc1bd39n%40googlegroups.com.

[james walker]

Hi Pierce, thank you for your time

Yes indeed I do not have much knowledge about python's asyncio and event loops. I chose tornado to do this mainly because it provides inbuilt websocket server, I have no requirements for other stuff.

1) As far as I can understand subprocess.Popen is the same as os.fork and os.exec.

2) But having a pseudoterminal is a requirement , as my goal is to  emulate a bash terminal or python interpreter inside a browser.

For anyone who comes here in future
1) Right tool for the job is node-pty  node-pty project from microsoft, which does all this in 20 lines of code.

2) If you insist on using python, which I don't recommend. It is far far simpler to do it with flask + flask-socketio, gevent

def read_pty_and_emit():

    while True:

        r,w,e = select.select([fd], [], [])

My initial thoughts that select is blocking is somewhat wrong too, select and poll/epoll are mechanisms to do asynchronous I/O on linux. https://man7.org/linux/man-pages/man2/select.2.html, 

To make the above method run in background, all you have to do is socketio.run_background_task(). (This still does rely on coroutines by the way that's what the gevent library is for)

No need to mess with async/await, PipeStreams, eventloops like I was trying to do through IOLoop.current.spawn_callback(), in my humble opinion this is just needless complexity.

Although as a self-respecting python dev, I should probably learn how to work with asyncio :P

[Rajdeep Rath]

Hello James,  just curious if one uses tornado interactive sub process  (read and write) with web sockets , wouldn’t it be able to achieve the same ? Definitely. It that simple but possible?

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/002b4546-051a-453a-88a0-7b75ff3792b2n%40googlegroups.com.

[Pierce Lopez]

I agree that this particular task is probably easier using other libraries. From time to time, people come to tornado just to try to get the websocket server functionality added to the other thing they are already trying to do, but are not familiar with async programming, and it is not practical to re-do all the other stuff for the async model, or to learn the somewhat more tricky techniques for interoperating between the ioloop and the other stuff running in separate threads.

select() is used for asynchronous programming on many operating systems, but if you do not give it an instantaneous timeout, it is blocking. If using an event-loop, your user code should not use select, the ioloop is what uses select() (but usually a more advanced variant like epoll on linux or kqueue on macOS) in order to check for readiness of all file descriptors at the same time, so it can continue whatever coroutine is ready. If one coroutine or user function uses select() with a non-instantaneous timeout, it blocks the ioloop, which makes everything else not work right.

subprocess is the same as fork+exec, but there can be tricky details between the fork and exec. (and it's more efficient, and compatible with windows, by using posix_spawn() instead, or the equivalent on windows, since windows doesn't have fork())

[Rajdeep Rath]

I seem I think I understand a little. But can you tell me if for example if we wanted to bridge the shell and browser using subprocess and websockets from tornado, is it not possible or just more work than in other similar frameworks?

--

You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CAOR_4NCwcTBuqFnrKy%3DgEfZxgFtums6%2BFuF%3DEs0ML08b%2B3t%2BDQ%40mail.gmail.com.

[Pierce Lopez]

Yes, it should be possible ... but not straightforward, if you're not already familiar with these things. A simple tornado subprocess should be more straightforward, then you just do all your code the asyncio way, you can try some different strategies and make progress. But add the pty stuff, then you have to figure out how to integrate that with asyncio. I'm not personally familiar enough to give very specific example of how to do it, and it seems not many others active on the list are either, so you need to be or become an "advanced" asyncio user to realistically be able to figure out and debug the right way. This is a lot to ask of someone who just wants to get this websocket working and move on.

By the way, a google search picked up an example strategy that I think is supposed to work: https://bugs.python.org/file47615/pty_test.py

- Pierce

[Rajdeep Rath]

👍

--

You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CAOR_4NDAMyM0GybMF6gCexV0nY5s-PfS3w7E6WhzOu9w0v26TQ%40mail.gmail.com.

[Pierce Lopez]

On Sat, Nov 21, 2020 at 2:45 PM Pierce Lopez <pierce...@gmail.com> wrote:

By the way, a google search picked up an example strategy that I think is supposed to work: https://bugs.python.org/file47615/pty_test.py

Actually, that doesn't help with using the pty for a subprocess, so nevermind :)

[Rajdeep Rath]

When I get a chance I will experiment with this using tornado process and websockets. Atleast theoretically bit is sound in the mind ;).

--

You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CAOR_4NDjbewGZC7dobnoS3gth%2BKtie4%3DM6EfUaxb3xuAAEw_qg%40mail.gmail.com.

[Ben Darnell]

On Wed, Nov 18, 2020 at 7:39 PM james walker <codeb...@gmail.com> wrote:

Hi All,

Python version: 3.8

Tornado version: 6.1/latest

I have a fd connected to a pseudo terminal, I am using select.select(fd, []. []) to read from this. Is there a way to use asyncio/tornado to call write_message once data is available?

rlist, wlist, xlist = select.select([self.application.settings["pty_master_fd"]], [], [], timeout_sec)

Basically when there is a value in rlist I need to read and send the data to websocket.

Let's come back to this original example. If you're already operating at the level of system calls like select.select and os.read, you're not far from Tornado's native capabilities. You can use PipeIOStream, but I think in this case it complicates more than it helps (tip: add `partial=True` to the `read_bytes()` call in one of your other examples. Without that, `read_bytes` will always wait to return exactly the requested number of bytes, instead of behaving like `os.read`).

A loop that calls `select()` can be replaced with `IOLoop.add_handler`. You'll get a notification that the file descriptor is ready to read, and you can use `os.read` exactly as you do here.

    def read_from_pty(self, fd, events):

        pty_output = os.read(fd, max_read_bytes)

        self.write_message(pty_output)

    def open(self):

        fd = self.application.settings["pty_master_fd"]

        os.set_blocking(fd, False)

        IOLoop.current().add_handler(fd, IOLoop.READ, self.read_from_pty)

    def on_close(self):

       IOLoop.current().remove_handler(self.application.settings["pty_master_fd"])

-Ben

 

I have seen the other thread and some stackoverflow answers , all of them suggest to keep a list of connected ws clients and loop over them to write_message. This seems a bit hackish to me.

Is there a way I can use something aysncio/tornado to run the read_emit_pty_otuput function in background and whenever data is available write that to websocket ?

Thanks,

PS: Please forgive me if this has already been answered many times.
I am very new to tornado and asyncio.

--

You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/7b5a61b5-73f4-4333-85bc-2c125597e148n%40googlegroups.com.

[james walker]

Hi, 

@Rajdeep as you noticed it sound possible to do it through subprocess.

I had read up a lot about simulating a python repl in a browser and having pseudo terminal is best for such purpose then you don't have to worry about special control sequences.(like ctrl +d).

In case someone is interested in learning about tty/pty a very thorough explanation is given here: http://www.linusakesson.net/programming/tty/index.php

@pierce gives quite an accurate statement why I did not try much "This is a lot to ask of someone who just wants to get this websocket working and move on."

I want to start from something that works, that I can understand, and is boring/old in that order. 

• node-pty is especially designed for simulating terminals and is developed open-source by microsoft. 
• I have a lot of experience with flask, gevent etc, (have delivered 5-6 microservices in flask at work). So if there's any issue I can probably find it on my own.
• It's hard to find answers/documentation for tornado as it used by much less people than flask.

@Ben Darnell Thank you for the input. I tried your solution and can confirm that it's working correctly.

[Rajdeep Rath]

👍

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/b3c5213c-e13a-4b2a-8017-db1afe196f4fn%40googlegroups.com.