How to effectively write a large file asynchronously in append mode?

[rajde...@gmail.com]

Hello , 

i am trying to see if a large file can be written to in append mode from tornado from a stream of incoming data without actually hampering tornado's core performance with other things. 

I have looked at aiofile but i cant find the proper examples of writing a file continuously. Can anyone provide some inputs?

Regards

[agn...@gmail.com]

look at this issues #231

[Rajdeep Rath]

Thanks for answering me 👏. However am little confused. Are you sure that is what will help me? Cause I am not looking to read data from post typically. My data could very well be In a file already or coming from a socket. I basically need to know how I can keep writing the data to large file in append mode. If the file already has data I would want to add to it and so on.

--


You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.


To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.


To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/b32775e3-0b04-4e24-8206-6c23562c2ffcn%40googlegroups.com.

[Shane Spencer]

aiofile is a good python module that addresses writing async

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CAP869y8ZWeWusUPxNxf91nheyhkSEjXxQOyHRuWDr7EzgCxO%3Dw%40mail.gmail.com.

--

Shane Spencer about.me/ShaneSpencer

[Michael DePalatis]

`stream_request_body` is precisely what you want. It is documented here: https://www.tornadoweb.org/en/stable/web.html#tornado.web.stream_request_body (there is a link to a file upload demo there). Here's the basic idea of usage:

from tornado.ioloop import IOLoop
from tornado.web import Application, RequestHandler, stream_request_body

@stream_request_body
class MyHandler(RequestHandler):
    def prepare(self):
        self.file_obj = open("output.txt", "ab")

    def on_finish(self):
        self.file_obj.close()

    def data_received(self, data):
        self.file_obj.write(data)

    def post(self):
        print("data received")
        self.write("ok")

if __name__ == "__main__":
    handlers = [("/", MyHandler)]
    app = Application(handlers)
    app.listen(8999)
    IOLoop.current().start()

You just need to implement the `data_received` which operates on each chunk of data as it comes in. Note that in this case there's no async/await, but `prepare`, `data_received`, and `post` may all be async if you want.

[Rajdeep Rath]

But I don’t see an example of writing large file in append mode anywhere.

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CANmUDYvfnbD8mTEp98xN%3DY44uEZXHdfRCg0tmceJkOJw_gwUEA%40mail.gmail.com.

[Rajdeep Rath]

Michael

Thanks. But what if I am not even using http post or get ? How will stream body help in writing data from somewhere else to a file in file system async.?

You received this message because you are subscribed to a topic in the Google Groups "Tornado Web Server" group.


To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-tornado/_at18vcqx_g/unsubscribe.


To unsubscribe from this group and all its topics, send an email to python-tornad...@googlegroups.com.


To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/df44fa27-99cf-4bbc-b442-623336de0b02n%40googlegroups.com.

[Shane Spencer]

There are async file libraries on pypi.

Tornado uses python asyncio loop now by default.  You can use async and await along with libs like aiofile to do async file stuffs.

Beyond that not sure how to better answer your question

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CAP869y9z01MJ9upuYh1jG7C5DyTpGvpv4gN%3DDvPgM2tO5ZN51g%40mail.gmail.com.

[Rajdeep Rath]

Ok thanks. I did check that out. I guess I have to ask the developer for a append mode example. Looks like that is what I need. Currently I can only see write and read examples.

Regards

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CANmUDYuAE2jX_1hbEWSbo6_OswgcEDSvzV%2B8PNYzh3HaH8r-HQ%40mail.gmail.com.

[Shane Spencer]

Read the source...  also w+ is common shorthand for append mode.

https://github.com/mosquito/aiofile/blob/master/aiofile/aio.py

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CAP869y_d-%3Dz%3D1yzpW8ChjNonWOqcyAZXh%3D%2B5bpH6bDCV5%3DJZbw%40mail.gmail.com.

[Shane Spencer]

Sorry.  I mean ‘a’.  

[Rajdeep Rath]

Ok thank you so much. I will check that out. 👍

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/CANmUDYsLoLQJ3cYBAOJLCZNSzzV2-9QFM1SG8skPCd37dB29Sw%40mail.gmail.com.