AsyncSSH pattern for copying text files
Nicholas Chammas
In the absence of direct support for SFTP, I’ve been experimenting with ways to copy text files around using AsyncSSH in a way that fits its model.
Here’s an example I came up with. I have some questions and notes in-line as comments. Ron, could you comment on each?
import asyncio
import asyncssh
@asyncio.coroutine
def put_text_file(host, local_path, remote_path):
# what is client useful for?
conn, client = yield from asyncssh.create_connection(host=host, client_factory=None)
with open(local_path) as f:
contents = f.read()
stdin, stdout, stderr = yield from conn.open_session(
"""
cat > {path}
""".format(path=remote_path))
stdin.write(contents) # cannot yield from here; problem for large files?
stdin.close() # necessary otherwise method hangs here
yield from stdout.channel.wait_closed()
conn.close() # necessary?
print(stdout.channel.get_exit_status()) # always prints None
return
asyncio.get_event_loop().run_until_complete(
put_text_file(
host='localhost',
local_path='/path/to/file.txt',
remote_path='/path/to/copy.txt'))
I have a slightly different version that seems to work better (I get a proper return code, for example), but seems like it might be a problem.
Basically, it looks mostly the same except I am using echo instead of cat, which feel clunky:
with open(local_path) as f:
contents = f.read()
stdin, stdout, stderr = yield from conn.open_session(
"""
echo {f} > {path}
""".format(
f=shlex.quote(contents),
path=remote_path))
Am I using AsyncSSH correctly here? What is a good way to use AsyncSSH to do this kind of ghetto file copy for text files?
Nick
Ron Frederick
In the absence of direct support for SFTP, I’ve been experimenting with ways to copy text files around using AsyncSSH in a way that fits its model.
Here’s an example I came up with. I have some questions and notes in-line as comments. Ron, could you comment on each?
import asyncio import asyncssh @asyncio.coroutine def put_text_file(host, local_path, remote_path): # what is client useful for? conn, client = yield from asyncssh.create_connection(host=host, client_factory=None)
with open(local_path) as f: contents = f.read() stdin, stdout, stderr = yield from conn.open_session( """ cat > {path} """.format(path=remote_path)) stdin.write(contents) # cannot yield from here; problem for large files?
stdin.close() # necessary otherwise method hangs here
yield from stdout.channel.wait_closed() conn.close() # necessary?
print(stdout.channel.get_exit_status()) # always prints None return asyncio.get_event_loop().run_until_complete( put_text_file( host='localhost', local_path='/path/to/file.txt', remote_path='/path/to/copy.txt'))I have a slightly different version that seems to work better (I get a proper return code, for example), but seems like it might be a problem.
Basically, it looks mostly the same except I am using
echoinstead ofcat, which feel clunky:with open(local_path) as f: contents = f.read() stdin, stdout, stderr = yield from conn.open_session( """ echo {f} > {path} """.format( f=shlex.quote(contents), path=remote_path))
Am I using AsyncSSH correctly here? What is a good way to use AsyncSSH to do this kind of ghetto file copy for text files?
Nicholas Chammas
I actually began work on adding SFTP support to AsyncSSH today. So far, I have some of the basic message parsing done and I’m able to generate and parse the FXP_INIT and FXP_VERSION messages. It’ll be a little while before I have something complete enough to put up on Github, but the work is in progress!
Oh, that’s good to hear! I’ll be glad to help test it out once you have something ready.
In the case where you pass client_factory=None, the return value “client” will give you the instance of the base SSHClient class that was created to handle the connection callbacks. Since you aren’t subclassing it, it’s not all that useful, though. The new asyncssh.connect() API we’ve been discussing is a better option now, and it will only return “conn” and not “client”.
Perfect. Another hurrah for that new API! :)
stdin.write(contents) # cannot yield from here; problem for large files?It’s not a problem that you don’t yield here, but it can be memory-intensive to read the entire file in and then write it out this way. A loop where you pass a maximum length to f.read() and call stdin.write() on pieces of the file might be a better option.
Hmm, so I know that Python’s file I/O is still the same old blocking file I/O library, so we’re stuck waiting to read the chunk of the file we request.
But when writing that chunk to a remote server via AsyncSSH, don’t we want to yield somehow until that server has received everything we’ve sent over? I’m imagining sending the same file to 100 servers at once.
(I’m still wrapping my head around how and when asyncio is best used, so forgive the n00b question, heh.)
You need the write_eof() here to get the cat command to exit. Without this, the stdout.channel.wait_closed() won’t complete as cat is still waiting for more input.
Makes perfect sense! This maps naturally to how one would use cat at the command line, sending ^D to terminate the input stream for example.
However, it is cleaner to do a proper close() of the connection before you exit.
Understood. Sounds like a good use case for a context manager, actually, right? I’ll post an example on the thread where you mentioned the new connect() method, since it seems like a good fit for that (though the example is probably obvious).
Nick
Ron Frederick
I actually began work on adding SFTP support to AsyncSSH today. So far, I have some of the basic message parsing done and I’m able to generate and parse the FXP_INIT and FXP_VERSION messages. It’ll be a little while before I have something complete enough to put up on Github, but the work is in progress!
Oh, that’s good to hear! I’ll be glad to help test it out once you have something ready.
In the case where you pass client_factory=None, the return value “client” will give you the instance of the base SSHClient class that was created to handle the connection callbacks. Since you aren’t subclassing it, it’s not all that useful, though. The new asyncssh.connect() API we’ve been discussing is a better option now, and it will only return “conn” and not “client”.
Perfect. Another hurrah for that new API! :)
stdin.write(contents) # cannot yield from here; problem for large files?It’s not a problem that you don’t yield here, but it can be memory-intensive to read the entire file in and then write it out this way. A loop where you pass a maximum length to f.read() and call stdin.write() on pieces of the file might be a better option.
Hmm, so I know that Python’s file I/O is still the same old blocking file I/O library, so we’re stuck waiting to read the chunk of the file we request.
But when writing that chunk to a remote server via AsyncSSH, don’t we want to yield somehow until that server has received everything we’ve sent over? I’m imagining sending the same file to 100 servers at once.
(I’m still wrapping my head around how and when asyncio is best used, so forgive the n00b question, heh.)
18.5.5.3. StreamWriter
- class asyncio.StreamWriter(transport, protocol, reader, loop)
- coroutine drain()
Let the write buffer of the underlying transport a chance to be flushed.
The intended use is to write:
When the size of the transport buffer reaches the high-water limit (the protocol is paused), block until the size of the buffer is drained down to the low-water limit and the protocol is resumed. When there is nothing to wait for, the yield-from continues immediately.
Yielding from drain() gives the opportunity for the loop to schedule the write operation and flush the buffer. It should especially be used when a possibly large amount of data is written to the transport, and the coroutine does not yield-from between calls to write().
This method is a coroutine.
- class
asyncssh.SSHWriter(session, datatype=None)[source] drain()[source]Wait until the write buffer on the channel is flushed
This method is a coroutine which blocks the caller if the stream is currently paused for writing, returning when enough data has been sent on the channel to allow writing to resume. This can be used to avoid buffering an excessive amount of data in the channel’s send buffer.
You need the write_eof() here to get the cat command to exit. Without this, the stdout.channel.wait_closed() won’t complete as cat is still waiting for more input.
Makes perfect sense! This maps naturally to how one would use cat at the command line, sending ^D to terminate the input stream for example.
However, it is cleaner to do a proper close() of the connection before you exit.
Understood. Sounds like a good use case for a context manager, actually, right? I’ll post an example on the thread where you mentioned the new
connect()method, since it seems like a good fit for that (though the example is probably obvious).
Nicholas Chammas
The AsyncSSH streams API is modeled after the Python asyncio streams API. There, the write() call is not a coroutine, as it just appends whatever data you give it to the end of a transmit buffer.
Ah, so in the hypothetical situation of transmitting a chunk of bytes to 100 hosts at once, we are not waiting for each host to receive the chunk before moving on. Python will just queue the chunk up to be transferred by placing it on this transmit buffer, and that is a “fast” operation in the sense that it does not depend directly on the responsiveness of the remote host.
So yielding on drain() is useful in the case where we are appending data too quickly to the buffer that at some point the write operation will actually block waiting for stuff to transfer out. Yielding on drain() allows that particular coroutine to yield control until there is more room in the buffer to continue writing.
Did I understand correctly?
Nick
Ron Frederick
The AsyncSSH streams API is modeled after the Python asyncio streams API. There, the write() call is not a coroutine, as it just appends whatever data you give it to the end of a transmit buffer.
Ah, so in the hypothetical situation of transmitting a chunk of bytes to 100 hosts at once, we are not waiting for each host to receive the chunk before moving on. Python will just queue the chunk up to be transferred by placing it on this transmit buffer, and that is a “fast” operation in the sense that it does not depend directly on the responsiveness of the remote host.
So yielding on
drain()is useful in the case where we are appending data too quickly to the buffer that at some point the write operation will actually block waiting for stuff to transfer out. Yielding ondrain()allows that particular coroutine to yield control until there is more room in the buffer to continue writing.Did I understand correctly?