[ANN] go-scp: a scp client library in go

[Hiroaki Nakamura]

Hi all,

I noticed the golang.org/x/crypto/ssh package exists, but the scp
package does not.
So I wrote a scp client library in go.
https://github.com/hnakamur/go-scp

I also wrote a sshd server just usable for testing go-scp.
https://github.com/hnakamur/go-sshd

Right now, go-scp only exports high level functions which are supposed
to be easy to use.
https://godoc.org/github.com/hnakamur/go-scp

However I wonder if there APIs can be improved. For example,
better function names and better arguments.

Could you tell me what you think?
Thanks!

Hiroaki Nakamura

[Paul Borman]

Adding an scp package is a nice addition.

You might want to consider simple names like:

Send - Sends from []byte to file on remote host

SendDir - Send files in dir to a remote host

SendFile - Sends the contents of a file to the remote host

Fetch - Fetches the contents of a file on remote host into memory

FetchFile - Fetches a file from remote host into file on local host 

FetchDir - Fetches the files in a directory from the remote host

These would translate in code to names like scp.SendFile, which is pretty descriptive all by itself.

For the directory copy, it might be better to have a function return the io.Writer to write the file to, rather than force the files into a directory.  This would make it easy to keep the contents in memory, change file names, or whatever.

Hiroaki Nakamura

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hiroaki Nakamura]

Hi Paul,
Thanks for your feedback!

2016-09-20 0:02 GMT+09:00 Paul Borman <bor...@google.com>:
> Adding an scp package is a nice addition.

I agree.
Should I send a pull request to https://github.com/golang/crypto?
If yes, what package? golang.org/x/crypto/scp or golang.org/x/crypto/ssh/scp?

> You might want to consider simple names like:
>
> Send - Sends from []byte to file on remote host
> SendDir - Send files in dir to a remote host
> SendFile - Sends the contents of a file to the remote host
> Fetch - Fetches the contents of a file on remote host into memory
> FetchFile - Fetches a file from remote host into file on local host
> FetchDir - Fetches the files in a directory from the remote host
>
> These would translate in code to names like scp.SendFile, which is pretty
> descriptive all by itself.

Thanks for simple and descriptive function names.
I renamed functions.
https://github.com/hnakamur/go-scp/commit/8cd6d9e5ab17187556e5efc3a666d29b4b561c78

>
> For the directory copy, it might be better to have a function return the
> io.Writer to write the file to, rather than force the files into a
> directory. This would make it easy to keep the contents in memory, change
> file names, or whatever.

Yes, I agree it might be better not to force the files in a directory.
However I don't think having a function return the io.Writer will do
since we need to read a reply for each scp protocol header or body.

I had read the article below and the openssh source code and
implemented my scp package https://github.com/hnakamur/go-scp.

"How the SCP protocol works (Jan Pechanec's weblog)"
https://blogs.oracle.com/janp/entry/how_the_scp_protocol_works

I built two structs sourceProtocol and sinkProtocol
https://github.com/hnakamur/go-scp/blob/master/protocol.go

I had thought to export these structs or make interfaces for that.
However the implementation of two functions SendDir and FetchDir
which using these structs become somewhat complex.

https://github.com/hnakamur/go-scp/blob/8cd6d9e5ab17187556e5efc3a666d29b4b561c78/source.go#L81-L175
https://github.com/hnakamur/go-scp/blob/8cd6d9e5ab17187556e5efc3a666d29b4b561c78/sink.go#L111-L223

So I was not confident about exporting sourceProtocol and sinkProtocol,
and I did not export them at the time.

If we can define structs, functions or interfaces which are easy to use,
I'm glad to export them.

Do you have an idea about such structs, functions or interfaces?
Thanks!

Hiroaki Nakamura

>
> On Fri, Sep 16, 2016 at 9:41 AM, Hiroaki Nakamura <hnak...@gmail.com>
> wrote:
>>
>> Hi all,
>>
>> I noticed the golang.org/x/crypto/ssh package exists, but the scp
>> package does not.
>> So I wrote a scp client library in go.
>> https://github.com/hnakamur/go-scp
>>
>> I also wrote a sshd server just usable for testing go-scp.
>> https://github.com/hnakamur/go-sshd
>>
>> Right now, go-scp only exports high level functions which are supposed
>> to be easy to use.
>> https://godoc.org/github.com/hnakamur/go-scp
>>
>> However I wonder if there APIs can be improved. For example,
>> better function names and better arguments.
>>
>> Could you tell me what you think?
>> Thanks!
>>
>> Hiroaki Nakamura
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to golang-nuts...@googlegroups.com.

[Paul Borman]

I would suggest just putting your package out there and see how it is received.  At some point, if it becomes the defacto package, moving it might make sense.

I actually wrote an internal SCP package that works with a ssh multiplexer built on top of crypto/ssh.  For various reasons, I am not in a position to OpenSource the package.

It provides a NewSCP that returns an SCP object.  You provide NewSCP with a configuration structure that specifies optional default timestamps, mode, if -p should be used, and if Start or Shell should be used to make the session (some SSH servers do not support exec mode (Start)). 

The SCP type only has 3 methods (and one is actually a convenience wrapper):

func (*SCP) Send(ctx context.Context, dst string, srcs ...string) error

Send sends the srcs to dst. If srcs has a length greater than 1, or any of the srcs is a Directory, dst must reference a directory on the remote host. Send returns any errors encountered. If an error is returned, it may contain multiple errors. User Errors(err) to retrieve the list of errors.

func (*SCP) Receive(ctx context.Context, src string, f func(*Incoming) error) error

Receive requests the directory or file src from the remote host. For each directory and file in src, f is called with a populated Incoming structure. If IsDir is not set then f must either populate W or return an error. In any event, if f returns an error then that file or directory's transfer is stopped. If the returned error is wrapped by Fatal{}, the entire SCP session is shutdown.

type Incoming struct {

Path    string      // Name of file on remote host

Mode    os.FileMode // Mode of the file on the remote host

MTime   time.Time   // Last modified time from the remote host

ATime   time.Time   // Last access time from the remote host

Length  int64       // Length of incoming file

IsDir   bool        // Set if receiving a directory

WErr    error       // Error encountered during write, if any

W       io.Writer   // Destination for read data

NoClose bool        // Do not close W when finished.

}

Incoming is where to send information about a file or directory. When an Incoming is provided to ReceiveFile, W must be set and IsDir must not be set. If the set W also implements io.Closer, W.Close is called after the fimal write to W, unless NoClose is set. NoClose is normally only set when using a single io.Writer for all data (such as os.Stdout).

The convenience function is:

func (*SCP) ReceiveFile(ctx context.Context, in *Incoming, src string) error

ReceiveFile requests the remote host send the file named src. ReceiveFile writes the received contents to in.W. Path, Mode, MTime and ATime are set prior to the first write to in.W. MTime and ATime may be the zero value for time.Time if not provided by the remote host.

If there is an error writing to in.W, in.WErr is set to the error. in.WErr is only valid after ReceiveFile returns. If there is an error receiving the file from the remote host, it is returned.

You could easily imagine a wrapper that called Receive and provided an internal function that did whatever needed to be done, or even a stock function that can be passed to Receive.

I hope this is helpful.

    -Paul

>> email to golang-nuts+unsubscribe@googlegroups.com.

[Hiroaki Nakamura]

2016-09-21 0:59 GMT+09:00 Paul Borman <bor...@google.com>:
> I would suggest just putting your package out there and see how it is
> received. At some point, if it becomes the defacto package, moving it might
> make sense.

OK, I understood.

>
> I actually wrote an internal SCP package that works with a ssh multiplexer
> built on top of crypto/ssh. For various reasons, I am not in a position to
> OpenSource the package.

I hope your SCP package will be released as an OpenSource.

Thanks for the spec about your SCP package.
I think it is more general than mine.

However most of my cases are copy a single file or a directory,
so I think I'd keep my APIs for the moment.

I added NewSCP and SCP
https://github.com/hnakamur/go-scp/commit/b43795fc10ef52b02a76653b643fa5aabb7b85a7

One thing I would like to add is the cancellation support using context.Context.
However ssh.Run does not take a context.Context as an argument,
so I don't know how to achieve that.
https://godoc.org/golang.org/x/crypto/ssh#Session.Run

Could you tell me how you implement the cancellation?

Thanks,
Hiroaki Nakamura

>> >> email to golang-nuts...@googlegroups.com.

[Paul Borman]

For cancellation you run the actual operation in a goroutine and return results on a channel.  You can then select on the Deadline from the context and the returned result.  If the deadline hits, you just shut down the SSH session.  Does that make sense?

    -Paul

>> >> email to golang-nuts+unsubscribe@googlegroups.com.

[Hiroaki Nakamura]

2016-09-21 8:31 GMT+09:00 Paul Borman <bor...@google.com>:
> For cancellation you run the actual operation in a goroutine and return
> results on a channel. You can then select on the Deadline from the context
> and the returned result. If the deadline hits, you just shut down the SSH
> session. Does that make sense?

Yes, thank you for your explanation!
I'll try to implement it in my spare time.

Thanks

>> >> >> email to golang-nuts...@googlegroups.com.