Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
SSH/SCP question: Does it do "reget"?
67 views
Skip to first unread message
Kenny McCormack
Jul 4, 2012, 8:47:17 AM7/4/12
to
By "reget", I mean the ability to resume an aborted transfer - e.g., suppose
you have successfully transferred all but the last few bytes of a large file
when the connection fails. You would like to be able to send the remaining
bytes and have it do the right thing (connecting it up with the existing
file sitting on the host). I believe that "wget" does this (or can, with
the right options set), but I could not find any such option in ssh/scp.
I'm hoping that I have merely overlooked something.
--
> No, I haven't, that's why I'm asking questions. If you won't help me,
> why don't you just go find your lost manhood elsewhere.
CLC in a nutshell.
you have successfully transferred all but the last few bytes of a large file
when the connection fails. You would like to be able to send the remaining
bytes and have it do the right thing (connecting it up with the existing
file sitting on the host). I believe that "wget" does this (or can, with
the right options set), but I could not find any such option in ssh/scp.
I'm hoping that I have merely overlooked something.
--
> No, I haven't, that's why I'm asking questions. If you won't help me,
> why don't you just go find your lost manhood elsewhere.
CLC in a nutshell.
pk
Jul 4, 2012, 9:06:05 AM7/4/12
to
On Wed, 4 Jul 2012 12:47:17 +0000 (UTC), gaz...@shell.xmission.com (Kenny
McCormack) wrote:
> By "reget", I mean the ability to resume an aborted transfer - e.g.,
> suppose you have successfully transferred all but the last few bytes of a
> large file when the connection fails. You would like to be able to send
> the remaining bytes and have it do the right thing (connecting it up with
> the existing file sitting on the host). I believe that "wget" does this
> (or can, with the right options set), but I could not find any such
> option in ssh/scp.
It doesn't do that, but rsync does.
McCormack) wrote:
> By "reget", I mean the ability to resume an aborted transfer - e.g.,
> suppose you have successfully transferred all but the last few bytes of a
> large file when the connection fails. You would like to be able to send
> the remaining bytes and have it do the right thing (connecting it up with
> the existing file sitting on the host). I believe that "wget" does this
> (or can, with the right options set), but I could not find any such
> option in ssh/scp.
Stachu 'Dozzie' K.
Jul 4, 2012, 9:22:28 AM7/4/12
to
On 2012-07-04, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> By "reget", I mean the ability to resume an aborted transfer - e.g., suppose
> you have successfully transferred all but the last few bytes of a large file
> when the connection fails. You would like to be able to send the remaining
> bytes and have it do the right thing (connecting it up with the existing
> file sitting on the host). I believe that "wget" does this (or can, with
> the right options set), but I could not find any such option in ssh/scp.
>
> I'm hoping that I have merely overlooked something.
OpenSSH's commands don't support re-get on their own, but lftp does (and
> By "reget", I mean the ability to resume an aborted transfer - e.g., suppose
> you have successfully transferred all but the last few bytes of a large file
> when the connection fails. You would like to be able to send the remaining
> bytes and have it do the right thing (connecting it up with the existing
> file sitting on the host). I believe that "wget" does this (or can, with
> the right options set), but I could not find any such option in ssh/scp.
>
> I'm hoping that I have merely overlooked something.
it supports sftp: URLs). rsync can resume transfer as well.
--
Secunia non olet.
Stanislaw Klekot
Kenny McCormack
Jul 4, 2012, 11:09:33 AM7/4/12
to
In article <slrnjv8gdh...@jarowit.net>,
Interesting. Maybe I will look into using either lftp or rsync.
Does it strike anyone else as odd that this (pretty obvious and normal)
functionality was forgotten in the SSH design & implementation?
--
Religion is regarded by the common people as true,
by the wise as foolish,
and by the rulers as useful.
(Seneca the Younger, 65 AD)
Does it strike anyone else as odd that this (pretty obvious and normal)
functionality was forgotten in the SSH design & implementation?
--
Religion is regarded by the common people as true,
by the wise as foolish,
and by the rulers as useful.
(Seneca the Younger, 65 AD)
Stephane Chazelas
Jul 4, 2012, 12:54:41 PM7/4/12
to
2012-07-04 15:09:33 +0000, Kenny McCormack:
The sftp protocol does support getting arbitrary chunk of files.
Otherwise things like sshfs wouldn't work. It's just the openssh
sftp client that has no interface to it.
Note that lftp and curl use the sftp subsystem, while rsync uses
ssh and relies on there being a rsync command on the remote end.
--
Stephane
> In article <slrnjv8gdh...@jarowit.net>,
> Stachu 'Dozzie' K. <doz...@go.eat.some.screws.spammer.invalid> wrote:
> >On 2012-07-04, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> >> By "reget", I mean the ability to resume an aborted transfer - e.g., suppose
> >> you have successfully transferred all but the last few bytes of a large file
> >> when the connection fails. You would like to be able to send the remaining
> >> bytes and have it do the right thing (connecting it up with the existing
> >> file sitting on the host). I believe that "wget" does this (or can, with
> >> the right options set), but I could not find any such option in ssh/scp.
> >>
> >> I'm hoping that I have merely overlooked something.
> >
> >OpenSSH's commands don't support re-get on their own, but lftp does (and
> >it supports sftp: URLs). rsync can resume transfer as well.
>
> Interesting. Maybe I will look into using either lftp or rsync.
>
> Does it strike anyone else as odd that this (pretty obvious and normal)
> functionality was forgotten in the SSH design & implementation?
[...]
> Stachu 'Dozzie' K. <doz...@go.eat.some.screws.spammer.invalid> wrote:
> >On 2012-07-04, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> >> By "reget", I mean the ability to resume an aborted transfer - e.g., suppose
> >> you have successfully transferred all but the last few bytes of a large file
> >> when the connection fails. You would like to be able to send the remaining
> >> bytes and have it do the right thing (connecting it up with the existing
> >> file sitting on the host). I believe that "wget" does this (or can, with
> >> the right options set), but I could not find any such option in ssh/scp.
> >>
> >> I'm hoping that I have merely overlooked something.
> >
> >OpenSSH's commands don't support re-get on their own, but lftp does (and
> >it supports sftp: URLs). rsync can resume transfer as well.
>
> Interesting. Maybe I will look into using either lftp or rsync.
>
> Does it strike anyone else as odd that this (pretty obvious and normal)
> functionality was forgotten in the SSH design & implementation?
The sftp protocol does support getting arbitrary chunk of files.
Otherwise things like sshfs wouldn't work. It's just the openssh
sftp client that has no interface to it.
Note that lftp and curl use the sftp subsystem, while rsync uses
ssh and relies on there being a rsync command on the remote end.
--
Stephane
Barry Margolin
Jul 4, 2012, 1:09:47 PM7/4/12
to
In article <jt1mbd$6j6$2...@news.xmission.com>,
similar to the normal Unix cp command. cp doesn't have a way to copy
parts of files, either.
Of course, you could argue that cp is less likely to be interrupted in
the middle of a file than scp, and even if it is the overhead of
starting over is much less. So it's often necessary to provide extra
functions for network applications than the analogous local apps.
--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
gaz...@shell.xmission.com (Kenny McCormack) wrote:
> Does it strike anyone else as odd that this (pretty obvious and normal)
> functionality was forgotten in the SSH design & implementation?
scp (and its predecessor, rcp) was intended to provide functionality
> Does it strike anyone else as odd that this (pretty obvious and normal)
> functionality was forgotten in the SSH design & implementation?
similar to the normal Unix cp command. cp doesn't have a way to copy
parts of files, either.
Of course, you could argue that cp is less likely to be interrupted in
the middle of a file than scp, and even if it is the overhead of
starting over is much less. So it's often necessary to provide extra
functions for network applications than the analogous local apps.
--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
Kenny McCormack
Jul 4, 2012, 1:16:04 PM7/4/12
to
In article <barmar-5CC42E....@news.eternal-september.org>,
wanting to preserve tradition (and traditional behaviors) and wanting to "do
things right". There is a strong tradition of "that's the way it has always
been done, so that's the way we're going to continue to do it".
As I've mentioned a couple of times now, in various threads, I think of ssh
and friends as not only doing it over securely, but also as an opportunity
to do it over "fixing all the old bugs". I'm disappointed when I find this
not to be the case - i.e., to find that they elected to not fix an old bug,
but rather to retain the old behavior.
--
Is God willing to prevent evil, but not able? Then he is not omnipotent.
Is he able, but not willing? Then he is malevolent.
Is he both able and willing? Then whence cometh evil?
Is he neither able nor willing? Then why call him God?
~ Epicurus
Barry Margolin <bar...@alum.mit.edu> wrote:
>In article <jt1mbd$6j6$2...@news.xmission.com>,
> gaz...@shell.xmission.com (Kenny McCormack) wrote:
>
>> Does it strike anyone else as odd that this (pretty obvious and normal)
>> functionality was forgotten in the SSH design & implementation?
>
>scp (and its predecessor, rcp) was intended to provide functionality
>similar to the normal Unix cp command. cp doesn't have a way to copy
>parts of files, either.
>
>Of course, you could argue that cp is less likely to be interrupted in
>the middle of a file than scp, and even if it is the overhead of
>starting over is much less. So it's often necessary to provide extra
>functions for network applications than the analogous local apps.
Yup. The design of (modern) Unix commands is often a tug-of-war between
>In article <jt1mbd$6j6$2...@news.xmission.com>,
> gaz...@shell.xmission.com (Kenny McCormack) wrote:
>
>> Does it strike anyone else as odd that this (pretty obvious and normal)
>> functionality was forgotten in the SSH design & implementation?
>
>scp (and its predecessor, rcp) was intended to provide functionality
>similar to the normal Unix cp command. cp doesn't have a way to copy
>parts of files, either.
>
>Of course, you could argue that cp is less likely to be interrupted in
>the middle of a file than scp, and even if it is the overhead of
>starting over is much less. So it's often necessary to provide extra
>functions for network applications than the analogous local apps.
wanting to preserve tradition (and traditional behaviors) and wanting to "do
things right". There is a strong tradition of "that's the way it has always
been done, so that's the way we're going to continue to do it".
As I've mentioned a couple of times now, in various threads, I think of ssh
and friends as not only doing it over securely, but also as an opportunity
to do it over "fixing all the old bugs". I'm disappointed when I find this
not to be the case - i.e., to find that they elected to not fix an old bug,
but rather to retain the old behavior.
--
Is God willing to prevent evil, but not able? Then he is not omnipotent.
Is he able, but not willing? Then he is malevolent.
Is he both able and willing? Then whence cometh evil?
Is he neither able nor willing? Then why call him God?
~ Epicurus
Barry Margolin
Jul 4, 2012, 2:27:32 PM7/4/12
to
In article <jt1tok$fst$2...@news.xmission.com>,
can elect to start from scratch, doing everything right. Or you can
take a more evolutionary approach, just replacing bits and pieces of the
design.
Thus, the designers of rcp probably said to themselves, "it's good
enough, except for its really bad security -- if we simply replace the
underlying transfer protocol with SSH we'll have something really nice."
gaz...@shell.xmission.com (Kenny McCormack) wrote:
> In article <barmar-5CC42E....@news.eternal-september.org>,
> Barry Margolin <bar...@alum.mit.edu> wrote:
> >In article <jt1mbd$6j6$2...@news.xmission.com>,
> > gaz...@shell.xmission.com (Kenny McCormack) wrote:
> >
> >> Does it strike anyone else as odd that this (pretty obvious and normal)
> >> functionality was forgotten in the SSH design & implementation?
> >
> >scp (and its predecessor, rcp) was intended to provide functionality
> >similar to the normal Unix cp command. cp doesn't have a way to copy
> >parts of files, either.
> >
> >Of course, you could argue that cp is less likely to be interrupted in
> >the middle of a file than scp, and even if it is the overhead of
> >starting over is much less. So it's often necessary to provide extra
> >functions for network applications than the analogous local apps.
>
> Yup. The design of (modern) Unix commands is often a tug-of-war between
> wanting to preserve tradition (and traditional behaviors) and wanting to "do
> things right". There is a strong tradition of "that's the way it has always
> been done, so that's the way we're going to continue to do it".
>
> As I've mentioned a couple of times now, in various threads, I think of ssh
> and friends as not only doing it over securely, but also as an opportunity
> to do it over "fixing all the old bugs". I'm disappointed when I find this
> not to be the case - i.e., to find that they elected to not fix an old bug,
> but rather to retain the old behavior.
As you say, there's a tug of war when designing things like this. You
> In article <barmar-5CC42E....@news.eternal-september.org>,
> Barry Margolin <bar...@alum.mit.edu> wrote:
> >In article <jt1mbd$6j6$2...@news.xmission.com>,
> > gaz...@shell.xmission.com (Kenny McCormack) wrote:
> >
> >> Does it strike anyone else as odd that this (pretty obvious and normal)
> >> functionality was forgotten in the SSH design & implementation?
> >
> >scp (and its predecessor, rcp) was intended to provide functionality
> >similar to the normal Unix cp command. cp doesn't have a way to copy
> >parts of files, either.
> >
> >Of course, you could argue that cp is less likely to be interrupted in
> >the middle of a file than scp, and even if it is the overhead of
> >starting over is much less. So it's often necessary to provide extra
> >functions for network applications than the analogous local apps.
>
> Yup. The design of (modern) Unix commands is often a tug-of-war between
> wanting to preserve tradition (and traditional behaviors) and wanting to "do
> things right". There is a strong tradition of "that's the way it has always
> been done, so that's the way we're going to continue to do it".
>
> As I've mentioned a couple of times now, in various threads, I think of ssh
> and friends as not only doing it over securely, but also as an opportunity
> to do it over "fixing all the old bugs". I'm disappointed when I find this
> not to be the case - i.e., to find that they elected to not fix an old bug,
> but rather to retain the old behavior.
can elect to start from scratch, doing everything right. Or you can
take a more evolutionary approach, just replacing bits and pieces of the
design.
Thus, the designers of rcp probably said to themselves, "it's good
enough, except for its really bad security -- if we simply replace the
underlying transfer protocol with SSH we'll have something really nice."
Christian Weisgerber
Jul 4, 2012, 2:20:47 PM7/4/12
to
Kenny McCormack <gaz...@shell.xmission.com> wrote:
> Does it strike anyone else as odd that this (pretty obvious and normal)
> functionality was forgotten in the SSH design & implementation?
No. scp(1) was designed as a drop-in replacement of Berkeley rcp(1)
> Does it strike anyone else as odd that this (pretty obvious and normal)
> functionality was forgotten in the SSH design & implementation?
and nothing more. By that standard it already suffers from bad
featuritis. As other people have already mentioned, if you want
advanced file transfer functions, use rsync over ssh.
--
Christian "naddy" Weisgerber na...@mips.inka.de
Kenny McCormack
Jul 4, 2012, 4:54:54 PM7/4/12
to
In article <jt21hv$17kh$1...@lorvorc.mips.inka.de>,
--
Here's a simple test for Fox viewers:
1) Sit back, close your eyes, and think (Yes, I know that's hard for you).
2) Think about and imagine all of your ridiculous fantasies about Barack Obama.
3) Now, imagine that he is white. Cogitate on how absurd your fantasies
seem now.
See? That wasn't hard, was it?
Christian Weisgerber <na...@mips.inka.de> wrote:
>Kenny McCormack <gaz...@shell.xmission.com> wrote:
>
>> Does it strike anyone else as odd that this (pretty obvious and normal)
>> functionality was forgotten in the SSH design & implementation?
>
>No. scp(1) was designed as a drop-in replacement of Berkeley rcp(1)
>and nothing more. By that standard it already suffers from bad
>featuritis. As other people have already mentioned, if you want
>advanced file transfer functions, use rsync over ssh.
So, we can mark you down as a traditionalist. Thank you.
>Kenny McCormack <gaz...@shell.xmission.com> wrote:
>
>> Does it strike anyone else as odd that this (pretty obvious and normal)
>> functionality was forgotten in the SSH design & implementation?
>
>No. scp(1) was designed as a drop-in replacement of Berkeley rcp(1)
>and nothing more. By that standard it already suffers from bad
>featuritis. As other people have already mentioned, if you want
>advanced file transfer functions, use rsync over ssh.
--
Here's a simple test for Fox viewers:
1) Sit back, close your eyes, and think (Yes, I know that's hard for you).
2) Think about and imagine all of your ridiculous fantasies about Barack Obama.
3) Now, imagine that he is white. Cogitate on how absurd your fantasies
seem now.
See? That wasn't hard, was it?
Stephane Chazelas
Jul 4, 2012, 4:55:43 PM7/4/12
to
2012-07-04 17:54:41 +0100, Stephane Chazelas:
[...]
And "scp" (as I had overlooked that the question was about scp
in the first place) also doesn't use the sftp subsystem but
relies on scp and runs a shell and the scp command on the remote
host as well.
And if we can run a shell on the remote host, we can do
ssh host tail -c +xxx file >> file
to resume a transfer.
--
Stephane
[...]
> > Does it strike anyone else as odd that this (pretty obvious and normal)
> > functionality was forgotten in the SSH design & implementation?
> [...]
>
> The sftp protocol does support getting arbitrary chunk of files.
> Otherwise things like sshfs wouldn't work. It's just the openssh
> sftp client that has no interface to it.
>
> Note that lftp and curl use the sftp subsystem, while rsync uses
> ssh and relies on there being a rsync command on the remote end.
[...]
> > functionality was forgotten in the SSH design & implementation?
> [...]
>
> The sftp protocol does support getting arbitrary chunk of files.
> Otherwise things like sshfs wouldn't work. It's just the openssh
> sftp client that has no interface to it.
>
> Note that lftp and curl use the sftp subsystem, while rsync uses
> ssh and relies on there being a rsync command on the remote end.
And "scp" (as I had overlooked that the question was about scp
in the first place) also doesn't use the sftp subsystem but
relies on scp and runs a shell and the scp command on the remote
host as well.
And if we can run a shell on the remote host, we can do
ssh host tail -c +xxx file >> file
to resume a transfer.
--
Stephane
Alan Curry
Jul 4, 2012, 6:05:05 PM7/4/12
to
In article <barmar-649B9D....@news.eternal-september.org>,
replacement for rsh, there would still be tons of rsh-using scripts running
all over the Internet today.
--
Alan Curry
Barry Margolin <bar...@alum.mit.edu> wrote:
>
>As you say, there's a tug of war when designing things like this. You
>can elect to start from scratch, doing everything right. Or you can
>take a more evolutionary approach, just replacing bits and pieces of the
>design.
>
>Thus, the designers of rcp probably said to themselves, "it's good
>enough, except for its really bad security -- if we simply replace the
>underlying transfer protocol with SSH we'll have something really nice."
If ssh had decided to start from scratch instead of being a drop-in
>
>As you say, there's a tug of war when designing things like this. You
>can elect to start from scratch, doing everything right. Or you can
>take a more evolutionary approach, just replacing bits and pieces of the
>design.
>
>Thus, the designers of rcp probably said to themselves, "it's good
>enough, except for its really bad security -- if we simply replace the
>underlying transfer protocol with SSH we'll have something really nice."
replacement for rsh, there would still be tons of rsh-using scripts running
all over the Internet today.
--
Alan Curry
Randal L. Schwartz
Jul 5, 2012, 12:22:09 AM7/5/12
to
>>>>> "Kenny" == Kenny McCormack <gaz...@shell.xmission.com> writes:
Kenny> Yup. The design of (modern) Unix commands is often a tug-of-war
Kenny> between wanting to preserve tradition (and traditional behaviors)
Kenny> and wanting to "do things right". There is a strong tradition of
Kenny> "that's the way it has always been done, so that's the way we're
Kenny> going to continue to do it".
One could argue that given the constraints of the time, "traditional
behaviors" were identical to "do things right". So please, don't
retroactively condemn traditional unix for doing both at the same time.
It did. When you add new constrants, different things emerge, but don't
blame the old tools for that.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<mer...@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.posterous.com/ for Smalltalk discussion
Kenny> Yup. The design of (modern) Unix commands is often a tug-of-war
Kenny> between wanting to preserve tradition (and traditional behaviors)
Kenny> and wanting to "do things right". There is a strong tradition of
Kenny> "that's the way it has always been done, so that's the way we're
Kenny> going to continue to do it".
One could argue that given the constraints of the time, "traditional
behaviors" were identical to "do things right". So please, don't
retroactively condemn traditional unix for doing both at the same time.
It did. When you add new constrants, different things emerge, but don't
blame the old tools for that.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<mer...@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.posterous.com/ for Smalltalk discussion
0 new messages