(SCP) /dev/fd/63: not a regular file

[Kenny McCormack]

Tried to do something like:

scp <(some command) somemachine:file

and got error message:

/dev/fd/63: not a regular file

Now, I understand how/why that message is generated, but what I don't
understand is why it should be that way. Is there any particular reason
why scp insists on a "regular file" ? I can't see any reason.

I could see it warning you about it - but there should be an option (like
"-f" or some such) that says "Go ahead and do it anyway".

Note that, if nothing else, scp could read the entire file into an internal
buffer, and then use that internal buffer as it sees fit. In this day and
age of cheap memory, this would not be a burden.

--
"If our country is going broke, let it be from
feeding the poor and caring for the elderly.
And not from pampering the rich and fighting
wars for them."

--Living Blue in a Red State--

[Casper H.S. Dik]

gaz...@shell.xmission.com (Kenny McCormack) writes:

>Tried to do something like:

> scp <(some command) somemachine:file

>and got error message:

> /dev/fd/63: not a regular file

>Now, I understand how/why that message is generated, but what I don't
>understand is why it should be that way. Is there any particular reason
>why scp insists on a "regular file" ? I can't see any reason.

It wants to know the size before it starts sending it.

>I could see it warning you about it - but there should be an option (like
>"-f" or some such) that says "Go ahead and do it anyway".

Because it can't.

>Note that, if nothing else, scp could read the entire file into an internal
>buffer, and then use that internal buffer as it sees fit. In this day and
>age of cheap memory, this would not be a burden.

Really? You're thinking too small. This would work with a few MB,
but would it work for a 100GB file?


Use:

some command | ssh somemachine 'cat > file'


Casper

[Kenny McCormack]

In article <56bae9de$0$23862$e4fe...@news.xs4all.nl>,

Casper H.S. Dik <Caspe...@OrSPaMcle.COM> wrote:
>gaz...@shell.xmission.com (Kenny McCormack) writes:
>
>>Tried to do something like:
>
>> scp <(some command) somemachine:file
>
>>and got error message:
>
>> /dev/fd/63: not a regular file
>
>>Now, I understand how/why that message is generated, but what I don't
>>understand is why it should be that way. Is there any particular reason
>>why scp insists on a "regular file" ? I can't see any reason.
>
>It wants to know the size before it starts sending it.

OK. Sensible.

>>I could see it warning you about it - but there should be an option (like
>>"-f" or some such) that says "Go ahead and do it anyway".
>
>Because it can't.

But see below.

>>Note that, if nothing else, scp could read the entire file into an internal
>>buffer, and then use that internal buffer as it sees fit. In this day and
>>age of cheap memory, this would not be a burden.
>
>Really? You're thinking too small. This would work with a few MB,
>but would it work for a 100GB file?

Many commands are documented as "No limit other than available memory".
This would/could be similar. The point is that the use cases for this
would generally involve relatively small files.

Bottom line: Although perhaps tricky in theory, in practice, it would work
right most of the time. The point is that a "-f" option carries with it a
"Well, we warned you, but you said to do it anyway; you're on your own now"
warning label.

>Use:
>
> some command | ssh somemachine 'cat > file'

That's good. Thanks. I knew there was a way to do it involving just
piping it on stdin, but couldn't recall the details.

--
Mike Huckabee has yet to consciously uncouple from Josh Duggar.

[Casper H.S. Dik]

gaz...@shell.xmission.com (Kenny McCormack) writes:

>>It wants to know the size before it starts sending it.

>OK. Sensible.

My experience is that if a file grows while you are
copying it with scp, it will not copy more than the
size at the beginning of the copy, but that is a very
long time ago.

>Many commands are documented as "No limit other than available memory".
>This would/could be similar. The point is that the use cases for this
>would generally involve relatively small files.

Yeah, well, that. But if the protocol was different it could use
a pipe or something else without having to copy the whole file into
its memory.

Casper

[Clark Smith]

On Wed, 10 Feb 2016 10:40:18 +0000, Casper H.S. Dik wrote:

> gaz...@shell.xmission.com (Kenny McCormack) writes:
>
>>>It wants to know the size before it starts sending it.
>
>>OK. Sensible.
>
> My experience is that if a file grows while you are copying it with scp,
> it will not copy more than the size at the beginning of the copy, but
> that is a very long time ago.

One of the first pieces of information that an SCP party sends
(SCP being a protocol that runs on top of SSH) is the name of the file,
its permissions, and its size. Hence what you observed.

[Ian Zimmerman]

On 2016-02-10 07:42 +0000, Casper H.S. Dik wrote:

> some command | ssh somemachine 'cat > file'

Does this really work? I never can get remote commands with
redirections to work the way I expect, despite trying sh -c , eval and
similar.

My workaround which has served me well is

some_command | ssh some_machine dd of=some_file

--
Please *no* private copies of mailing list or newsgroup messages.
Rule 420: All persons more than eight miles high to leave the court.

[Casper H.S. Dik]

Ian Zimmerman <i...@buug.org> writes:

>On 2016-02-10 07:42 +0000, Casper H.S. Dik wrote:

>> some command | ssh somemachine 'cat > file'

>Does this really work? I never can get remote commands with
>redirections to work the way I expect, despite trying sh -c , eval and
>similar.

If you also invoke "sh -c" you likely run into the double or
triple evaluation of the quotes:

- evaluated once locally
- evaluated once more remotely
- evaluated a second time remotely when using sh -c.

>My workaround which has served me well is

>some_command | ssh some_machine dd of=some_file

There is often a problem with dd; it reads blocks and if
it reads an incomplete block it will zero fill before
writing it out (there are options in dd that make sure
it works reliable in that case)

Casper

[Kaz Kylheku]

On 2016-02-10, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> In article <56bae9de$0$23862$e4fe...@news.xs4all.nl>,
> Casper H.S. Dik <Caspe...@OrSPaMcle.COM> wrote:
>>gaz...@shell.xmission.com (Kenny McCormack) writes:
>>
>>>Tried to do something like:
>>
>>> scp <(some command) somemachine:file
>>
>>>and got error message:
>>
>>> /dev/fd/63: not a regular file
>>
>>>Now, I understand how/why that message is generated, but what I don't
>>>understand is why it should be that way. Is there any particular reason
>>>why scp insists on a "regular file" ? I can't see any reason.
>>
>>It wants to know the size before it starts sending it.
>
> OK. Sensible.

No it isn't. It could handle the case for objects that don't have a
size.

The underlying network stream certainly doesn't need to know any
up-front size.

We can do:

some command | ssh user@host 'cat > file'

works fine without knowing the size up front.

Also, "cp <(some command) file" works!

The scp protocol evidently has a format which requires the size. Doing
an strace on a scp job (copying a Makefile), I see this:

22508 open("Makefile", O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 3
22508 fstat64(3, {st_mode=S_IFREG|0644, st_size=496, ...}) = 0
22508 fcntl64(3, F_GETFL) = 0x8800 (flags
O_RDONLY|O_NONBLOCK|O_LARGEFIL
22508 fcntl64(3, F_SETFL, O_RDONLY|O_LARGEFILE) = 0
22508 write(6, "C0644 496 Makefile\n", 19 <unfinished ...>

The line

C0644 496 Makefile

is sent to the remote host, where 0644 is obviously the perms in octal,
and 496 is the size of the file.

> That's good. Thanks. I knew there was a way to do it involving just

> piping it on stdin, but couldn't recall the details..

It may be worth experimenting to see whether rsync over SSH has the same
problem as scp.

It has different problems:

$ rsync -auv --rsh=ssh <(echo blah) kaz@localhost:blah
sending incremental file list
63 -> pipe:[1859716]

sent 56 bytes received 15 bytes 47.33 bytes/sec
total size is 64 speedup is 0.90

Okay, looks promising.

$ cat ~blah
cat: /home/kaz/blah: No such file or directory

Where the hell is it?

$ ls -l1t ~ | head -3
total 14736
lrwxrwxrwx 1 kaz kaz 14 Feb 15 07:46 blah -> pipe:[1859716]
drwxr-xr-x 20 kaz kaz 4096 Feb 15 07:43 test

Oh, that's where! It created a symlink! But that's our fault, due to the
"rsync -a" habit. We asked rsync to preserve everything, and that means
symlinks are copied as symlinks.

Let's not do that:

$ rsync --rsh=ssh <(echo blah) kaz@localhost:blah
skipping non-regular file "63"

Whoops! Maybe we can RTFM around this in the rsync man page:

By default, symbolic links are not transferred at all. A message "skipping
non-regular" file is emitted for any symlinks that exist.

If --links is specified, then symlinks are recreated with the same target on
the destination. Note that --archive implies --links.

If --copy-links is specified, then symlinks are "collapsed" by copying their
referent, rather than the symlink.

Third attempt (using the -L shorthand for --copy-links):

$ rsync -L --rsh=ssh <(echo blah) kaz@localhost:blah
skipping non-regular file "63"

No luck! Sure the symlink is being dereferenced now ... but what it refers to
isn't a regular file.

[Kaz Kylheku]

On 2016-02-10, Clark Smith <noad...@nowhere.net> wrote:
> On Wed, 10 Feb 2016 10:40:18 +0000, Casper H.S. Dik wrote:
>
>> gaz...@shell.xmission.com (Kenny McCormack) writes:
>>
>>>>It wants to know the size before it starts sending it.
>>
>>>OK. Sensible.
>>
>> My experience is that if a file grows while you are copying it with scp,
>> it will not copy more than the size at the beginning of the copy, but
>> that is a very long time ago.
>
> One of the first pieces of information that an SCP party sends
> (SCP being a protocol that runs on top of SSH) is the name of the file,
> its permissions, and its size. Hence what you observed.

Even so, this problem could be fixed without substantially changing
the structure of the protocol. When sending a file, the stream looks like:

C<perms> <size> <filename>
DATA....

Now suppose a new form is added,

A<size> <filename>
DATA...

which appends data to a file.

A pipe could be handled by reading small-buffer-sized chunks and
transmitting a C unit, followed by, if necessary, one or more A
units.

[Stephane Chazelas]

2016-02-09 23:45:26 +0000, Kenny McCormack:

> Tried to do something like:
>
> scp <(some command) somemachine:file
>
> and got error message:
>
> /dev/fd/63: not a regular file

[...]

Note that if using zsh, you can always use the:

scp =(some command) host:file

form of process substitution which uses a temporary file instead
of a pipe.

Note that scp attempts to copy the permissions, which in this
case will be very restricted.

--
Stephane

[Casper H.S. Dik]

Kaz Kylheku <330-70...@kylheku.com> writes:

>On 2016-02-10, Kenny McCormack <gaz...@shell.xmission.com> wrote:
>> In article <56bae9de$0$23862$e4fe...@news.xs4all.nl>,
>> Casper H.S. Dik <Caspe...@OrSPaMcle.COM> wrote:
>>>gaz...@shell.xmission.com (Kenny McCormack) writes:
>>>
>>>>Tried to do something like:
>>>
>>>> scp <(some command) somemachine:file
>>>
>>>>and got error message:
>>>
>>>> /dev/fd/63: not a regular file
>>>
>>>>Now, I understand how/why that message is generated, but what I don't
>>>>understand is why it should be that way. Is there any particular reason
>>>>why scp insists on a "regular file" ? I can't see any reason.
>>>
>>>It wants to know the size before it starts sending it.
>>
>> OK. Sensible.

>No it isn't. It could handle the case for objects that don't have a
>size.

It is defined in such a way in the protocol; not perhaps that isn't
the best way to define a protocol but that is what it is.

While you can change the protocol, doing to requires to add negatiotion
of the new variant of the protocol and an implementation and
support on both sides.

Casper

[Clark Smith]

Not likely to happen. The SCP protocol is not documented - it is,
essentially, what the BSD rcp code does, but on top of SSH.