Can one connect a named pipe to a remote filesystem using ssh?

[carolus]

I have a program that writes several files to user-specified names and
paths. I would like to write these to a remote filesystem. Is there some
way to combine named pipes with ssh to do this?

[Grant Taylor]

I would try creating a fifo, and writing to it. I'd have something
running in the background like the following:

cat /path/to/fifo | ssh us...@remote.host cat \> /path/to/remote/file

I'd also check to see if sshfs might work for what was needed or not. I
think it would be (slightly) less hacky. (Not that hacky is bad, just
more difficult to support.)



--
Grant. . . .
unix || die

[carolus]

Your suggestion seems to work for the test case:

mkfifo mypipe
cat mypipe |ssh $D8XB cat \> junk/pipe.out &
echo hello > mypipe

I don't quite understand it and I'll have to play with it a bit more,
but it looks like I can use the named pipe in lieu of a filename for
program output. sshfs should also serve the purpose, but this way I
don't have to set up a shared folder, and unlike sshfs this will work
from Cygwin.

Thanks.

[carolus]

On 12/4/2017 1:04 AM, carolus wrote:
> On 12/3/2017 5:51 PM, Grant Taylor wrote:
>> On 12/03/2017 08:34 AM, carolus wrote:
>>> I have a program that writes several files to user-specified names
>>> and paths. I would like to write these to a remote filesystem. Is
>>> there some way to combine named pipes with ssh to do this?
>>
>> I would try creating a fifo, and writing to it.  I'd have something
>> running in the background like the following:
>>
>> cat /path/to/fifo | ssh us...@remote.host cat \> /path/to/remote/file
>>

> Your suggestion seems to work for the test case:
>
> mkfifo mypipe
> cat mypipe |ssh $D8XB cat \> junk/pipe.out &
> echo hello > mypipe
>

Unfortunately this fails for a test case that is closer to my intended
use, redirecting program output to a remote filesystem:

$mkfifo mypipe
$cat mypipe | ssh $D8XB cat \> junk/hello.o &
$gcc -c -o mypipe hello.c
Assembler messages:
Fatal error: can't create mypipe: Device or resource busy

Yet "echo hello > mypipe" writes "hello" to remote ~/junk/hello.o

[Ian Zimmerman]

For the sake of simplicity and maintainability, I would write to local
temporary files first, then have a phase with rsync (or similar) to get
them to the remote location.

This under the assupmtion that the files aren't so huge that they'll
fill your local filesystem.

--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet, fetch the TXT record for the domain.

[carolus]

On 12/4/2017 1:46 PM, Ian Zimmerman wrote:
> On 2017-12-03 09:34, carolus wrote:
>
>> I have a program that writes several files to user-specified names and
>> paths. I would like to write these to a remote filesystem. Is there
>> some way to combine named pipes with ssh to do this?
>
> For the sake of simplicity and maintainability, I would write to local
> temporary files first, then have a phase with rsync (or similar) to get
> them to the remote location.
>
> This under the assupmtion that the files aren't so huge that they'll
> fill your local filesystem.

Big files are in fact the problem: I want to write someplace where I
have a big drive. These are occasional tasks, so simplicity is a
priority and maintenance is of no concern.

After the experiments posted above, I suspect that named pipes are not
the answer. It looks like writing to a named pipe may depend on program
internals beyond my control, presumably on how the output files are
opened. That leaves sshfs or some other means of file sharing.

At least I learned how to connect a named pipe to a remote filesystem, a
potentially useful trick, even if I can't always write to the named pipe.

[Kaz Kylheku]

On 2017-12-04, carolus <sendme...@bellsouth.net> wrote:
> Big files are in fact the problem: I want to write someplace where I
> have a big drive. These are occasional tasks, so simplicity is a
> priority and maintenance is of no concern.

Is this is just logging? Logging to remote servers is a solved problem;
e.g. rsyslogd.

[Grant Taylor]

On 12/04/2017 12:04 AM, carolus wrote:
> Your suggestion seems to work for the test case:

Good.

> mkfifo mypipe
> cat mypipe |ssh $D8XB cat \> junk/pipe.out &
> echo hello > mypipe
>
> I don't quite understand it and I'll have to play with it a bit more,

The named pipe (a.k.a. FIFO) is a file that you can write to. Think of
it as a FIFO buffer.

So, you are creating what appears to be a file that you can write to and
then accessing it (reading from it) via cat.

cat's STDOUT is passed to ssh's STDIN which comes out the other end of
the ssh connection.

You are then redirecting STDOUT on the far end into a file.

Thus you write to what appears to be a file on the local end, and the
contents that you write are pushed to the remote end where they are
dumped to a file.

> but it looks like I can use the named pipe in lieu of a filename for
> program output.

Good.

> sshfs should also serve the purpose, but this way I
> don't have to set up a shared folder, and unlike sshfs this will work
> from Cygwin.

I've not used sshfs in quite a while, but I didn't think there was any
aspect of a shared folder. Maybe I'm conflating a couple of issues.

> Thanks.

You're welcome.

[Grant Taylor]

On 12/04/2017 11:40 AM, carolus wrote:
> Unfortunately this fails for a test case that is closer to my intended
> use, redirecting program output to a remote filesystem:
>
> $mkfifo mypipe
> $cat mypipe | ssh $D8XB cat \> junk/hello.o &
> $gcc -c -o mypipe hello.c
>   Assembler messages:
>   Fatal error: can't create mypipe: Device or resource busy

I'm guessing that gcc is expecting the file that it's going to create to
not exist.

I don't know if there are command line options to change that.

I also don't know if gcc will run into a problem in that it may not be
able to read from the FIFO like it expects to.

This may be the first of what may prove to be multiple failures.

> Yet "echo hello > mypipe" writes "hello" to remote ~/junk/hello.o

It has to do with the complexity of and assumptions about what you're doing.

echo / STDOUT / STDIN is rather dumb and does not make any assumptions.

gcc is rather smart and will make a LOT of assumptions. Some of which
hinder what you want to do.

[Kaz Kylheku]

On 2017-12-05, Grant Taylor <gta...@tnetconsulting.net> wrote:
> On 12/04/2017 12:04 AM, carolus wrote:
>> Your suggestion seems to work for the test case:
>
> Good.
>
>> mkfifo mypipe
>> cat mypipe |ssh $D8XB cat \> junk/pipe.out &
>> echo hello > mypipe
>>
>> I don't quite understand it and I'll have to play with it a bit more,
>
> The named pipe (a.k.a. FIFO) is a file that you can write to. Think of
> it as a FIFO buffer.

A named FIFO is an object in the filesystem which serves as an IPC
rendezvous point.

This rendezvous point generates zero or more pipes (unidirectional IPC
objects) in response to being open by pairs of readers and writers.

If a FIFO is open for reading, it will block until it is also open for
writing and vice versa. When both opens take place, then the two
descriptors which are returned to these two calls are the endpoints of a
newly created pipe.

The rendezvous can repeat; a new reader-writer pair can come along and
open the pipe even though the previously created pipe still exists.

If fifty processes open for reading at about the same time as fifty
processes for writing, they will be matched up somehow and fifty pipes
will result.

In other words, the situation can potentially turn into a mess.

If you want to use a FIFO for multiple transfers, you typically want
one direction to be treated with a service loop, and the other direction
then freely used by clients.

For example, to use the FIFO mechanism effectively for sending files to
a remote service, you basically need a process which sits on one end of
the pipe and continuously opens it for reading. Each time it opens it,
it then runs a function to service the pipe descriptor (perhaps in its
own process). Clients open the FIFO for writing. Each such open produces
a new session for a new file; the client dumps the data in there and
closes.

The other end somehow needs to know what is being written; for instance,
you likely need an *in-band protocol* for indicating the file name.

It's very similar to doing an accept() on a socket, and then dispatching
a client-servicing fucntion over the result file descriptor, while going
back to the accept() to get more clients.

A possible protocol is to use a master service pipe where the client
doesn't write the data itself but only the *name* of a file (or FIFO!)
to send.

I.e. the client writes a path name into the service FIFO. Server opens
the path, reads the data, sends it off, and deletes the path.
The path can be a regular file or itself a FIFO: client's choice.

[carolus]

On 12/4/2017 7:01 PM, Grant Taylor wrote:
> On 12/04/2017 11:40 AM, carolus wrote:
>> Unfortunately this fails for a test case that is closer to my intended
>> use, redirecting program output to a remote filesystem:
>>
>> $mkfifo mypipe
>> $cat mypipe | ssh $D8XB cat \> junk/hello.o &
>> $gcc -c -o mypipe hello.c
>>    Assembler messages:
>>    Fatal error: can't create mypipe: Device or resource busy
>
> I'm guessing that gcc is expecting the file that it's going to create to
> not exist.
>

My guess from the error message was that the "type" option in the
fopen(filename,type) that was used to open the output file was not
permissive enough. That is internal to the program and inaccessible to
me, so that at least for this program (gcc) I cannot simply substitute
the pipe name for a file name. So much for the generality of the method.

>
> I also don't know if gcc will run into a problem in that it may not be
> able to read from the FIFO like it expects to.

Why would gcc read from the pipe? My understanding of the above code
was that gcc would write to the pipe, and "cat" on the remote host would
read from the pipe.

[carolus]

On 12/4/2017 6:58 PM, Grant Taylor wrote:
>> sshfs should also serve the purpose, but this way I don't have to set
>> up a shared folder, and unlike sshfs this will work from Cygwin.
>
> I've not used sshfs in quite a while, but I didn't think there was any
> aspect of a shared folder.

Running this from Debian mounts an existing folder on a remote Windows
machine that is running sshd under Cygwin:

# mount remote
sshfs cdr@asus10:/home/cdr /mnt/cdr_remote
fusermount: user has no write access to mountpoint /mnt/cdr_remote
sudo chown cdr /mnt/cdr_remote #enables write access
# connect to remote
sshfs cdr@asus10:/home/cdr /mnt/cdr_remote
ls /mnt/cdr_remote #checks OK

(Other than verifying that "ls" and "touch" work in the remote
directory, I have not yet actually used this.) Pretty easy, but it would
be even simpler if a named pipe was interchangeable with an output filename.

[Casper H.S. Dik]

carolus <sendme...@bellsouth.net> writes:

>After the experiments posted above, I suspect that named pipes are not
>the answer. It looks like writing to a named pipe may depend on program
>internals beyond my control, presumably on how the output files are
>opened. That leaves sshfs or some other means of file sharing.

Named pipes are local to the system; they can't be used to write them
on one system and read them from another. I'd use something like "tar"
and pack the files on one end, and send the output through ssh:

tar cf .. | ssh host 'cd store; tar xf -'

Casper

[Geoff Clare]

Kaz Kylheku wrote:

> A named FIFO is an object in the filesystem which serves as an IPC
> rendezvous point.
>
> This rendezvous point generates zero or more pipes (unidirectional IPC
> objects) in response to being open by pairs of readers and writers.
>
> If a FIFO is open for reading, it will block until it is also open for
> writing and vice versa. When both opens take place, then the two
> descriptors which are returned to these two calls are the endpoints of a
> newly created pipe.
>
> The rendezvous can repeat; a new reader-writer pair can come along and
> open the pipe even though the previously created pipe still exists.

Do you actually know of a system that works like that?

I had previously thought that the behaviour implemented by everyone
was that multiple opens of a FIFO all open the same pipe.

--
Geoff Clare <net...@gclare.org.uk>

[carolus]

On 12/4/2017 10:57 PM, carolus wrote:
>  My guess from the error message was that the  "type" option in the
> fopen(filename,type) that was used to open the output file was not
> permissive enough.

Bad guess. I tried opening a file of type "w+" with a named pipe as the
filename, and it segfaulted.

[Thomas 'PointedEars' Lahn]

Yes.

--
PointedEars

Twitter: @PointedEars2
Please do not cc me. /Bitte keine Kopien per E-Mail.

[Thomas 'PointedEars' Lahn]

Grant Taylor wrote:

> I would try creating a fifo, and writing to it. I'd have something
> running in the background like the following:
>
> cat /path/to/fifo | ssh us...@remote.host cat \> /path/to/remote/file

UUOC.

Copy local data to remote:

ssh us...@remote.host.example < /local/path 'cat > /remote/path'

Copy remote data to local:

ssh us...@remote.host.example 'cat /remote/path' > /local/path

[This is basically what SCP does, but by contrast to SCP it works with
all files that are not directories, including FIFOs. Compression (e.g.
gzip) can be used on either end or both ends to improve performance.
“>>” and other or no redirection can be used as well.]

Neither command runs “in the background” without “&” appended.

[Thomas 'PointedEars' Lahn]

carolus wrote:

> Unfortunately this fails for a test case that is closer to my intended
> use, redirecting program output to a remote filesystem:
>
> $mkfifo mypipe
> $cat mypipe | ssh $D8XB cat \> junk/hello.o &
> $gcc -c -o mypipe hello.c
> Assembler messages:
> Fatal error: can't create mypipe: Device or resource busy

gcc(1) does not work on FIFOs, regardless where they are.
If you think about what gcc(1) is, you should realize why.

Get a real name and address.

[Thomas 'PointedEars' Lahn]

carolus wrote:

> Big files are in fact the problem: I want to write someplace where I
> have a big drive. These are occasional tasks, so simplicity is a
> priority and maintenance is of no concern.
>
> After the experiments posted above, I suspect that named pipes are not
> the answer.

Probably correct; probably you do not need named pipes for this at all.

> It looks like writing to a named pipe may depend on program
> internals beyond my control, presumably on how the output files are
> opened. That leaves sshfs or some other means of file sharing.

ISTM that you have an X–Y problem.

<https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem>

> At least I learned how to connect a named pipe to a remote filesystem, a
> potentially useful trick, even if I can't always write to the named pipe.

But you can always write an output *stream* to a (named) pipe.

[Helmut Waitzmann]

Kaz Kylheku <217-67...@kylheku.com>:

> On 2017-12-05, Grant Taylor <gta...@tnetconsulting.net> wrote:
>> On 12/04/2017 12:04 AM, carolus wrote:
>>> Your suggestion seems to work for the test case:
>>
>> Good.
>>
>>> mkfifo mypipe
>>> cat mypipe |ssh $D8XB cat \> junk/pipe.out &
>>> echo hello > mypipe
>>>
>>> I don't quite understand it and I'll have to play with it a bit more,
>>
>> The named pipe (a.k.a. FIFO) is a file that you can write to. Think of
>> it as a FIFO buffer.
>
> A named FIFO is an object in the filesystem which serves as an IPC
> rendezvous point.
>
> This rendezvous point generates zero or more pipes (unidirectional IPC
> objects) in response to being open by pairs of readers and writers.
>
> If a FIFO is open for reading, it will block until it is also open for
> writing and vice versa. When both opens take place, then the two
> descriptors which are returned to these two calls are the endpoints of a
> newly created pipe.
>
> The rendezvous can repeat; a new reader-writer pair can come along and
> open the pipe even though the previously created pipe still exists.

Interesting.

On my Debian system, a named pipe, even if opened by more than one
reader‐writer pair, seems to give one FIFO only. In the
command line

mkfifo -- fifo &&
(
writer()
{
while sleep "${1:?}"
do
printf '%s\n' "${2?}"
done
} &&
reader()
(
while sleep -- "${1:?}" && IFS= read -r line
do
printf '%s\n' "$line"
done |
sed -e 's/^/'"${2?}"': /' --
) &&
{ writer 4 1 >> fifo & } &&
{ reader 6 1 < fifo & } &&
sleep -- 1 &&
{ writer 6 2 >> fifo & } &&
{ reader 4 2 < fifo & } &&
wait
)

“reader 4 2” happens to read data written by “writer 4 1” as well
as “writer 6 2”. The same holds for “reader 6 1”.

[Thomas 'PointedEars' Lahn]

Thomas 'PointedEars' Lahn wrote:

> Grant Taylor wrote:
>> I would try creating a fifo, and writing to it. I'd have something
>> running in the background like the following:
>>
>> cat /path/to/fifo | ssh us...@remote.host cat \> /path/to/remote/file
>
> UUOC.

(useless use of cat)

> Copy local data to remote:
>
> ssh us...@remote.host.example < /local/path 'cat > /remote/path'

> […]

> [This is basically what SCP does, but by contrast to SCP it works with
> all files that are not directories, including FIFOs. Compression
> (e.g. gzip) can be used on either end or both ends to improve
> performance. “>>” and other or no redirection can be used as well.]
>
> Neither command runs “in the background” without “&” appended.

A common use of this pattern is

ssh us...@remote.host.example < ~/.ssh/id….pub 'cat >> .ssh/authorized_keys'

to append the local user’s public key contained in the file ~/.ssh/id….pub
to the remote user “user”’s list of keys of users authorized to connect as
the remote user via SSH with the “public-key” authentication method (instead
of the less secure “password” method).

[Grant Taylor]

On 12/06/2017 08:40 PM, Thomas 'PointedEars' Lahn wrote:
> UUOC.

I disagree.

I do agree that it may be an inefficient and unneeded use of cat, but it
is not useless.

I have spent the better part of the last 10 years teaching a lot of
people how to do things in unix. I've found that redirection stumps a
LOT of people. At least people early in their unix learning.

As such, I have taken to using the not-quite-UUOC approach and using cat
anyway.

Many people start by looking at a file via cat, and then iterating and
organically evolving the command. - I find that switching from "cat …
| $filter … " to "filter … < file | …" takes thought. Thought that I
would rather new students to put towards building a functional command
string that accomplishes their goal instead of worrying about the
optimal way to achieve it. - Let's get there and then optimize when
needed. (How many quick one off commands will ever be run again?)

In the purest sens, you are correct. In the practical sense, it doesn't
matter.

Much like Vim golf, you are only really playing against yourself. - Do
you really want to play in the middle of an emergency? Or do you just
want to get the job done?

So … ride your UUOC high horse somewhere else please.

> Copy local data to remote:
>
> ssh us...@remote.host.example < /local/path 'cat > /remote/path'
>
> Copy remote data to local:
>
> ssh us...@remote.host.example 'cat /remote/path' > /local/path

I agree that these are perfectly viable alternatives.

> Neither command runs “in the background” without “&” appended.



--

[Thomas 'PointedEars' Lahn]

Grant Taylor wrote:

> On 12/06/2017 08:40 PM, Thomas 'PointedEars' Lahn wrote:
>> UUOC.
>
> I disagree.
>
> I do agree that it may be an inefficient and unneeded use of cat, but it
> is not useless.

<http://porkmail.org/era/unix/award.html>
<http://linuxwiki.de/UselessUseOfCat>
<https://sanctum.geek.nz/arabesque/useless-use-of-cat/>

etc.

> [pointless argument]

> So … ride your UUOC high horse somewhere else please.

Get a life. And learn to quote.

[Ian Zimmerman]

On 2017-12-16 15:33, Grant Taylor wrote:

> > Copy local data to remote:
> >
> > ssh us...@remote.host.example < /local/path 'cat > /remote/path'
> >
> > Copy remote data to local:
> >
> > ssh us...@remote.host.example 'cat /remote/path' > /local/path
>
> I agree that these are perfectly viable alternatives.

Also without any confusing ssh command quoting:

</local/path ssh us...@remote.host.example dd of=/remote/path

I think this one would win the golf round, if it weren't for scp :P

[Helmut Waitzmann]

Ian Zimmerman <i...@no-use.mooo.com>:

> On 2017-12-16 15:33, Grant Taylor wrote:
>
>> > Copy local data to remote:
>> >
>> > ssh us...@remote.host.example < /local/path 'cat > /remote/path'

[…]

> Also without any confusing ssh command quoting:
>
> </local/path ssh us...@remote.host.example dd of=/remote/path

I don't think, that the “dd” version of the command will differ
w.r.t. quoting from the “cat” version.

An equal variant of the “cat” version:

</local/path ssh us...@remote.host.example cat '>' /remote/path

AFAIK “ssh” just glues the given words together with spaces in
between to compose the command line for the remote shell. It
cannot and won't do any shell quoting, which would be required, if
one would like to compose a command line resembling an invocation
of a shell simple command out of given words, because it doesn't
make any assumptions about the remote shell (and its quoting
rules) other than that, that the option “-c” will tell it to
expect a command line as the first non‐option argument.

For example,

</local/path ssh us...@remote.host.example cat '>' \
'/remote/some funny name'

as well as

</local/path ssh us...@remote.host.example dd \
of='/remote/some funny name'

will fail to write to the file “/remote/some funny name” at the
remote site.

But maybe I understand you wrong. Could you explain more
verbosely?

[Thomas 'PointedEars' Lahn]

Helmut Waitzmann wrote:

> An equal variant of the “cat” version:
>
> </local/path ssh us...@remote.host.example cat '>' /remote/path

Considering the direction in which the “<” character is pointing, I find
that order of tokens harder to understand than what I proposed.

> […] “ssh” just glues the given words together with spaces in

> between to compose the command line for the remote shell.

Yes, appearantly that is so:

| $ ssh "$REMOTE_HOST" printf '%s\n' 1 2 3
| 1n2n3n

Fascinating.

> But maybe I understand you wrong. Could you explain more
> verbosely?

(I would use “I _misunderstand you_.)

[Helmut Waitzmann]

Thomas 'PointedEars' Lahn <Point...@web.de>:

> Helmut Waitzmann wrote:
>
>> An equal variant of the “cat” version:
>>
>> </local/path ssh us...@remote.host.example cat '>' /remote/path

[…]

>> […] “ssh” just glues the given words together with spaces in
>> between to compose the command line for the remote shell.
>
> Yes, appearantly that is so:
>
> | $ ssh "$REMOTE_HOST" printf '%s\n' 1 2 3
> | 1n2n3n
>
> Fascinating.

More verbose explanation:

“ssh” is given the command words “printf”, “%s\n”, “1”, “2”, and
“3”. It glues them together with spaces in between:

“printf %s\n 1 2”,

and manages to invoke the shell on the remote host, that is
configured for the remote user (for example, a POSIX‐compliant
shell, or a “tcsh”, or some different one), with the command line
“printf %s\n 1 2”.

If the remote shell is a POSIX‐compliant one, “\n”, when parsed,
yields the same like “n”: “n”. So, eventually, the utility
“printf” is invoked with the parameters “%sn”, “1”, and “2”.

“ssh user@host ...” faces the same problems like
“sh -c -- ...” or “eval ...”: All of them expect a command line
rather than an “execve()” argument vector.

So, if one likes to invoke a utility with its parameters, one has
to reconstruct a shell command line out of the utility name and
the parameters. If the command line is to be given to a remote
“bash”, that can be done by using the “bash”‐built‐in „printf”
with the format specification “%q ”:

quote_words_for_bash()
{
bash -c -- '"$@"' bash printf '%q ' "$@"
}
bash_command_line="$(
quote_words_for_bash printf '%s\n' 1 2
)" &&
ssh "$REMOTE_HOST" "$bash_command_line"

If the local shell is a “bash”, too, the function
“quote_words_for_bash” can be simplified to

quote_words_for_bash()
{
printf '%q ' "$@"
}


To come back to the problem, that

</local/path ssh us...@remote.host.example cat '>' \
'/remote/some funny name'

as well as

</local/path ssh us...@remote.host.example dd \
of='/remote/some funny name'

won't work:

If the remote shell is a “bash”, one can do:

</local/path ssh us...@remote.host.example \
"$(
quote_words_for_bash dd of='/remote/some funny name'
)"

With “cat” and output redirection, it's slightly more complicated,
because the redirection operator “>” must not be quoted in the
command line for the remote shell:

</local/path ssh us...@remote.host.example \
"$( quote_words_for_bash cat )" '>' \
"$( quote_words_for_bash '/remote/some funny name' )"

In this two examples, the words “dd” and “cat” don't need any
quoting, because there are no funny characters in them. So, the
commands might be simplified to

</local/path ssh us...@remote.host.example dd \
"$(
quote_words_for_bash of='/remote/some funny name'
)"

and

</local/path ssh us...@remote.host.example cat '>' \
"$( quote_words_for_bash '/remote/some funny name' )"

respectively.

Finally, if the remote shell is a POSIX‐compliant shell (only)
rather than a “bash”, or, if at the command line constructing
site, there is no “bash” available, a function for a
POSIX‐compliant shell (a “bash” will do as well) can be defined,
which quotes its parameters in a way suitable for a
POSIX‐compliant shell (as well as a “bash”). See the function
“my_quote_words_for_shells” at the end of this posting.

>> But maybe I understand you wrong. Could you explain more
>> verbosely?
>
> (I would use “I _misunderstand you_.)

(My English may not be good enough. I apologize.)

With quoting for the POSIX‐compliant shell's command line, there
are some rules:

Every sequence of characters, except apostrophes and maybe NULL
characters, can be quoted by surrounding that sequence with a pair
of apostrophes.

A sequence of apostrophes can be quoted by either surrounding it
with a pair of quotation marks or by preceding each apostrophe
with a backslash.

Any string, that is to be quoted, can be quoted by splitting it
into any two parts, then quoting each of the parts and finally
gluing the quoted parts together without anything in between.
For example, the string

“This string's purpose is to say "Hello, world!"”

could be quoted by splitting it before and after the apostrophe,
then quoting each of the three parts, and finally gluing them
together:

Splitting:

“This string”, “'”, “s purpose is to say "Hello, world!"”

Quoting each of the three parts according to the rules above:

“'This string'”, “"'"”, “'s purpose is to say "Hello, world!"'”
A A QaQ A q qA

Legend:
A marks a quoting apostrophe
Q marks a quoting quotation mark
a marks a quoted apostrophe
q marks a quoted quotation mark

Gluing them together:

“'This string'"'"'s purpose is to say "Hello, world!"'”
A AQaQA q qA

For example a command line containing a “printf” invocation to
output the sentence

“This message's purpose is to say "Hello, world!"”,

could be written as

“printf '%s\n' \
'This message'"'"'s purpose is to say "Hello, world!"'
”

To invoke that command in a remote POSIX‐compliant shell, one
could write:

“command_line="$( my_quote_words_for_shells \
printf '%s\n' \
'This message'"'"'s purpose is to say "Hello, world!"' \
)" &&
ssh us...@host.remote.example "$command_line"
”

This is the function:

my_quote_words_for_shells()
(
# Outputs a (part of) a command line resembling a 'simple command'
# to be used by a POSIX-compliant shell.
#
# Usage:
#
# my_quote_words_for_shells words to be quoted...
#
# exit code: 0, if the command line has been successfully constructed,
# !=0, if the function failed to construct the command line.
#
# Examples:
#
# Put a word list resembling a command invocation into a command line:
#
# if command_line="$(my_quote_words_for_shells \
# program with parameters)"
# then
# # "eval" it:
# eval "$command_line"
# # or invoke a new shell:
# sh -c "$command_line" sh
# # or use it remotely:
# ssh us...@remote.host.example "$command_line"
# else
# # failed to compute the command line.
# fi
#
#
# Store the shell positional parameters in a variable and restore
# them later:
#
# if args="$(my_quote_words_for_shells "$@")"
# then
# # the positional parameters may be changed for any purpose
# # ...
#
# # restore the positional parameters:
#
# set '' "$args" && shift
# else
# # failed to save the positional parameters.
# fi


# "wordsep" contains a separator used for the list of the quoted
# words. The first need not to be preceded by a separater:

wordsep=
for word
do
# "$wordsep" separates each word (except the first one)
# from its precedessor:
printf '%s' "$wordsep"
if test -z "$word"
then
# The word is empty. Then the result is "''":
printf '%s' "''"
else
# The word is not empty.
while
{
prefix="${word%%\'*}"
word="${word#"$prefix"}"
# "$prefix" is the longest beginning part
# of "$word", that does not contain any
# apostrophes; it is removed from "$word".
# Conclusion: "$word" is either empty or
# begins with an apostrophe.
if test -n "$prefix"
then
# "$prefix" consists of one or more
# characters. Put them in between
# of two apostrophes:
printf \''%s'\' "${prefix}"
fi
test -n "$word" &&
{
# "$word" is not empty.
# Conclusion: "$word" begins with
# one or more apostrophes.
apostr="${word%%[!\']*}"
# "$apostr" ist the longest
# beginning part of "$word", that
# contains apostrophes only.
# Put it in between of two double
# quotes:
printf '"%s"' "${apostr}"
# Remove the beginning apostrophes
# from "$word":
word="${word#"$apostr"}"
# If nothing is left, we are done:
${word:+:} false
}
}
do
:
done
fi
# All following words (except the first one) have to be
# separated from their precedessor with a blank:
wordsep=' '
done
printf '\n'
)

[Ian Zimmerman]

On 2017-12-20 18:48, Helmut Waitzmann wrote:

[quoted material reordered]

> > Also without any confusing ssh command quoting:
> >
> > </local/path ssh us...@remote.host.example dd of=/remote/path

> maybe I understand you wrong. Could you explain more verbosely?

> An equal variant of the “cat” version:
>
> </local/path ssh us...@remote.host.example cat '>' /remote/path
>
> AFAIK “ssh” just glues the given words together with spaces in between
> to compose the command line for the remote shell. It cannot and won't
> do any shell quoting, which would be required, if one would like to
> compose a command line resembling an invocation of a shell simple
> command out of given words, because it doesn't make any assumptions
> about the remote shell (and its quoting rules) other than that, that
> the option “-c” will tell it to expect a command line as the first
> non‐option argument.

This is just what I mean - having to think about this (on top of the
normal shell quoting problems) is a distraction. The dd version frees
me from that.

[Helmut Waitzmann]

Ian Zimmerman <i...@no-use.mooo.com>:

> On 2017-12-20 18:48, Helmut Waitzmann wrote:
>
> [quoted material reordered]
>
>> > Also without any confusing ssh command quoting:
>> >
>> > </local/path ssh us...@remote.host.example dd of=/remote/path
>
>> maybe I understand you wrong. Could you explain more verbosely?
>
>> An equal variant of the “cat” version:
>>
>> </local/path ssh us...@remote.host.example cat '>' /remote/path
>>
>> AFAIK “ssh” just glues the given words together with spaces in between
>> to compose the command line for the remote shell. It cannot and won't
>> do any shell quoting, which would be required, if one would like to
>> compose a command line resembling an invocation of a shell simple
>> command out of given words, because it doesn't make any assumptions
>> about the remote shell (and its quoting rules) other than that, that
>> the option “-c” will tell it to expect a command line as the first
>> non‐option argument.
>
> This is just what I mean - having to think about this (on top of the
> normal shell quoting problems) is a distraction. The dd version frees
> me from that.

I don't think so. The “dd” version won't you free from that.
Using “cat” and output redirection with funny path names will
fail:

</local/path ssh us...@remote.host.example cat '>' \
'/remote/funny path name'

This command tells the remote host to “cat” the files “path”
and “name” and write the result into the file “/remote/funny”.
The file “/local/path” is not read.

Using “dd” (without output redirection) with funny path names will
fail as well:

</local/path ssh us...@remote.host.example dd \
of='/remote/funny path name'

This command line tells the remote host to read the data supplied
by the local host from the file “/local/path” and write it to the
file “/remote/funny”. The additional arguments “path” and “name”
given to the remote “dd” command will probably make “dd” fail.

Both commands fail, because of the missing quoting. To get it
right, one would have to do (for example)

</local/path ssh us...@remote.host.example cat '>' \
\''/remote/funny path name'\'

and

</local/path ssh us...@remote.host.example dd \
of=\''/remote/funny path name'\'

respectively.

[Thomas 'PointedEars' Lahn]

Helmut Waitzmann wrote:

> Both commands fail, because of the missing quoting. To get it
> right, one would have to do (for example)
>
> </local/path ssh us...@remote.host.example cat '>' \
> \''/remote/funny path name'\'
>
> and
>
> </local/path ssh us...@remote.host.example dd \
> of=\''/remote/funny path name'\'
>
> respectively.

</local/path ssh us...@remote.host.example cat '>' \
\'/remote/funny path name\'

and

</local/path ssh us...@remote.host.example dd \

of=\'/remote/funny path name\'

would suffice, unless the remote file patch contains characters that are
special to the local shell. That is, single-quoting the remote file path
does not add another value. One should therefore, for consistency, precede
with backslash all non-whitespace characters that should be passed verbatim
to the remote shell: “\>”.

[Helmut Waitzmann]

Thomas 'PointedEars' Lahn <Point...@web.de>:

> Helmut Waitzmann wrote:
>
>> Both commands fail, because of the missing quoting. To get it
>> right, one would have to do (for example)
>>
>> </local/path ssh us...@remote.host.example cat '>' \
>> \''/remote/funny path name'\'
>>
>> and
>>
>> </local/path ssh us...@remote.host.example dd \
>> of=\''/remote/funny path name'\'
>>
>> respectively.
>
> </local/path ssh us...@remote.host.example cat '>' \
> \'/remote/funny path name\'
>
> and
>
> </local/path ssh us...@remote.host.example dd \
> of=\'/remote/funny path name\'
>
> would suffice,

Yes.

> unless the remote file patch contains characters that are
> special to the local shell. That is, single-quoting the remote
> file path does not add another value. One should therefore, for
> consistency, precede with backslash all non-whitespace
> characters that should be passed verbatim to the remote shell:
> “\>”.

Could you explain more verbose, please?

Do I understand you correctly in that you are saying, if, for
example, the remote path name contained whitespace other than
single spaces, like

“/remote/containing two consecutive spaces and a
newline”,

that the “ssh” command line using “dd” should be

“</local/path ssh us...@remote.host.example dd \
of=\'/remote/containing two consecutive blanks and a
newline\'”?