named pipe across machines
Russ P.
across Linux machines on the same LAN. Is that possible? If so, what
is the simplest way to achieve it? Thanks.
Russ P.
place for the following question, but I'll ask it anyway because I'm
not sure where else to ask it (suggestions welcomed).
I have set up a server with a TCP/IP socket, and it works fine. but I
am now realizing that I may need to get the input from one process and
send the output to another. I figure I'll need two sockets, with each
on a separate thread. Has anyone else solved this problem before? Am I
on the right track?
Russ P.
On Mar 17, 12:00 am, Michael Vilain <vil...@NOspamcop.net> wrote:
> In article
> <1e48a2cb-d927-4a9d-9940-3cd6a5031...@d26g2000prn.googlegroups.com>,
> Nope. Pipes are strictly on the same machine because it's part of the
> filesystem. Use a socket.
>
> http://www.linuxhq.com/guides/LPG/node8.htmlhttp://en.wikipedia.org/wiki/Internet_sockethttp://en.wikipedia.org/wiki/STREAMS
>
> --
> DeeDee, don't press that button! DeeDee! NO! Dee...
> [I filter all Goggle Groups posts, so any reply may be automatically ignored]
Ben Bacarisse
> Thanks. I'm spinning my wheels here. This is probably not the right
> place for the following question, but I'll ask it anyway because I'm
> not sure where else to ask it (suggestions welcomed).
>
> I have set up a server with a TCP/IP socket, and it works fine. but I
> am now realizing that I may need to get the input from one process and
> send the output to another. I figure I'll need two sockets, with each
> on a separate thread. Has anyone else solved this problem before? Am I
> on the right track?
There are lots of very flexible tools for "socket plumbing". For
example netcat (often just called nc) and socat. It's hard to know if
such a think is what you need without knowing more details.
BTW, "interleaved" or plain bottom posting is usually preferred here.
--
Ben.
Russ P.
Netcat looks very useful. I wish I'd know about it a long time ago.
Thanks for the tip.
--Russ P.
Russ P.
I have a question about netcat. Say I have a “server” that reads text
from standard input and writes text to standard output. Now I want to
use netcat to set things up so the server gets its input from one
program and sends its output to another program. Can that be done? If
so, any suggestion how? Thanks.
Russ P.
pk
wrote:
> I have a question about netcat. Say I have a “server” that reads text
> from standard input and writes text to standard output. Now I want to
> use netcat to set things up so the server gets its input from one
> program and sends its output to another program. Can that be done? If
> so, any suggestion how? Thanks.
Why do you need netcat at all?
program1 | server | program2
Russ P.
> On Fri, 18 Mar 2011 12:27:18 -0700 (PDT) "Russ P." <russ.paie...@gmail.com>
> wrote:
>
> > I have a question about netcat. Say I have a server that reads text
> > from standard input and writes text to standard output. Now I want to
> > use netcat to set things up so the server gets its input from one
> > program and sends its output to another program. Can that be done? If
> > so, any suggestion how? Thanks.
>
> Why do you need netcat at all?
>
> program1 | server | program2
Will that work across machines?
Ben Bacarisse
Not on it's own, no, but you could run these commands using ssh and use
the fact that ssh will handle stdin and stdout:
program1 | ssh user@host1 server | ssh user@host2 program2
This may not be what you want (what are all these programs in reality?)
so you could use netcat. You'd start program2 (on host2) like this:
nc -l 9999 | program2
and run the server (on host1) like this:
nc -l 9999 | server | nc host2 9999
and program1 like this:
program1 | nc host1 9999
I've not had time to test it out, so no doubt I've got something wrong.
With luck the gist of it is correct
--
Ben.
Russ P.
Hey, thanks a lot, Ben. My "server" is actually an R&D prototype for
air traffic control. The main inputs are surveillance data (aircraft
positions and velocities), and the outputs are the predicted conflicts
and (if enabled) resolution maneuvers to resolve those conflicts. I am
going to try your suggestions.
--Russ P.
Russ P.
Actually, there's one more twist to this story. The program that
provides the inputs to my server already has outputs going to stdout
(mainly for monitoring, I believe), and I am unlikely to get that
changed. The alternative, I guess, is to have the outputs that I need
(the inputs to my server) sent to a pipe, then connect the pipe to my
server. Does that make sense? Any suggestions on how to do it?
--Russ P.
Russ P.
I just spoke with a colleague about the idea of using stdin/stdout and
unix pipes (or netcat) for a safety-critical real-time system (or at
least one that will be safety-critical if and when it is deployed). I
think it is an elegantly simple approach, but she thinks it is too
basic for such a system. To my way of thinking, the simplest approach
that solves the problem is usually the best one. I'd be interested in
what others think if you have any opinion on the matter. Thanks.
--Russ P.
Ben Bacarisse
<snip>
> I just spoke with a colleague about the idea of using stdin/stdout and
> unix pipes (or netcat) for a safety-critical real-time system (or at
> least one that will be safety-critical if and when it is deployed). I
> think it is an elegantly simple approach, but she thinks it is too
> basic for such a system.
I think your colleague is showing admirable restraint! I don't think
there is any way that gluing components together with netcat can be
considered for a "safety-critical real-time system".
> To my way of thinking, the simplest approach
> that solves the problem is usually the best one.
Yes, but what is the problem? If it to get this going any way at all so
it can be tested then, yes, you have a simple solution, but if the real
problem is making the system highly reliable and responsive in real
time, then you have no solution at all.
> I'd be interested in
> what others think if you have any opinion on the matter. Thanks.
I will declare right off that I am not an expert on anything real-time
not anything safety-critical though, many eons ago, I did know a fair
amount about measuring and modelling the reliability of distributed
systems. I think you need to talk to people who are experts in
real-time and safety-critical systems.
--
Ben.
Russ P.
> "Russ P." <russ.paie...@gmail.com> writes:
>
> <snip>
>
> > I just spoke with a colleague about the idea of using stdin/stdout and
> > unix pipes (or netcat) for a safety-critical real-time system (or at
> > least one that will be safety-critical if and when it is deployed). I
> > think it is an elegantly simple approach, but she thinks it is too
> > basic for such a system.
>
> I think your colleague is showing admirable restraint! I don't think
> there is any way that gluing components together with netcat can be
> considered for a "safety-critical real-time system".
>
> > To my way of thinking, the simplest approach
> > that solves the problem is usually the best one.
>
> Yes, but what is the problem? If it to get this going any way at all so
> it can be tested then, yes, you have a simple solution, but if the real
> problem is making the system highly reliable and responsive in real
> time, then you have no solution at all.
Thanks for the feedback, Ben. Yes, my main short-term objective is to
test my program, but I would like to test it with the same interface
that it can or will have in the field, if possible.
If netcat is not reliable enough, then I would think that some other
more reliable (commercial?) program that performs the same basic
function could be substituted for netcat. On the other hand, if the
entire stdin/stdout/pipe idea is itself unacceptable for safety-
critical systems, I'd like to know why. How is it any less reliable
than sockets?
--Russ P.
Ben Bacarisse
It's a complex area and I am no expert so these are just some random
views.
You can't control the buffering with netcat. That makes it a problem
for real-time systems. The data will all get read and written but you
will presumably need data to arrive in a timely manner. Netcat is not
designed to do that.
From the reliability point of view, you have to ask why three machines
are involved. They don't provide redundancy (which can increase
reliability) they each provide another independent failure point (more
than one each if you include the possibilities like unplugging the
network cable). I'd want to try to minimise the total number of
resources that need to be continuously available for the system to
operate.
The netcat setup can't have its components restarted "on the fly" nor
can it recover from a broken connection. This is because (in part) the
connection is made only one. Any break in it and the whole "pipeline"
has to restart. A robust three-component system would be designed to
re-establish communication between the parts automatically. Even is
restarting of components were never an option, you'd want to be able to
treat a broken connection as a transient "glitch" and re-connect
automatically.
Finally (by which I mean this is all that come to mind at this time of
night!) you can't design a good transaction mechanism with one-way
sockets. In a critical system I'd want the components to be able to
acknowledge receipt of data (or notify each other about non-receipt)
which requires two-way communication and full control over timeouts and
so on.
--
Ben.
Russ P.
Thanks again for the feedback. I have a few comments in reply below.
> You can't control the buffering with netcat. That makes it a problem
> for real-time systems. The data will all get read and written but you
> will presumably need data to arrive in a timely manner. Netcat is not
> designed to do that.
This is not a particularly stringent "real-time" application. A second
or so of delay here and there will hardly be noticed.
> From the reliability point of view, you have to ask why three machines
> are involved. They don't provide redundancy (which can increase
> reliability) they each provide another independent failure point (more
> than one each if you include the possibilities like unplugging the
> network cable). I'd want to try to minimise the total number of
> resources that need to be continuously available for the system to
> operate.
Actually, I don't even know if three separate machines will be
involved in the final deployment, but I don't want to assume that
everything will be on one machine. If I plan for separate machines,
doing it on one machine will be easy, but if I plan for one machine,
adapting it to separate machines could be more difficult.
If you're wondering why I am so uninformed about the "target
environment," it's because the time between development and deployment
is typically many years in ATC, and the software may be completely
rewritten along the way. So why am I worrying about this stuff, you
ask? Because I'd like to reduce the chance that my software will need
to be completely rewritten. In my position (as an aeronautical
research engineer, not a software engineer per se), I'm supposed to
worry more about the algorithms and logic than the particular software
implementation, but I like to pay some attention to the latter too. In
any case, I should find out more about the target environment. I have
some contacts who know all about that, so I'll ask them about it.
> The netcat setup can't have its components restarted "on the fly" nor
> can it recover from a broken connection. This is because (in part) the
> connection is made only one. Any break in it and the whole "pipeline"
> has to restart. A robust three-component system would be designed to
> re-establish communication between the parts automatically. Even is
> restarting of components were never an option, you'd want to be able to
> treat a broken connection as a transient "glitch" and re-connect
> automatically.
Interesting points. I hadn't really thought of that.
> Finally (by which I mean this is all that come to mind at this time of
> night!) you can't design a good transaction mechanism with one-way
> sockets. In a critical system I'd want the components to be able to
> acknowledge receipt of data (or notify each other about non-receipt)
> which requires two-way communication and full control over timeouts and
> so on.
Thanks again for the feedback.
--Russ P.
Ben Bacarisse
> On Mar 18, 9:04 pm, Ben Bacarisse <ben.use...@bsb.me.uk> wrote:
<snip>
>> It's a complex area and I am no expert so these are just some random
>> views.
>
> Thanks again for the feedback. I have a few comments in reply below.
>
>> You can't control the buffering with netcat. That makes it a problem
>> for real-time systems. The data will all get read and written but you
>> will presumably need data to arrive in a timely manner. Netcat is not
>> designed to do that.
>
> This is not a particularly stringent "real-time" application. A second
> or so of delay here and there will hardly be noticed.
You may be OK there then. I don't think nc offers any assurance at all
about delivery times (i.e. an implementation could decide to wait for
"enough data" rather than sending data pretty much as it arrives) but
the implementation I have seems to be reasonably prompt.
>> From the reliability point of view, you have to ask why three machines
>> are involved. They don't provide redundancy (which can increase
>> reliability) they each provide another independent failure point (more
>> than one each if you include the possibilities like unplugging the
>> network cable). I'd want to try to minimise the total number of
>> resources that need to be continuously available for the system to
>> operate.
>
> Actually, I don't even know if three separate machines will be
> involved in the final deployment, but I don't want to assume that
> everything will be on one machine. If I plan for separate machines,
> doing it on one machine will be easy, but if I plan for one machine,
> adapting it to separate machines could be more difficult.
Agreed, but that's not quite the same issue. It's probably good to
design the system as components that communicate -- the issue is whether
nc is a good enough tool to enable a simple stdin/stdout plus glue IO
model.
<snip>
>> The netcat setup can't have its components restarted "on the fly" nor
>> can it recover from a broken connection. This is because (in part) the
>> connection is made only one. Any break in it and the whole "pipeline"
>> has to restart. A robust three-component system would be designed to
>> re-establish communication between the parts automatically. Even is
>> restarting of components were never an option, you'd want to be able to
>> treat a broken connection as a transient "glitch" and re-connect
>> automatically.
>
> Interesting points. I hadn't really thought of that.
Turns out I was rather hard on nc in this respect: the -k flag tells it
to repeatedly listen for incoming connections. There may be all sorts
of issues to do with partial communications and so on, but it can be run
in a "restarting" mode.
>> Finally (by which I mean this is all that come to mind at this time of
>> night!) you can't design a good transaction mechanism with one-way
>> sockets. In a critical system I'd want the components to be able to
>> acknowledge receipt of data (or notify each other about non-receipt)
>> which requires two-way communication and full control over timeouts and
>> so on.
>
> Thanks again for the feedback.
To summarise: I think I am particularly worried about missing the
opportunity to design the connections in a way that is tailored to the
needs of the application. It is possible that nc will provide just
enough functionality to do what you want but it's not clear what
opportunities will have been missed along the way.
--
Ben.
Kees Nuyt
<russ.p...@gmail.com> wrote:
>
>I just spoke with a colleague about the idea of using stdin/stdout and
>unix pipes (or netcat) for a safety-critical real-time system (or at
>least one that will be safety-critical if and when it is deployed). I
>think it is an elegantly simple approach, but she thinks it is too
>basic for such a system.
She is right.
If you need (near) real time, restartable, guaranteed
delivery, stale message expiration and the like, you
should take a look at message queues (MQ).
>To my way of thinking, the simplest approach
>that solves the problem is usually the best one.
Sure, but you have to take care of automatic error
recovery somehow.
> I'd be interested in
>what others think if you have any opinion on the matter. Thanks.
Best regards,
--
( Kees Nuyt
)
c[_]
Russ P.
> On Fri, 18 Mar 2011 15:44:28 -0700 (PDT), "Russ P."
>
Thanks for the tip. I did a quick google on message queues, and they
may well be what I need. I'll take a closer look when I get a chance.
For now, however, I'd like to see if I can make my socket-based setup
work with minor changes. My original problem, if you'll recall, was
that I need my server to be able to get input from one program and
send output to another (rather than sending the output back to the
same program, as is normally done in a client/server setup). It
occurred to me that what I need is really very simple (to state,
anyway). I need for my server to be able to connect through the same
socket to multiple clients in a sort of "party line" setup. The "party
line" would allow any of the clients to send input to the server, and
it would send the resulting output to all the clients. Is that
possible?
--Russ P.
Robert Bonomi
Russ P. <russ.p...@gmail.com> wrote:
>
>If you're wondering why I am so uninformed about the "target
>environment," it's because the time between development and deployment
>is typically many years in ATC, and the software may be completely
>rewritten along the way. So why am I worrying about this stuff, you
>ask? Because I'd like to reduce the chance that my software will need
>to be completely rewritten. In my position (as an aeronautical
>research engineer, not a software engineer per se), I'm supposed to
>worry more about the algorithms and logic than the particular software
>implementation, but I like to pay some attention to the latter too. In
>any case, I should find out more about the target environment. I have
>some contacts who know all about that, so I'll ask them about it.
The *right* way to do this, is to write the 'detection/avoidance' software
as a _true_ network `'server' application.
One that allows _multiple_ sources of aircraft speed/direction data -- i.e
will 'accept(2)' multiple connections from other processes and uses
select(2) to monitor all of them simultaneously,
And, similarly, accepts multiple connections from processes desiring to
receive the avoidance instructions -- the analyzer simply iterates across
all the receivers sending the same avoidance data over each connection.
You can either implement the two directions on two different ports, or
have the clients (the source data sender(s), and the avoidance receiver(s))
send a first line that indicates what they will use the connection for.
Note: by virtue of 'direct' TCP socket connections between the source(s),
analyzer process, and receiver(s), you automatically get the failure
detection capabilities, and can easily handle automatic restart/reconnect
programmatically -- points the prior poster raised.
Aside: if you use 'shared memory' for input/output buffers, and for all
the critical analyzer data, you can have analyzer crash/restart *without*
losing any 'state' _or_ the connections to sender(s)/receiver(s).
(let me know if you want professional consulting on this.)
Robert Bonomi
Absolutely, "yes". see my other article in this thread.
Russ P.
wrote:
> In article <760067ba-4a56-4010-8b89-c2a553eee...@j35g2000prb.googlegroups.com>,
I would love to get professional consulting on this, but I don't think
I can arrange for it. (I work for the federal gov't, and as far as I
know we cannot get sole-source contracts for personal consulting
services.) But if you would be willing to fill in the outline a bit
more, I will surely appreciate it. You can think of it as a doing
voluntary service for your country. 8^)