"Dummy" IO object to push and pull data?
Shay Hawkins
I'm relatively new to advanced Ruby programming, and I seem to be in
need of a "dummy" IO object that does nothing except allowing reading
and writing to it. (I suppose it could do more, but I'd like to keep it
as simple as possible.) What I mean is, I need to use a select statement
to check an array two IO objects that could have data to be read.
However, in other threads running, I will be needing to push data to
these objects.
I'm using Windows Vista, and what I've found out about IO.pipe doesn't
seem to work like I intended - at least, not on my operating system.
Here's an example of what I'm trying to do:
$objectA = ???
$objectB = ???
Thread.new() do
$objectA.puts("stuff")
sleep(10)
end
Thread.new() do
$objectB.puts("stuff")
sleep(15)
end
while(1)
ready = select([$objectA,$objectB])
next if(ready == nil)
for object in ready[0]
if(object == $objectA)
puts("10 milliseconds have passed in the first thread.")
puts(">> with message: #{object.gets()}")
elsif(object == $objectB)
puts("15 milliseconds have passed in the second thread.")
puts(">> with message: #{object.gets()}")
end
end
end
Does anyone know of a way I could accomplish this? If I'm not explaining
what I'm trying to do clearly enough, just ask what more you'd like to
know. Also, please don't comment on any of the conventions I use, unless
it directly pertains to the IO question - the above is an example, and
barely resembles the actual code.
--
Posted via http://www.ruby-forum.com/.
Ryan Davis
On Jan 1, 2010, at 16:40 , Shay Hawkins wrote:
> I'm relatively new to advanced Ruby programming, and I seem to be in
> need of a "dummy" IO object that does nothing except allowing reading
> and writing to it. (I suppose it could do more, but I'd like to keep it
> as simple as possible.) What I mean is, I need to use a select statement
> to check an array two IO objects that could have data to be read.
> However, in other threads running, I will be needing to push data to
> these objects.
ri StringIO
Shay Hawkins
> ri StringIO
Thanks, but I think I'm missing something:
> test.rb:15:in 'select': can't convert StringIO to IO (TypeError)
objectA = StringIO.new("")
objectB = StringIO.new("")
..
ready = select([objectA,objectB])
What would be the alternative to using select on them?
I apologize if these are basic questions, but I've tried Google and
looked on both the IO and StringIO documentation pages with no luck. If
there's a better way for me to go about searching, do tell - I'm all for
nagging you guys less.
andrew mcelroy
> Ryan Davis wrote:
>> ri StringIO
>
> Thanks, but I think I'm missing something:
>
Depending on your application, fakefs might be worth while.
Are you needing fake files?
Andrew McElroy
>> test.rb:15:in 'select': can't convert StringIO to IO (TypeError)
>
> objectA = StringIO.new("")
> objectB = StringIO.new("")
> ...
Shay Hawkins
isn't the best practice. It involves me creating a local TCPServer that
simply echos any lines it receives back to the socket, and creating a
bunch of TCPSockets. Then the sockets can use puts() in delayed threads
while they're all being checked in a loop { select() } to see if they
have data ready to be gets()'d.
Robert Klemme
If your scenario is such that for any particular thread you are always
either reading *or* writing then you should probably look at class Queue.
require 'thread'
io = Queue.new # well, this isn't an IO
writers = (1..5).map do
Thread.new do
rand(10).times do |i|
io.enq i
end
end
end
readers = (1..2).map do
Thread.new do
while (x=io.deq)
puts x
end
end
end
writers.each {|th| th.join}
readers.size.times {io.enq nil}
readers.each {|th| th.join}
Kind regards
robert
--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
Brian Candler
> Ryan Davis wrote:
>> ri StringIO
>
> Thanks, but I think I'm missing something:
>
>> test.rb:15:in 'select': can't convert StringIO to IO (TypeError)
>
> objectA = StringIO.new("")
> objectB = StringIO.new("")
> ready = select([objectA,objectB])
Ah, then you want something with a real underlying O/S file descriptor.
require 'socket'
a, b = Socket.pair(Socket::AF_UNIX, Socket::SOCK_STREAM, 0)
Then you can select on a while in another thread you write to b (or vice
versa)
Brian Candler
flavours.
>> rd, wr = IO.pipe
=> [#<IO:0x7f8a4c31d0e8>, #<IO:0x7f8a4c31d098>]
>> wr.puts "hello"
=> nil
>> rd.gets
=> "hello\n"
Shay Hawkins
> Oh, and there is also IO.pipe, which is unidirectional under most Unix
> flavours.
I've tried pipes, but it seems to always think there's data to be read.
Here's the code I was testing it with:
pipeAA, pipeAB = IO.pipe()
pipeBA, pipeBB = IO.pipe()
Thread.new() do
loop do
sleep(1)
pipeAB.puts("1 sec")
end
end
Thread.new() do
loop do
sleep(5)
pipeBB.puts("5 sec")
end
end
loop do
# Should wait for one (or both) to have data, right?
ready = select([pipeAA,pipeBA])
next if(ready == nil)
# But it ALWAYS thinks there's data to be read...
for i in ready[0]
if(i == pipeAA)
puts("AA:")
puts(pipeAA.gets()) # ...meaning this locks the program up!
elsif(i == pipeBA)
puts("BA:")
puts(pipeBA.gets())
end
end
end
The output of that is a single line, "AA:", which is just before it
attempts to get the data from pipeAA. It doesn't continue, because it's
waiting for data from pipeAA, but it shouldn't have gotten to that step
if there wasn't data ready in the first place.
Tony Arcieri
On Fri, Jan 1, 2010 at 5:40 PM, Shay Hawkins <gohe...@comcast.net> wrote:
> Hello,
>
> I'm relatively new to advanced Ruby programming, and I seem to be in
> need of a "dummy" IO object that does nothing except allowing reading
> and writing to it. (I suppose it could do more, but I'd like to keep it
> as simple as possible.) What I mean is, I need to use a select statement
> to check an array two IO objects that could have data to be read.
> However, in other threads running, I will be needing to push data to
> these objects.
>
If this is all just for thread synchronization, you shouldn't be using IO
primitives at all; you should be using thread synchronization primitives.
Try ri ConditionVariable and see if that accomplishes what you're trying to
do.
--
Tony Arcieri
Medioh! A Kudelski Brand
Caleb Clausen
> Brian Candler wrote:
>> Oh, and there is also IO.pipe, which is unidirectional under most Unix
>> flavours.
>
> I've tried pipes, but it seems to always think there's data to be read.
> Here's the code I was testing it with:
My recollection is that on windows, select only works on sockets. It
doesn't do anything useful with pipes. You are far from being the
first programmer to regret that this is the case.
Shay Hawkins
> My recollection is that on windows, select only works on sockets. It
> doesn't do anything useful with pipes. You are far from being the
> first programmer to regret that this is the case.
To make sure that I've been using it right, select is supposed to check
if the IO objects have data ready, and if so, return an array of the
ready objects?
require("socket")
server = TCPServer.new("localhost",0)
insock = TCPSocket.new(server.addr[3],server.addr[1])
outsock = server.accept()
server.close()
puts("Select starting...")
ready = select([outsock])
puts("Select is over.")
I created a quick test program to make sure that select was working how
I thought it did. When this program is run, it outputs "Select
starting...", and then locks up, because it's waiting for outsock to
receive data (which it never will). What I hoped it would do is output
"Select starting..." and then output "Select is over.", because no data
was ready to be read from outsock.
If I'm using it correctly, is there an alternative to this on Windows
machines?
Caleb Clausen
> Caleb Clausen wrote:
>> My recollection is that on windows, select only works on sockets. It
>> doesn't do anything useful with pipes. You are far from being the
>> first programmer to regret that this is the case.
>
> To make sure that I've been using it right, select is supposed to check
> if the IO objects have data ready, and if so, return an array of the
> ready objects?
It returns an _array_ of arrays of ready file handles.
> require("socket")
>
> server = TCPServer.new("localhost",0)
> insock = TCPSocket.new(server.addr[3],server.addr[1])
> outsock = server.accept()
> server.close()
>
> puts("Select starting...")
> ready = select([outsock])
> puts("Select is over.")
>
> I created a quick test program to make sure that select was working how
> I thought it did. When this program is run, it outputs "Select
> starting...", and then locks up, because it's waiting for outsock to
> receive data (which it never will). What I hoped it would do is output
> "Select starting..." and then output "Select is over.", because no data
> was ready to be read from outsock.
Right. Select never returns because nothing ever happens on any of the
file handles you asked it about. If you want a nonblocking select,
pass 0 for its timeout (final parameter). Even better, make some kind
of event for select to read. Either write some data to insock, or
close insock so the select there's some event to be read (even if just
end of file). (BTW, above snippet has unfortunate variable names;
insock and outsock should be swapped.)
Tony Arcieri
On Sun, Jan 3, 2010 at 3:07 PM, Shay Hawkins <gohe...@comcast.net> wrote:
> If I'm using it correctly, is there an alternative to this on Windows
> machines?
>
I'd again ask if you're trying to do I/O or thread synchronization here. It
sure sounds like the latter to me.
ConditionVariables will work on Windows just fine.
Shay Hawkins
> I'd again ask if you're trying to do I/O or thread synchronization here.
> It
> sure sounds like the latter to me.
>
> ConditionVariables will work on Windows just fine.
Well, it seems the problem is primarily occuring with Open3.popen3
(which I downloaded the support for Windows from RAA
(http://raa.ruby-lang.org/project/win32-open3/)). When I am attempting
to call .gets() on the STDERR line, it won't let me input anything into
STDIN via .puts() until the .gets() method is over (they are being
called in seperate threads). I tried using select on STDERR so that it
wouldn't lock up with .gets() if there was no data, but as mentioned
earlier in this topic, select only works properly with sockets on
Windows machines.
If you think ConditionVariables would help me in this case, could you
please give me an example of how they would go about fixing this
problem?
Brian Candler
> Well, it seems the problem is primarily occuring with Open3.popen3
> (which I downloaded the support for Windows from RAA
> (http://raa.ruby-lang.org/project/win32-open3/)). When I am attempting
> to call .gets() on the STDERR line, it won't let me input anything into
> STDIN via .puts() until the .gets() method is over (they are being
> called in seperate threads).
Sorry, I've read that several times and I still don't understand. Can
you post a small example which demonstrates the problem, preferably
standalone?
You should have three IO objects from open3: one writable (connected to
stdin on the child), and two readable (connected to stdout and stderr on
the child).
That means you should be able to do
select([child_out, child_err])
to find out which, if any, has data available for reading. It will block
if neither is readable, unless you add a timeout:
select([child_out, child_err], nil, nil, 0)
If that's broken on Windows, your best option is to upgrade to a real
operating system :-)
However, I believe you should also be able to read from child_out and
child_err in two separate Ruby threads, which may be the simplest way to
multiplex them. Try it and see:
Thread.new(child_out) do |fd|
while line = fd.gets
puts line
end
end
Thread.new(child_err) do |fd|
while line = fd.gets
puts line
end
end
I don't think condition variables will help you here, because you're
really talking over I/O to a separate process, whilst a ruby
ConditionVariable is something within a single process.
By the way, remember that gets will block until it gets a newline.
select() tells you that some data is available, but not how much, nor
whether there's a newline. So if the other process has sent a prompt
which doesn't end with newline, gets will hang.
You could try replacing xxx.gets with xxx.readpartial(1024) to read
whatever data is available.
Robert Klemme
> If you think ConditionVariables would help me in this case, could you
> please give me an example of how they would go about fixing this
> problem?
I gave an example using Queue (also a tool for thread synchronization)
already. How about describing what problem you are trying to solve. So
far I have only read that there are some threads that want to write
something and other threads that want to read. For that you can
certainly use a Queue as demonstrated. What things do you want to
"write" and "read" and what do all your threads do otherwise (i.e. when
not reading or writing)?
Shay Hawkins
> Sorry, I've read that several times and I still don't understand. Can
> you post a small example which demonstrates the problem, preferably
> standalone?
My apologies: I'll see if I can elaborate a bit...
Well, I originally was trying to find a workaround to this, but it seems
the root of the problem is in the process' streams. I have a seperate
program that I did not create myself (and therefore cannot change the
code of) - this program puts output into stderr, and accepts input
through stdin.
I call Open3.popen3 to create the three streams. In a loop, I want to
constantly read input from stderr. (Assuming I assigned the pipes to
stdin, stdout, and stderr variable names.)
loop do
line = stderr.gets()
puts("Output: #{line}")
end
Then, if I receive an output of say, 0, I want to input something to the
program through stdin five seconds later.
loop do
line = stderr.gets().strip()
puts("Output: #{line}")
if(line == "0")
Thread.new() do
sleep(5)
stdin.puts("Received 0 five seconds ago.")
end
end
end
It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr. However, when it
receives a new set of input from stderr, since the stream is momentarily
being unused, it can then grab the stdin stream and properly input the
old line.
As I mentioned before, I tried using select on stderr so that it
wouldn't tie up the stream, yet I've been told select only works
properly with sockets on Windows.
Brian Candler wrote:
> If that's broken on Windows, your best option is to upgrade to a real
> operating system :-)
Can't argue with you there :)
Robert Klemme wrote:
> What things do you want to "write" and "read" and what do all your
> threads do otherwise (i.e. when not reading or writing)?
See: the above example. If you need a different one, just ask.
Shay Hawkins
> It creates the thread properly, but when the sleep expires and it is
> ready to call stdin.puts(), it cannot do anything, because the loop
> continued around and it is reading from stderr. However, when it
> receives a new set of input from stderr, since the stream is momentarily
> being unused, it can then grab the stdin stream and properly input the
> old line.
Hm, let me give an example of what this would output. The process that
I'm creating the stream for outputs something every 10 seconds. As I
mentioned, if it outputs a 0, I want to give it an input of "Received 0
five seconds ago." five seconds later.
0:00:00 - Program started.
0:00:10 - Output: 3
0:00:20 - Output: 0
0:00:30 - Output: 2
0:00:30 - Input: Received 0 five seconds ago.
It's a rough example, yes, but the point is that stdin seems to wait
until stderr is free. I understand this may be normal, but is there an
alternative to select() that will prevent me from tying up the streams
if there's no ready data in stderr?
Brian Candler
> I call Open3.popen3 to create the three streams. In a loop, I want to
> constantly read input from stderr. (Assuming I assigned the pipes to
> stdin, stdout, and stderr variable names.)
>
> loop do
> line = stderr.gets()
> puts("Output: #{line}")
> end
>
> Then, if I receive an output of say, 0, I want to input something to the
> program through stdin five seconds later.
>
> loop do
> line = stderr.gets().strip()
> puts("Output: #{line}")
>
> if(line == "0")
> Thread.new() do
> sleep(5)
> stdin.puts("Received 0 five seconds ago.")
> end
> end
> end
>
> It creates the thread properly, but when the sleep expires and it is
> ready to call stdin.puts(), it cannot do anything, because the loop
> continued around and it is reading from stderr.
Perhaps it would be helpful to distinguish symptoms from your assumed
causes.
Firstly, are you saying that the stdin.puts line never completes? You
need to prove this. I suggest:
STDERR.puts "About to send..."
stdin.puts "Received 0"
STDERR.puts "Finished sending."
Secondly, if the puts never completes, are you sure that an exception is
not being raised within your thread? Such exceptions are usually silent,
unless you either join the thread (wait for it to complete), or you set
Thread.abort_on_exception = true
at the top of your program. I suggest you do the latter, to help rule
out a possible cause.
Thirdly, because of buffering, the stuff you wrote to stdin might not
actually be sent until you do
stdin.flush
Or you could do 'stdin.sync = true' after opening with popen3, so that
all writes to stdin are flushed automatically.
(So if you see "Finished sending." but the other program doesn't appear
to respond, that could be the underlying problem)
> However, when it
> receives a new set of input from stderr, since the stream is momentarily
> being unused, it can then grab the stdin stream and properly input the
> old line.
So prove that; e.g. the STDERR.puts "Finished sending." I suggested
before.
> As I mentioned before, I tried using select on stderr so that it
> wouldn't tie up the stream, yet I've been told select only works
> properly with sockets on Windows.
No, if you are using threads, you do not need to do select.
Actually, Ruby will be using select() behind the scenes, and so if
Windows really is that broken, then you would never be able to run
multiple threads talking on different file descriptions; but I didn't
think it was quite that broken (you need a Windows expert though, I am
not one)
If the program really only outputs to stderr and nothing to stdout, then
it may be simpler just to redirect stderr to stdout, and then talk to it
using a normal IO.popen. That would be "yourcmd 2>&1" under Linux; I
don't know what the Windows equivalent would be.
Of course, if it weren't for the "5 second delay" requirement, then it
could be written quite happily without a thread:
while line = stderr.gets
line.chomp!
puts "Output: #{line}"
if line == "0"
stdin.puts "Sending something now"
stdin.flush
end
end
Shay Hawkins
> Perhaps it would be helpful to distinguish symptoms from your assumed
> causes.
>
> Firstly, are you saying that the stdin.puts line never completes? You
> need to prove this. I suggest:
>
> STDERR.puts "About to send..."
> stdin.puts "Received 0"
> STDERR.puts "Finished sending."
>
> actually be sent until you do
>
> stdin.flush
I've actually tried these myself; it doesn't get past the stdin.puts
line, but rather only outputs the first line (where I'm using puts(),
not STDERR.puts()). As for flush, I've tried calling it in a few
different places - before the stdin.puts, after the stdin.puts,
inbetween loops, but it doesn't seem to do anything.
I'll look into the other things you mentioned and post my results.
Brian Candler wrote:
> Of course, if it weren't for the "5 second delay" requirement, then it
> could be written quite happily without a thread:
This is true; it works fine without a delay. The delay, unfortunately,
is the crucial aspect that I've been trying to get to work from the
beginning.
Shay Hawkins
# Required support, go to process dir
require("open3.so")
Dir.chdir("..")
# Initiate the process and its streams
command = "********"
inline, outline, errline = Open3.popen3(command)
# Quick function for short timestamps
require("time")
def tstamp(line)
puts("#{Time.now().to_s().split(" ")[3]} #{line}")
end
# Read from stderr, thread stdin input
loop do
tstamp("> Awaiting data...")
line = errline.gets().strip()
tstamp("Output: #{line}")
if(line == "0")
tstamp("> Received the required line.")
Thread.new() do
tstamp("> The thread has been created.")
sleep(5)
tstamp("> Time's up! Sending input...")
inline.puts("EXIT")
tstamp("> The input has been sent.")
end
end
end
Output:
11:35:03 > Awaiting data...
11:35:03 Output: 2
11:35:03 > Awaiting data...
11:35:13 Output: 3
11:35:13 > Awaiting data...
11:35:23 Output: 0
11:35:23 > Received the required line.
11:35:23 > The thread has been created.
11:35:23 > Awaiting data...
11:35:33 Output: 4
11:35:33 > Awaiting data...
11:35:43 Output: 1
11:35:43 > Awaiting data...
Caleb Clausen
> It creates the thread properly, but when the sleep expires and it is
> ready to call stdin.puts(), it cannot do anything, because the loop
> continued around and it is reading from stderr. However, when it
> receives a new set of input from stderr, since the stream is momentarily
> being unused, it can then grab the stdin stream and properly input the
> old line.
Ah, that's right. On windows, there's a special method you have to use
instead of select to check if input is available on a pipe. Let's see
.... I believe the method you're looking for is kbhit (or _kbhit
maybe? ms likes their underscores...) Check this page on MSDN:
http://msdn.microsoft.com/en-us/library/58w7c94c(VS.80).aspx
There should be some library somewhere that makes this available to
you in rubyland.... Unfortunately, it does mean you have to poll the
pipe for available input instead of letting it notify you.
Brian Candler
> Test program:
..
> tstamp("> Time's up! Sending input...")
But that message never appears in the output you showed - and this line
occurs *before* the puts. So at the moment looks either (a) sleep(5) is
sleeping indefinitely, or (b) the whole thread has terminated with an
exception.
A better test would be a standalone pair of ruby programs, because then
you have something which anyone can reproduce the problem with (*).
Here's your test program, very slightly modified, with a partner
program.
==> prg1.rb <==
Thread.abort_on_exception = true
require "open3"
command = "ruby prg2.rb"
inline, outline, errline = Open3.popen3(command)
def tstamp(line)
puts("#{Time.now().to_s().split(" ")[3]} #{line}")
end
# Read from stderr, thread stdin input
loop do
tstamp("> Awaiting data...")
line = errline.gets().strip()
tstamp("Output: #{line}")
if(line == "0")
tstamp("> Received the required line.")
Thread.new() do
tstamp("> The thread has been created.")
sleep(5)
tstamp("> Time's up! Sending input...")
inline.puts("EXIT")
tstamp("> The input has been sent.")
end
end
end
==> prg2.rb <==
i = -2
loop do
$stderr.puts i
i += 1
if select([$stdin],nil,nil,2)
line = $stdin.gets.chomp
$stderr.puts ">>> Received #{line} <<<"
end
end
And if I run it under Linux, here's what I get:
$ ruby prg1.rb
19:58:05 > Awaiting data...
19:58:05 Output: -2
19:58:05 > Awaiting data...
19:58:07 Output: -1
19:58:07 > Awaiting data...
19:58:09 Output: 0
19:58:09 > Received the required line.
19:58:09 > The thread has been created.
19:58:09 > Awaiting data...
19:58:11 Output: 1
19:58:11 > Awaiting data...
19:58:13 Output: 2
19:58:13 > Awaiting data...
19:58:14 > Time's up! Sending input...
19:58:14 > The input has been sent.
19:58:14 Output: >>> Received EXIT <<<
19:58:14 > Awaiting data...
19:58:14 Output: 3
19:58:14 > Awaiting data...
19:58:16 Output: 4
19:58:16 > Awaiting data...
19:58:18 Output: 5
19:58:18 > Awaiting data...
^Cprg1.rb:13:in `gets': Interrupt
from prg1.rb:13
from prg1.rb:11:in `loop'
from prg1.rb:11
What happens if you run this under Windows?
Now, this slightly muddies the water because prg2.rb also depends on
being able to select(), as I wanted it to be able to sleep *and* receive
data on stdin, the same as your main program does. So it would be also
interesting to know what happens if you do "ruby prg2.rb" by itself. It
*should* print a new number every 2 seconds, and if you type something
onto stdin, it should echo it back as >>> Received ... <<<
What I'm trying to say is, if you can make prg2.rb work properly (or
make another of version of prg2 work using threads), then you ought to
be able to make prg1.rb work properly too.
(*) Incidentally: for someone to reproduce this problem, they'll need to
know exactly what platform you have. You said "Windows Vista" but you
didn't say which version of Ruby, nor which package (Cygwin? One-Click
Installer? Other?)
Regards,
Brian.
Caleb Clausen
> On 1/4/10, Shay Hawkins <gohe...@comcast.net> wrote:
>> It creates the thread properly, but when the sleep expires and it is
>> ready to call stdin.puts(), it cannot do anything, because the loop
>> continued around and it is reading from stderr. However, when it
>> receives a new set of input from stderr, since the stream is momentarily
>> being unused, it can then grab the stdin stream and properly input the
>> old line.
>
> Ah, that's right. On windows, there's a special method you have to use
> instead of select to check if input is available on a pipe. Let's see
> maybe? ms likes their underscores...) Check this page on MSDN:
> http://msdn.microsoft.com/en-us/library/58w7c94c(VS.80).aspx
>
> There should be some library somewhere that makes this available to
> you in rubyland.... Unfortunately, it does mean you have to poll the
> pipe for available input instead of letting it notify you.
Except when I actually read the msdn link above, I see it only works
with stdin... that may not help you. I think there might be some other
windows-specific call you can make that checks any fd for input
availability... you might have to hunt around to find it, or ask on a
windows programming list.
Shay Hawkins
> What happens if you run this under Windows?
C:\Users\Shay\Desktop>ruby prg1.rb
15:51:20 > Awaiting data...
15:51:21 Output: -2
15:51:21 > Awaiting data...
Then nothing else happens. I'm 90% sure this is because Windows, being
the operating system that it is, holds up everything when .gets() is
called on stdin until something is provided.
Brian Candler wrote:
> Now, this slightly muddies the water because prg2.rb also depends on
> being able to select(), as I wanted it to be able to sleep *and* receive
> data on stdin, the same as your main program does. So it would be also
> interesting to know what happens if you do "ruby prg2.rb" by itself. It
> *should* print a new number every 2 seconds, and if you type something
> onto stdin, it should echo it back as >>> Received ... <<<
C:\Users\Shay\Desktop>ruby prg2.rb
-2
C:\Users\Shay\Desktop>Hello World!
>>> Received Hello World! <<<
-1
C:\Users\Shay\Desktop>Blah~
>>> Received Blah~ <<<
0
And so forth and so on.
Brian Candler wrote:
> (*) Incidentally: for someone to reproduce this problem, they'll need to
> know exactly what platform you have. You said "Windows Vista" but you
> didn't say which version of Ruby, nor which package (Cygwin? One-Click
> Installer? Other?)
C:\Users\Shay\Desktop>ruby -v
ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]
The program I used to install it (or at least what I have in my
downloads folder) is ruby186-27_rc2 from rubyinstaller.rubyforge.org.
Brian Candler
> C:\Users\Shay\Desktop>ruby prg2.rb
> -2
> C:\Users\Shay\Desktop>Hello World!
>>>> Received Hello World! <<<
> -1
> C:\Users\Shay\Desktop>Blah~
>>>> Received Blah~ <<<
> 0
>
> And so forth and so on.
It returns you to a command prompt? How odd.
What about a threaded version of prg2.rb, does it behave any
differently?
Thread.new do
while line = $stdin.gets
$stderr.puts ">>> Received #{line.chomp} <<<"
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end
Shay Hawkins
> It returns you to a command prompt? How odd.
>
> What about a threaded version of prg2.rb, does it behave any
> differently?
>
> Thread.new do
> while line = $stdin.gets
> $stderr.puts ">>> Received #{line.chomp} <<<"
> end
> end
> i = -2
> loop do
> $stderr.puts i
> i += 1
> sleep 2
> end
Oh, my apologies, I had put the user info there to show that I was
giving input (as if @echo hadn't been turned off). As for that program,
it behaves almost the same way - locking up until user gives input, then
displaying i if the 2 seconds have expired.
Brian Candler
> giving input (as if @echo hadn't been turned off). As for that program,
> it behaves almost the same way - locking up until user gives input, then
> displaying i if the 2 seconds have expired.
OK. I've now got a Windows XP box beside me. I tried the threaded
version of prg2.rb with:
(1) rubyinstaller-1.8.6-p383-rc1.exe from
rubyforge.org/projects/rubyinstaller
(this is about a year newer than the 186-27_rc2 you have)
(2) cygwin-1.7.1 which has ruby-1.8.7-p72-2 package under 'interpreters'
(3) ruby 1.8.7-p72 binary from www.ruby-lang.org/en/downloads/
The first fails in the way you described: that is, the sleeping thread
doesn't "wake up" until some input has been given to stdin.
However the second and third work just fine, in the same way as it does
under Linux. That is, I see a new number every 2 seconds, and I can type
in input whenever I like and get it echoed back immediately.
So clearly the problem is with how ruby is built in the one-click
installer. You may wish to raise a ticket against it, using this as a
demonstration of the problem. But in any case there are two other
working versions of ruby you can use instead.
HTH,
Brian.
Robert Klemme
That can't normally be, because the thread runs independently of the
loop. I am rather suspecting that you are having an issue with not
waiting for all threads to proper finish or do not make sure pipes are
read properly. You could also have created a dead lock that way, i.e.
by not reading all pipes properly because a process that tries to write
to a filled pipe is blocked.
I have created a pair of test scripts which I will place below.
Strangely enough, there seems to be an issue with detecting closing of
the stderr pipe which I need to research separately. If you just invoke
the client script with "./clnt.rb 15 x" you will see the interaction
properly working. If you, for example, increase the number and omit the
reading of sin in the client the server will eventually block in one of
the puts or printf statements. It may be that this is what you are
observing.
Kind regards
robert
Place both files in the same directory.
file clnt.rb:
#!/usr/local/bin/ruby19
# "client" which starts the server and every time a
# zero is read from the server's stderr pipe we send
# a notification to the server's stdin.
# server's stdout is used for logging state and is mirrored
# to this process's stdout with prefix "From server".
require 'open3'
# for debugging
Thread.abort_on_exception = true
Open3.popen3 File.join(File.dirname($0), "serv.rb"), *ARGV do |sin,
sout, serr|
sin.sync = true
# we must make sure all pipes are either read or closed!
r = Thread.new do
sout.each do |line|
puts "From server: #{line}"
end
end
threads = []
serr.each do |line|
printf "%-20s read %p\n", Time.now, line
line.chomp!
if line == "0"
threads << Thread.new do
sleep 5
sin.puts "Received 0 five seconds ago."
end
elsif /finish/i =~ line
# interestingly enough EOF detection does not
# work with serv.rb
break
end
end
puts "finished reading"
threads.each {|th| th.join}
puts "finished notifying"
# init shutdown sequence
sin.close
r.join
end
file serv.rb:
#!/usr/local/bin/ruby19
# "server" process which writes a fixed amount of numbers
# to stderr and then closes stderr to indicate it's finished.
# first arg is no of repetitions
# second arg if present says indicate end with a particular
# message to stderr.
$stdout.sync = $stderr.sync = true
puts "started"
th = Thread.new do
$stdin.each do |line|
printf "%-30s read %p\n", Time.now, line
end
end
rep = (ARGV.shift || 10).to_i
ind = ARGV.shift
rep.times do
$stderr.puts rand(3)
sleep 1
end
puts "finished writing"
# indicate we are done
$stderr.puts "Finish" if ind
$stderr.close
puts "stderr.closed? #{$stderr.closed?}"
puts "finishing"
th.join
puts "finished"
Brian Candler
>> It creates the thread properly, but when the sleep expires and it is
>> ready to call stdin.puts(), it cannot do anything, because the loop
>> continued around and it is reading from stderr.
>
> That can't normally be, because the thread runs independently of the
> loop.
He's right, and I have replicated the problem. It occurs with with 1.8.6
one-click installer under Windows. With this interpreter, threads are
borked: when one thread is waiting on I/O, it prevents all other threads
from running.
The "official" 1.8.7 Windows build is OK, and so is a cygwin 1.8.7.
You can replicate the problem with this:
---------------------------------------
Thread.new do
while line = $stdin.gets
$stderr.puts ">>> Received #{line.chomp} <<<"
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end
---------------------------------------
Under the one-click installer 1.8.6, the printing loop freezes until you
give it some input.
Phillip Gawlowski
For the heck of it, I've tested 1.8.6 and 1.9.1 compiled with MinGW:
PS C:\Scripts> ruby -v
ruby 1.9.1p243 (2009-07-16 revision 24175) [i386-mingw32]
PS C:\Scripts> c:\ruby\bin\ruby -v
ruby 1.8.6 (2009-08-04 patchlevel 383) [i386-mingw32]
Ruby 1.9.1 performs as expected: The threads start, and i gets
incremented every 2 seconds:
PS C:\Scripts> ruby .\thread.rb
-2
-1
0
1
2
3
14
1
>>> Received 11 <<<
5
6
:q
>>> Received :q <<<
7
8
Same script with 1.8.6:
PS C:\Scripts> c:\ruby\bin\ruby .\thread.rb
-2
>>> Received <<<
1
>>> Received 1 <<<
-1
2
>>> Received 2 <<<
0
:q
>>> Received :q <<<
1
2
(Yeah, vim user here...)
It looks like 1.8.7 introduced (or rather got backported from 1.9) a
change that makes threads work independent from the underlying OS.
--
Phillip Gawlowski
Robert Klemme
> Brian Candler wrote:
>> What happens if you run this under Windows?
>
> C:\Users\Shay\Desktop>ruby prg1.rb
> 15:51:20 > Awaiting data...
> 15:51:21 Output: -2
> 15:51:21 > Awaiting data...
>
> Then nothing else happens. I'm 90% sure this is because Windows, being
> the operating system that it is, holds up everything when .gets() is
> called on stdin until something is provided.
Actually part of that functionality is in Ruby itself because IO uses
buffering by default. For tests like these it helps to place a line
like this at the beginning of the script:
$stdout.sync = $stderr.sync = true
As far as I can see this wasn't done in Brian's test scripts so
repeating with that line introduced will likely yield more accurate results.
Kind regards
robert
Brian Candler
> For tests like these it helps to place a line
> like this at the beginning of the script:
>
> $stdout.sync = $stderr.sync = true
I agree that's a good idea, but I don't think it actually makes a
difference here.
For one thing, the first output line ("-2") was shown on the terminal
immediately, so there's no reason why the second line ("-1") should not
be.
For another thing, the same program worked fine on the two 1.8.7
platforms but not the 1.8.6, on the same machine.
And finally, $stderr is usually unbuffered anyway (on Unix; I know I
can't make such assumptions about Windows).
Shay Hawkins
> (3) ruby 1.8.7-p72 binary from www.ruby-lang.org/en/downloads/
Ah, that'd be it. Everything works fine now - Thank you all for your
large amount of help.