Executing system commands in threads under Ruby 1.8.6
vhaerun vh
external binary is called from a different thread, but, under 1.8.6,
this doesn't seem to work. Everything is executing in a sequential-like
manner.
The guys from StackOverflow said this was because of Ruby's thread
implementation ( and when I tried the same code under JRuby, it worked
).
Is there some way to make my script work under 1.8.6, or is upgrading to
1.9 the only solution?
--
Posted via http://www.ruby-forum.com/.
Joel VanderWerf
What do you mean by external binary? Something you invoke with #system
or the like?
$ ruby -e 't=Thread.new {system "echo foo; sleep 1; echo bar"}; system
"echo baz; sleep 1; echo quux"; t.join'
foo
baz
bar
quux
Note the order of output: baz before bar. So it's not sequential.
--
vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
Robert Klemme
I don't think that is necessary. Can you provide a example of the
phenomenon you describe? Normally, external programs are separate
processes and work independently. It may be though that if you do the
IO handling for external processes not properly that it looks like
they are executed sequentially because they are blocked in IO
operations.
Kind regards
robert
--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
vhaerun vh
http://stackoverflow.com/questions/1383470/why-is-this-running-like-it-isnt-threaded
I'm executing the calls using backticks.
Eleanor McHugh
> Here's a link to the question I asked on SO:
>
> http://stackoverflow.com/questions/1383470/why-is-this-running-like-it-isnt-threaded
>
> I'm executing the calls using backticks.
Backticks are blocking calls which return the entirety of their stdout
as a single string on completion of the subprocess.
Ellie
Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net
----
raise ArgumentError unless @reality.responds_to? :reason
Eleanor McHugh
> On 7 Sep 2009, at 09:55, vhaerun vh wrote:
>> Here's a link to the question I asked on SO:
>>
>> http://stackoverflow.com/questions/1383470/why-is-this-running-like-it-isnt-threaded
>>
>> I'm executing the calls using backticks.
>
> Backticks are blocking calls which return the entirety of their
> stdout as a single string on completion of the subprocess.
You may also find my "Ruby Plumber's Guide to *nix" presentation
useful. It's available from the link in my sig.
Bertram Scharpf
Am Montag, 07. Sep 2009, 20:06:32 +0900 schrieb Eleanor McHugh:
> On 7 Sep 2009, at 09:55, vhaerun vh wrote:
>> Here's a link to the question I asked on SO:
>>
>> http://stackoverflow.com/questions/1383470/why-is-this-running-like-it-isnt-threaded
>>
>> I'm executing the calls using backticks.
>
> Backticks are blocking calls which return the entirety of their stdout as a
> single string on completion of the subprocess.
I cannot open the above link. So I rewrite Joel's code snippet
using backticks:
t = Thread.new {
puts `echo foo; sleep 1; echo bar`
}
puts `echo baz; sleep 1; echo quux`
t.join
Stricly spoken it's not blocking. The "baz" is written to its pipe
before "bar". But it is stored until "quux" is echoed and puts is
asked to write it out.
Bertram
--
Bertram Scharpf
Stuttgart, Deutschland/Germany
http://www.bertram-scharpf.de
geo ssscripting
the host on an ip range from 1 to 254 like in the following line :
`ping #{ip_addr}`
This happened from inside each individual thread. How could I replace
the backticks so that they won't block ?
Robert Klemme
> The commands executed in the script at the SO link was simply pinging
> the host on an ip range from 1 to 254 like in the following line :
>
> `ping #{ip_addr}`
>
> This happened from inside each individual thread. How could I replace
> the backticks so that they won't block ?
You can use one of the popen methods, e.g.
IO.popen ["ping", ip_addr] do |io|
io.each {|l| puts l}
end
Example
15:37:19 ~$ ruby <<EOF
> threads = (1..2).map do |i|
> Thread.new i do |ii|
> IO.popen 'ping 192.168.110.74' do |io|
> io.each do |line|
> printf "%2d %s", ii, line
> end
> end
> end
> end
> threads.map do |th|
> th.value
> end
> EOF
1
1 Pinging 192.168.110.74 with 32 bytes of data:
1
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2
2 Pinging 192.168.110.74 with 32 bytes of data:
2
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1
1 Ping statistics for 192.168.110.74:
1 Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
1 Approximate round trip times in milli-seconds:
1 Minimum = 0ms, Maximum = 0ms, Average = 0ms
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2
2 Ping statistics for 192.168.110.74:
2 Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
2 Approximate round trip times in milli-seconds:
2 Minimum = 0ms, Maximum = 0ms, Average = 0ms
15:37:38 ~$ ruby -version
ruby 1.8.7 (2008-08-11 patchlevel 72) [i386-cygwin]
-e:1: undefined local variable or method `rsion' for main:Object (NameError)
15:37:53 ~$
Cheers
Bertram Scharpf
Am Montag, 07. Sep 2009, 21:04:53 +0900 schrieb geo ssscripting:
> The commands executed in the script at the SO link was simply pinging
> the host on an ip range from 1 to 254 like in the following line :
>
> `ping #{ip_addr}`
>
> This happened from inside each individual thread. How could I replace
> the backticks so that they won't block ?
You could fork-exec-wait as described in
<http://www.ruby-doc.org/docs/ProgrammingRuby/html/ref_m_process.html#Process.waitpid>.
To retrieve the exit status call waitpid2 as described below
waitpid.
Just fill an array with pids and you don't even need threading.
Eleanor McHugh
> I cannot open the above link. So I rewrite Joel's code snippet
> using backticks:
>
> t = Thread.new {
> puts `echo foo; sleep 1; echo bar`
> }
> puts `echo baz; sleep 1; echo quux`
> t.join
>
> Stricly spoken it's not blocking. The "baz" is written to its pipe
> before "bar". But it is stored until "quux" is echoed and puts is
> asked to write it out.
From Ruby's perspective the backtick is definitely a blocking IO
operation, just as it is in shell script.
geo ssscripting
threads = []
(1..254).each do |i|
puts "pinging #{i}"
threads << Thread.new {
content = ""
IO.popen("ping 192.168.0.#{i}") do |io|
io.each { |l| content << l }
end
content
}
end
threads.each do |t|
t.join
puts t.value
end
This doesn't behave different. My Ruby -v outputs this :
ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]
I'm running Windows XP.
Bertram Scharpf
Am Dienstag, 08. Sep 2009, 01:11:23 +0900 schrieb Eleanor McHugh:
> On 7 Sep 2009, at 12:56, Bertram Scharpf wrote:
>>
>> t = Thread.new {
>> puts `echo foo; sleep 1; echo bar`
>> }
>> puts `echo baz; sleep 1; echo quux`
>> t.join
>>
>> Stricly spoken it's not blocking. The "baz" is written to its pipe
>> before "bar". But it is stored until "quux" is echoed and puts is
>> asked to write it out.
>
> From Ruby's perspective the backtick is definitely a blocking IO operation,
> just as it is in shell script.
Yes, of course. I just examined the interpreter's source. The
waitpid function is called _without_ the WNOHANG flag. Before that
the child processes output is read and appended to a string until
the stream is closed what usually happens when the program
terminates.
What I meant was just that the child process is really running and
writing "foo" and "baz" to the pipes 1 second before the output
can by noticed in the parent.
Robert Klemme
> Here's the rewrite using IO.popen:
>
> threads = []
>
> (1..254).each do |i|
> puts "pinging #{i}"
> threads << Thread.new {
> content = ""
> IO.popen("ping 192.168.0.#{i}") do |io|
> io.each { |l| content << l }
> end
> content
> }
> end
>
> threads.each do |t|
> t.join
#join is superfluous when using #value.
> puts t.value
> end
>
> This doesn't behave different. My Ruby -v outputs this :
What exactly do you mean? What do you expect? If you refer to seeing
the output of both ping commands sequentially: with the code you
presented you always will get your output sequentially simply because
you wait until all threads finish and then you'll iterate them and print
the output of one thread at a time. Your ping commands will run in
parallel.
> ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]
>
> I'm running Windows XP.
robert
geo ssscripting
everything runs as if only one thing is executed at a time. I get the
same behaviour no matter how many times I try to run it.
Bertram Scharpf
Am Dienstag, 08. Sep 2009, 02:05:03 +0900 schrieb Bertram Scharpf:
> What I meant was just that the child process is really running and
> writing "foo" and "baz" to the pipes 1 second before the output
> can by noticed in the parent.
In other words: the child processes don't block each other.
Is that ok?
Eleanor McHugh
> Am Dienstag, 08. Sep 2009, 02:05:03 +0900 schrieb Bertram Scharpf:
>> What I meant was just that the child process is really running and
>> writing "foo" and "baz" to the pipes 1 second before the output
>> can by noticed in the parent.
>
> In other words: the child processes don't block each other.
> Is that ok?
Rather let's say that child processes won't block each other due to
the underlying implementation of backtick. It is however still
possible to introduce blocking behaviour by using operations in those
children which would cause blocking, such as accessing a semaphore or
other blocking system resource.