Net::HTTP Closes STDIN
James Edward Gray II
into it. Oddly, it seems to happen when interacting with the
Net::HTTP library. I've narrowed it done to the following example on
my box:
$ cat stdin_closed_issue.rb
require 'net/http'
require 'io/wait'
Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
$ ruby stdin_closed_issue.rb
true
Can anyone explain why $stdin is closed after the page read?
James Edward Gray II
Eric Hodel
$stdin isn't closed, its at the end of file. Use #closed? to test if
an IO has been closed or not.
$ cat test.rb
require 'net/http'
require 'io/wait'
Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.closed?
p $stdin.eof?
$ ruby test.rb
false # hit ^D to close $stdin on the sending side.
true
$
$ echo hi | ruby test.rb
false
false
--
Eric Hodel - drb...@segment7.net - http://blog.segment7.net
I LIT YOUR GEM ON FIRE!
James Edward Gray II
> $stdin isn't closed, its at the end of file. Use #closed? to test
> if an IO has been closed or not.
Right, good point.
However, when I call eof?() before the Net::HTTP call it behaves
differently (stalls and prints false). Why does it not behave the
same after that page read?
James Edward Gray II
Eric Hodel
$ cat test.rb
puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?
require 'net/http'
require 'io/wait'
Net::HTTP.start('localhost', 80) do |http|
body = http.get('/').body
end
puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?
$ ruby test.rb
closed? false
type some text here
eof? false
closed? false
eof? false
$
For the first #eof? no data written, so Ruby waits until something's
been flushed. I typed some text and hit return to flush the
terminal's stdout (Ruby's $stdin). At the second #eof? no input on
$stdin has been consumed, so Ruby doesn't need to check for #eof?
again by waiting.
$ cat test.rb
puts "eof? %p" % $stdin.eof?
gets
puts "eof? %p" % $stdin.eof?
$ ruby test.rb
aoeu
eof? false
aoeu
eof? false
James Edward Gray II
OK, that makes sense. However, why does moving that eof?() check
below a Net::HTTP page fetch change this? In that case, why does
Ruby not need for something to be flushed in that case and why is eof?
() then +true+?
James Edward Gray II
Eric Hodel
> On Jan 27, 2007, at 8:48 PM, Eric Hodel wrote:
>> On Jan 27, 2007, at 18:38, James Edward Gray II wrote:
>>> On Jan 27, 2007, at 7:39 PM, Eric Hodel wrote:
>>>
>>>> $stdin isn't closed, its at the end of file. Use #closed? to
>>>> test if an IO has been closed or not.
>>>
>>> Right, good point.
>>>
>>> However, when I call eof?() before the Net::HTTP call it behaves
>>> differently (stalls and prints false). Why does it not behave
>>> the same after that page read?
>>
>> closed? false
>> type some text here
>> eof? false
>> closed? false
>> eof? false
>> $
>>
>> For the first #eof? no data written, so Ruby waits until
>> something's been flushed.
>
> OK, that makes sense. However, why does moving that eof?() check
> below a Net::HTTP page fetch change this?
I don't see this behavior. With your original example I have to hit
^D to get a prompt back.
$ ruby -v
ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]
> In that case, why does Ruby not need for something to be flushed
Because #eof? reads a character if feof(3) is not true and there's no
data pending:
rb_io_eof(io)
VALUE io;
{
OpenFile *fptr;
int ch;
GetOpenFile(io, fptr);
rb_io_check_readable(fptr);
if (feof(fptr->f)) return Qtrue;
if (READ_DATA_PENDING(fptr->f)) return Qfalse;
READ_CHECK(fptr->f);
clearerr(fptr->f);
TRAP_BEG;
ch = getc(fptr->f);
^^^^^^^^^^^^^^^^^^^
Analysis of sampling pid 15640 every 10.000000 milliseconds
Call graph:
[...]
300 rb_io_eof
300 getc
[...]
> in that case and why is eof?() then +true+?
#eof? will only be true when there's no more data to read.
James Edward Gray II
> On Jan 27, 2007, at 22:01, James Edward Gray II wrote:
>> OK, that makes sense. However, why does moving that eof?() check
>> below a Net::HTTP page fetch change this?
>
> I don't see this behavior. With your original example I have to
> hit ^D to get a prompt back.
>
> $ ruby -v
> ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]
I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don't get prompted for input and it still tells me $stdin
is closed:
Firefly:~/Desktop$ ruby -v
ruby 1.8.5 (2006-12-25 patchlevel 12) [i686-darwin8.8.1]
Firefly:~/Desktop$ cat stdin_eof_issue.rb
require 'net/http'
require 'io/wait'
Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
Firefly:~/Desktop$ ruby stdin_eof_issue.rb
true
James Edward Gray II
James Edward Gray II
> I still don't get prompted for input and it still tells me $stdin
> is closed:
I'm sorry, I meant it still claims $stdin is at eof?().
James Edward Gray II
Jacob Fugal
> I just upgraded to Ruby 1.8.5 to see if it would make a difference,
> but I still don't get prompted for input and it still tells me $stdin
I can confirm on my machine:
lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ cat test.rb
require 'net/http'
require 'io/wait'
Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
lukfugl@hephaestus:~$ ruby test.rb
true
One note: the first time I ran it, I thought it was waiting for input
too, but it was really just a few second delay while fetching
http://www.ruby-lang.org/en/license.txt.
Jacob Fugal
Eric Hodel
> On 1/29/07, James Edward Gray II <ja...@grayproductions.net> wrote:
>> I just upgraded to Ruby 1.8.5 to see if it would make a difference,
>> but I still don't get prompted for input and it still tells me $stdin
>> is [eof]:
>
> I can confirm on my machine:
Then your getc may do something different than mine (BSD-ish).
James Edward Gray II
In other words, you don't feel this is a Ruby issue, right?
James Edward Gray II
gwt...@mac.com
On Jan 27, 2007, at 8:13 PM, James Edward Gray II wrote:
> Kenneth Kalmer has brought up a HighLine issue and I'm trying to
> look into it. Oddly, it seems to happen when interacting with the
> Net::HTTP library. I've narrowed it done to the following example
> on my box:
Here is a simpler example that illustrates the issue:
Thread.new { }
p $stdin.eof?
As Eric Hodel pointed out, EOF on a terminal device is detected by
trying to read from
the device. It appears that *prior* to the creation of a thread, a
read on a terminal
device will simply block waiting for some activity on the device;
either actual data
or indication of an eof condition (someone typing control-d).
*After* a thread has been created, Ruby's green thread implementation
kicks in. This
means that the read caused by IO#eof? on the terminal device is not
allowed to block.
Instead it fails as an interrupted system call, which is interpreted
by IO#eof? as an
end of file condition. This is true even if the main thread is the
only running thread.
James' original example used Net::HTTP, which uses Timeout, which
creates a thread to
gain control of code that may block (i.e., an HTTP connection to a
remote host).
I don't know enough about Ruby's thread implementation to know if
this is a bug or
a feature. My hunch is that IO#eof? needs to be fixed so that an
interrupted system
call doesn't get interpreted as an end of file condition. Another
change would be to
turn off the software interrupt that is used to multiplex IO activity
by threads when
only the main thread is running.
Gary Wright
ara.t....@noaa.gov
sounds quite reasonable. however, on my box:
harp:~ > cat a.rb
Thread.new { }
p $stdin.eof?
harp:~ > ruby a.rb # and then type ctrl-d
true
harp:~ > ruby -v
ruby 1.8.4 (2005-12-01) [i686-linux]
harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686
so looks to be a bsd/mac issue?
-a
--
we can deny everything, except that we have the possibility of being better.
simply reflect on that.
- the dalai lama
Jacob Fugal
> ... on my box:
>
> harp:~ > cat a.rb
> Thread.new { }
> p $stdin.eof?
>
> harp:~ > ruby a.rb # and then type ctrl-d
> true
>
> harp:~ > ruby -v
> ruby 1.8.4 (2005-12-01) [i686-linux]
>
> harp:~ > uname -srm
> Linux 2.4.21-47.0.1.EL i686
>
> so looks to be a bsd/mac issue?
On my box:
lukfugl@hephaestus:~$ cat a.rb
Thread.new { }
p $stdin.eof?
lukfugl@hephaestus:~$ ruby a.rb # no CTRL-D, terminates immediately
true
lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ uname -srm
Linux 2.6.17-2-686 i686
So it's present for me with ruby 1.8.5 on linux 2.6.17. I did just
notice that my ruby is compiled i486-linux -- that *may* have
something to do with it, but I don't think so.
Jacob Fugal
gwt...@mac.com
On Jan 30, 2007, at 5:35 PM, ara.t....@noaa.gov wrote:
> On Wed, 31 Jan 2007 gwt...@mac.com wrote:
>> Here is a simpler example that illustrates the issue:
>>
>> Thread.new { }
>> p $stdin.eof?
>
>
> harp:~ > cat a.rb
> Thread.new { }
> p $stdin.eof?
>
> harp:~ > ruby a.rb # and then type ctrl-d
> true
>
> harp:~ > ruby -v
> ruby 1.8.4 (2005-12-01) [i686-linux]
>
> harp:~ > uname -srm
> Linux 2.4.21-47.0.1.EL i686
>
> so looks to be a bsd/mac issue?
My post was based on ruby 1.8.5 on Darwin 8.8.0 Power Macintosh
The anomaly does not occur with ruby-1.9 on the same OS/hardware.
Gary Wright
Eric Hodel
> On Jan 27, 2007, at 8:13 PM, James Edward Gray II wrote:
>> Kenneth Kalmer has brought up a HighLine issue and I'm trying to
>> look into it. Oddly, it seems to happen when interacting with the
>> Net::HTTP library. I've narrowed it done to the following example
>> on my box:
>
> Here is a simpler example that illustrates the issue:
>
> Thread.new { }
> p $stdin.eof?
>
> As Eric Hodel pointed out, EOF on a terminal device is detected by
> trying to read from the device. It appears that *prior* to the
> creation of a thread, a read on a terminal device will simply block
> waiting for some activity on the device; either actual data or
> indication of an eof condition (someone typing control-d).
>
> *After* a thread has been created, Ruby's green thread
> implementation kicks in. This means that the read caused by
> IO#eof? on the terminal device is not allowed to block.
If there are other threads and the IO isn't readable, ruby will run
those until they complete. Your thread immediately stopped.
Since the thread in your example has completed there's no threads to
switch to, so the main thread blocks waiting for a character (on a
BSD-ish libc). If there was a thread running, the main thread would
remain blocked (as if it were blocked on getc(3)) and the other
threads were running.
To figure out what your platform is doing use a tool that gives
system-call information like ktrace.
ara.t....@noaa.gov
> On my box:
>
> lukfugl@hephaestus:~$ cat a.rb
> Thread.new { }
> p $stdin.eof?
>
> lukfugl@hephaestus:~$ ruby a.rb # no CTRL-D, terminates immediately
> true
>
> lukfugl@hephaestus:~$ ruby -v
> ruby 1.8.5 (2006-08-25) [i486-linux]
>
> lukfugl@hephaestus:~$ uname -srm
> Linux 2.6.17-2-686 i686
>
> So it's present for me with ruby 1.8.5 on linux 2.6.17. I did just notice
> that my ruby is compiled i486-linux -- that *may* have something to do with
> it, but I don't think so.
strange:
[nrt@anchor ~]$ ruby -v -e' Thread.new{} and p STDIN.eof? ' # ctrl-d
ruby 1.8.5 (2006-12-11 patchlevel 5000) [i686-linux]
true
[nrt@anchor ~]$ uname -srm
Linux 2.6.9-42.0.3.ELsmp i686
and:
harp:~ > ruby19 -v -e' Thread.new{} and p STDIN.eof? ' # ctrl-d
ruby 1.9.0 (2006-07-26) [i686-linux]
true
harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686
this could get tricky to debug! none of my linux versions or ruby versions
show this...
just to be clear (to everyone): have you all compiled ruby from source? all
my installs are from source using nothing but '--prefix' given to ./configure.
if anyone has a package (cough) manager or vendor (cough cough) install i'd
reccomend compiling from source without any options and testing again.
kind regards.
gwt...@mac.com
On Jan 30, 2007, at 6:17 PM, Eric Hodel wrote:
> If there are other threads and the IO isn't readable, ruby will run
> those until they complete. Your thread immediately stopped.
I coded it that way on purpose to see if the issue depended on
whether another thread was running. On my platform it doesn't
matter. It also doesn't matter if the main thread joins the now dead
thread. It is the initiation of
another thread that changes the behavior of IO#eof? on certain
platforms. The two platforms that have been reported
are:
ruby 1.8.5 Linux 2.6.17-2-686 i686
ruby 1.8.5 Darwin 8.8.0 Power Macintosh (Mac OS X 10.4.8)
> Since the thread in your example has completed there's no threads
> to switch to, so the main thread blocks waiting for a character (on
> a BSD-ish libc). If there was a thread running, the main thread
> would remain blocked (as if it were blocked on getc(3)) and the
> other threads were running.
But my point (and James' point) was that on my (his) platform it
*doesn't* block when it logically should.
> To figure out what your platform is doing use a tool that gives
> system-call information like ktrace.
I did that and came up with the simpler example in the process. If
you run just:
p $stdin.eof?
Ruby blocks waiting for input. If you first create a thread
which immediately terminates (my simple example) then Ruby doesn't
block (ruby 1.8.4 on Mac OS X) instead the read call is interrupted
with a software interrupt, which I interpreted as being party of Ruby's
thread implementation.
Gary Wright
James Edward Gray II
> this could get tricky to debug!
Bingo. It's actually a pretty significant problem for HighLine
because after you do something like fetching a web page, HighLine
believes $stdin to e closed and won't allow you to ask the user any
more questions. I fear we will need to disable HighLine's eof?()
check the work around this.
> just to be clear (to everyone): have you all compiled ruby from
> source?
I compiled from source. I did --enable-pthread as well.
James Edward Gray II
James Edward Gray II
>
> On Jan 30, 2007, at 6:17 PM, Eric Hodel wrote:
>> Since the thread in your example has completed there's no threads
>> to switch to, so the main thread blocks waiting for a character
>> (on a BSD-ish libc). If there was a thread running, the main
>> thread would remain blocked (as if it were blocked on getc(3)) and
>> the other threads were running.
>
> But my point (and James' point) was that on my (his) platform it
> *doesn't* block when it logically should.
Thank you. I was beginning to think I was going crazy. ;)
James Edward Gray II
Eric Hodel
> On Jan 30, 2007, at 5:28 PM, ara.t....@noaa.gov wrote:
>> this could get tricky to debug!
>
> Bingo. It's actually a pretty significant problem for HighLine
> because after you do something like fetching a web page, HighLine
> believes $stdin to e closed and won't allow you to ask the user any
> more questions. I fear we will need to disable HighLine's eof?()
> check the work around this.
Then it sounds like HighLine needs to check #closed? or continue
reading until IOError. If #eof? returns true you've read everything
you could up to now. Nothing prevents more data from appearing on
the socket after you've checked for #eof?.
James Edward Gray II
> On Jan 30, 2007, at 19:08, James Edward Gray II wrote:
>> On Jan 30, 2007, at 5:28 PM, ara.t....@noaa.gov wrote:
>>> this could get tricky to debug!
>>
>> Bingo. It's actually a pretty significant problem for HighLine
>> because after you do something like fetching a web page, HighLine
>> believes $stdin to e closed and won't allow you to ask the user
>> any more questions. I fear we will need to disable HighLine's eof?
>> () check the work around this.
>
> Then it sounds like HighLine needs to check #closed? or continue
> reading until IOError.
But, as you pointed out, closed?() is not the same as eof?().
Watching for an IOError also isn't very reliable since HighLine's
input stream may be anything from STDIN, to a Socket, with IO and
StringIO in between.
For now, I've added a workaround so user code can disable HighLine's
EOF checks as needed.
Thank to all for helping me to understand this issue.
James Edward Gray II
ara.t....@noaa.gov
>
> Thank to all for helping me to understand this issue.
>
great - that make one of us! ;-)
gwt...@mac.com
On Jan 31, 2007, at 2:32 PM, ara.t....@noaa.gov wrote:
> On Thu, 1 Feb 2007, James Edward Gray II wrote:
>>
>> Thank to all for helping me to understand this issue.
>>
>
> great - that make one of us! ;-)
I spent a bit of time looking at the code in io.c. This
file has changed radically from the 1.8 branch to the
1.9 (trunk) branch. But...
On my platform (Darwin 8.8.0 Power Macintosh):
1.9 from sources works
1.8 (head) from source works
1.8.5 from darwin ports broken
1.8.3 from source works
So perhaps there is something to Ara's suggestion that
the packaged version of ruby for Mac OS X is broken. I
don't have access to linux.
I'm rebuilding 1.8.5 from source now...
Gary Wright
gwt...@mac.com
Is the Darwin Ports (macports) maintainer for Ruby on this list?
The problem that James originally reported only seems
to exist in the Darwin Ports version of 1.8.5:
$ ruby -v
ruby 1.8.5 (2006-08-25) [powerpc-darwin8.7.0]
I just built from the current 1.8.5 branch and the problem doesn't
exist:
$ ./ruby -v
ruby 1.8.5 (2007-01-27 patchlevel 17) [powerpc-darwin8.8.0]
$ ./ruby -e 'Thread.new{} and p $stdin.eof?' # pause until ctrl-D
entered
true
Gary Wright
James Edward Gray II
> The problem that James originally reported only seems
> to exist in the Darwin Ports version of 1.8.5:
Na. I don't use MacPorts. I built Ruby from source.
James Edward Gray II
James Edward Gray II
> I just built from the current 1.8.5 branch and the problem doesn't
> exist:
>
> $ ./ruby -v
> ruby 1.8.5 (2007-01-27 patchlevel 17) [powerpc-darwin8.8.0]
I wonder if this is related to the --enable-pthread option. I used
it and wouldn't be surprised to hear that MacPorts did too. Did you
use this option when you did this test?
James Edward Gray II
gwt...@mac.com
I just typed:
autoconf; ./configure; make
I'll try:
/configure --enable-pthread; make
and report back...
Gary Wright
gwt...@mac.com
On Feb 1, 2007, at 3:46 PM, James Edward Gray II wrote:
>
> I wonder if this is related to the --enable-pthread option. I used
> it and wouldn't be surprised to hear that MacPorts did too. Did
> you use this option when you did this test?
Sure enough. When I compiled 1.8.5 with --enable-pthread the test
failed.
Ruby 1.9 wouldn't compile with --enable-pthread but works without.
Gary Wright
James Edward Gray II
>
> On Feb 1, 2007, at 3:46 PM, James Edward Gray II wrote:
>>
>> I wonder if this is related to the --enable-pthread option. I
>> used it and wouldn't be surprised to hear that MacPorts did too.
>> Did you use this option when you did this test?
>
> Sure enough. When I compiled 1.8.5 with --enable-pthread the test
> failed.
Ah, so it's likely a pthread issue. Most of the instruction articles
floating around the web for installing Ruby on Mac OS X recommend
adding that option, which is probably why we see this on that platform.
What is the advantage of pthread? I don't know.
James Edward Gray II