Unicorn on Rubinius?

[Eric Wong]

Hi all,

I've been working on the Unicorn HTTP server mostly in Ruby but the
C/Ragel HTTP parser is descended from the Mongrel one. I've made the
parser compatible with Rubinius 0.13 as of commit
c89ce4fb958d79009feb18cea39b14ddf8b11ff5 in unicorn.git, however the
pure-Ruby parts do not appear to pass tests at the moment.

The Ruby parts of Unicorn are extremely *nix-oriented and uses a lot of
things that only work on *nix-like systems. However, everything we do
is currently known to be working under 1.9.1, 1.8.7 and 1.8.6 (and I'm
committed to continue supporting Unicorn on MRI).

Since I'm already short on time/resources and unfamiliar with Rubinius
internals (or C++), but I thought you guys might be interested in
getting Unicorn to run under Rubinius since all the currently failing
parts are still pure Ruby, just not commonly-used Ruby.

Unicorn does a lot of Unix-only things that are uncommon in most Ruby
code:

* Working with unlinked Files (empty ones are also shared across
parent+child and we do fchmod() on them). We also buffer
large uploads to unlinked File objects. This seems to fail
immediately at startup once the master forks off the worker
since each worker gets a file descriptor it shares with
the master.

* Iterating through ObjectSpace for File objects with the O_APPEND
flag and File#sync=true, and then doing File#reopen on them.
This seems to fail under test/unit/test_util.rb

* Signal handlers, I keep them short and process them later in
the main event loops. test_signals.rb seems to get bogged
down and weird, and test_exec.rb is basically shell script
written in Ruby.

* {TCP,UNIX}Server socket objects may be duplicated as bare Socket
objects in the IO_PURGATORY array. We can inherit non-FD_CLOEXEC
file descriptors from our parent, and first create an intermediate
bare Socket object before finally "casting" it into a TCPServer
or UNIXServer socket object. We keep the original Socket in
IO_PURGATORY to prevent the GC from nuking the underlying
descriptor. Not sure how much of a problem this is...

A good chunk of the so-called "unit" tests are actually messy+ugly
integration tests, and they'll fork off all over the place; even run
curl or dd.

I basically do not trust my own knowledge/use of Ruby, especially
after what 1.9 did with encodings; so some tests use curl/dd to write
data to sockets.

Some parts definitely over-test some things, but the tests still run
pretty quickly because I can run them in parallel with GNU make[1].

If you want, please grab the latest code off
git://git.bogomips.org/unicorn and poke around it yourselves

There's the HACKING document in there and I've tried to ensure the weird
parts lib/unicorn.rb and lib/unicorn/util.rb are well-commented. Please
let me know if you have any questions or need clarification for it, I'll
be glad to answer. Just don't ask me to look at and debug C++ :) I'm
very glad your Capi seems to work well and all the HTTP parser tests
pass.


[1] - I'm not afraid to admit I *like* programming with gmake :>

--
Eric Wong

[evanphx]

Hi Eric!

Thanks for giving Rubinius a ride on the Unicorn. Sounds like you're
using a lot of features that we probably haven't yet sussed the bugs
out of, I'll address them more below.

On Nov 22, 12:18 am, Eric Wong <normalper...@yhbt.net> wrote:
> Hi all,
>
> I've been working on the Unicorn HTTP server mostly in Ruby but the
> C/Ragel HTTP parser is descended from the Mongrel one.  I've made the
> parser compatible with Rubinius 0.13 as of commit
> c89ce4fb958d79009feb18cea39b14ddf8b11ff5 in unicorn.git, however the
> pure-Ruby parts do not appear to pass tests at the moment.
>
> The Ruby parts of Unicorn are extremely *nix-oriented and uses a lot of
> things that only work on *nix-like systems.  However, everything we do
> is currently known to be working under 1.9.1, 1.8.7 and 1.8.6 (and I'm
> committed to continue supporting Unicorn on MRI).
>
> Since I'm already short on time/resources and unfamiliar with Rubinius
> internals (or C++), but I thought you guys might be interested in
> getting Unicorn to run under Rubinius since all the currently failing
> parts are still pure Ruby, just not commonly-used Ruby.

Yeah, it sounds like you're hitting some behavior right at the edge of
a bunch of classes. It's pretty likely we haven't yet hit this (no
specs for it). Any of these behaviors though it would be great if you
could write rubyspecs for. That would make it super easy for us to see
your expectations for the behavior and have an isolated test case.

>
> Unicorn does a lot of Unix-only things that are uncommon in most Ruby
> code:
>
> * Working with unlinked Files (empty ones are also shared across
>   parent+child and we do fchmod() on them).  We also buffer
>   large uploads to unlinked File objects.  This seems to fail
>   immediately at startup once the master forks off the worker
>   since each worker gets a file descriptor it shares with
>   the master.

This does not happen on MRI? Child processes to share file descriptors
(and thusly unix kernel file objects) with the parent, so this is how
I'd expect it to work. What behavior are you expecting?

>
> * Iterating through ObjectSpace for File objects with the O_APPEND
>   flag and File#sync=true, and then doing File#reopen on them.
>   This seems to fail under test/unit/test_util.rb

Zoinks. This sounds like bad news. We don't yet support
ObjectSpace#each_object, and even when we do, this code sounds mighty
dangerous. You could easily see and manipulate File objects that
rubinius kernel is using internally. Why are you doing this?

>
> * Signal handlers, I keep them short and process them later in
>   the main event loops.  test_signals.rb seems to get bogged
>   down and weird, and test_exec.rb is basically shell script
>   written in Ruby.

We haven't pushed hard on Signal handlers, I'll take a look at this.

>
> * {TCP,UNIX}Server socket objects may be duplicated as bare Socket
>   objects in the IO_PURGATORY array.  We can inherit non-FD_CLOEXEC
>   file descriptors from our parent, and first create an intermediate
>   bare Socket object before finally "casting" it into a TCPServer
>   or UNIXServer socket object.  We keep the original Socket in
>   IO_PURGATORY to prevent the GC from nuking the underlying
>   descriptor.  Not sure how much of a problem this is...

This should be fine, though if you're duping a Socket, perhaps there
is a bug there.

>
> A good chunk of the so-called "unit" tests are actually messy+ugly
> integration tests, and they'll fork off all over the place; even run
> curl or dd.
>
>   I basically do not trust my own knowledge/use of Ruby, especially
>   after what 1.9 did with encodings; so some tests use curl/dd to write
>   data to sockets.
>
>   Some parts definitely over-test some things, but the tests still run
>   pretty quickly because I can run them in parallel with GNU make[1].
>
> If you want, please grab the latest code off
> git://git.bogomips.org/unicorn and poke around it yourselves
>
> There's the HACKING document in there and I've tried to ensure the weird
> parts lib/unicorn.rb and lib/unicorn/util.rb are well-commented.  Please
> let me know if you have any questions or need clarification for it, I'll
> be glad to answer.  Just don't ask me to look at and debug C++ :)  I'm
> very glad your Capi seems to work well and all the HTTP parser tests
> pass.

Sounds like we should be able to get this sorted out, just need a bit
more info on the questions I've asked above.

[Eric Wong]

evanphx <evan...@gmail.com> wrote:
> Hi Eric!
>
> Thanks for giving Rubinius a ride on the Unicorn. Sounds like you're
> using a lot of features that we probably haven't yet sussed the bugs
> out of, I'll address them more below.
>
> On Nov 22, 12:18�am, Eric Wong <normalper...@yhbt.net> wrote:
> > Hi all,

> > Since I'm already short on time/resources and unfamiliar with Rubinius
> > internals (or C++), but I thought you guys might be interested in
> > getting Unicorn to run under Rubinius since all the currently failing
> > parts are still pure Ruby, just not commonly-used Ruby.
>
> Yeah, it sounds like you're hitting some behavior right at the edge of
> a bunch of classes. It's pretty likely we haven't yet hit this (no
> specs for it). Any of these behaviors though it would be great if you
> could write rubyspecs for. That would make it super easy for us to see
> your expectations for the behavior and have an isolated test case.

OK, it might take me a few days to find time to get familiar with
rubyspecs (the spec DSL weirds me out a bit). I'll be sure to let you
guys know if/when I get around to it.

> > Unicorn does a lot of Unix-only things that are uncommon in most Ruby
> > code:
> >
> > * Working with unlinked Files (empty ones are also shared across
> > � parent+child and we do fchmod() on them). �We also buffer
> > � large uploads to unlinked File objects. �This seems to fail
> > � immediately at startup once the master forks off the worker
> > � since each worker gets a file descriptor it shares with
> > � the master.
>
> This does not happen on MRI? Child processes to share file descriptors
> (and thusly unix kernel file objects) with the parent, so this is how
> I'd expect it to work. What behavior are you expecting?

Actually, File#stat seems to be failing in the parent, and I don't even
need fork to reproduce it. I just reproduced this while zoning out in a
meeting:

------------------------------- 8< ----------------------------
require 'tempfile'

files = (1..10).map do
fp = Tempfile.new(nil)
File.unlink(fp.path)
fp
end
puts 'starting stat loop'

loop do
files.each do |fp|
p [ fp.path, fp.stat ]
end
end
------------------------------- 8< ----------------------------
$ ./bin/rbx t.unlinked.rb
starting stat loop
An exception occurred running t.unlinked.rb
No such file or directory - No such file or directory - /tmp/20091123-20573-1nogd72-0 (ENOENT)

Backtrace:
Errno.handle at kernel/common/errno.rb:24
File::Stat#initialize at kernel/common/file.rb:915
File#stat at kernel/common/file.rb:805
#<Class:0x2ac>(Tempfile)#stat at (eval):2
main.__script__ {} at t.unlinked.rb:12
Array#each at kernel/bootstrap/array.rb:156
main.__script__ at t.unlinked.rb:11
Rubinius::CompiledMethod#as_script at kernel/common/compiled_method.rb:230
Requirer::Utils.single_load at kernel/delta/requirer.rb:236
Requirer::Utils.load_from_extension at kernel/delta/requirer.rb:321
Rubinius::Loader#script at kernel/loader.rb:334
Rubinius::Loader#main at kernel/loader.rb:448
Object#__script__ at kernel/loader.rb:452
------------------------------- 8< ----------------------------

I'm running 0.13 on Debian GNU/Linux stable, x86_64

> > * Iterating through ObjectSpace for File objects with the O_APPEND
> > � flag and File#sync=true, and then doing File#reopen on them.
> > � This seems to fail under test/unit/test_util.rb
>
> Zoinks. This sounds like bad news. We don't yet support
> ObjectSpace#each_object, and even when we do, this code sounds mighty
> dangerous. You could easily see and manipulate File objects that
> rubinius kernel is using internally. Why are you doing this?

I was a bit worried at first about it, too, but it's been used pretty
heavily already with MRI and it seems to work well.

We need to reopen logs that have been rotated via rename(2) with
logrotate or similar tools, and we believe copytruncate is evil :>

For $stderr and $stdout it's easy, but for stuff like Rails
production.log or any application/framework-specific log files, Unicorn
doesn't know about the File objects for them, so we iterate through
ObjectSpace

ref: http://git.bogomips.org/cgit/unicorn.git/tree/lib/unicorn/util.rb

> > * Signal handlers, I keep them short and process them later in
> > � the main event loops. �test_signals.rb seems to get bogged
> > � down and weird, and test_exec.rb is basically shell script
> > � written in Ruby.
>
> We haven't pushed hard on Signal handlers, I'll take a look at this.

Cool.

> Sounds like we should be able to get this sorted out, just need a bit
> more info on the questions I've asked above.

Thanks!

--
Eric Wong

[Evan Phoenix]

Ah! We're using the wrong stat for File. Here is File#stat:

def stat
Stat.new @path
end

IE, it's going to be using stat(), which since the file is unlinked, fails. We need to be using fdstat(). Easy fix. Will go in later today.

>
>>> * Iterating through ObjectSpace for File objects with the O_APPEND
>>> flag and File#sync=true, and then doing File#reopen on them.
>>> This seems to fail under test/unit/test_util.rb
>>
>> Zoinks. This sounds like bad news. We don't yet support
>> ObjectSpace#each_object, and even when we do, this code sounds mighty
>> dangerous. You could easily see and manipulate File objects that
>> rubinius kernel is using internally. Why are you doing this?
>
> I was a bit worried at first about it, too, but it's been used pretty
> heavily already with MRI and it seems to work well.

The problem is it depends on ObjectSpace#each_object, which we don't have implemented. And when we do implement it, it will be a lot slower than MRI because of having a moving GC.

Just had idea actually. We could specifically support ObjectSpace.each_object(File) by keeping a WeakRef array of all IOs in the system and pass them up to you when you pass File to each_object. It's similar to the cheat we use for each_object(Class).

>
> We need to reopen logs that have been rotated via rename(2) with
> logrotate or similar tools, and we believe copytruncate is evil :>
>
> For $stderr and $stdout it's easy, but for stuff like Rails
> production.log or any application/framework-specific log files, Unicorn
> doesn't know about the File objects for them, so we iterate through
> ObjectSpace

Hm, so you're trying to support all possible log files without having to know where to look for them and not requiring an application to register it's log file with you. A lofty goal.

As I said, this has serious problems though because what keeps you from screwing up non-log files?

>
> ref: http://git.bogomips.org/cgit/unicorn.git/tree/lib/unicorn/util.rb
>
>>> * Signal handlers, I keep them short and process them later in
>>> the main event loops. test_signals.rb seems to get bogged
>>> down and weird, and test_exec.rb is basically shell script
>>> written in Ruby.
>>
>> We haven't pushed hard on Signal handlers, I'll take a look at this.
>
> Cool.
>
>> Sounds like we should be able to get this sorted out, just need a bit
>> more info on the questions I've asked above.
>
> Thanks!
>
> --
> Eric Wong
>

> --
> --- !ruby/object:MailingList
> name: rubinius-dev
> view: http://groups.google.com/group/rubinius-dev?hl=en
> post: rubini...@googlegroups.com
> unsubscribe: rubinius-dev...@googlegroups.com
>

[Eric Wong]

Evan Phoenix <ev...@fallingsnow.net> wrote:
> On Nov 23, 2009, at 10:32 AM, Eric Wong wrote:
> > evanphx <evan...@gmail.com> wrote:
> Ah! We're using the wrong stat for File. Here is File#stat:
>
> def stat
> Stat.new @path
> end
>

> IE, it's going to be using stat(), which since the file is unlinked,
> fails. We need to be using fdstat(). Easy fix. Will go in later today.

Thanks!

> >>> * Iterating through ObjectSpace for File objects with the O_APPEND
> >>> flag and File#sync=true, and then doing File#reopen on them.
> >>> This seems to fail under test/unit/test_util.rb
> >>
> >> Zoinks. This sounds like bad news. We don't yet support
> >> ObjectSpace#each_object, and even when we do, this code sounds mighty
> >> dangerous. You could easily see and manipulate File objects that
> >> rubinius kernel is using internally. Why are you doing this?
> >
> > I was a bit worried at first about it, too, but it's been used pretty
> > heavily already with MRI and it seems to work well.
>
> The problem is it depends on ObjectSpace#each_object, which we don't
> have implemented. And when we do implement it, it will be a lot slower
> than MRI because of having a moving GC.
>
> Just had idea actually. We could specifically support
> ObjectSpace.each_object(File) by keeping a WeakRef array of all IOs in
> the system and pass them up to you when you pass File to each_object.
> It's similar to the cheat we use for each_object(Class).

That should work.

> >
> > We need to reopen logs that have been rotated via rename(2) with
> > logrotate or similar tools, and we believe copytruncate is evil :>
> >
> > For $stderr and $stdout it's easy, but for stuff like Rails
> > production.log or any application/framework-specific log files, Unicorn
> > doesn't know about the File objects for them, so we iterate through
> > ObjectSpace
>
> Hm, so you're trying to support all possible log files without having
> to know where to look for them and not requiring an application to
> register it's log file with you. A lofty goal.

Yup. It's actually not *too* difficult, though.

> As I said, this has serious problems though because what keeps you
> from screwing up non-log files?

We actually check several things and are quite paranoid about it:

* fstat(2) != stat(2) (comparing inode + device)
* (O_APPEND | O_WRONLY) from fcntl(F_GETFL)
* path is absolute
* sync=true (Logger sets this)
* not closed

All of these are potentially racy[1], but rare since Unicorn doesn't do
threads.

You can check the code here:

> > ref: http://git.bogomips.org/cgit/unicorn.git/tree/lib/unicorn/util.rb

[1] It could be a problem with Rainbows![2] but in practice I expect log
files to be created once when the application is loaded and generally
untouched otherwise. People using the built-in rotation functionality
in Logger are out of luck, but that functionality is broken in the
presence of multiple processes anyways.

[2] - Unicorn + multiple levels of craz^H^H^H^Hconcurrency:
http://rainbows.rubyforge.org/

--
Eric Wong