RFC Patches for Ruby 1.9 runtime for debuggers, profilers, and beyond

[rocky]

The following was sent to ruby-core. In case there readers here that
do not follow that...

I'd like to offer for public review some patches for the Ruby 1.9 run-
time environment. The patches attempt to support better run-time
introspection, especially of VM instruction sequences, tracing, and
promote better support for writing tools like debuggers or profilers.

I use these changes to Ruby in an experimental C extension which
provides a run time call-stack object (http://github.com/rocky/rb-
threadframe). In turn, that extension is used in an experimental
debugger (http://github.com/rocky/rbdbgr).

The patches come in 3 variations.

1. A single patch file for the trunk (1.9.3),
http://github.com/rocky/rb-threadframe/blob/master/patches/ruby-trunk-combined.patch

2. The above broken out into 14 individual patches. There are 15,
but the last one, 14-eval-iseq-name.patch is not currently used.

3. A single patch file for 1.9.2 rc2. This is analogous to 1.

The patches are a bit too large to post here, but you can find them in
the threadframe project http://github.com/rocky/rb-threadframe/tree/master/patches.
Each of the 14 separate patches has text at the beginning describing
what the patch does.

A synopsis of them is given below.

0. changes to allow a C extension to get access to some Ruby methods
and fields defined in vm_core.h.
1. Fixes rb_vm_get_sourceline() so it returns the correct line number
when the VM PC passed in is 0.
2. Adds the ability to set tracing on or off per call frame.
3. Add more access to VM opcodes and finer control of VM disassembly.
4. Adds access to instruction sequences via a mechanism similar to
SCRIPT_LINES__ via SCRIPT_ISEQS__ and ISEQS__
5. Allows an extension to create an RubyVM::InstructionSequence object
from
a rb_seq_t pointer. Useful in adding Method#iseq and Proc#iseq
6. Adds ability to get the number of parameters passed to a C
function. Like
"arity" but is the *actual* number of parameters passed.
7. Support for fast breakpoints. Used in the experimental 1.9
debugger.
8. Provides a way to get the exception object in a trace hook on a
"raise" event callback.
9. Adds an optional trace masks trace hooks, and chaining trace hooks
for threads. Adjusts the location slightly in C-function callbacks for
calls and returns so they have access to the call stack. Allows a hook
to change the return value on a C return.
10. Changes the instruction-sequence name of the top-most instruction
sequence to something less generic, e.g. <top /tmp/ruby-program.rb>
vs. <top (required)>
11. Adds RubyVM::InstructionSequence#arity and allows a C extension to
return a binding thread- and control-frame parameters. Used in rb-
threadframe to provide
RubyVM::ThreadFrame#binding.
12. Adds a new trace event mask to provide the ability to step VM
instructions
13. Keeps a trace hook which raises an exception in fielding a C
return from looping infinitely.

I look forward to hearing your comments and suggestions.

Thanks.

[Roger Pack]

Go for it!
One thought I had recently was that it would be nice for
Proc#source_location to also return a line number...
Also do you patches provide for people to get the AST back from
methods/procs (or at least a source string representation...)?
Thanks.
-r

[Rocky Bernstein]

On Sat, Jul 31, 2010 at 9:22 AM, Roger Pack <roger...@gmail.com> wrote:

Go for it!
One thought I had recently was that it would be nice for
Proc#source_location to also return a line number...

This currently is part of rb-threadframe. In some cases where patches couldn't be tested individually, I moved code from rb-threadframe into Ruby. For example I did that for breakpoint support.  In this case, code was not moved.

In general, I'd rather not put code into Ruby if I don't have to.  Ruby is already rather large. And by keeping extensions separate, we can test them independently, adjust them, and not be tied to a Ruby release schedule.

Once there is movement of some of these patches, more of rb-threadframe can be moved into Ruby. In fact one might argue that the entire frame object should be.

Also, note that source_location is an array of items, not a line number. One reason it is an array is to be able to accommodate a range pairs (start location, stop location) where location could be a line number, or byte offset of a container, or a nested pair of line start and column numbers.

Also do you patches provide for people to get the AST back from
methods/procs (or at least a source string representation...)?

Not yet fully. See the project rb-parsetree19 for work in progress for that.

These patches are just the first step. In my view it had to be large enough to be worth the effort. (And by large enough I mean something to be able to start to support debuggers and profilers in a more reasonable way). But on the other hand I did not want the first set of patches to be too big or too entangled that it is hard to sift over the individual features.

If we make progress in this step, then another step should be taken.

[Rocky Bernstein]

Having written this (all good stuff), I see that Proc#source_location and Method#source_location is in Ruby 1.9.2 rc2 and trunk. It too returns an array rather than a line number, but the array it returns is a pair of file name and line number, which I think less is than optimal: in rb-threadframe I break a position into a "container" and a "location".

[Roger Pack]

The following was sent to ruby-core. In case there readers here that
do not follow that...

 

Maybe you should just ask for commit access so you can commit them?

-r

 

[Rocky Bernstein]

Thanks for the suggestion and for following up.

Koichi has objections with the patches. At the last Ruby Kaigi in Japan, I met with him and his student, Testu. (Tetsu gave a wonderful talk about a memory profiler he wrote). I understand Koichi-san's position and there is much that I agree with too. Given this, it would be unwise to merge the patches as is into Ruby 1.9.3.

Right now, I'd say things are at an impasse with respect to getting changes into Ruby trunk. But lest that sound be negative, I don't think of it that way or feel bad about it. It's just they way things are. 

Ruby 1.9.3 is the product of a lot of different people and there is a large community that depends on it. Therefore, getting large changes into Ruby is necessarily hard and requires a lot of coordination.

In my opinion, there is has been long-term neglect of debugging issues. This means there needs to be (again in my opinion) larger and in some cases deeper changes. As I said above, more of a commitment is needed to address all of the issues in a way everyone will agree upon. One can try to work through each of the 15 or so suggested patches which cover this little aspect or that. (For example a recent one is http://redmine.ruby-lang.org/issues/show/4046). In Japan, Koichi, Tetsu and I tried a little of that. I think all of us were exhausted after an hour. I don't have the persistence and patience to track progress of each of the 15 or so changes.

I feel my position is similar to a one attributed to Matz regarding writing specs for Ruby 1.9. He is alleged as saying something like, "Well, I could spend my time writing specs or I could spend my time writing code and designing language features. Which do you think is more fun?" 

In my case, I could spend my time maintaining 15 or so individual patch files, writing redmine tickets, updating debugger specs, and RFCs. And then follow up on the those are ignored, revise the proposals, comment on tickets and modify debugger code based on what eventually gets done and what doesn't. Or... I could just work on the code.

I've gone through something like this once before. A decade or so, I forked bash in order to get better debugging run-time support. It is really only through the persistence of Masatake YAMATO in petitioning Chet Ramey that these changes got into what became the next major release of bash. I should also note that my patches underwent modification before they were incorporated.

In closing, for now the easiest and most satisfying approach for me is to:

1) provide patches for 1.9.2. 1.9.2 doesn't change that often and the changes aren't as drastic. 

2) move forward in understanding how to better provide debugging in Rubinius which, for now, is more agile and more amenable to providing better debugging support.

But again, thanks for following up and for your suggestion.