Patches as github forks?

[Jay Levitt]

Have you considered architecting the rb-threadframe as a github fork of https://github.com/ruby/ruby instead of as patches? I see several advantages:

1. Git can intelligently rebase and merge; you wouldn't have to maintain the patches against frequent ruby trunk changes just to keep line numbers in sync. Right now, the 1.9.3 patch is, um, 5000 commits behind.  (I'm sure there were only a few breaking changes, but they definitely broke.)

2. It would make it easier for those of us who want to use multiple patchsets; for instance, I would like to use both rb-threadframe and railsbench patchsets. If I convince both of you to maintain your patches as forks, I can fork ruby too and merge your forks into mine.

3. There's a slight possibility that it could pave the way to get some of these patches into core. I saw your frustrated post from December, and it really bugs me (architecturally, aesthetically and pragmatically) that I need to apply patches to make Ruby an operationally-sound component of my app. I'm old school, and if I can't profile and debug, I don't have a production app, period.  While ruby-core still doesn't really use github pull requests as a patch queue, it's possible that they'll grow into it - and it'd let multiple people attempt different combinations of "OK, well, how about THIS small subset of the patch?" simply by forking and sending pull requests to core.  It would crowdsource the push into core. As you said, "I could spend my time maintaining 15 or so individual patch files, writing redmine tickets.. Or... I could just work on the code."  Maybe the community can pitch in here.

The only downside is that you'd probably want to frequently rebase the fork, which means folks would have to rebase, not pull, from github.  I suspect that anyone technical enough to care about alternative debuggers and garbage collection is cool with that.

Are you interested? My C is at the "Crash? Try an ampersand.  OK, so try an asterisk." level, but I could do the grunt work of turning your patches into cleanly-applicable ruby forks.

[Rocky Bernstein]

I understand where you are coming from and understand your concerns.  But I don't see myself maintaining patches off of the github fork of Ruby (especially HEAD) at this point.  Of course, one of the ideas of github which I like is that it doesn't matter what I do (or any individual does), because anyone can decide to fork code.

However, permit me to explain where things stand with respect to my efforts. 

It is too much work for me to patch or fork off of HEAD or 1.9.3. My experience with trying to keep things up date of HEAD when it was going into the 1.9.2 release cycle is that it was far too time consuming. There seemed to be a flurry of activity for release and the code that I need it to change, e.g. thread.c is a bit large, a bit low-level. Also, there were too many changes to it for a numerous things having nothing to do with my concerns while at the same time impinging on changes I had made. (To be blunt and perhaps a bit unfair, there is a bit of C code that should be rewritten or reorganized to be made more modular which would reduce the impact of one change having an effect my changes which sometimes were independent.)

Given this, I work off of the last stable release. It changes less, making it better suited to amount of effort I want to put into this at present. 

In other words, I'm not concerned with what goes on in YARV 1.9.3 until sometime after it is finalized. And even then, perhaps I will wait a bit to make sure the release is really settled down so there are few changes. At that point, I have the luxury of reassessing what's gone on and how to add better run-time support for debugging (tracing, run-time introspection and source-level profiling). Delaying working this also gives me a better perspective: I tend to get more experience from other Ruby environments such as Rubinius and environments from other similar Dynamic languages, such as Perl. 

That doesn't address why I do not branch off of the 1.9.2 branch in git. Some of reasons I don't want to do this right now:

• Ruby is developed in svn, not git; Therefore it means relying on the git fork to be correct. Possibly the least of my concerns,
• In the past, too many conflicts were not simple automerges, so git doesn't help much here. Likewise, in those cases where automerge is possible, I really don't mind doing it by hand. 
• I am not a git guru yet or have confidence in getting git to do what I want without oversight. (Perhaps this is a restatement of the previous point)

I hope this covers why things are the way they are. Alas, reading this though I haven't covered all of your concerns so some very specific ones I will try to address inline below:

On Sat, Oct 1, 2011 at 12:57 PM, Jay Levitt <j...@jay.fm> wrote:

Have you considered architecting the rb-threadframe as a github fork of https://github.com/ruby/ruby instead of as patches? I see several advantages:

1. Git can intelligently rebase and merge; you wouldn't have to maintain the patches against frequent ruby trunk changes just to keep line numbers in sync. Right now, the 1.9.3 patch is, um, 5000 commits behind.  (I'm sure there were only a few breaking changes, but they definitely broke.)

 

Yep, 1.9.3 patching is not supported right now. 

2. It would make it easier for those of us who want to use multiple patchsets; for instance, I would like to use both rb-threadframe and railsbench patchsets. If I convince both of you to maintain your patches as forks, I can fork ruby too and merge your forks into mine.

One can to whatever kind of git branching you want on your own and just apply my patches on top of your branch(es).  I don't envision any more significant changes to the run-time for YARV 1.9.2. For YARV 1.9.3, (as I hope I have conveyed above), a reassessment may go on at some vague time after it is released. For that, it might turn out to be best thought off as a totally new effort based on experience form the past rather than incremental changes from YARV 1.9.2. It depends on how YARV 1.9.3 has changed and how my experience has changed. 

 

3. There's a slight possibility that it could pave the way to get some of these patches into core.

I am not currently optimistic about this. To date, I have not seen any significant effort in getting previous patches into core, no matter how trivial the patches are. What's needed is a "project manager" to shepherd this in. (To be fair, there my be some significant changes and compromises I need to make; but at this point I'd rather have the freedom to get 100% of what I want which I can do with the patches. Of course this comes at the expense of it not being in core and that not everyone will be able to use it.) 

 

I saw your frustrated post from December, and it really bugs me (architecturally, aesthetically and pragmatically) that I need to apply patches to make Ruby an operationally-sound component of my app. I'm old school, and if I can't profile and debug, I don't have a production app, period.  While ruby-core still doesn't really use github pull requests as a patch queue, it's possible that they'll grow into it - and it'd let multiple people attempt different combinations of "OK, well, how about THIS small subset of the patch?" simply by forking and sending pull requests to core.  It would crowdsource the push into core. As you said, "I could spend my time maintaining 15 or so individual patch files, writing redmine tickets.. Or... I could just work on the code."  Maybe the community can pitch in here.

Again, I am sorry to be a little bit pessimistic. I haven't seen much push from the community that has translated to any motion in moving things forward in an observable way to an end-user Ruby programmer. Individuals or groups of individuals are free to take what's there already and to move things forward. I'm willing to help out here.

To that end, I've already made an effort to break the changes into 15 or so different aspects. None of them have moved forward. ("None" seems a little strong, but if I'm wrong, someone please correct me.)

The only downside is that you'd probably want to frequently rebase the fork, which means folks would have to rebase, not pull, from github.  I suspect that anyone technical enough to care about alternative debuggers and garbage collection is cool with that.

Are you interested? My C is at the "Crash? Try an ampersand.  OK, so try an asterisk." level, but I could do the grunt work of turning your patches into cleanly-applicable ruby forks.

If you want to help, that would be fantastic and I would appreciate it.  I don't think initially any great experience in C is needed. What I'd suggest is taking what is in rb-threadframe/patches/trunk and pick something you think easy to get into core. Initially it may mean updating those patches as I am not maintaining that these patches any more. Instead, I am only maintaining patches/ruby-1.9.2-combined.patch. The amount of C knowledge that I think one would need to do this is minimal. 

[Jay Levitt]

Wow - thanks for the detailed and thoughtful response. I understand the context a lot better now. 

For one, I didn't realize that 1.9.3 was unsupported; I saw the first pass at it in the code, and assumed it ought to be working.  I'll stick to 1.9.2 for now (and you'll see a bug report in a moment, since I can't build it cleanly.)  I got all excited at the 1.9.3 preview release, and wanted all my shiny new toys to play together. You might want to add that to the README.

I completely understand your desire to wait till 1.9.3 stabilizes, and especially your point that the longer you wait, the more you learn elsewhere.  One of my favorite things about being an on-again off-again Rails developer is that every time I come back, something that used to bug me is now a solved problem.

You're right - what you need is a project manager to shepherd the changes into core.  So that is what I will volunteer to do.  I don't think there's much value in doing so with the 1.9.2 patches, since core is focused on getting 1.9.3 out the door, but once that ships and you bring rb-threadframe up to speed, I will start opening core tickets to see if at least some of it can get into 1.9.4.  I'll also be interested in what parts of core could be reorganized for modularity; perhaps that's the first patch.  Maybe you could drop me a line when you start working on that?  My real e-mail is jay at jay dot eff em.

[Rocky Bernstein]

On Mon, Oct 3, 2011 at 11:49 AM, Jay Levitt <j...@jay.fm> wrote:

Wow - thanks for the detailed and thoughtful response. I understand the context a lot better now. 

For one, I didn't realize that 1.9.3 was unsupported; I saw the first pass at it in the code, and assumed it ought to be working.  I'll stick to 1.9.2 for now (and you'll see a bug report in a moment, since I can't build it cleanly.)  I got all excited at the 1.9.3 preview release, and wanted all my shiny new toys to play together. You might want to add that to the README.

Mention of 1.9.2 (and no other version) was in both the wiki page and the github home page (via README.md). It was implicit that this was not for other versions, so I've made this explicit now; I have also revised the patch-ruby.sh shell script to explain on this. If it is currently not clear enough, let me know.

...

You're right - what you need is a project manager to shepherd the changes into core.  So that is what I will volunteer to do.  

Great! Thanks. 

 

I don't think there's much value in doing so with the 1.9.2 patches, since core is focused on getting 1.9.3 out the door, but once that ships and you bring rb-threadframe up to speed, I will start opening core tickets to see if at least some of it can get into 1.9.4.  I'll also be interested in what parts of core could be reorganized for modularity; perhaps that's the first patch.

This is not a simple matter. In some cases the code is a matter of style, and my style is more smaller testable files rather than larger ones. (thread.c is almost 4500 lines long, compile.c is almost 5500 lines.)

I just took a look at thread.c: I see hook code which has very little to do with the guts of thread (it is just called from a thread), and the *_coverages functions which also have very little to do with threads. There are some conditionally-compiled rb_fd_* functions. That said, I would not undertake to revise this code lightly. It is too much at the core of things and too likely to cause breakage. 

 

 Maybe you could drop me a line when you start working on that?  My real e-mail is jay at jay dot eff em.

Ok. But as ko1 said a long while ago after 1.9.1 was released with respect to adding better debugging support (at that time the current idea was to add stepping by instruction replacement like rubinius had at the time but no longer does): no timeline. 

[SASADA Koichi]

(2011/10/04 3:35), Rocky Bernstein wrote:
>
>> > Maybe you could drop me a line when you start working on that? My real
>> > e-mail is jay at jay dot eff em.
>> >
> Ok. But as ko1 said a long while ago after 1.9.1 was released with respect
> to adding better debugging support (at that time the current idea was to add
> stepping by instruction replacement like rubinius had at the time but no
> longer does): no timeline.
>

Next 2.0 ...

Simply, we will add same mechanism as rb-threadframe. However, we can't
introduce features such as "replace return value", and so on.

In other words, we support frame "querying" API.

Sorry for my lazyness.

Thanks,
Koichi
--
// SASADA Koichi at atdot dot net

[Rocky Bernstein]

Koichi - 

Thank you for keeping us updated on the current plans ('timeline") in YARV 1.9.3 for runtime debugging and call-frame support.

My comment, which was poorly worded, simply meant I that was not making any strong commitment or giving any time estimate for a RubyVM::Threadframe object (the github rb-threadframe project) in Ruby 1.9.3. It certainly could not happen before the final YARV 1.9.3 is released.