Well, there were two separate issues that were both complex and both breaking render :layout.
The first was fixed in r2871122. It was more or less a Rails bug, although I'm not sure what a fix on the Rails side would look like. The issue stems from the fact that Haml needs to keep track of whether the view object is currently in a Haml context or not. If it is, Haml overrides all sorts of ActionView methods to make them Haml-compatible, and if it's not, it delegates the methods to the standard handlers (see all the calls to #is_haml?). Whenever a Haml template is entered, is_haml? becomes true. However, since it's possible that any sub-template could use another system, whenever #render is called is_haml? becomes false until it returns. If it ends up rendering a Haml template, it bec omes true again within the template.
For the most part, this system works well. However, there's a "dead zone" in there between when #render is called and when the new template gets rendered where is_haml? is false but it's actually in a Haml template, so there's no template system taking responsibility for anything. Since #render just returns a string, that's not a problem, because there's no template-system calls being made in the dead zone.
Except that #render doesn't always return a string. In a cruel quirk of the Rails APIs, render :layout with a block concatenates directly to the template. To do so, it calls #concat in the dead zone, which tries to access #output_buffer, which Haml isn't overriding and which thus returns nil. The hack for this in r2871122 is to not set is_haml? to false when doing render :layout with a block.
The second issue was fixed in r936d9c1. This was definitely a Haml problem. When render :layout has a block, it stores it in @_proc_for_layout. This block was created in the context of the parent template (call it T1). Its _hamlout local variable, the variable containing the buffer to which the text should be output, is pointing to the T1 buffer. When the child template (call it T2) yields, capture_haml is called on the proc, so the proc is run and all text added to the current buffer is sliced off and returned. Unfotunately, the T2 buffer is current but the proc added text to the T1 buffer, so no text is returned at all and the results of the proc are at the wrong place in the template.
My fix for this in r936d9c1 was to set the _hamlout variable of the block as the current buffer in capture_haml. There are at least two alternatives: we could compile the Haml templates so that the original code uses @haml_buffer rather than _hamlout (I just thought of that one - it's probably a better solution); or we could tweak the compiled Haml code so that all Ruby blocks have their own buffer and return the string result, thus rendering capture_haml obsolete (Yehuda Katz thought of that - it's more work but might end up being a big speed gain).
Don't worry about not figuring it out. It took me a good long time and a surprising amount of knowledge of both the Haml and ActionView internals.
- Nathan