runtime changes

[Rob Pike]

There is a blizzard of changes to the runtime: allocator, scheduler,
garbage collector, maps, and more. I can't keep up and although a few
pieces have been described outside of CLs mostly all that's happening
is non-stop change without a visible plan. I would like to see some
order brought to this chaos.

What is the plan for each of the major components? What are the goals?
What are the milestones along the way? Who is responsible for each
part? When will it be done?

-rob

[⚛]

I speak only for the garbage collector and allocator.

There is competition among several people for code in mgc0.c.

As far as I understand, Dmitry's proposal to replace the garbage collector and to make changes in the allocator isn't touching any code now. It is in the state of a proposal.

If there is an agreement to accept https://codereview.appspot.com/9754044 when the code settles down, I can close https://codereview.appspot.com/9858043.

Here is my personal schedule:

- waiting for approval or rejection of the gctool (https://codereview.appspot.com/9666049/)

- remove buffering from runtime·settype (https://codereview.appspot.com/9716045/)

- remove object and pointer buffers from mgc0.c (not started yet, 1 day to implement)

- remove calls to runtime·mallocgc() from runtime·settype() (not started yet, 1 week to implement)

- optimize scanblock() in mgc0.c (not started yet, this is going to be a bigger change and it may be controversial)

- I have no intentions right now to touch code related to precise GC of stack frames

[Rob Pike]

Thanks, but what are the goals of these changes? What do they improve?
Performance? Precision? Design? Code quality?

-rob

[⚛]

1. remove buffering from runtime·settype (https://codereview.appspot.com/9716045/)

Goal = simpler code, same performance

2. remove object and pointer buffers from mgc0.c (not started yet, 1 day to implement)

Goal = simpler code, preparation for step 4

3. remove calls to runtime·mallocgc() from runtime·settype() (not started yet, 1 week to implement)

Goal = more heap space available to 32-bit Go programs, slightly improved performance

4. optimize scanblock() in mgc0.c (not started yet, this is going to be a bigger change and it may be controversial)

Goal = performance, redesign

The goal of https://codereview.appspot.com/9754044 is to slightly improve GC performance (although the improvement won't be measurable), and to fix GC precision issues on 386.

[Keith Randall]

On Thu, May 30, 2013 at 8:51 AM, ⚛ <0xe2.0x...@gmail.com> wrote:

1. remove buffering from runtime·settype (https://codereview.appspot.com/9716045/)

Goal = simpler code, same performance

2. remove object and pointer buffers from mgc0.c (not started yet, 1 day to implement)

Goal = simpler code, preparation for step 4

3. remove calls to runtime·mallocgc() from runtime·settype() (not started yet, 1 week to implement)

Goal = more heap space available to 32-bit Go programs, slightly improved performance

4. optimize scanblock() in mgc0.c (not started yet, this is going to be a bigger change and it may be controversial)

Goal = performance, redesign

The goal of https://codereview.appspot.com/9754044 is to slightly improve GC performance (although the improvement won't be measurable), and to fix GC precision issues on 386.

My goal for 9754044 is that it is required for copyable stacks.  The other two benefits don't hurt...

 

On Thursday, May 30, 2013 5:29:55 PM UTC+2, Rob Pike wrote:

Thanks, but what are the goals of these changes? What do they improve?
Performance? Precision? Design? Code quality?

-rob

--
 
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[⚛]

On Thursday, May 30, 2013 6:05:38 PM UTC+2, Keith Randall wrote:

My goal for 9754044 is that it is required for copyable stacks.

I am not sure I understand why to copy stacks.

[Ian Lance Taylor]

There are a few different somewhat overlapping proposals at the moment.

Dmitriy has a plan for preemptive scheduling:
https://docs.google.com/a/google.com/document/d/1ETuA2IOmnaQ4j81AtTGT40Y4_Jr6_IDASEKg0t0dBR8/edit

Dmitriy has a plan for more efficient memory allocation and GC:
https://docs.google.com/document/d/1HCPu3WKyCX3ZRYxmIMKTk0Ik1dePxKW1p02k3uhcft4/edit?usp=sharing

Keith has a plan for copying stacks:
https://docs.google.com/a/google.com/document/d/101GUdflCuKLeoOHzA_6ruWxCDftDQ23Zh00ZIbUMmlM/pub

Carl has a plan for more precise GC:
https://docs.google.com/document/d/13v_u3UrN2pgUtPnH4y-qfmlXwEEryikFu0SQiwk35SA/pub

The overall goal of all of these proposals is increased performance,
particularly of GC.

There is no master plan at present.

Ian

[Russ Cox]

Copyable stacks remove the performance cliffs caused by the current segmented stack implementation. It has happened repeatedly that innocent changes that happen to make one stack frame larger or smaller have a big effect on performance because they make a frequently called function trigger stack growth/shrinking on each call.

Russ

[Keith Randall]

The gc first grabs the range of the stack, then scans within that range.  I want to make sure the stack is not copied between those two points, or we'd end up scanning garbage.  (This happens in mgc0.c when addframeroots or addroot ends up triggering a stack copy when called from addstackroots.)

[⚛]

On Thursday, May 30, 2013 6:22:09 PM UTC+2, rsc wrote:

Copyable stacks remove the performance cliffs caused by the current segmented stack implementation. It has happened repeatedly that innocent changes that happen to make one stack frame larger or smaller have a big effect on performance because they make a frequently called function trigger stack growth/shrinking on each call.

Russ

Doesn't copyable stacks interfere with pointer escape analysis?

[Dmitry Vyukov]

On Thu, May 30, 2013 at 8:21 PM, Ian Lance Taylor <ia...@golang.org> wrote:
> On Thu, May 30, 2013 at 7:32 AM, Rob Pike <r...@golang.org> wrote:
>> There is a blizzard of changes to the runtime: allocator, scheduler,
>> garbage collector, maps, and more. I can't keep up and although a few
>> pieces have been described outside of CLs mostly all that's happening
>> is non-stop change without a visible plan. I would like to see some
>> order brought to this chaos.
>>
>> What is the plan for each of the major components? What are the goals?
>> What are the milestones along the way? Who is responsible for each
>> part? When will it be done?
>
> There are a few different somewhat overlapping proposals at the moment.
>
> Dmitriy has a plan for preemptive scheduling:
> https://docs.google.com/a/google.com/document/d/1ETuA2IOmnaQ4j81AtTGT40Y4_Jr6_IDASEKg0t0dBR8/edit

Most of my current changes are related to the preemptive scheduler. As
far as I udnerstand it was LGTMed.
The goals are to provide more habitual environment for users (threads
in most other systems are preemptive) and to eliminate some
degenerative cases with GC when it takes minutes.
The timeframe is: improve preemption for GC -- Go1.2; general
preemption of long running goroutines -- most likely Go1.2 (simple to
implement, but decision is not yet made); finer-grained preemption
(e.g. for {}) -- post Go1.2 (jury is still out).

> Dmitriy has a plan for more efficient memory allocation and GC:
> https://docs.google.com/document/d/1HCPu3WKyCX3ZRYxmIMKTk0Ik1dePxKW1p02k3uhcft4/edit?usp=sharing

It's more like declaration of intention, I don't have the exact
design, not saying about LGTM.
However, for Go1.2 I am doing some non-questionable spot optimizations
of malloc.
Goals: (1) does not require users to make destructive optimizations
for the sake of malloc/GC performance, (2) reduce GC pause time.
Timeframe: the bulk of the changes is post Go1.2.

> Keith has a plan for copying stacks:
> https://docs.google.com/a/google.com/document/d/101GUdflCuKLeoOHzA_6ruWxCDftDQ23Zh00ZIbUMmlM/pub
>
> Carl has a plan for more precise GC:
> https://docs.google.com/document/d/13v_u3UrN2pgUtPnH4y-qfmlXwEEryikFu0SQiwk35SA/pub
>
> The overall goal of all of these proposals is increased performance,
> particularly of GC.
>
> There is no master plan at present.
>
> Ian
>

[Keith Randall]

No, that is taken into account.  Read the design doc:  https://docs.google.com/document/d/1wAaf1rYoM4S4gtnPh0zOlGzWtrZFQ5suE8qr2sD8uWQ/pub

[⚛]

I have decided to abandon the projects mentioned below.