Planning the next release

[Rob Pike]

As the release date for Go 1.2 approaches, thoughts have begun to move
to 1.3. But not too fast. The idea of a regular release cycle is in
part to organize time as well as code. A period of development is
followed by a time of testing, soaking and refinement, which is
followed by quiet period of stability before the release. That quiet
time provides an opportunity for people working on the project to use
Go rather than work on it, and also to think about long-term plans.

We're in the quiet period now, with the release scheduled for December
1. Even if we think the release is ready before then, we will wait
until December 1 to make it official, so we have time to work and plan
before code reviews start coming in.

The 1.2 release is the first one we've run within this new, regular
schedule[1], and we are sure to refine the process as we proceed. One
thing we'd like to try for 1.3 is to do a little more up-front
planning. The 1.1 round was mostly about performance, with 1.2 having
some focus on tools, but neither release was well-orchestrated. As the
Go community grows it's important to be clear about our goals.

There are several substantial changes slated to appear in 1.3,
including an entirely new implementation of stacks[2] and more
precision in the garbage collector. These changes affect fundamental
components of the compilers and run-time, and it would help to get
these rolled out early in the cycle. Since much of the work has
already been done, we'd like to start the review process going soon,
before December 1, even though the code won't be checked in until
after the release goes live. Once the release is live, we'll be open
to new CLs but want to concentrate on getting these fundamental
changes in early, even scheduling them so the code can move gracefully
without introducing merge conflicts and other complexities. If anyone
else has other wide-ranging, core work like this that would make sense
for 1.3, please say so now so it can be scheduled as well.

Over the next few weeks we'll work with you to sketch out the
large-scale priorities for 1.3 and pencil in the sequence of changes
to satisfy them. If we do this right, the next release will be even
smoother than for 1.2 despite having some sweeping, important changes.

To summarize, we want to bring a little more order to the Go
development process so we have a clearer idea of what the next release
will hold and so the flow of contributions focuses on the goals we
have for the release, for the language and libraries, and for the
project as a whole. This is not a profound change, just a slight
course adjustment. The open source community remains a key component
of the Go ecosystem and we want that to continue. Our goal here is to
give the project more focus and some necessary planning as it grows.

-rob, for the Go team

[1] http://golang.org/s/releasesched
[2] http://golang.org/s/contigstacks

[Kyle Lemons]

After having written and reviewed more code than I think I could safely even estimate, I've started to notice some patterns.  In particular, some patterns that I've seen people implement incorrectly, and/or implemented incorrectly myself by accident.  I'm sure I'm not the only one, and I would be interested in starting a discussion on which of these sorts of things, if any, deserve to be brought into the standard library (preferably along with gofix modules or something that can identify where you're using them or where you should be using them) to encourage consistency and correctness.

Some specific ideas:

- An easy way to sanity-check deferred closes; one option: func "log".Closer(io.Closer) io.Closer // returns a closer that will log all errors on close before returning them; example usage: defer log.Closer(f).Close()

- A way to easily and correctly implement ReaderFrom/WriterTo, in particular properly counting the N returned.  one option: "io/ioutil".CountingReader(n *int, io.Reader) io.Reader // and corresponding Writer

- An easy way to easily and correctly buffer HTTP responses with the option to subsequently choose to display an error.  one option: "net/http/httputil".ResponseBuffer

- A way to prefix errors that exposes a way to retrieve the underlying error; some options: defer "errors".Prefix(prefix string, ptr *error); or "fmt".ErrorPrefix(base error, format string, args...) error

I'm sure there are a lot of other patterns that are starting to become widespread.  I don't know what the bar should be for such patterns being internalized into the stdlib, but I suspect it includes some amount of prevalence and difficulty to implement correctly.

--

---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dave Cheney]

Good progress is being made on the sunos/amd64 port [1].

Aram has done great work figuring out how to interface with the sunos
syscall API and recently got to the point of building and running
helloworld.go.

If things continue to progress at this rate I would like to propose
integrating the sunos/amd64 port during the 1.3 cycle.

Cheers

Dave

[1] https://bitbucket.org/4ad/go-sunos

[Aram Hăvărneanu]

On Thu, Nov 7, 2013 at 9:32 AM, Dave Cheney <da...@cheney.net> wrote:
> Good progress is being made on the sunos/amd64 port [1].
>
> Aram has done great work figuring out how to interface with the sunos
> syscall API and recently got to the point of building and running
> helloworld.go.
>
> If things continue to progress at this rate I would like to propose
> integrating the sunos/amd64 port during the 1.3 cycle.
>
> Cheers
>
> Dave
>
> [1] https://bitbucket.org/4ad/go-sunos

We have also secured a builder, donation from Joyent, and there's
commercial (and personal interest) interest in this port, so it
will not be abandoned.

--
Aram Hăvărneanu

[Elias Naur]

If new GOOS targets are wide ranging enough for this planning work, this is the revised proposal for Android support I was going to make public once 1.3 development opened.

https://docs.google.com/document/d/1QcMuZYgEHLBQSGcFXT7YwUkvI1hRxxvXWjOo6MQ_F9k/edit?usp=sharing

 - elias

[Dave Cheney]

If it has not already been proposed, I would like to see the runtime
poller exposed a bit better so code outside the net package could use
it.

[Dmitry Vyukov]

Just exposing it is simple. But we need a good stable API that is usable for variety of use cases on different OSes.
Do you have a use case?
One of the use cases may be file io on windows.
I am not sure we can make it work for both network/file on Linux.

A safer way to do it is to have third party package that uses c/asm to call private functions in runtime. Then try to use it. Then make it standard.

sent from phone

[Keith Randall]

I've put together a list of changes I'm planning for the contiguous stack implementation.  I hope to start sending out CLs for review this week.  Nothing will be checked in until 1.2 is released.

1. Get rid of C vararg calls from Go code.  Mostly converting arguments and return values from T to *T so the pointerness of their arguments can be described succinctly.

1. CL 14794043 (go-mapnovararg) for maps

2. CL 23310043 (go-iconv) for interface conversions

3. CL TBD (go-assert) for interface assertions

4. CL TBD (not started) for channels

5. …stdcall, deferproc, deferreturn, newproc, concatstring

2. Get all C code off of the G stack.  This is in preference to teaching 6c how to emit pointer-map information for its local variables.

1. CL 23250043 (go-mapong0) is an experiment to measure the performance cost of switching stacks (answer: 3-6 ns/call)

3. fix stack unwind bug 6619

4. Treat cast to unsafe.Pointer as escaping (other option - fix reflect.Value and hope no one else is storing nonpointer info in an unsafe.Pointer)

1. CL 22290044 (go-unsafestack)

5. Provide funcdata for necessary asm routines (callXX, others?)

1. part of #6 for now, but I can separate it out

6. copyable stacks, including panic reimplementation

1. CL 9029047 (go-movestack) (cgo portions incomplete)

#2 depends on #1.  #6 depends on all four.

(go-* are the names of my clients that these changes are staged in)

[Dave Cheney]

On Wed, Nov 13, 2013 at 1:46 AM, Dmitry Vyukov <dvy...@google.com> wrote:
> Just exposing it is simple. But we need a good stable API that is usable for
> variety of use cases on different OSes.
> Do you have a use case?

My use case is reading devices like serial terminals and odd file like
things in /sys/class/gpio.

This can be done already, but concurrent read and close operations are
undefined, so using a poller type operation is required to get that
behaviour.

> One of the use cases may be file io on windows.
> I am not sure we can make it work for both network/file on Linux.

Probably not as cleanly as everyone would want.

>
> A safer way to do it is to have third party package that uses c/asm to call
> private functions in runtime. Then try to use it. Then make it standard.

Sounds like a good idea.

[Rob Pike]

One topic that's been bothering me lately is the size of Go binaries.
I'm sure they can be smaller. We need better tools to understand what
makes them large and how the pieces work together. There is probably
duplication. Runtime changes in the pipeline will likely make them
even bigger; we should also schedule work to counteract that.

-rob

[Gustavo Niemeyer]

On Fri, Nov 1, 2013 at 2:59 PM, Rob Pike <r...@golang.org> wrote:
> If anyone else has other wide-ranging, core work like this that would
> make sense for 1.3, please say so now so it can be scheduled as well.

There are a few patches in progress for building shared objects from
Go code that may be imported into C/C++ applications. Is there a
chance these might see some further attention within the 1.3 cycle?


gustavo @ http://niemeyer.net

[Ian Lance Taylor]

On Wed, Nov 13, 2013 at 9:44 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
>
> There are a few patches in progress for building shared objects from
> Go code that may be imported into C/C++ applications. Is there a
> chance these might see some further attention within the 1.3 cycle?

Yes, I think that would be good.

What those patches really need is somebody to write a design for how
to handle initialization for Go code in shared libraries. Also how to
handle multiple Go shared libraries linked into a single executable.
Also it would be nice to discuss dlopen and dlclose.

The last time I looked at the patches these issues were essentially
"whatever the code happens to do." I think that at the very very
least we must write down what the code does.

Ian

[james4k]

Support for Objective-C files for cgo, along the same vein as C++.

https://code.google.com/p/go/issues/detail?id=6536

Small feature, but a big step for gettable desktop-app packages.

[Dmitry Vyukov]

On Wed, Nov 13, 2013 at 4:40 AM, Dave Cheney <da...@cheney.net> wrote:

On Wed, Nov 13, 2013 at 1:46 AM, Dmitry Vyukov <dvy...@google.com> wrote:
> Just exposing it is simple. But we need a good stable API that is usable for
> variety of use cases on different OSes.
> Do you have a use case?

My use case is reading devices like serial terminals and odd file like
things in /sys/class/gpio.

Do these file descriptors work with epoll?

 

This can be done already, but concurrent read and close operations are
undefined, so using a poller type operation is required to get that
behaviour.

This functionality is not provided by the poller. The poller requires that there are no races between read/close. This functionality is provided by fdMutex:

https://code.google.com/p/go/source/browse/src/pkg/net/fd_mutex.go

[Dmitry Vyukov]

It would also be nice to catch regressions with binary size. The performance dashboard can be helpful here.

 

[Dave Cheney]

On Thu, Nov 14, 2013 at 5:20 AM, Dmitry Vyukov <dvy...@google.com> wrote:
> On Wed, Nov 13, 2013 at 4:40 AM, Dave Cheney <da...@cheney.net> wrote:
>>
>> On Wed, Nov 13, 2013 at 1:46 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>> > Just exposing it is simple. But we need a good stable API that is usable
>> > for
>> > variety of use cases on different OSes.
>> > Do you have a use case?
>>
>> My use case is reading devices like serial terminals and odd file like
>> things in /sys/class/gpio.
>
>
>
> Do these file descriptors work with epoll?

Yes, but only on linux, which is ok because the /sys filesystem is
linux only. However, polling on serial terminals would need to work on
kqueue/epoll/select, etc systems.

>
>
>>
>> This can be done already, but concurrent read and close operations are
>> undefined, so using a poller type operation is required to get that
>> behaviour.
>
>
>
> This functionality is not provided by the poller. The poller requires that
> there are no races between read/close. This functionality is provided by
> fdMutex:
> https://code.google.com/p/go/source/browse/src/pkg/net/fd_mutex.go

Then that pushes me strongly to think this doesn't need to be
integrated into the runtime.

[Dave Cheney]

Seconded.

[Michael Hudson-Doyle]

Similarly, the arm64 port should be on the radar although I guess it's a
bit unclear at this stage how ready it will be for 1.3.

Cheers,
mwh

[Rob Pike]

Arm64 will not be ready for 1.3.

[Dmitry Vyukov]

On Thu, Nov 14, 2013 at 12:03 AM, Dave Cheney <da...@cheney.net> wrote:

On Thu, Nov 14, 2013 at 5:20 AM, Dmitry Vyukov <dvy...@google.com> wrote:
> On Wed, Nov 13, 2013 at 4:40 AM, Dave Cheney <da...@cheney.net> wrote:
>>
>> On Wed, Nov 13, 2013 at 1:46 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>> > Just exposing it is simple. But we need a good stable API that is usable
>> > for
>> > variety of use cases on different OSes.
>> > Do you have a use case?
>>
>> My use case is reading devices like serial terminals and odd file like
>> things in /sys/class/gpio.
>
>
>
> Do these file descriptors work with epoll?

Yes, but only on linux, which is ok because the /sys filesystem is
linux only. However, polling on serial terminals would need to work on
kqueue/epoll/select, etc systems.

Please open tracking issue for this.

 

>> This can be done already, but concurrent read and close operations are
>> undefined, so using a poller type operation is required to get that
>> behaviour.
>
>
>
> This functionality is not provided by the poller. The poller requires that
> there are no races between read/close. This functionality is provided by
> fdMutex:
> https://code.google.com/p/go/source/browse/src/pkg/net/fd_mutex.go

Then that pushes me strongly to think this doesn't need to be
integrated into the runtime.

What exactly do you mean? fdMutex is not part of runtime.

[Dave Cheney]

> What exactly do you mean? fdMutex is not part of runtime.

s/runtime/standard library/

[Dmitry Vyukov]

I still do not understand what change you want.

[minux]

On Thu, Nov 14, 2013 at 1:01 AM, Dmitry Vyukov <dvy...@google.com> wrote:

I still do not understand what change you want.

as i understand it, he (actually I) probably want API like this:

// PollFd lets the runtime integrated poller take care of waiting on fd becames

// ready for PollerMode, and return the readiness from the returned channel.

// fd must be a valid pollable fd, otherwise the behavior is undefined.

func PollFd(fd uintptr, mode PollerMode) <- chan PollerInfo

// UnPollFd unregisters the fd from the poller.

func UnPollFd(fd uintptr) // unregisters fd from the poller

where PollerMode is something like WaitForReadable, WaitForWritable, and so forth.

Of course, the names and semantics are just examples, I'm pretty sure there are

better alternatives.

For the semantics of the API, I think probably we might also want to integrate a buffer

into it, and also return the data through the channel. There are many possibilities, but the

essence is the same: poll a user supplied fd. (the write part is more difficult, perhaps we

can first design the read part)

I think APIs like that makes a lot of problems easier for Go programs, and people might

use goroutines more to structure their program if user supplied fd could be polled by the

runtime (implementing this in user space is very awkward, for example, how to do this

properly and idiomatically in Go?

set the tty into raw mode, wait for input for 3 seconds, if it times out or interrupt received,

restore the terminal mode and continue (or exit, in case of interruption).

How could I structure the program and don't lose key presses if the user input something

after the timer fires? How to interrupt the Read system call if the it times out?

Notice that making the terminal read in a separate goroutine is not acceptable for library

function like this (the example above could be a ReadTerminalWIthTimeout library function),

and it must not change the way tty io works (e.g. force the user to read tty inputs from

a channel after calling this function).

[Dmitry Vyukov]

On Thu, Nov 14, 2013 at 10:19 AM, minux <minu...@gmail.com> wrote:

On Thu, Nov 14, 2013 at 1:01 AM, Dmitry Vyukov <dvy...@google.com> wrote:

I still do not understand what change you want.

as i understand it, he (actually I) probably want API like this:

// PollFd lets the runtime integrated poller take care of waiting on fd becames

// ready for PollerMode, and return the readiness from the returned channel.

// fd must be a valid pollable fd, otherwise the behavior is undefined.

func PollFd(fd uintptr, mode PollerMode) <- chan PollerInfo

// UnPollFd unregisters the fd from the poller.

func UnPollFd(fd uintptr) // unregisters fd from the poller

Why don't you want API similar to the current one? I.e.

WaitRead(fd) // blocks until fd is ready for reading

WaitWrite(fd) // blocks until fd is ready for writing

The channels may be slower and less idiomatic for IO.

[Dave Cheney]

yes, sorry for not being clear, I want WaitRead(fd) WaitWrite(fd)
similar to what we have now for network fds

[Dmitry Vyukov]

and what about fdMutex (concurrent read/close)?

[Dave Cheney]

Yes, I want that too, but I thought you said that was part of the net package, so I was going to solve that part of the problem with the copy pasta button. 

[minux]

On Nov 14, 2013 1:42 AM, "Dmitry Vyukov" <dvy...@google.com> wrote:
>
> On Thu, Nov 14, 2013 at 10:19 AM, minux <minu...@gmail.com> wrote:
>>
>>
>> On Thu, Nov 14, 2013 at 1:01 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>>>
>>> I still do not understand what change you want.
>>
>> as i understand it, he (actually I) probably want API like this:
>>
>> // PollFd lets the runtime integrated poller take care of waiting on fd becames
>> // ready for PollerMode, and return the readiness from the returned channel.
>> // fd must be a valid pollable fd, otherwise the behavior is undefined.
>> func PollFd(fd uintptr, mode PollerMode) <- chan PollerInfo
>> // UnPollFd unregisters the fd from the poller.
>> func UnPollFd(fd uintptr) // unregisters fd from the poller
> Why don't you want API similar to the current one? I.e.
> WaitRead(fd) // blocks until fd is ready for reading
> WaitWrite(fd) // blocks until fd is ready for writing
> The channels may be slower and less idiomatic for IO.

but we need to figure out a way to wait on several things, e.g.
1. terminal could read something
2. not timed out
3. not interrupted.

i think we can achieve 1 and 2 with the current net poller API, but not sure about 3.
(it could be implemented if the current net poller is exposed, but not very simple. i.e. the behaviour i want can be simulated with the current API with another goroutine)

And I'm not sure whether we can devise a more general (and efficient) API than the current if we want to expose the poller.

[minux]

On Nov 14, 2013 1:46 AM, "Dmitry Vyukov" <dvy...@google.com> wrote:
>
> and what about fdMutex (concurrent read/close)?

i think for non-network IO concurrent read/close is almost always an error.

[Dmitry Vyukov]

What do you mean by 'interrupted'?

The same holds for network connections as well, right? A single goroutine waits only for net IO [+ optionally for timeout]. If your use case different?

Somebody needs to open an issue and collect all use cases.

[Dmitry Vyukov]

On Thu, Nov 14, 2013 at 10:58 AM, Dave Cheney <da...@cheney.net> wrote:

Yes, I want that too, but I thought you said that was part of the net package, so I was going to solve that part of the problem with the copy pasta button. 

It relies on runtime magic a bit (runtime_Semacquire). You probably can do the same with some other sync primitive. But we need to understand whether it's a part of polling API or not.

[Dmitry Vyukov]

On Thu, Nov 14, 2013 at 10:58 AM, minux <minu...@gmail.com> wrote:

Concurrent read/close is a way to shutdown. If you have a goroutine blocked on IO (w/o timeout or large timeout), there is no way you can abort it asynchronously otherwise.

[minux]

yes, that the problem. but you probably don't want to close stdin/tty to

force the trad to return.

this is why i suggest the alternative API based on channel, so people could

choose to continue if they need. 

[Dave Cheney]

But you do want to close /dev/ttyUSB0 if you're cleaning up after programming an AVR microcontroller (to pick one example) 

I will raise an issue. 

[minux]

see my earlier post for a complete description.

interrupt here means notification from os/signal. 

let me summarize the problem here:

// it sets the terminal to raw mode, what for deadline before restoring

// the terminal mode and return the input read. If the user pressed

// ctrl-c, it will immediately restore the mode and return.

func WaitRawTerminalInputWithTimeout(deadline time.Time) (data []byte, interrupted bool)

i'm replying on phone, so i left out some detail in the func docs, but

i think you can get the idea.

the essential question here is this func can't do a read of the terminal

before knowing it's readable.

[minux]

On Thursday, November 14, 2013, Dave Cheney wrote:

But you do want to close /dev/ttyUSB0 if you're cleaning up after programming an AVR microcontroller (to pick one example) 

i mean you do not want to close the fd just to break

a blocked Read/Writecall. You certainly need to close

it when done.

I will raise an issue. 

thank you, please cc me. 

[Dmitry Vyukov]

You can't do this with net conn as well.

You need a separate goroutine that waits for os.Signal and then closes the connection.

[Dmitry Vyukov]

net does the same, it allows concurrent close/read, but the close syscall is not executed concurrently with read syscall (otherwise it would corrupt data)

so the concurrent close merely unblocks all pending operations in poller

it seems that you want a stripped-down netFD

would something similar to os.NewFile(fd uintptr, name string) *File work for you? I mean you give it raw fd and it returns something similar to netFD connected to poller and handling concurrent close/read

[Dave Cheney]

On Thu, Nov 14, 2013 at 6:54 PM, Dmitry Vyukov <dvy...@google.com> wrote:
> net does the same, it allows concurrent close/read, but the close syscall is
> not executed concurrently with read syscall (otherwise it would corrupt
> data)
> so the concurrent close merely unblocks all pending operations in poller
>
> it seems that you want a stripped-down netFD

Yes, that is correct. I was mistaken to call it concurrent, but I do
want closing an FD to unblock all other readers/writers of that
thread.

> would something similar to os.NewFile(fd uintptr, name string) *File work
> for you? I mean you give it raw fd and it returns something similar to netFD
> connected to poller and handling concurrent close/read

It's really the concurrent close/read operation that I want.

[Rob Pike]

This thread has been hijacked by the discussion of net polling. Maybe
that should be another thread or even a separate document.

-rob

[Elias Naur]

Along with my Android proposal, I'd be interested in getting darwin/arm aka ios support into mainline. It depends on 4717 (cgo enabled cross compiling) just like my proposed Android port. It also depends on 4069 which I think need very little work to cover darwin/arm (5l already supports external linking on linux/arm).

minux is the author of the iOS port, and I'm willing to help out with any work to get it merged.

 - elias

[capnm]

On Thursday, 14 November 2013 18:17:03 UTC+1, Elias Naur wrote:

Along with my Android proposal, I'd be interested in getting darwin/arm aka ios support into mainline. It depends on 4717 (cgo enabled cross compiling) just like my proposed Android port. It also depends on 4069 which I think need very little work to cover darwin/arm (5l already supports external linking on linux/arm).

minux is the author of the iOS port, and I'm willing to help out with any work to get it merged.

IIRC rsc vetoed this as the support would be against the will of the device manufacturer.

~ martin

[Elias Naur]

IIRC, he only vetoed it as long as the port required a jailbroken ios device to run. Apple had historically banned ios apps in languages not supported by the official SDK, but has since loosened that requirement. Consider MonoTouch, the iOS port of .NET, which is allowed by Apple.

The largest road block to running on regular ios devices was the requirement on runtime code generation. This requirement was removed as part of https://docs.google.com/document/d/1bMwCey-gmqZVTpRax-ESeVuZGmjwbocYs1iHplK-cjo/pub so it is my belief that the ios port can now be merged. The other large block was code signing, but that is marked as done on the iOS port wiki. In any case, 5l supporting external linking should make it much easier to support codesigning.

 - elias

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/846QFpppXUo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

[Russ Cox]

I would like to shuffle some code around to reduce the amount of time spent in the linker. http://golang.org/s/go13linker has my plans.

Russ

[Dave Cheney]

I like everything about this plan.

I'm keen to see some parts of the toolchain written in Go, but as
you've highlighted, giving up the ease of bootstrapping using just a C
compiler is a large price to pay. Is it possible to retain the old C
linker in a form that gets to go_bootstrap then we use the full
fledged toolchain to compile from there ?.

Using gccgo to compile the Go based linker sounds like an interesting
solution, but would me projects like BSDPorts, PkgSrc, and Homebrew
would need to to a compile of gccgo to build Go. This feels like a
large additional cost to lay on end users who happen to use a platform
that builds 3rd party code from source.

Cheers

Dave

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Russ Cox]

On Tue, Nov 26, 2013 at 4:30 PM, Dave Cheney <da...@cheney.net> wrote:

I like everything about this plan.

Thanks. I moved the reply to this email (and hopefully any other replies to the linker plan) to a new thread on golang-dev.

Russ

[Ugorji]

Can we look into the following:

- non-parallel channels aka co-routines support. This will help formalize the pattern of using channels without requiring a round-robin on the goroutine scheduler for many use cases e.g. efficiently doing a range over a non-exported value. See https://groups.google.com/d/msg/golang-dev/VoswK7OBckY/OXtgDJFUxmAJ

- cpu-local cache/pool. See https://groups.google.com/d/msg/golang-dev/QtSIOFsNFrQ/N89_5dZWsdIJ and https://code.google.com/p/go/issues/detail?id=4720

- proc-local objects: This allows users build more complex pools, caches, etc

[Dmitry Vyukov]

On Wed, Nov 27, 2013 at 4:16 AM, Ugorji <ugo...@gmail.com> wrote:
> Can we look into the following:
> - non-parallel channels aka co-routines support. This will help formalize
> the pattern of using channels without requiring a round-robin on the
> goroutine scheduler for many use cases e.g. efficiently doing a range over a
> non-exported value. See
> https://groups.google.com/d/msg/golang-dev/VoswK7OBckY/OXtgDJFUxmAJ

This is somewhat experimental, and there are some unanswered questions
in the discussion. I was not planning to prototype this for Go1.3.

> - cpu-local cache/pool. See
> https://groups.google.com/d/msg/golang-dev/QtSIOFsNFrQ/N89_5dZWsdIJ and
> https://code.google.com/p/go/issues/detail?id=4720

I think it would be useful.
Russ' bottomline was:
"In brief: I don't believe the cache sizing can be done correctly for
most uses. I don't believe that there are enough valid uses. I don't
want the library to fill up with little hidden pools, because they are
very hard to do both transparently and safely (see the discussion
about the bufio pool for example). The only truly safe use I can think
of for this is fmt, and fmt is probably fast enough already.
We have been doing a lot of higher-priority things for Go 1.2, which
means we don't have time for a long discussion about sync.Pool."

> - proc-local objects: This allows users build more complex pools, caches,

We need to decide on sync.Pool first.

[shka...@gmail.com]

Can sync.Pool be opened to the public as a third party library or experimental package? It is indeed very useful data structure, but custom cache size would help tremendously.

Thanks,

Serge

[Russ Cox]

I would like to bring fsnotify into the main repo (with a revised API) and then make cmd/go use it, so that 'go test' does not need to stat every source file the test depends on before starting to build the test binary. It would also enable things like 'go test -watch', which could be fun.

http://golang.org/s/go13fsnotify has my plans.

Russ

[Russ Cox]

I collected the "big" things from the responses from this thread and related threads and added them to golang.org/s/go13todo.

I also started the dashboard for 1.3 at research.swtch.com/dashboard/Go1.3. I tagged all the issues that did not have a release tag already and were not Priority-Someday with Go1.3Maybe. We can cull from there.

Russ

[Dave Cheney]

SGTM. 

Of the three (possibly more) fs notify impls in common use; yours, the one in go.exp, and chowey's one, which in your view represents the closest starting point for this package?

On BSD this would be implemented with kqueue, and Linux epoll (I think), given the other discussions around exposing the runtime poller, do you think that this proposal strengthens the case for exposing the runtime poller to Go code, or is that merely an implementation detail?

Cheers

Dave

--
 
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Oleku Konko]

I noticed https://codereview.appspot.com/15930045/ was not on your dashboard .. Nice proposal fnotify

[Dmitry Vyukov]

On Wed, Nov 13, 2013 at 9:25 PM, Rob Pike <r...@golang.org> wrote:
> One topic that's been bothering me lately is the size of Go binaries.
> I'm sure they can be smaller. We need better tools to understand what
> makes them large and how the pieces work together. There is probably
> duplication. Runtime changes in the pipeline will likely make them
> even bigger; we should also schedule work to counteract that.


FYI, here is info about binary size, build speed and memory
consumption since Aug 14. Lots of things have happened.
http://goperfd.appspot.com/perfgraph?builder=linux-amd64-perf&benchmark=build-1&metric=binary-size&metric=rss&metric=runtime
Difference between min and max during this period:
binary-size: 14.53%
runtime: 17.96%
rss: 11.80%

[Rob Pike]

https://code.google.com/p/go/issues/detail?id=6853

[Oleku Konko]

Hello, can you change the font on the stats page from light gray to dark gray or black. It has low visibility.

Thanks

[Rob Pike]

If you're talking about the 1.2 page showing pending bugs, they're all
grey because they are not blocking. (Nothing is blocking at this
point, by definition). If a blocking issue existed, it would appear in
black.

-rob

[Dmitry Vyukov]

noted

[Oleku Konko]

Thanks for clarification on that too, but i was actually referring to http://goperfd.appspot.com/perf 

[Egon Elbre]

It has low visibility by design. I made the styling such that almost-zero stats wouldn't get in the way. I don't think they add anything useful for the display; e.g. +0.07% improvement could just as well be -0.03% or 0.0%. I tried to make them so light that if you look quickly you won't notice them; but dark enough, so you could still read them.

+ egon

[Dmitry Vyukov]

On Wed, Nov 27, 2013 at 11:43 PM, Russ Cox <r...@golang.org> wrote:
> I collected the "big" things from the responses from this thread and related
> threads and added them to golang.org/s/go13todo.

Since we don't have issues that can be marked as "started", I reserve
the following items here:

runtime: reexamine defer cache
runtime: garbage collector concurrent sweep phase
runtime: move timers into scheduler, per-P, eliminating timer thread
runtime: lock-free run queue
runtime: lock-free network polling
sync: add Pool if we can agree on what it means (remaining work)
collect more test/bench/go1 benchmarks: scheduler, timers, channels,
“real system”

[Dave Cheney]

SGTM. There is no reason you cannot raise an issue for each of these
and mark them for 1.3, you already have the title and can link to this
discussion thread.

> --
>
> ---

> You received this message because you are subscribed to the Google Groups "golang-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Dmitry Vyukov]

On Sat, Dec 28, 2013 at 8:40 PM, Dmitry Vyukov <dvy...@google.com> wrote:
> On Wed, Nov 27, 2013 at 11:43 PM, Russ Cox <r...@golang.org> wrote:
>> I collected the "big" things from the responses from this thread and related
>> threads and added them to golang.org/s/go13todo.
>
> Since we don't have issues that can be marked as "started", I reserve
> the following items here:

a quick update

> runtime: reexamine defer cache

done

> runtime: garbage collector concurrent sweep phase

mailed CL

> runtime: move timers into scheduler, per-P, eliminating timer thread

not sure whether I will be able to do this
this requires more thought, it's easy to distribute state, but then
difficult to restore global predicates (the nearest timer)

> runtime: lock-free run queue

done

> runtime: lock-free network polling

mailed CL

> sync: add Pool if we can agree on what it means (remaining work)

Brad and Russ done the bulk, I've mailed CL with faster implementation

> collect more test/bench/go1 benchmarks: scheduler, timers, channels,
> “real system”

I am doing this as part of "performance dashboard".


maybe I will also do partially lock-free chans