Planning the 1.5 release

23,872 views
Skip to first unread message

Rob Pike

unread,
Dec 1, 2014, 11:51:46 PM12/1/14
to golan...@googlegroups.com

Go 1.4 is almost done; we just need to soak the new release candidate and, if it's OK, push the button. Once again it is time to start thinking about the next release, 1.5, which is scheduled for launch in 6 months.


What will be in 1.5?


The biggest visible change will be that repository is stored in Git, not Mercurial, but that has minimal bearing on the contents of the repository.


Regarding the source itself, a few major changes are planned:


First, the garbage collector in 1.5 will be fully concurrent, with minimal pauses. This work is well underway, and is described here.


Second, the compiler, assembler, linker, and runtime for 1.5 will be entirely in Go. some of this work is done already, some will be automated, some will be done by hand, but the goal is to eliminate C from the source tree so thatexcept for programs that use cgo—no C compiler is necessary to build or use Go. This was proposed a while back, but is finally happening.


Finally, new ports will include 64-bit Power PC, more complete support for Android, and possibly 64-bit ARM.


And to repeat what was said last time:


Once the 1.4 release is done, we'll be open to new CLs but want to concentrate on these fundamental changes, even scheduling them so the code can move gracefully without introducing merge conflicts and other complexities. If anyone else has other wide-ranging, core work like this that would make sense for 1.5, please say so now so it can be scheduled as well.


Over the next few weeks we'll work with you to sketch out the  large-scale priorities for 1.5 and pencil in the sequence of changes to satisfy them.


Please remember that the open source community is vital to the Go ecosystem. We can't thank you enough for your contributions.


-rob, for the Go team


小菜

unread,
Dec 2, 2014, 3:07:15 AM12/2/14
to golan...@googlegroups.com
Great! I Still hope that golang provides an official GUI package!


在 2014年12月2日星期二UTC+8下午12时51分46秒,Rob Pike写道:

adrian....@gmail.com

unread,
Dec 2, 2014, 3:32:06 AM12/2/14
to golan...@googlegroups.com

Qian Qiao

unread,
Dec 2, 2014, 3:55:55 AM12/2/14
to 小菜, golan...@googlegroups.com
On Tue Dec 02 2014 at 3:07:17 AM 小菜 <yxw.2...@gmail.com> wrote:
Great! I Still hope that golang provides an official GUI package!

If we take the actual underlying UI toolkit out of the equation, I actually think Go would be a cool choice for GUI applications too. Although it's difficult to take that out of the equation.

The fan-out -> fan-in pattern would probably bridge nicely between the concurrent part of the app and the GUI event loop.

Speaking of the event loop approach, it reminded me of this article[1]. Maybe, just maybe, Go would be that "radically new and better approach" the author had hoped for. :)

Axel Wagner

unread,
Dec 2, 2014, 4:06:43 AM12/2/14
to golan...@googlegroups.com
I don't have a vote, but if I would, I would vote for dynamic loading of go code [1]. Is there any plan to maybe start moving in that direction? (it sounds like the kind of wide-ranging core work you where referring to)

[1] https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit

Jeremy Jackins

unread,
Dec 2, 2014, 4:19:18 AM12/2/14
to Axel Wagner, golan...@googlegroups.com
He is asking for people to declare their plans, not to vote or request features.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

denghu...@gmail.com

unread,
Dec 2, 2014, 4:29:56 AM12/2/14
to golan...@googlegroups.com
after 1.5, no more "STW"

在 2014年12月2日星期二UTC+8下午12时51分46秒,Rob Pike写道:

Dmitry Vyukov

unread,
Dec 2, 2014, 4:36:08 AM12/2/14
to Rob Pike, golang-dev
I have the execution tracer. Russ can you confirm that it is approved for 1.5?

The rest will probably conflict with the GC work.

I may also do some race detector related work, it should not conflict
with other work.

Nick Li

unread,
Dec 2, 2014, 4:36:26 AM12/2/14
to golan...@googlegroups.com
>more complete support for Android
Wow

On Tuesday, December 2, 2014 12:51:46 PM UTC+8, Rob Pike wrote:

rafa...@gmail.com

unread,
Dec 2, 2014, 4:47:51 AM12/2/14
to golan...@googlegroups.com
Thanks Rob Pike, I'm your fan!

minux

unread,
Dec 2, 2014, 5:06:16 AM12/2/14
to Rob Pike, golang-dev
On Mon, Dec 1, 2014 at 11:51 PM, Rob Pike <r...@golang.org> wrote:

Finally, new ports will include 64-bit Power PC, more complete support for Android, and possibly 64-bit ARM.

I want to upstream my Darwin/ARM port, and possibly also a port for Darwin/ARM64
depending on the progress of the Linux/ARM64 port.
(I just learnt that new Apps uploaded to the store must contain 64-bit code starting
Feb 2015, so only having Darwin/ARM port in Go 1.5 is unfortunately not enough
to write iOS apps anymore).

Aram Hăvărneanu

unread,
Dec 2, 2014, 6:23:36 AM12/2/14
to minux, Rob Pike, golang-dev
Yes, GOARCH=ARM64 is coming in 1.5. 1.4 went slower than expected
since hardware was late, and when it finally come it was mighty buggy.
But now I have my own ARM64 right next to me so I expect it to
progress much faster.

--
Aram Hăvărneanu

Brad Fitzpatrick

unread,
Dec 2, 2014, 10:21:23 AM12/2/14
to Rob Pike, golang-dev
I've always wanted to return to https://code.google.com/p/go/issues/detail?id=5147 (optimizing two+ map lookups with the same key).



beancin...@gmail.com

unread,
Dec 2, 2014, 10:28:38 AM12/2/14
to golan...@googlegroups.com
This is great; looking forward to the concurrent GC and other work in the Runtime.

A quick question about the scheduler: Given the work that's been done towards the runtime scheduler per Vyukov's work at https://docs.google.com/document/d/1TTj4T2JO42uD5ID9e89oa0sLKhJYD0Y_kqxDv3I3XMw/edit - can we look forward to either G to P, or P to M affinity (points 4, 5 in the Potential Further Improvements section)? This would be a boon for GOMAXPROCS > 1 performance.

Thank you all for your hard work,

Mo

Brendan Tracey

unread,
Dec 2, 2014, 10:46:09 AM12/2/14
to beancin...@gmail.com, golan...@googlegroups.com
Russ has stated elsewhere (issue 9129) that he doesn’t want to see big changes to the runtime while there are so many other changes already occurring in 1.5

beancin...@gmail.com

unread,
Dec 2, 2014, 10:56:02 AM12/2/14
to golan...@googlegroups.com, beancin...@gmail.com
That's fair; I'd like to know if these further improvements are on the roadmap/radar at all? At the moment, it's common knowledge that most applications will be most performant on GOMAXPROCS = 1, which is unfortunate, but understandable as the scheduler is still being improved.

henz...@gmail.com

unread,
Dec 2, 2014, 11:06:50 AM12/2/14
to golan...@googlegroups.com
All the pieces are falling into place, for me, concurrent GC, is the missing piece on the server side; that I might be able to use Go on android in a limited, NDK, fashion is a huge bonus.

I think that over time, people are letting go of their 'my language feature' phobia and are starting to realize the simple model that Go offers gives them an incredibly powerful, general purpose tool.  Eliminate excessive GC pauses, and we can add Java folks to the Python / Rubiest arriving at  golang Ellis Island.

Keep up the phenomenal work.

Michael Hudson-Doyle

unread,
Dec 2, 2014, 5:26:46 PM12/2/14
to Rob Pike, golan...@googlegroups.com
Rob Pike <r...@golang.org> writes:

> Go 1.4 is almost done; we just need to soak the new release candidate and,
> if it's OK, push the button. Once again it is time to start thinking about
> the next release, 1.5, which is scheduled for launch in 6 months.
>
> What will be in 1.5?
>
> The biggest visible change will be that repository is stored in Git, not
> Mercurial, but that has minimal bearing on the contents of the repository.
>
> Regarding the source itself, a few major changes are planned:
>
> First, the garbage collector in 1.5 will be fully concurrent, with minimal
> pauses. This work is well underway, and is described here
> <http://golang.org/s/go14gc>.
>
> Second, the compiler, assembler, linker, and runtime for 1.5 will be
> entirely in Go. some of this work is done already, some will be automated,
> some will be done by hand, but the goal is to eliminate C from the source
> tree so that—except for programs that use cgo—no C compiler is necessary to
> build or use Go. This was proposed <http://golang.org/s/go13compiler> a
> while back, but is finally happening.
>
> Finally, new ports will include 64-bit Power PC, more complete support for
> Android, and possibly 64-bit ARM.
>
> And to repeat what was said last time:
>
> Once the 1.4 release is done, we'll be open to new CLs but want to
> concentrate on these fundamental changes, even scheduling them so the code
> can move gracefully without introducing merge conflicts and other
> complexities. If anyone else has other wide-ranging, core work like this
> that would make sense for 1.5, please say so now so it can be scheduled as
> well.

Canonical (in practice, me) would like to work on building and
distributing Go code as shared libraries. I have a proposal at:

https://docs.google.com/document/d/1PxhXNhsdqwBjteW7poqv4Vf3PhtTmYuQKkOk_JNWeh0/edit?usp=sharing

This obviously has non-trivial interaction with the work to convert the
toolchain to go. I think Canonical would be happy for me to spend some
time helping with that work first. (I would also much rather work on a
Go toolchain than a C one!)

This does not cover the "loading Go code" use cases other people are
interested in, but it is a step on the way.

If you are not scared of Debian packaging details, I have another
document describing how the packaging of Go code as a shared library
might work:

https://docs.google.com/a/canonical.com/document/d/1XEVdhZynO5gO48w_jYa018arxdbY7emolTEcT94tj-U/edit#

Cheers,
mwh

>
> Over the next few weeks we'll work with you to sketch out the large-scale
> priorities for 1.5 and pencil in the sequence of changes to satisfy them.
>
> Please remember that the open source community is vital to the Go
> ecosystem. We can't thank you enough for your contributions.
>
> -rob, for the Go team
>

Josh Bleecher Snyder

unread,
Dec 2, 2014, 5:28:02 PM12/2/14
to Rob Pike, golan...@googlegroups.com
> Once the 1.4 release is done, we'll be open to new CLs but want to
> concentrate on these fundamental changes, even scheduling them so the code
> can move gracefully without introducing merge conflicts and other
> complexities. If anyone else has other wide-ranging, core work like this
> that would make sense for 1.5, please say so now so it can be scheduled as
> well.

Due to previous freezes, I have several months of queued and WIP
compiler CLs. (It's less than it sounds like.) They're not
wide-ranging, but I would like to start to get feedback so that I can
learn and improve them, even if they can't actually go in until later.

I've just returned from a long leave and I'll likely be away again for
significant stretches over the coming months, so urgency is low, but
I'd rather not wait until 1.6.

Also, Keith's call for help converting the G stack code to Go was a
lot of fun, at least for myself and for other contributors I have
talked to. I know that there's significant overhead when you open the
floodgates and that the flurry of activity ended up being somewhat
disorderly. Nevertheless, I hope that the team will keep an eye out
for opportunities to ask the community for help. There's a lot of
energy and excitement out here waiting to be tapped. :)

-josh

Anthony Martin

unread,
Dec 2, 2014, 7:44:20 PM12/2/14
to Josh Bleecher Snyder, Rob Pike, golan...@googlegroups.com
Josh Bleecher Snyder <josh...@gmail.com> once said:
> Also, Keith's call for help converting the G stack code to Go was a
> lot of fun, at least for myself and for other contributors I have
> talked to. I know that there's significant overhead when you open the
> floodgates and that the flurry of activity ended up being somewhat
> disorderly. Nevertheless, I hope that the team will keep an eye out
> for opportunities to ask the community for help. There's a lot of
> energy and excitement out here waiting to be tapped. :)

I agree with all of this.

Anthony

Anthony Martin

unread,
Dec 2, 2014, 7:49:50 PM12/2/14
to Rob Pike, golan...@googlegroups.com
Rob Pike <r...@golang.org> once said:
> Finally, new ports will include 64-bit Power PC, more complete support for
> Android, and possibly 64-bit ARM.

I will have the plan9/arm port ready for this cycle.
David has kindly volunteered to run the builder.

Anthony

zuxiong lin

unread,
Dec 2, 2014, 8:06:31 PM12/2/14
to golan...@googlegroups.com

mattn

unread,
Dec 2, 2014, 8:19:35 PM12/2/14
to golan...@googlegroups.com

Roger Pack

unread,
Dec 3, 2014, 9:44:06 AM12/3/14
to golan...@googlegroups.com


On Monday, December 1, 2014 9:51:46 PM UTC-7, Rob Pike wrote:

Go 1.4 is almost done; we just need to soak the new release candidate and, if it's OK, push the button. Once again it is time to start thinking about the next release, 1.5, which is scheduled for launch in 6 months.


What will be in 1.5?


The biggest visible change will be that repository is stored in Git, not Mercurial, but that has minimal bearing on the contents of the repository.


Thanks for doing this :)
-roger- 

Dave Cheney

unread,
Dec 4, 2014, 4:52:55 AM12/4/14
to Rob Pike, golang-dev
Proposal:

I would like to raise the suggestion of switching to the native Go
based DNS resolver in the net package for GOOS=linux only.

History:

https://code.google.com/p/go/source/detail?r=cc0f39d02e93

This commit on 20 April 2011 introduced a cgo based resolver and made
it the default, replacing the existing Go based dns resolver in the
net package.

The justifications for this change was to resolve unpleasant
interactions with the OSX firewall dialog boxes if the libc dns
resolver was not used. Some other minor justifications for this
change, possibly by me, were access to things like avaihis .local
domain on linux.

Outcome:

For linux, which this proposal is only concerned with, the results
have been problematic. Because of various latent bugs and limitations
in libc in current and older linux distros, Go users on linux have
found that dns resolution when building crawlers, or proxies, to name
two examples, remains problematic.

I believe that the use case of Go programmers writing crawlers and
other network intensive clients is valid, and in fact something Go
should excel at, and the continuing issues with libc's dns resolver on
linux spoils that promise.

I have not provided details of the issues raised over the years in the
issues tracker, irc, or mailing list, but if you care about this
proposal, you've probably got a passing familiarity with the problem.
I can find references and issues if it'll help.

Proposal:

For Go 1.5, for GOOS=linux only, I propose that we change the net
package to default to the pure Go dns resolver. An option for building
the net package _with_ libc's dns resolver should also be provided.

Thank you for your time.

Dave

Sebastien Binet

unread,
Dec 4, 2014, 5:53:58 AM12/4/14
to mattn, golan...@googlegroups.com
On Wed, Dec 3, 2014 at 2:19 AM, mattn <matt...@gmail.com> wrote:
> I'm waiting for execution mode.
>
> https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit

also wainting on this:
- it would allow "my" go-python package and people using it to write
their python extension modules in go - and provide another way to
sneak go into the mix,
- it would also enable interesting things for "my" go interpreter
(eg: loading/importing new native go packages from the interpreter
prompt)

and I am willing to give a hand for this to happen for 1.5.

-s

Ian Lance Taylor

unread,
Dec 4, 2014, 12:14:31 PM12/4/14
to Dave Cheney, Rob Pike, golang-dev
On Thu, Dec 4, 2014 at 1:52 AM, Dave Cheney <da...@cheney.net> wrote:
>
> For Go 1.5, for GOOS=linux only, I propose that we change the net
> package to default to the pure Go dns resolver. An option for building
> the net package _with_ libc's dns resolver should also be provided.

My concern is this: you are describing problems that happen in
relatively extreme cases to programs that should in principle be run
by people who understand networking and who work closely with the
people who develop the networking environment in which the programs
run. The cost of changing to the Go resolver is causing confounding
results to naïve developers working in complex network environments
maintained by other people. That is, Go programs run in environments
with custom configurations of /etc/nsswitch.conf will behave
strangely. I would usually prefer to make experts work a little bit
harder to get the right results than to cause confounding results for
naïve users, because experts have a greater ability to figure out what
the problem is.

If you buy that argument, I propose that we instead provide an easier
fallback for experts. Perhaps Yet Another Environment Variable to
prefer the Go resolver.

Ian

Brad Fitzpatrick

unread,
Dec 4, 2014, 12:17:31 PM12/4/14
to Ian Taylor, Rob Pike, Dave Cheney, golang-dev

Or use the native resolver on Linux if the resolve.conf is of the normal boring variety as it is almost everywhere?

That is, only fall back to libc if the host config suggests we need to.

Ian Lance Taylor

unread,
Dec 4, 2014, 12:31:23 PM12/4/14
to Brad Fitzpatrick, Rob Pike, Dave Cheney, golang-dev
On Thu, Dec 4, 2014 at 9:17 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
>
> Or use the native resolver on Linux if the resolve.conf is of the normal
> boring variety as it is almost everywhere?

Definitely a possibility but note that we would need to check
/etc/nsswitch.conf also.

Ian

Brad Fitzpatrick

unread,
Dec 4, 2014, 12:47:00 PM12/4/14
to Ian Lance Taylor, Rob Pike, Dave Cheney, golang-dev
On Thu, Dec 4, 2014 at 9:31 AM, Ian Lance Taylor <ia...@golang.org> wrote:
On Thu, Dec 4, 2014 at 9:17 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
>
> Or use the native resolver on Linux if the resolve.conf is of the normal
> boring variety as it is almost everywhere?

Definitely a possibility but note that we would need to check
/etc/nsswitch.conf also.

Right. Hadn't had coffee yet. :)

 

Dave Cheney

unread,
Dec 4, 2014, 3:57:47 PM12/4/14
to Ian Lance Taylor, Rob Pike, golang-dev
On Fri, Dec 5, 2014 at 4:14 AM, Ian Lance Taylor <ia...@golang.org> wrote:
> On Thu, Dec 4, 2014 at 1:52 AM, Dave Cheney <da...@cheney.net> wrote:
>>
>> For Go 1.5, for GOOS=linux only, I propose that we change the net
>> package to default to the pure Go dns resolver. An option for building
>> the net package _with_ libc's dns resolver should also be provided.
>
> My concern is this: you are describing problems that happen in
> relatively extreme cases to programs that should in principle be run
> by people who understand networking and who work closely with the
> people who develop the networking environment in which the programs
> run. The cost of changing to the Go resolver is causing confounding
> results to naïve developers working in complex network environments
> maintained by other people. That is, Go programs run in environments
> with custom configurations of /etc/nsswitch.conf will behave
> strangely. I would usually prefer to make experts work a little bit
> harder to get the right results than to cause confounding results for
> naïve users, because experts have a greater ability to figure out what
> the problem is.

I accept that argument and agree that dns issues are not the only
problem you'll have writing a high performance proxy. With that said,
the most common failure that people see is a failure to resolve names,
which is related to fd exhaustion (or sometimes no free fd's in the
first 1024) but doesn't return the error "dns resolution failed: file
descriptors exhausted", dns resolution succeeds, but returns no
results.

This error is entirely libc's fault, but because we us libc for dns
resolution, this becomes our problem.

Brad and others have improved the situation limiting the number of in
fly requests (1.2 I think) but the problem still isn't solved and then
helping people on the mailng list becomes

"oh, you want to do a _lot_ of dns queries, well, yeah, you need to
rebuild your program using go install -a -tags netgo std" which often
raises more support issues.

I understand there is resistance to this suggestion as it is aimed as
solving a problem which should only be attempted by experts, and I
totally agree with that, high volume crawling is harder than just
spawning a bunch of goroutines. With that said, writing a crawler is
one of the first things new Go programmers attempt, and leads to a
poor experience because "no host found" makes no sense when they can
prove that the host is resolvable.

Using the cgo resolver on OSX makes total sense, but on the balance of
the last few years, I don't think using the cgo resolver on linux as
the default pays for the cost and complexity in support load.

For the extreme cases, those should be the ones rebuilding the net
package with -tag netcgo (or something) to explicitly choose to get
into bed with libc.

> If you buy that argument, I propose that we instead provide an easier
> fallback for experts. Perhaps Yet Another Environment Variable to
> prefer the Go resolver.

Nah, we already have the -tag netgo workaround. I don't want to see
another environment variable.

> Ian

toffa...@gmail.com

unread,
Dec 4, 2014, 6:34:37 PM12/4/14
to golan...@googlegroups.com, ia...@golang.org, r...@golang.org


On Thursday, December 4, 2014 12:57:47 PM UTC-8, Dave Cheney wrote:
> If you buy that argument, I propose that we instead provide an easier
> fallback for experts.  Perhaps Yet Another Environment Variable to
> prefer the Go resolver.

Nah, we already have the -tag netgo workaround. I don't want to see
another environment variable.

All of the DNS functions are in the form:

if r, ok := cgoLookup(); !ok {
  r = goLookup()
}

And right now the cgoLookup functions just return true for the last value. It seems like a good compromise to change that to return a value read from the environment at startup.
Message has been deleted

Dave Cheney

unread,
Dec 4, 2014, 11:00:23 PM12/4/14
to toffa...@gmail.com, golang-dev, Rob 'Commander' Pike, ia...@golang.org

I'm not interested in adding another configuration variable.

Go works well with a sensible default, I just want to change that default for Linux.

--

Russ Cox

unread,
Dec 4, 2014, 11:21:50 PM12/4/14
to Dave Cheney, toffa...@gmail.com, golang-dev, Rob 'Commander' Pike, Ian Lance Taylor
I think the 'standard config' suggestion is a good one. I suggest that dnsclient_unix.go add a bit to the dnsConfig type saying whether it had anything nonstandard (that we didn't understand) in it. Then the code that decides to call cgo or not can look at that bit. It will have to read nsswitch.conf too, just to decide the bit. On OS X, the nonstandard bit should always be set (we must always try cgo first).

Please don't do anything that hard-codes a different behavior for Linux. There are other Unix systems.

Russ

Florian Weimer

unread,
Dec 5, 2014, 5:28:46 AM12/5/14
to Dave Cheney, Ian Lance Taylor, Rob Pike, golang-dev
On 12/04/2014 09:57 PM, Dave Cheney wrote:
> I accept that argument and agree that dns issues are not the only
> problem you'll have writing a high performance proxy. With that said,
> the most common failure that people see is a failure to resolve names,
> which is related to fd exhaustion (or sometimes no free fd's in the
> first 1024) but doesn't return the error "dns resolution failed: file
> descriptors exhausted", dns resolution succeeds, but returns no
> results.
>
> This error is entirely libc's fault, but because we us libc for dns
> resolution, this becomes our problem.

Is this on OS X because it uses select instead of poll (poll is broken
for UDP at the kernel level)?

--
Florian Weimer / Red Hat Product Security

Dave Cheney

unread,
Dec 5, 2014, 5:35:38 AM12/5/14
to Florian Weimer, Ian Lance Taylor, Rob Pike, golang-dev
Turns out that if you do your own DNS resolution the OS X firewall freaks out and punishes you with many modal dialogs.

Dave Cheney

unread,
Dec 5, 2014, 8:09:20 PM12/5/14
to Russ Cox, toffa...@gmail.com, golang-dev, Rob 'Commander' Pike, Ian Lance Taylor
Thanks for your reply Russ.

I was wrong to suggest that linux should be treated differently than
other OSs. This was motivated by the fact that linux appears to be the
most affected by this issue, but I appreciate that other *BSD's may
have their own, unknown, faults.

I'm not keen on making a program (using the net package) change its
behaviour based on the contents of /etc/nssswitch.conf when it is
moved from machine to machine, especially as the nssswitch.conf may
not be under the control of the programmer who is deploying their go
application.

Sure, you could argue that dev, test, stage and prod environments
should be identical, but this would still be a nasty gotcha if dns
resolution behaved differently between machines.

As a counter proposal, what about changing all the unix platforms,
*except* OSX, to use the go dns resolver? Again the option to enable
the cgo resolver would be via build tag.

Thank you for your time.

Liam

unread,
Dec 5, 2014, 8:24:31 PM12/5/14
to golan...@googlegroups.com


On Monday, December 1, 2014 8:51:46 PM UTC-8, Rob Pike wrote:

Second, the compiler, assembler, linker, and runtime for 1.5 will be entirely in Go.

The compiler proposal states: "Phase 3. Use some tools, perhaps derived from gofix and the Go oracle to split the compiler into packages"

Would one of those packages be accessible to a go app that wishes to compile go source, as an extension/plugin mechanism?

Russ Cox

unread,
Dec 5, 2014, 8:24:38 PM12/5/14
to Dave Cheney, toffa...@gmail.com, golang-dev, Rob 'Commander' Pike, Ian Lance Taylor
On Fri, Dec 5, 2014 at 8:09 PM, Dave Cheney <da...@cheney.net> wrote:
Thanks for your reply Russ.

I was wrong to suggest that linux should be treated differently than
other OSs. This was motivated by the fact that linux appears to be the
most affected by this issue, but I appreciate that other *BSD's may
have their own, unknown, faults.

I'm not keen on making a program (using the net package) change its
behaviour based on the contents of /etc/nssswitch.conf when it is
moved from machine to machine, especially as the nssswitch.conf may
not be under the control of the programmer who is deploying their go
application.

I understand that, but I think you're making too much of this. Wait until it happens and then we can stop dealing in abstractions.
  
Sure, you could argue that dev, test, stage and prod environments
should be identical, but this would still be a nasty gotcha if dns
resolution behaved differently between machines.

As a counter proposal, what about changing all the unix platforms,
*except* OSX, to use the go dns resolver? Again the option to enable
the cgo resolver would be via build tag.

Sorry, but no. I've run avahi on Linux and it should work with Go.

Russ

Dave Cheney

unread,
Dec 5, 2014, 8:33:43 PM12/5/14
to Russ Cox, toffa...@gmail.com, golang-dev, Rob 'Commander' Pike, Ian Lance Taylor
On Sat, Dec 6, 2014 at 12:24 PM, Russ Cox <r...@golang.org> wrote:
> On Fri, Dec 5, 2014 at 8:09 PM, Dave Cheney <da...@cheney.net> wrote:
>>
>> Thanks for your reply Russ.
>>
>> I was wrong to suggest that linux should be treated differently than
>> other OSs. This was motivated by the fact that linux appears to be the
>> most affected by this issue, but I appreciate that other *BSD's may
>> have their own, unknown, faults.
>>
>> I'm not keen on making a program (using the net package) change its
>> behaviour based on the contents of /etc/nssswitch.conf when it is
>> moved from machine to machine, especially as the nssswitch.conf may
>> not be under the control of the programmer who is deploying their go
>> application.
>
>
> I understand that, but I think you're making too much of this. Wait until it
> happens and then we can stop dealing in abstractions.

Ok.

Thanks for the discussion, I'll open an issue for 1.5 for your suggestion.

Michael MacInnis

unread,
Dec 8, 2014, 9:51:10 AM12/8/14
to golan...@googlegroups.com
For Go 1.5, I would like to make a second attempt at changes that would allow launching a process, on Unix, and having:

- the process join a specific process group;
- the process's group become the foreground process group.

(My initial attempt at these changes had an unfortunate and embarrassing bug. I apologize for any frustration this caused).

I would like to make further changes to allow signals to be ignored. Currently this is not possible in Go without using cgo. It is possible to catch a signal and do nothing but I don't believe this is the same as ignoring a signal (as others have stated: https://groups.google.com/forum/#!msg/golang-nuts/HAexIfRKbAg/548PCWR19nYJ).

I would also like to add the Go equivalents of the isatty, tcgetpgrp and tcsetpgrp functions to the Unix go.sys package.

I'm hoping to make these changes across all Unix platforms.

These changes will allow me to implement a job control shell in Go. I could be mistaken but I believe that these changes are also useful outside of this particular context.

I would be grateful for any assistance with these changes. Particularly from those who know more about Go or this area of Unix.

I don't think these qualify as wide-ranging changes but I hope it is still appropriate to mention them here.

Michael.

jbra...@gmail.com

unread,
Dec 9, 2014, 10:05:41 AM12/9/14
to golan...@googlegroups.com
I was looking to port the Plan9 compiler suite to a new architecture to provide support.  Would this work be fruitless since everything will soon be written in Go?  There is support for gccgo already which could be used for bootstrapping.  

-Jonathan

Aram Hăvărneanu

unread,
Dec 9, 2014, 12:20:22 PM12/9/14
to jbra...@gmail.com, golang-dev
The runtime has already been rewritten in Go, and the Plan 9-based C
compiler have been removed, see the dev.cc branch on mercurial or the
master branch in Git.

If you want to port Go to a new architecture, I suggest starting with
the Go toolchain, not Plan 9. Out of curiosity, what architecture is
this?

--
Aram Hăvărneanu

Brian Marete

unread,
Dec 9, 2014, 2:51:18 PM12/9/14
to golan...@googlegroups.com
On Tuesday, December 2, 2014 7:51:46 AM UTC+3, Rob Pike wrote:

Go 1.4 is almost done; we just need to soak the new release candidate and, if it's OK, push the button. Once again it is time to start thinking about the next release, 1.5, which is scheduled for launch in 6 months.


What will be in 1.5?


The biggest visible change will be that repository is stored in Git, not Mercurial, but that has minimal bearing on the contents of the repository.


Regarding the source itself, a few major changes are planned:


First, the garbage collector in 1.5 will be fully concurrent, with minimal pauses. This work is well underway, and is described here.


Second, the compiler, assembler, linker, and runtime for 1.5 will be entirely in Go. some of this work is done already, some will be automated, some will be done by hand, but the goal is to eliminate C from the source tree so thatexcept for programs that use cgo—no C compiler is necessary to build or use Go. This was proposed a while back, but is finally happening.


Finally, new ports will include 64-bit Power PC, more complete support for Android, and possibly 64-bit ARM.


And to repeat what was said last time:


Once the 1.4 release is done, we'll be open to new CLs but want to concentrate on these fundamental changes, even scheduling them so the code can move gracefully without introducing merge conflicts and other complexities. If anyone else has other wide-ranging, core work like this that would make sense for 1.5, please say so now so it can be scheduled as well.


Over the next few weeks we'll work with you to sketch out the  large-scale priorities for 1.5 and pencil in the sequence of changes to satisfy them.


Please remember that the open source community is vital to the Go ecosystem. We can't thank you enough for your contributions.


-rob, for the Go team



Above all, please keep Go simple and rapidly understandable. It is a small language in that sense, fitting all in the mind --  as it does now -- after a few hours or days of looking at it (for an experienced programmer). There is really no point in pulling in every exotic feature of C#, C++ or Java. Those languages are available for those who like generics and similar features.

Keep the language as it is. Any improvements in the runtime or std library are, of course, a welcome bonus.

BGM.

dum...@gmail.com

unread,
Dec 12, 2014, 1:19:26 AM12/12/14
to golan...@googlegroups.com
if android can be supported, that would be very very very great.

在 2014年12月2日星期二UTC+8下午12时51分46秒,Rob Pike写道:

var...@gmail.com

unread,
Dec 16, 2014, 10:40:21 AM12/16/14
to golan...@googlegroups.com
Guys, 

What about execution tracer by Dmitry Vyukov ?
It was promised to be included in 1.5 release.... 

Dmitry Vyukov

unread,
Dec 17, 2014, 2:45:42 AM12/17/14
to var...@gmail.com, golang-dev
I've already mailed most of the tracer changes. They seem to be going into 1.5.

Kevin Gillette

unread,
Dec 19, 2014, 2:55:21 PM12/19/14
to golan...@googlegroups.com
There are some minor transparent improvements I'd like to make (or see made) to the net/http and websocket packages, as well as fix a few inaccuracies/inconsistencies I've seen in various internal line-comments throughout the standard library.

One example of low-hanging fruit in the net/http package is a 3-way type switch in request.go's NewRequest function just to get at a `Len() int` method; just type asserting to an `interface { Len() int }` would cover the existing use case but also accommodate additional types. I suspect semantic collisions with sort.Interface to be unlikely.

I would also like to explore providing a way to opt-out of full buffering when using json.Encoder (https://github.com/golang/go/issues/7872#issuecomment-66094201), and allowing the `json:",string"` modifier to apply to []byte as well.


On Monday, December 1, 2014 9:51:46 PM UTC-7, Rob Pike wrote:

jothi...@gmail.com

unread,
Jan 9, 2015, 7:17:08 AM1/9/15
to golan...@googlegroups.com
Hello,

I’m thinking of using Go application in embedded system running in ARM. The major point that blocks us from using it is 


It is now marked for Go1.5. I see that it was earlier marked for go1.3 and then 1.4 and is now moved. 

It would be awesome if this issue is resolved with Go1.5 (or at least the size is reduced significantly to an acceptable level).

Hopefully as the compiler, assembler, linker are planned to be written in Go1.5 with Go, this size reduction could be achieved.

Best Regards,  
Joe.

小菜

unread,
Jan 11, 2015, 8:20:47 AM1/11/15
to golan...@googlegroups.com

Very interesting article!



在 2014年12月2日星期二 UTC+8下午12:51:46,Rob Pike写道:

Aram Hăvărneanu

unread,
Jan 27, 2015, 11:50:20 AM1/27/15
to Dave Cheney, Rob Pike, golang-dev
On Thu, Dec 4, 2014 at 10:52 AM, Dave Cheney <da...@cheney.net> wrote:
> Proposal:
>
> I would like to raise the suggestion of switching to the native Go
> based DNS resolver in the net package for GOOS=linux only.

Perhaps this is relevant:

https://sourceware.org/bugzilla/show_bug.cgi?id=15014
https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2015-0235
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776391
https://translate.google.com/translate?hl=en&sl=fr&tl=en&u=http%3A%2F%2Fwww.frsag.org%2Fpipermail%2Ffrsag%2F2015-January%2F005722.html

--
Aram Hăvărneanu

Brad Fitzpatrick

unread,
Jan 27, 2015, 11:51:30 AM1/27/15
to Dave Cheney, Rob Pike, golang-dev
More motivation for a Go-based DNS resolver: https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2015-0235


On Thu, Dec 4, 2014 at 1:52 AM, Dave Cheney <da...@cheney.net> wrote:
Proposal:

I would like to raise the suggestion of switching to the native Go
based DNS resolver in the net package for GOOS=linux only.

History:

https://code.google.com/p/go/source/detail?r=cc0f39d02e93

This commit on 20 April 2011 introduced a cgo based resolver and made
it the default, replacing the existing Go based dns resolver in the
net package.

The justifications for this change was to resolve unpleasant
interactions with the OSX firewall dialog boxes if the libc dns
resolver was not used. Some other minor justifications for this
change, possibly by me, were access to things like avaihis .local
domain on linux.

Outcome:

For linux, which this proposal is only concerned with, the results
have been problematic. Because of various latent bugs and limitations
in libc in current and older linux distros, Go users on linux have
found that dns resolution when building crawlers, or proxies, to name
two examples, remains problematic.

I believe that the use case of Go programmers writing crawlers and
other network intensive clients is valid, and in fact something Go
should excel at, and the continuing issues with libc's dns resolver on
linux spoils that promise.

I have not provided details of the issues raised over the years in the
issues tracker, irc, or mailing list, but if you care about this
proposal, you've probably got a passing familiarity with the problem.
I can find references and issues if it'll help.

Proposal:


For Go 1.5, for GOOS=linux only, I propose that we change the net
package to default to the pure Go dns resolver. An option for building
the net package _with_ libc's dns resolver should also be provided.

Thank you for your time.

Dave

David du Colombier

unread,
Jan 27, 2015, 11:56:43 AM1/27/15
to Brad Fitzpatrick, Dave Cheney, Rob Pike, golang-dev
Well, at least Go on Unix use getaddrinfo, not the deprecated gethostbyname.

--
David du Colombier

David du Colombier

unread,
Jan 27, 2015, 12:03:49 PM1/27/15
to Brad Fitzpatrick, Dave Cheney, Rob Pike, golang-dev
FWIW, POSIX 2001 discourages the use of gethostbyname
and it has been completely removed from POSIX 2008.

http://pubs.opengroup.org/onlinepubs/009695399/functions/gethostbyname.html

--
David du Colombier

Brad Fitzpatrick

unread,
Jan 27, 2015, 12:05:06 PM1/27/15
to David du Colombier, Dave Cheney, Rob Pike, golang-dev
Which API function it was isn't the point. The point is having a dependency on an unsafe language taints the whole program as potentially unsafe and exploitable.

David du Colombier

unread,
Jan 27, 2015, 12:08:13 PM1/27/15
to Brad Fitzpatrick, Dave Cheney, Rob Pike, golang-dev
> Which API function it was isn't the point. The point is having a dependency
> on an unsafe language taints the whole program as potentially unsafe and
> exploitable.

Yes, I was just clarifying that Go isn't directly affected by this
particular issue.
I agree with your point.

--
David du Colombier

David Crawshaw

unread,
Jan 27, 2015, 12:38:15 PM1/27/15
to Brad Fitzpatrick, Dave Cheney, Rob Pike, golang-dev
It seems systemd (which I have no experience with) now has a DNS
resolver available over dbus.

http://lwn.net/Articles/609740/
http://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html

I'm not sure it is widespread enough to matter yet, but a Go-based DNS
resolver could use it if present without cgo.

Russ Cox

unread,
Jan 28, 2015, 2:17:32 PM1/28/15
to David Crawshaw, Brad Fitzpatrick, Dave Cheney, Rob Pike, golang-dev
I already forked this into a separate thread about pure Go DNS. Please use that one.

Florian Weimer

unread,
Feb 9, 2015, 11:20:54 AM2/9/15
to golan...@googlegroups.com
On 01/27/2015 06:38 PM, David Crawshaw wrote:
> It seems systemd (which I have no experience with) now has a DNS
> resolver available over dbus.

It's caching behavior is quite broken. Writing DNS caches is very hard,
and the RFCs don't provide much guidance.

> I'm not sure it is widespread enough to matter yet, but a Go-based DNS
> resolver could use it if present without cgo.

It depends how many installations have something else besides “files
dns” listed for the hosts database in /etc/nsswitch.conf. If people use
NIS, LDAP or something proprietary to access host information, then a
pure Go approach is not likely to work. For the non-hosts database, I
really don't see a way around NSS, for this reason.

(Sorry, I couldn't find the Go DNS thread.)

Brad Fitzpatrick

unread,
Apr 16, 2015, 5:39:27 PM4/16/15
to Ian Lance Taylor, Rob Pike, Dave Cheney, golang-dev


On Thu, Dec 4, 2014 at 9:31 AM, Ian Lance Taylor <ia...@golang.org> wrote:
On Thu, Dec 4, 2014 at 9:17 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
>
> Or use the native resolver on Linux if the resolve.conf is of the normal
> boring variety as it is almost everywhere?

Definitely a possibility but note that we would need to check
/etc/nsswitch.conf also.


It needs more tests from more systems' nsswitch.conf files, but seems like it should work.
 

Reply all
Reply to author
Forward
0 new messages