weekly.2011-09-01
Nigel Tao
can update by running:
hg pull
hg update weekly
This weekly contains performance improvements and bug fixes.
The gc compiler now does escape analysis, which improves program performance
by placing variables on the call stack instead of the heap when it is safe to
do so.
The container/vector package is deprecated and will be removed at some point
in the future.
Other changes:
* archive/tar: support symlinks. (thanks Mike Rosset)
* big: fix nat.scan bug. (thanks Evan Shaw)
* bufio: handle a "\r\n" that straddles the buffer.
add openbsd.
avoid redundant bss declarations.
fix unused parameters.
fix windows/amd64 build with newest mingw-w64. (thanks Hector Chu)
* bytes: clarify that NewBuffer is not for beginners.
* cgo: explain how to free something.
fix GoBytes. (thanks Gustavo Niemeyer)
fixes callback for windows amd64. (thanks Wei Guangjing)
note that CString result must be freed. (thanks Gustavo Niemeyer)
* cov: remove tautological #defines. (thanks Lucio De Re)
* dashboard: yet another utf-8 fix.
* doc/codelab/wiki: fix Makefile.
* doc/progs: fix windows/amd64. (thanks Jaroslavas Počepko)
* doc/tmpltohtml: update to new template package.
* doc: emphasize that environment variables are optional.
* effective_go: convert to use tmpltohtml.
* exp/norm: reduced the size of the byte buffer used by reorderBuffer
by half by reusing space when combining.
a few minor fixes to support the implementation of norm.
added implementation for []byte versions of methods.
* exp/template/html: add some tests for ">" attributes.
added handling for URL attributes.
differentiate URL-valued attributes (such as href).
reworked escapeText to recognize attr boundaries.
* exp/template: moved from exp to the main tree.
* exp/wingui: made compatible with windows/amd64. (thanks Jaroslavas Počepko)
* flag: add Parsed, restore Usage.
* gc: add openbsd.
escape analysis.
fix build on Plan 9. (thanks Lucio De Re)
fix div bug.
fix pc/line table. (thanks Julian Phillips)
fix some spurious leaks.
make static initialization more static.
remove JCXZ; add JCXZW, JCXZL, and JCXZQ instructions. (thanks
Jaroslavas Počepko)
shuffle #includes.
simplify escape analysis recursion.
tweak and enable escape analysis.
* go/ast cleanup: base File/PackageExports on FilterFile/FilterPackage code.
adjustments to filter function.
fix ast.MergePackageFiles to collect infos about imports. (thanks
Sebastien Binet)
generalize ast.FilterFile.
* go/build: add test support & use in gotest.
separate test imports out when scanning. (thanks Gustavo Niemeyer)
* go/parser: fix type switch scoping.
fix type switch scoping.
* gob: explain that Debug isn't useful unless it's compiled in.
* gobuilder: increase log limit.
* godashboard: fix utf-8 in user names.
* godoc: first step towards reducing index size.
add dummy playground.js to silence godoc warning at start-up.
added systematic throttling to indexing goroutine.
fix bug in zip.go.
support for reading/writing (splitted) index files.
use virtual file system when generating package synopses.
* gofix: forgot to rename the URL type.
osopen: fixed=true when changing O_CREAT. (thanks Tarmigan Casebolt)
* goinstall: error out with paths that end with '/'. (thanks Tarmigan Casebolt)
report lack of $GOPATH on errors. (thanks Gustavo Niemeyer)
select the tag that is closest to runtime.Version.
* gotry: add missing $. (thanks Tarmigan Casebolt)
* http: add MaxBytesReader to limit request body size.
add file protocol transport.
adjust test threshold for larger suse buffers.
delete error kludge.
on invalid request, send 400 response.
return 413 instead of 400 when the request body is too large. (thanks
Dave Cheney)
support setting Transport's TLS client config.
* image/tiff: add a decode benchmark. (thanks Benny Siegert)
decoder optimization. (thanks Benny Siegert)
* image: add PalettedImage interface, and make image/png recognize it.
(thanks Jaroslavas Počepko)
* io: add TeeReader. (thanks Hector Chu)
* json: add struct tag option to wrap literals in strings.
calculate Offset for Indent correctly. (thanks Jeff Hodges)
fix decode bug with struct tag names with ,opts being ignored.
* ld: handle Plan 9 ar format. (thanks Lucio De Re)
remove duplicate bss definitions.
* libmach: support reading symbols from Windows .exe for nm. (thanks
Mateusz Czapliński)
* math: fix Pow10 loop. (thanks Volker Dobler)
* mime: ParseMediaType returns os.Error now, not a nil map.
media type formatter. (thanks Pascal S. de Kloe)
text charset defaults. (thanks Pascal S. de Kloe)
* misc/dashboard: remove limit for json package list.
* misc/emacs: refine label detection.
* net: add ParseMAC function. (thanks Paul Borman)
change the internal form of IPMask for IPv4. (thanks Mikio Hara)
disable "tcp" test on openbsd.
fix windows build. (thanks Alex Brainman)
join and leave a IPv6 group address, on a specific interface. (thanks
Mikio Hara)
make use of IPv4len, IPv6len. (thanks Mikio Hara)
move internal string manipulation routines to parse.go. (thanks Mikio Hara)
* os: disable Hostname test on OpenBSD.
fix WNOHANG Waitmsg. (thanks Gustavo Niemeyer)
* reflect: add Value.Bytes, Value.SetBytes methods.
* rpc: add benchmark for async rpc calls.
* runtime: add openbsd 386 defs.h.
add runtime support for openbsd 386.
add runtime· prefix to showframe.
ctrlhandler for windows amd64. (thanks Wei Guangjing)
fix stack cleanup on windows/amd64. (thanks Hector Chu)
fix void warnings.
go interface to cdecl calbacks. (thanks Jaroslavas Počepko)
handle string + char literals in goc2c.
make arm work on Ubuntu Natty qemu.
openbsd thread tweaks.
simplify stack traces.
speed up cgo calls. (thanks Alex Brainman)
use cgo runtime functions to call windows syscalls. (thanks Alex Brainman)
windows/amd64 callbacks fixed and syscall fixed to allow using it in
callbacks. (thanks Jaroslavas Počepko)
* strconv: put decimal on stack.
* spec: update section on Implementation Differences.
* syscall: SOMAXCONN should be 0x7fffffff at winsock2. (thanks
Yasuhiro Matsumoto)
add openbsd 386.
handle RTM_NEWROUTE in ParseNetlinkRouteAttr on Linux. (thanks Albert
Strasheim)
handle routing entry in ParseRoutingSockaddr on BSD variants. (thanks
Mikio Hara)
openbsd amd64 syscall support.
use the vdso page on linux x86 for faster syscalls instead of int
$0x80. (thanks Yuval Pavel Zholkover)
* template/parse: give if, range, and with a common representation.
* template: grammar fix for template documentation. (thanks Bill Neubauer)
range over channel.
remove else and end nodes from public view.
* test: put GOROOT/bin before all others in run.
* time: fix Plan 9 build. (thanks Fazlul Shahriar)
fix zone during windows test.
* type switches: test for pathological case.
* version.bash: update VERSION on -save if already present. (thanks
Gustavo Niemeyer)
* websocket: implements new version of WebSocket protocol. (thanks
Fumitoshi Ukai)
* windows/386: clean stack after syscall. (thanks Jaroslavas Počepko)
* xml: marshal "parent>child" tags correctly. (thanks Ross Light)
Apologies if we missed anyone in the list above. We appreciate all your help.
To see a full list of changes between this and the previous weekly,
after updating, run:
hg log -r weekly.2011-08-17:weekly.2011-09-01
Strewth,
Nigel.
Sammi Song
Nigel Tao
> test runtime/pprof
> TEST FAIL runtime/pprof
Can you also give me the results of "env|grep ^GO"?
Also, you did run src/all.bash, right?
Jan Mercl
* gc: add openbsd.
escape analysis.
Sammi Song
On 2011-09-01, at 2:51 AM, Nigel Tao wrote:
> On 1 September 2011 18:07, Sammi Song <songyun...@gmail.com> wrote:
>> test runtime/pprof
>> TEST FAIL runtime/pprof
>
> Can you also give me the results of "env|grep ^GO"?
[yunhuine@vm bots]$ env|grep ^GO
GOBIN=/home/yunhuine/go/bin
GOARCH=386
GOROOT=/home/yunhuine/go
GOOS=linux
>
> Also, you did run src/all.bash, right?
Yes, I ran ./all.bash
P.S
uname -a
Linux vm.qlist.ca 2.6.40-4.fc15.i686 #1 SMP Fri Jul 29 18:54:39 UTC 2011 i686 i686 i386 GNU/Linux
Sammi
Eleanor McHugh
>
> * gc: add openbsd.
I've just built weekly on an old 386 box I have lying around with OpenBSD 4.7 and everything builds fine. However all binaries created with the 8g toolchain are seg faulting.
I'll be upgrading to OpenBSD 4.9 later as it's long overdue anyway and will report back on whether this makes a difference.
Ellie
Eleanor McHugh
Games With Brains
http://feyeleanor.tel
----
if _, ok := reality.(Reasonable); !ok { panic(reality) }
Brandon Mercer
In fact I think you need a 5.0-ish snapshot for things to work. HTH,
Brandon
Brandon Mercer
allow for more open files for the user you're building/testing/running
as. HTH! Cheers,
Brandon
On Thu, Sep 1, 2011 at 11:15 AM, David Fleming <dcf...@gmail.com> wrote:
> Tried building default branch on a 5.0 - current (8/17 i386 snapshot)
> with kern.rthreads=1. Build succeeded, but tests failed on the http
> package. I've included the relevant portion of the build file. Please
> let me know if there's anything else i can do.
>
>
> test hash/adler32
> test hash/crc32
> test hash/crc64
> test hash/fnv
> test html
> test http
> TEST FAIL http
> gmake[1]: Entering directory `/home/david/go/src/pkg/http'
> gotest -test.short -test.timeout=120
> rm -f _test/http.a
> 8g -o _gotest_.8 chunked.go client.go cookie.go dump.go
> filetransport.go fs.go header.go lex.go persist.go request.go
> response.go reverseproxy.go server.go sniff.go status.go transfer.go
> transport.go cookie_test.go export_test.go header_test.go lex_test.go
> proxy_test.go range_test.go readrequest_test.go requestwrite_test.go
> response_test.go responsewrite_test.go
> rm -f _test/http.a
> gopack grc _test/http.a _gotest_.8
> 2011/09/01 10:31:25 http: Accept error: accept tcp 127.0.0.1:37143:
> too many open files
> 2011/09/01 10:31:25 http: Accept error: accept tcp 127.0.0.1:37143:
> too many open files
> 2011/09/01 10:31:25 http: Accept error: accept tcp 127.0.0.1:37143:
> too many open files
> 2011/09/01 10:31:25 http: Accept error: accept tcp 127.0.0.1:37143:
> too many open files
> 2011/09/01 10:31:25 http: Accept error: accept tcp 127.0.0.1:37143:
> too many open files
> .
> .
> (last line repeats)
>
Florian Uekermann
workarounds for incredibly slow strconv.Atof64.
David Fleming
David
David Fleming
Brandon Mercer
folks searching about the list could find it, this will simply be
another place they can search. :) Cheers,
Brandon
Han-Wen Nienhuys
> We've just tagged a new Go weekly, weekly.2011-09-01. As usual, you
> can update by running:
> hg pull
> hg update weekly
>
> This weekly contains performance improvements and bug fixes.
The 9/1 release gives the following in a test that runs fine on the
8/17 release:
011/09/01 18:25:15 saving hash 96c15c2bb2921193bf290df8cd85e2ba
panic: runtime error: invalid memory address or nil pointer
dereference [recovered]
panic: interface conversion: interface is runtime.errorString,
not gob.gobError
[signal 0xb code=0x1 addr=0x0 pc=0x40aa16]
goroutine 166 [running]:
gob.catchError(0xf84003e780, 0x7f792a354100)
/usr/local/google/home/hanwen/vc/go/src/pkg/gob/error.go:39 +0x67
----- stack segment boundary -----
bytes.(*Buffer).WriteString(0xf84003e720, 0x0, 0xf8401cc180,
0xf800000010, 0x0, ...)
/usr/local/google/home/hanwen/vc/go/src/pkg/bytes/buffer.go:109 +0x9d
gob.encString(0xf840033de0, 0xf840348a20, 0xf840180e30, 0x42a354501,
0xf840348a20, ...)
/usr/local/google/home/hanwen/vc/go/src/pkg/gob/encode.go:321 +0x95
gob.(*Encoder).encodeStruct(0xf84003e6e0, 0xf84003e720, 0xf840156100,
0xf840180e00, 0xf8401a5fe2, ...)
/usr/local/google/home/hanwen/vc/go/src/pkg/gob/encode.go:371 +0xea
gob._func_016(0xf8401772e8, 0x5052d4, 0x0, 0xf8403489f0, 0xf840180e00, ...)
..
(this is running the test suite of Termite.) - Full traces on request.
This is coming from a test that runs fine in isolation (-test.run
TestEndToEndNegativeNotify). I don't really know where to start
debugging or reducing this.
--
Han-Wen Nienhuys
Google Engineering Belo Horizonte
han...@google.com
Rob 'Commander' Pike
with 6g -s and see if the bug goes away? It's a broad test but would
provide a clue.
-rob
Han-Wen Nienhuys
--
Nigel Tao
> Is there a flag to pass to gotest that does this ?
You can pass "-s" to 6g, but I think that you'll want to rebuild your
entire tree with escape analysis disabled. Open src/cmd/gc/lex.c and
comment out the section that says:
// Phase 3b: escape analysis.
if(!debug['s'])
escapes();
Rob 'Commander' Pike
work either way.
-rob
Michael Jones
Jan Mercl
you can also just update Make.pkg, but it's about the same amount of
work either way.
Nigel Tao
> profile too short: [0x0 0x3 0x0 0x2710 0x0]
Hmm... is your computer really fast? src/pkg/runtime/pprof/pprof_test.go says:
// This loop takes about a quarter second on a 2 GHz laptop.
// We only need to get one 100 Hz clock tick, so we've got
// a 25x safety buffer.
for i := 0; i < 1000; i++ {
crc32.ChecksumIEEE(buf)
}
Does the test pass if you change 1000 to something bigger?
Dave Cheney
Sent from my iPad
Nigel Tao
> On 2011-09-01, at 2:51 AM, Nigel Tao wrote:
>> On 1 September 2011 18:07, Sammi Song <songyun...@gmail.com> wrote:
>>> test runtime/pprof
>>> TEST FAIL runtime/pprof
>>
>> Can you also give me the results of "env|grep ^GO"?
>
> [yunhuine@vm bots]$ env|grep ^GO
> GOBIN=/home/yunhuine/go/bin
> GOARCH=386
> GOROOT=/home/yunhuine/go
> GOOS=linux
I forgot to add that, as a data point, I've freshly synced to tip, and
all.bash runs fine on my GOARCH=386 GOOS=linux laptop.
Russ Cox
> May I please propose, if possible, to
> backport http://code.google.com/p/go/source/detail?r=3d378fb6c4dd to
> weekly.2011-09-01 and make it say weekly.2011-09-01.1?
Maintaining sub-weekly updates is probably overkill here.
The weekly snapshots are snapshots. It's great
to be on the cutting edge but you have to be willing
to tolerate a bug or two. If you need something
super stable, you should stay on the 'release' tags.
I believe that Han-Wen's bug is different from the
one fixed in that CL anyway, so it would just turn into
a succession of point releases.
That said, we very much appreciate the bug reports.
Please keep them coming. It helps us improve the
code before doing the next release.
Thanks.
Russ
Russ Cox
> You can pass "-s" to 6g, but I think that you'll want to rebuild your
> entire tree with escape analysis disabled. Open src/cmd/gc/lex.c and
> comment out the section that says:
>
> // Phase 3b: escape analysis.
> if(!debug['s'])
> escapes();
This change will break the compiler badly.
There are other code paths that the -s flag affects too.
If you are going to hard-code it in the 6g binary,
the way to change a flag is to put
debug['s'] = 1;
right before the ARGBEGIN line in gc/lex.c.
Or edit src/Make.inc's GC definition.
Russ
Dan Rzeppa
Russ Cox
> The weekly snapshots are snapshots. It's great
> to be on the cutting edge but you have to be willing
> to tolerate a bug or two. If you need something
> super stable, you should stay on the 'release' tags.
>
> I believe that Han-Wen's bug is different from the
> one fixed in that CL anyway
Having now found Han-Wen's bug, I want to reiterate this:
the weekly snapshots are snapshots. You should use
them to get an early look at what will be in a later release,
but you should not depend on them to be bug-free.
We maintain the 'release' branches as something you
can depend on for production use.
The bug that caused the error Han-Wen reported is more
serious than the exponential-time escape analysis one
that I fixed yesterday. It means that just the right kind of slice
literal will end up with uninitialized memory in the backing array,
which can lead to all sorts of interesting yet undesirable
behaviors. The conditions are a bit narrow but deterministic,
so if you get hit, you're going to get hit basically every time
(unless the stack happens to have zeros on it, which is rare).
If this is a problem, use weekly.2011-08-17 or, better,
use release.r59.
Thanks again for the help finding bugs.
Russ
Sammi Song
TEST FAIL runtime/pprof
make[1]: Entering directory `/home/yunhuine/go/src/pkg/runtime/pprof'
gotest -test.short -test.timeout=120
rm -f _test/runtime/pprof.a
8g -o _gotest_.8 pprof.go
rm -f _test/runtime/pprof.a
gopack grc _test/runtime/pprof.a _gotest_.8
--- FAIL: pprof_test.TestCPUProfile (42.57 seconds)
profile too short: [0x0 0x3 0x0 0x2710 0x0]
FAIL
gotest: "./8.out -test.short=true -test.timeout=120" failed: exit status 1
make[1]: *** [testshort] Error 2
make[1]: Leaving directory `/home/yunhuine/go/src/pkg/runtime/pprof'
make: *** [runtime/pprof.testshort] Error 1
Russ Cox
> Yes, that is fedora15 inside a virtualbox, the host os is mac mini.
The runtime/pprof support depends on setitimer's
ITIMER_PROF mode to send a SIGPROF for it to
handle. It is possible that for whatever reason
your configuration does not support that, so no
signal arrives, so no profile data is generated.
It might be interesting to run the 8.out under
strace -f -e setitimer so that you can see the call
to setitimer and whether it is returning some kind
of error.
Russ
Sammi Song
>>> TEST FAIL runtime/pprof
Brad Fitzpatrick
Scott Pakin
> The container/vector package is deprecated and will be removed at some point
> in the future.
represent a resizable sequence of elements?
Thanks,
-- Scott
peterGo
Yes, but! The golang.org site is updated to the latest release. For
the weekly, use this unofficial clone site: weekly.goneat.org. For the
tip, use: tip.goneat.org.
Peter
Brad Fitzpatrick
Scott Pakin
> a slice and append.
>
> var foo []int
> foo = append(foo, 1)
> foo = append(foo, 2)
> foo = append(foo, 3)
>
> http://golang.org/doc/effective_go.html#append
-- Scott
Eleanor McHugh
Or if you need something more heavyweight than container/vector I've written packages for Ruby-style arrays [0], Lisp-style cons lists [1] and linked lists [2].
Ellie
[0] http://github.com/feyeleanor/slices
[1] http://github.com/feyeleanor/chain
[2] http://github.com/feyeleanor/lists
Eleanor McHugh
Games With Brains
http://feyeleanor.tel
----
if _, ok := reality.(Reasonable); !ok { panic(reality) }