fatal error: sweep increased allocation count, go1.9.x

[matthe...@gmail.com]

Hello,

I've encountered a crash with the signature "fatal error: sweep increased allocation count" that only appears for me in the go.1.9.x series, testing on linux/amd64. The memory problem appears to be triggered in this function:

func TruncatedAbsPathsForKind(the Kind, from AbsPoint, with Orientation) AbsPathSetMap {
        absmap := make(AbsPathSetMap)
        for movetype, paths := range RelPathMapForKind(the) {
                availablepaths := make(AbsPathSet)
                for path, _ := range paths {
                        availablepath := AbsPath{
                                Points: make([]AbsPoint, 0, len(*path)),
                        }
                        truncated := false
                        for _, point := range *path {
                                absfile := int8(from.File) + point.XOffset
                                if (absfile > 7) || (absfile < 0) {
                                        truncated = true
                                        break
                                }
                                var absrank int8
                                if with == White {
                                        absrank = int8(from.Rank) + point.YOffset
                                } else {
                                        absrank = int8(from.Rank) - point.YOffset
                                }
                                if (absrank > 7) || (absrank < 0) {
                                        truncated = true
                                        break
                                }
                                availablepath.Points = append(availablepath.Points, AbsPoint{File: uint8(absfile), Rank: uint8(absrank)})
                        }
                        availablepath.Truncated = truncated
                        if len(availablepath.Points) != 0 {
                                availablepaths[&availablepath] = struct{}{}
                        }
                }
                if len(availablepaths) != 0 {
                        absmap[movetype] = availablepaths
                }
        }
        return absmap
}

Taking the address of availablepath seems to cause a memory management problem - is there something I'm doing wrong here? The game does work when played manually so logically I believe this is correct, otherwise the moves would be wrong.

The program is a chess server and the crash only appears when doing a load test simulating clients concurrently playing more games than there are cores in my server (4) as fast as can be played by a computer. My concern is for hitting this crash when a significant number of real players would be playing near a point where a future version would need to start another instance.

A writeup of the bug is here with regular traces and GODEBUG=gccheckmark=1 traces: https://github.com/pciet/wichess/issues/19

And the code can be found here: https://github.com/pciet/wichess

The type definitions are here: https://github.com/pciet/wichess/blob/master/wichessing/path.go

Currently sets are represented with maps but I do have plans to change to slices as profiling shows map iteration taking significant time, but for now I am getting this logically complete implementation working before doing any optimization effort.

Any help is appreciated.

Thanks!

Matt

[Ian Lance Taylor]

On Thu, Nov 16, 2017 at 11:28 AM, <matthe...@gmail.com> wrote:
>
> I've encountered a crash with the signature "fatal error: sweep increased
> allocation count" that only appears for me in the go.1.9.x series, testing
> on linux/amd64. The memory problem appears to be triggered in this function:

Thanks for the report. Please open an issue for this at
https://golang.org/issue. Thanks.

Ian

[matthe...@gmail.com]

https://github.com/golang/go/issues/22781

I'll try to put together an easier failing case with less code and without the database. Thanks for the help.

Matt