"Sending on a closed channel will cause a panic"

[Brendan Tracey]

The above quote is from the go tour. It makes sense, but I'm wondering if it's the whole story. My code has a nil channel dereference, and I think the the cause is that the receiver closes the channel and not the sender. However, there is one receive per send, so I shouldn't be ever writing to the closed channel. Is it possible there's a panic from a nil channel even if there isn't an actual send? Secondly, assuming the above is true, in my current model, there is an input channel and a result channel. I launch a bunch of infinite loop go-routines, who wait for something to be sent on the input channel. When there is something to receive, they process it, and send it back on the result channel. Given that there are some number of goroutines who can send on this channel, what is the proper way to close it? It seems like a quit signal to one of them would cause the same issue in the other ones. Is this just a bad model? Should I have a channel of channels that talk to the other goroutines?

I think this code is functionally equivalent to my code, http://play.golang.org/p/J_rZ2NaurT  , but it works fine in the playground. My code is crashing on close(net.trainResultCh). Help would be appreciated, thanks

[Brendan Tracey]

Running go 1.0.3 on darwin, 

actual test script  http://play.golang.org/p/cOhsuv6NLy

actual error

mygo$ go test -v -run TestParLossAndDeriv  nnet

=== RUN TestParLossAndDeriv-8

In close goroutines

Closed trainInputCh

panic: runtime error: invalid memory address or nil pointer dereference

[signal 0xb code=0x1 addr=0x0 pc=0x31fed]

goroutine 4 [running]:

nnet._func_002(0x42289330, 0x0)

/Users/brendan/Documents/mygo/src/nnet/par.go:64 +0x78

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

goroutine 1 [chan receive]:

testing.RunTests(0x2000, 0x2066e8, 0x800000008, 0x1, 0x1f84a0, ...)

/usr/local/go/src/pkg/testing/testing.go:369 +0x7b8

testing.Main(0x2000, 0x2066e8, 0x800000008, 0x209a68, 0x0, ...)

/usr/local/go/src/pkg/testing/testing.go:304 +0x7a

main.main()

nnet/_test/_testmain.go:57 +0x91

goroutine 2 [syscall]:

created by runtime.main

/usr/local/go/src/pkg/runtime/proc.c:221

goroutine 3 [runnable]:

syscall.Syscall()

/usr/local/go/src/pkg/syscall/asm_darwin_amd64.s:34 +0x61

syscall.Write(0x1, 0x422baf80, 0x8000000014, 0x4227c7d0, 0x0, ...)

/usr/local/go/src/pkg/syscall/zsyscall_darwin_amd64.go:1279 +0x78

os.(*File).write(0x42289008, 0x422baf80, 0x8000000014, 0x0, 0x0, ...)

/usr/local/go/src/pkg/os/file_unix.go:188 +0x69

os.(*File).Write(0x42289008, 0x422baf80, 0x8000000014, 0x422c0101, 0x0, ...)

/usr/local/go/src/pkg/os/file.go:139 +0x83

fmt.Fprintln(0x422993c0, 0x42289008, 0x442268e00, 0x100000001, 0x42289008, ...)

/usr/local/go/src/pkg/fmt/print.go:280 +0x94

fmt.Println(0x442268e00, 0x100000001, 0x736f6c4300000013, 0x0, 0x0, ...)

/usr/local/go/src/pkg/fmt/print.go:289 +0x7b

nnet.(*Net).CloseGoroutines(0x422b9000, 0x17314c)

/Users/brendan/Documents/mygo/src/nnet/par.go:27 +0x134

nnet.TestParLossAndDeriv(0x422ac180, 0x8be1920)

/Users/brendan/Documents/mygo/src/nnet/nnet_test.go:594 +0x3dc

testing.tRunner(0x422ac180, 0x206778, 0x0, 0x0)

/usr/local/go/src/pkg/testing/testing.go:292 +0x6f

created by testing.RunTests

/usr/local/go/src/pkg/testing/testing.go:368 +0x795

goroutine 5 [running]:

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

goroutine 6 [running]:

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

goroutine 7 [running]:

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

goroutine 8 [running]:

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

goroutine 9 [running]:

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

goroutine 10 [running]:

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

goroutine 11 [running]:

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

panic: runtime error: invalid memory address or nil pointer dereference

[signal 0xb code=0x1 addr=0x0 pc=0x31fed]

goroutine 11 [running]:

nnet._func_002(0x42289330, 0x0)

/Users/brendan/Documents/mygo/src/nnet/par.go:64 +0x78

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

panic: runtime error: invalid memory address or nil pointer dereference

[signal 0xb code=0x1 addr=0x0 pc=0x31fed]

goroutine 5 [running]:

nnet._func_002(0x42289330, 0x0)

/Users/brendan/Documents/mygo/src/nnet/par.go:64 +0x78

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

panic: runtime error: invalid memory address or nil pointer dereference

[signal 0xb code=0x1 addr=0x0 pc=0x31fed]

goroutine 9 [running]:

nnet._func_002(0x42289330, 0x0)

/Users/brendan/Documents/mygo/src/nnet/par.go:64 +0x78

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

panic: runtime error: invalid memory address or nil pointer dereference

[signal 0xb code=0x1 addr=0x0 pc=0x31fed]

goroutine 8 [running]:

nnet._func_002(0x42289330, 0x0)

/Users/brendan/Documents/mygo/src/nnet/par.go:64 +0x78

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

panic: runtime error: invalid memory address or nil pointer dereference

[signal 0xb code=0x1 addr=0x0 pc=0x31fed]

goroutine 6 [running]:

nnet._func_002(0x42289330, 0x0)

/Users/brendan/Documents/mygo/src/nnet/par.go:64 +0x78

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

panic: runtime error: invalid memory address or nil pointer dereference

[signal 0xb code=0x1 addr=0x0 pc=0x31fed]

goroutine 10 [running]:

nnet._func_002(0x42289330, 0x0)

/Users/brendan/Documents/mygo/src/nnet/par.go:64 +0x78

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

panic: runtime error: invalid memory address or nil pointer dereference

[signal 0xb code=0x1 addr=0x0 pc=0x31fed]

goroutine 7 [running]:

nnet._func_002(0x42289330, 0x0)

/Users/brendan/Documents/mygo/src/nnet/par.go:64 +0x78

created by nnet.(*Net).launchGoroutines

/Users/brendan/Documents/mygo/src/nnet/par.go:75 +0x117

exit status 2

FAIL nnet 0.134s

[Ian Lance Taylor]

On Thu, Apr 11, 2013 at 9:29 AM, Brendan Tracey
<tracey....@gmail.com> wrote:
> The above quote is from the go tour. It makes sense, but I'm wondering if
> it's the whole story. My code has a nil channel dereference, and I think the
> the cause is that the receiver closes the channel and not the sender.

I don't know what is happening in your program, but note that a closed
channel is not a nil channel. Closing a channel does not make it
become nil.

Ian

[Brendan Tracey]

Huh. Thanks. Adding some more print statements, http://play.golang.org/p/EKcz_87g7K

the code seems to be failing at the second "fmt.Println(net.trainResultCh)".

Trying to only close the sending channels http://play.golang.org/p/QE84xJZ0pt , fails trying to print predictInputCh the second time. I don't understand why closing trainInputCh would cause the other channels to become bad memory

[Dan Kortschak]

What do you mean failing at the second print? After, before or during? (I can't see how that code would panic during). Seeing CloseGorourtines would help.

[Brendan Tracey]

It had been during (or possibly concurrent with?), but after moving to go1.1beta2, that appears to no longer be the case.

Now I get a runtime error at the end of CloseRoutines. 

This is code that I think is functionally equivalent to the full code, but it clearly isn't, because this code runs successfully, while the full code does not

http://play.golang.org/p/qUASZyV6Lb

I just made a bunch of methods public and tried to run the script. I'm still getting a nil pointer dereference error, but the goroutine output is different, so I'm going to keep hunting

[Dan Kortschak]

On Thu, 2013-04-11 at 15:38 -0700, Brendan Tracey wrote:
> It had been during (or possibly concurrent with?), but after moving
> to
> go1.1beta2, that appears to no longer be the case.
> Now I get a runtime error at the end of CloseRoutines.
>

Can you paste the panic? and preferably also a link to the actual code?

[Brendan Tracey]

I've isolated the error (sort of)

Code A:

http://play.golang.org/p/cTsntN4Pvd

Code B  (Snippet from the test script)

http://play.golang.org/p/ue7br78zOA

Code A runs successfully, Code B has a runtime error. I do not see the difference between these two scripts, aside from the fact that one is being run with go test and the other with go run.

Here is the output from running the scripts

http://play.golang.org/p/5X4j7eV-hG

The code is not online at the moment. I could copy and paste the code for you if you'd like, otherwise I could try to get it online. I have an account on GitHub, but haven't used it before.  I'd like to get it on github eventually, but it might take a bit.

Thank you very much for your help, I really appreciate it.

[Dave Cheney]

Brendan, please try to make the complete code available. These
snippets cannot be run as is, so we cannot being to reproduce your
problem.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Brendan Tracey]

Will do, I just wasn't sure if it was preferred that I give a list of play.golang.org  links or if I should upload it to github

[Dave Cheney]

I think the tool is less important that a runnable example.

[Dan Kortschak]

On Thu, 2013-04-11 at 17:00 -0700, Brendan Tracey wrote:
> Will do, I just wasn't sure if it was preferred that I give a list of
> play.golang.org links or if I should upload it to github
>

Context is a wonderful thing. The playground is good when a minimal
example can be played with there, not so much when it's more complicated
than that. Even just a gist of the relevant files would do.

[Brendan Tracey]

Yes. I agree it's important.Sorry for the delay, I was trying to condense the files into something more manageable. If you'd like, I'd be happy to spend more time cleaning the code up. I would try to post a smaller example, but I did the best I could above, and it does not reproduce the error.

Main package  nnet

http://play.golang.org/p/eRRoKT_1GK  nnet.go

http://play.golang.org/p/vooYHLrwJS    other.go

http://play.golang.org/p/Xw7jY6oEdx   par.go

http://play.golang.org/p/RGp6SnGzue  nnet_test.go

Sub-package scale

http://play.golang.org/p/Q-S0dzR9YA   scale.go

Caller script (also shared above)

http://play.golang.org/p/cTsntN4Pvd   nettest.go

Thanks again

[Dan Kortschak]

When you close the goroutines you close the training and input
prediction channels when means those cases can proceed. This gives you p
or t == nil.

func (net *Net) LaunchGoroutines(nRoutines int) {
net.numGoroutines = nRoutines
net.createGoroutineMemory()
net.trainInputCh = make(chan *trainStruct, nRoutines+2)
net.trainResultCh = make(chan *trainStruct, nRoutines)
net.predictInputCh = make(chan *predictStruct, 10)
net.predictResultCh = make(chan *predictStruct, nRoutines)
for i := 0; i < nRoutines; i++ {
go func() {
for {
select {
case t := <-net.trainInputCh:
t.loss = net.seqLossAndDerivative(
t.inputs,
t.trueOutputs,
t.derivative,
t.tmpMemory,
)
net.trainResultCh <- t
case p := <-net.predictInputCh:
net.seqPredict(p.inputs, p.predictions, p.tmpMemory)
net.predictResultCh <- p
}
}
}()
}
}

func (net *Net) CloseGoroutines() {
close(net.trainInputCh)
close(net.predictInputCh)
}

func TestLaunchAndCloseGoroutines(t *testing.T) {
net := &DefaultNet
err := net.Create("Regression", 5, 5, 6, 5)
if err != nil {
t.Errorf("Error during create in test launch and close")
}
net.LaunchGoroutines(runtime.NumCPU())
net.CloseGoroutines()
}

If you want to use this approach you'll need to have a specific quit
chan that you close and then clean up after the goroutine has sort
itself out. Perhaps look at launchpad.net/tomb for ideas about this.

This was harder than it should have been for one simple reason: !go fmt.
If you go fmt the code you run and the code you publish it's very easy
to correlate the two.

As an aside, I'd recommend the use of table driven tests where possible.

[Dan Kortschak]

The reason you weren't seeing the error on the standalone was that main
was completing before the goroutine had a chance to receive the close.

http://play.golang.org/p/Fht0HN68d3 Remove the select to feel less panicy.

[Brendan Tracey]

The code on my computer is formatted with gofmt. I didn't realize that the formatting didn't translate over. I'm sorry, I should have paid more attention.

I still don't understand though. Now that you point it out, I understand that I'm not actually closing the goroutines, because there's still no way for them to exit. However, as Ian points out above, channels don't become nil when you close them, so I don't understand what is nil and being dereferenced. Thanks for the pointer on table driven tests. I know my testing script isn't good at the moment, but I hadn't decided on a good way to do it. I tried having a quit channel before, but was getting "all goroutines are asleep" panics, and then things started working in the main script, so I thought that wasn't the problem and took it out for simplicity. I'll look at tomb. 

[Dan Kortschak]

This shows what is nil (the values passed via the channel):
http://play.golang.org/p/L4j0rauhJt

This is how you can terminate the goroutines:
http://play.golang.org/p/91JBoQwS-A

[Brendan Tracey]

Oh, I see now. I had forgotten "Receiving from a closed channel always succeeds, immediately returning the element type's zero value." 

Thanks for the help. Sorry that was such an ordeal.

[Dan Kortschak]

Looking more at the code, where you take a string to determine network
type it makes more sense to accept an interface { Activate(float64)
float64; DActivateDSum(float64, float64) float64 } since that's what
you're defining.

[Brendan Tracey]

I thought about that, and I'm not sure what the right answer is. You'll notice that the actual neurons of the net are implemented as an interface with a custom input argument (NeuronActivator), and there are a number of possible good ones there. The finalLayerActivator is just for transforming the final hidden layer outputs into a prediction. For regression, that's a real number and so shouldn't be touched, but for classification that's a number between 0 and 1 so it should be piped through a sigmoid. I guess there's no harm in keeping it as an interface, and then defining two variables DefaultRegression and DefaultClassification, and then if people want they can change it to be something else.

Thanks for the input