go1.8 bad type for XXX_extensions field

[si...@intesoft.net]

I came across one other thing that fails with go1.8 but works with go1

Unfortunately, I can't tell exactly where it's happening so I can't provide anything but the stack trace but I'm going to try cutting code out until it works to try and identify it and will update this when I do.

panic: bad type for XXX_extensions field: 

goroutine 17 [running]:

appengine_internal.(*context).Call(0xc00844bb80, 0x1c67389, 0xd, 0x1c65c6e, 0xc, 0x2a87e04c8af0, 0xc008416930, 0x2a87e04c8b28, 0xc0085f6780, 0xc00864d430, ...)

go/src/appengine_internal/api_prod.go:596 +0x321

google.golang.org/appengine/internal.Call(0x2a87e04c8928, 0xc008438ea0, 0x1c67389, 0xd, 0x1c65c6e, 0xc, 0x20a8620, 0xc008416930, 0x20a85e0, 0xc0085f6780, ...)

google.golang.org/appengine/internal/api_classic.go:119 +0x35f

google.golang.org/appengine/socket.DialTimeout(0x2a87e04c8560, 0xc0084cd530, 0x1c5dc8f, 0x3, 0x1c7341e, 0x19, 0x4a817ab29, 0x0, 0x0, 0x0)

google.golang.org/appengine/socket/socket_classic.go:93 +0x6df

google.golang.org/api/transport.init.1.func1.1(0x1c7341e, 0x19, 0x4a817ab29, 0xed1001587, 0xac5e28b, 0x20feac0, 0x4a817ab29)

google.golang.org/api/transport/dial_appengine.go:33 +0x6a

google.golang.org/grpc.WithDialer.func1.1(0x2a87e04c8928, 0xc008438ba0, 0x1c7341e, 0x19, 0x36, 0xc0086418a0, 0x12ba396, 0x2a87e04c2148)

google.golang.org/grpc/clientconn.go:229 +0xd1

google.golang.org/grpc/transport.dial(0x2a87e04c8928, 0xc008438ba0, 0xc00864c820, 0x1c7341e, 0x19, 0xc008641900, 0x12ba396, 0x2a87e045ec78, 0xa)

google.golang.org/grpc/transport/http2_client.go:128 +0x5a

google.golang.org/grpc/transport.newHTTP2Client(0x2a87e04c8928, 0xc008438ba0, 0x1c7341e, 0x19, 0x1aa2ee0, 0xc00864c880, 0x1c6aeba, 0x11, 0x0, 0x0, ...)

google.golang.org/grpc/transport/http2_client.go:166 +0xbf

google.golang.org/grpc/transport.NewClientTransport(0x2a87e04c8928, 0xc008438ba0, 0x1c7341e, 0x19, 0x1aa2ee0, 0xc00864c880, 0x1c6aeba, 0x11, 0x0, 0x0, ...)

google.golang.org/grpc/transport/transport.go:421 +0xa1

google.golang.org/grpc.(*addrConn).resetTransport(0xc00866c820, 0x0, 0x0, 0x0)

google.golang.org/grpc/clientconn.go:800 +0x3b3

google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc00866c820)

google.golang.org/grpc/clientconn.go:599 +0x34

created by google.golang.org/grpc.(*ClientConn).resetAddrConn

google.golang.org/grpc/clientconn.go:608 +0x715

[Steven Buss]

This feels like the same Context issue that cbro@ fixed (but which hasn't yet rolled out). If you can give me a code sample I can try it on our QA environment to verify.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsubscribe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/6c5f3a77-339c-499a-9159-dbd36e5fcf7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Guild]

I'm also seeing this on a fresh deployment. Luckily, this wasn't to production.

panic: bad type for XXX_InternalExtensions field: XXX_InternalExtensions
goroutine 21 [running]:
panic(0x1562bc0, 0xc010709970)
go/src/runtime/panic.go:443 +0x4e9
appengine_internal.(*aecontext).Call(0xc01074ff80, 0x1a32a28, 0x6, 0x1a2b808, 0x6, 0x2b16ab6df7e0, 0xc0106f0fc0, 0x2b16ab6df818, 0xc0106ff500, 0x0, ...)
go/src/appengine_internal/api_prod.go:574 +0x915
google.golang.org/appengine/internal.Call(0x2b16ab6da938, 0xc0105bde30, 0x1a32a28, 0x6, 0x1a2b808, 0x6, 0x2b16ab6da3e8, 0xc0106f0fc0, 0x2b16ab6da4c8, 0xc0106ff500, ...)
google.golang.org/appengine/internal/api_classic.go:125 +0x5f2
google.golang.org/appengine/search.moreSearch(0xc0107292c0, 0x0, 0x0)
google.golang.org/appengine/search/search.go:432 +0xa2a
google.golang.org/appengine/search.(*Iterator).fetchMore(0xc0107292c0)

... How to fix?

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Ben Guild]

Also, this is on Go 1.6 for us.

[jrum...@shift.com]

If it helps with debugging, we did three deploys to GAE. 

1. app_verison: go1
2. app_version: go1.6
3. app_version: go1.8

and all three had the same above error. 

[Steven Buss]

Thanks for the additional error reports and tracebacks. Looks like proto.Unmarshal is encountering a type it doesn't like and panicking. I'm looking in to this today and will hopefully track it down soon.

Simon: Does this only happen on 1.8 for you? Or are you also seeing this in fresh 1.6 deploys?

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsubscribe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/6c5f3a77-339c-499a-9159-dbd36e5fcf7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/f31d89a5-379a-489a-a84d-889f41d16266%40googlegroups.com.

[Ben Guild]

I’m seeing it on fresh 1.6 deploys. Since the issue is inside `appengine_internal` it’s pretty clear that whatever is happening is external to whatever we’re dealing with client-side, pre-deployment.

Just in case, I tried updating all of my local packages, but didn’t see any difference. Luckily, existing deployments aren’t affected, but it’d be great if in the future you guys could give us a way to target a stable version rather than just releasing these changes. Otherwise, we have to maintain extra prior deployments/versions just in case and that could get messy, especially if these changes appear mid-deployment in a situation where there are multiple modules that interact with shared data structures.

[si...@intesoft.net]

Sorry for not getting back, I ended up busy with other projects

Yes, it's only happening on 1,8 - if I switch back to 1.6 then it works fine (no code changes other than the version in app.yaml).

I did start trying to cut out some code to narrow it down to what might be causing it but it was a chunk that's a bit involved so wasn't easy to do. I only started on this new piece I added though.

I'll try and get further with it today and let you know if I can help focus in on anything.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/6c5f3a77-339c-499a-9159-dbd36e5fcf7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Guild]

Yep. App Engine Search is down in production because of whatever this is in `appengine_internal`. — Hope nobody needs to deploy or did deploy anything critical. :(

[Steven Buss]

We were in the process of rolling out a new toolchain for the go compiler, which means we do a rolling update to the compiler jobs in progressively larger stages. It looks like the new compiler toolchain contains a type mismatch with our internal and external proto libs. We will halt this rollout and roll back the experimental compiler.

When you deploy your app you will get routed to a random compiler service. If you can't wait for the rollback to complete (shouldn't be more than an hour or two), I suggest you attempt another deploy; you may be routed to a compiler server without this bug.

[Ben Guild]

Cool, thanks for the update and explanation. That totally makes sense, just wish it hadn’t happened. Good to know that in the future we can potentially do “deploy roulette” or something if there is a diff.

[Steven Buss]

Rollback has completed. Please re deploy your apps if they were affected.

[Ronoaldo José de Lana Pereira]

We just did a deploy of our prod app and that particular deploy started to crash with the same erros!

This is using the same tools we always used! Go SDK (outside Gcloud) version . Same deployments made yesterday were not affected by this bug:

$ goapp version:

go version go1.6.3 (appengine-1.9.48) linux/amd64

$ gcloud version

Google Cloud SDK 155.0.0

bq 2.0.24

core 2017.05.10

gcloud 

gsutil 4.26

Any work-around or hotfix are appreciated, as we now are unable to make new deployments!

2017-07-20 14:28 GMT-03:00 Ben Guild <b...@benguild.com>:

I’m seeing it on fresh 1.6 deploys. Since the issue is inside `appengine_internal` it’s pretty clear that whatever is happening is external to whatever we’re dealing with client-side, pre-deployment.

Just in case, I tried updating all of my local packages, but didn’t see any difference. Luckily, existing deployments aren’t affected, but it’d be great if in the future you guys could give us a way to target a stable version rather than just releasing these changes. Otherwise, we have to maintain extra prior deployments/versions just in case and that could get messy, especially if these changes appear mid-deployment in a situation where there are multiple modules that interact with shared data structures.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/BDE22719-9DE2-4AF8-B9B0-87D95C53F89B%40benguild.com.

For more options, visit https://groups.google.com/d/optout.

--

Ronoaldo Pereira

ronoaldo.com

[ke...@picatic.com]

I just started deploying to Standard in the Go1.8 an hour ago and I'm seeing this same error.
Not sure if that means the rollback worked or it wasn't rollback far enough.

Hope this helps.

-Kevin

[Steven Buss]

The rollback was because this issue started showing up in Go 1.6. It has, unfortunately, been present in 1.8 for several weeks. I'm working on root causing this and writing a fix this week.

--

You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/22e9c26d-435d-4491-9274-878fa81eb112%40googlegroups.com.

[Karan Misra]

Getting a similar error when deploying our Go 1.6 stable app with Go 1.8:

panic: bad type for XXX_extensions field: goroutine 499 [running]: appengine_internal.(*context).Call(0xc008627d00, 0x1f99e11, 0xd, 0x1f9721d, 0xc, 0x2a95b4eaa5b0, 0xc0087b0fc0, 0x2a95b4eaa5e8, 0xc00868d380, 0xc008adbdd0, ...) go/src/appengine_internal/api_prod.go:596 +0x321 google.golang.org/appengine/internal.Call(0x2a95b4c1bab8, 0xc008ad1320, 0x1f99e11, 0xd, 0x1f9721d, 0xc, 0x286b260, 0xc0087b0fc0, 0x286b220, 0xc00868d380, ...) google.golang.org/appengine/internal/api_classic.go:125 +0x35f google.golang.org/appengine/socket.DialTimeout(0x2a95b4c1b430, 0xc0093bdda0, 0x1f8a309, 0x3, 0x1fb2e1e, 0x19, 0x4a817ad59, 0x0, 0x0, 0x0) google.golang.org/appengine/socket/socket_classic.go:93 +0x6df google.golang.org/api/transport.init.1.func1.1(0x1fb2e1e, 0x19, 0x4a817ad59, 0xed10d78bf, 0xc03b6a38ec, 0x28ed9a0, 0x4a817ad59) google.golang.org/api/transport/dial_appengine.go:33 +0x6a google.golang.org/grpc.WithDialer.func1.1(0x2a95b4c1bab8, 0xc008ad1200, 0x1fb2e1e, 0x19, 0x1bac860, 0xc8e60, 0xc008c91e00, 0xc008d23850) google.golang.org/grpc/clientconn.go:240 +0xd1 google.golang.org/grpc/transport.dial(0x2a95b4c1bab8, 0xc008ad1200, 0xc008adb710, 0x1fb2e1e, 0x19, 0xc008d238c0, 0x107c71e, 0xc0091a0340, 0xc008709340) google.golang.org/grpc/transport/http2_client.go:122 +0x5a google.golang.org/grpc/transport.newHTTP2Client(0x2a95b4c1bab8, 0xc008ad1200, 0x1fb2e1e, 0x19, 0x1bf7e40, 0xc008adb740, 0x1fa0de5, 0x11, 0x0, 0x0, ...) google.golang.org/grpc/transport/http2_client.go:160 +0xbf google.golang.org/grpc/transport.NewClientTransport(0x2a95b4c1bab8, 0xc008ad1200, 0x1fb2e1e, 0x19, 0x1bf7e40, 0xc008adb740, 0x1fa0de5, 0x11, 0x0, 0x0, ...) google.golang.org/grpc/transport/transport.go:505 +0xae google.golang.org/grpc.(*addrConn).resetTransport(0xc009349dc0, 0x0, 0xc0089b4d20, 0x102945f) google.golang.org/grpc/clientconn.go:862 +0x3b2 google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc009349dc0) google.golang.org/grpc/clientconn.go:634 +0x34 created by google.golang.org/grpc.(*ClientConn).resetAddrConn google.golang.org/grpc/clientconn.go:643 +0x786

[Steven Buss]

This is a known issue with the Go 1.8 toolchain that I'm still working to fix. My apologies for the long turnaround.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsubscribe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/22e9c26d-435d-4491-9274-878fa81eb112%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/9184dbe0-8435-41c7-8eb1-c8af4fed65ed%40googlegroups.com.

[thomas....@gmail.com]

Any update on this ?

Attached is a stripped down snippet that will consistently trigger this panic.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/22e9c26d-435d-4491-9274-878fa81eb112%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Steven Buss]

I was perhaps a little optimistic with the release timeline. I have fixed the issue but I'm waiting in line for a release.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsubscribe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/22e9c26d-435d-4491-9274-878fa81eb112%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsubscribe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/9184dbe0-8435-41c7-8eb1-c8af4fed65ed%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/2d72286d-e2a8-45bd-a7d5-018a294ec76a%40googlegroups.com.

[namu...@gmail.com]

the issue has been fixed now, thanks.