Go version

[Elliott Stoneham]

Hi,

I've been experimenting with creating Go apps on both AppScale and AppEngine. 

I notice that the Go version is different between them, with AppEngine using go1.6 but AppScale using go1.4.

Are there plans to bring the two versions into line?

Also, perhaps related, the AppScale Go app creation process requires a layout of Go code that is different from AppEngine.

This is because included go packages are not automatically copied by "tar --exclude='app.tar.gz' -czvf app.tar.gz *".

This makes AppScale very awkward to use, is there a tool available to automate this process?

Thanks. 

[Elliott Stoneham]

By way of explanation, Go 1.6 introduced the ability to vendor go package dependencies. See the extract below of the AppEngine release notes:

March 24, 2016 - Version 1.9.35

App Engine notes

• App Engine Managed VMs is renamed to App Engine flexible environment.
• Fixes trace timestamps to match log timestamps.

Go runtime notes

• This release is based on Go 1.6.
• The SDK now includes support for vendoring external dependencies. See the go command documentation for more details.

 

[Chris Donati]

Hi Elliott,

I think it will be fairly easy to update AppScale's Go support to 1.6.

However, I'm new to vendoring in Go, and I am having trouble testing it with the SDK. It cannot find my vendored package, but it seems to just be looking in GOROOT and GOPATH.

Would you mind pointing me to an example application that uses the vendor directory and works with the latest SDK?

Thanks,

Chris

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.
To post to this group, send email to appscale_community@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Elliott Stoneham]

Many thanks for your prompt response Chris.

Please find a working (at least on app engine, both locally and online) example at https://github.com/elliott5/sample-apps/tree/master/go

Because of the way that appengine handles vendoring, the application consists of two directories:

- "vendortest" contains the app.yaml and a stub application, which just includes "vendortestcode/app".

- "vendortestcode" contains both the "vendor" subdirectory and an "app" subdirectory with the actual code in it.

This has been a learning exercise for me too, eventually discovering how to do it from http://stackoverflow.com/questions/39948027/how-do-i-make-vendoring-work-with-google-app-engine

Note that the include path for the "app" directory in the stub application points to "appscale/sample-apps" rather than "elliott5/sample-apps" so I will send that repository a pull request.

I look forward to hearing how you get on.

Thanks,

Elliott

[Chris Donati]

Thank you for the clear sample code. It works for me, and I can confirm that it doesn't yet work on AppScale.

It looks like the SDK uses an included `go-app-builder` binary to discover the location of dependencies that need to be included in the app tarball. I've created redmine tickets for the two issues that need to be addressed in order to support the app that you've provided: Add Support for Go 1.6 and Include Go Application Dependencies.

Adding support for Go 1.6 should be fairly simple, and we may get it in for AppScale 3.2 if we have some extra time this sprint. Discovering the dependencies is a trickier problem. I don't think we want to include the go-app-builder binary in our appscale-tools Python package, and implementing the same functionality in Python would be a challenge. We might be able to allow the user to specify the location of an existing Go SDK directory that the tools can use to discover these.

[Elliott Stoneham]

Thank you Chris, that is wonderful news!

You can get a json description of the go code that will be included by using the command "$ go list -json goPackageDir".

For example I typed  "go list -json ./vendortest" which returned:

{

"Dir": "/Users/elliott/go/src/github.com/appscale/sample-apps/go/vendortest/vendortest",

"ImportPath": "github.com/appscale/sample-apps/go/vendortest/vendortest",

"Name": "vendortest",

"Target": "/Users/elliott/go/pkg/darwin_amd64/github.com/appscale/sample-apps/go/vendortest/vendortest.a",

"Stale": true,

"Root": "/Users/elliott/go",

"GoFiles": [

"vendortest.go"

],

"Imports": [

"github.com/appscale/sample-apps/go/vendortestcode/app"

],

"Deps": [

"bufio",

"bytes",

"compress/flate",

"compress/gzip",

"container/list",

"crypto",

"crypto/aes",

"crypto/cipher",

"crypto/des",

"crypto/dsa",

"crypto/ecdsa",

"crypto/elliptic",

"crypto/hmac",

"crypto/md5",

"crypto/rand",

"crypto/rc4",

"crypto/rsa",

"crypto/sha1",

"crypto/sha256",

"crypto/sha512",

"crypto/subtle",

"crypto/tls",

"crypto/x509",

"crypto/x509/pkix",

"encoding/asn1",

"encoding/base64",

"encoding/binary",

"encoding/hex",

"encoding/pem",

"errors",

"fmt",

"github.com/appscale/sample-apps/go/vendortestcode/app",

"github.com/appscale/sample-apps/go/vendortestcode/vendor/my.vendor.test/foo/bar",

"hash",

"hash/crc32",

"internal/golang.org/x/net/http2/hpack",

"internal/race",

"internal/singleflight",

"io",

"io/ioutil",

"log",

"math",

"math/big",

"math/rand",

"mime",

"mime/multipart",

"mime/quotedprintable",

"net",

"net/http",

"net/http/internal",

"net/textproto",

"net/url",

"os",

"os/exec",

"path",

"path/filepath",

"reflect",

"runtime",

"runtime/cgo",

"runtime/internal/atomic",

"runtime/internal/sys",

"sort",

"strconv",

"strings",

"sync",

"sync/atomic",

"syscall",

"time",

"unicode",

"unicode/utf8",

"unsafe"

]

}

You will see a slightly different list, as I ran the above using go1.6.5, rather than go1.6.2 which AppEngine seems to use currently, see https://cloud.google.com/appengine/docs/go/release-notes

Prior to running this test I was using go1.8rc1, so an upgrade of AppEngine's go version must be due early this year.

Hope that helps!

Elliott

 

[Chris Donati]

That's helpful, Elliott. The only issue with the output of `go list` is that it includes a lot of extra packages that the server does not need in order to compile the app. For example, the sample app you provided only requires the two Go source code files in vendortestcode to be present in the GOPATH when compiling.

We could probably use the output of `go list` and filter out unneeded packages. But for now, I've taken the easy route and just used the binary included in the SDK. Using this appscale branch and this branch from the tools, I was able to get your sample app working with `GOROOT=/path/to/sdk/goroot GOPATH=/my/gopath appscale deploy vendortest`.

It's not the most elegant solution, but I think it's a step in the right direction.

--

[Elliott Stoneham]

Well done Chris, seems like you've found a very pragmatic solution to the problem.

The solution of using the binary in the SDK is likely to work well in the long-term, since it minimizes the version differences.

If you want a more substantial Go app to test against, can I suggest:  https://github.com/cayleygraph/cayley/tree/master/appengine 

Thanks again, Elliott