How to get vendoring to work?

[John Beckett]

I have an app that uses google.golang.org/cloud/storage, which was recently changed.  So my app builds locally (where I have the older version), but not on my automated testing platform (which uses the newest version).  I presume that vendoring is the solution to this, but locally when I set up the correct files and directories in the /vendor directory, I get a number of build errors.  This does not happen if I don't use the vendor folder and structure.

Has anyone managed to get vendoring to work on AppEngine? 

I'm using version 1.9.38.

[Constantin Konstantinidis]

The link mentions backward compatible possible problems. It seems that vendoring won't help in this case. Otherwise, you can read https://github.com/golang/go/wiki/PackageManagementTools where a list of tools is available. Please note that you should be in Go 1.6 where vendoring in natively supported.

[Strom]

I've successfully gotten vendoring to work on my App Engine apps. I had to move the app to be inside $GOPATH though, so make sure that's the case for you as well.

[Constantin Konstantinidis]

Thanks for your e-mail. I suggest to move to the latest release for SDK as for Go (1.6.2 at least if not 1.7beta) unless you have strict reasons. You risk to look for issues already solved in later release. I read here and there that vendoring was improved lately. You can read https://github.com/golang/go/issues/13961 where a deprecated feature is mentioned.

On Friday, July 1, 2016 at 3:54:09 PM UTC+2, John Beckett wrote:

[John Beckett]

I'm using 1.9.38, which as far as I'm aware is the latest version of the SDK.  From what I understand the SDK uses its own version of go (1.6.1 in this case), even if my system version of go is different.  If I'm missing something, please let me know.

[Constantin Konstantinidis]

The following issue looks very similar to yours : https://github.com/golang/go/issues/15478

There are many questions and possibilities. Can you check and confirm how your proceed and if issue is similar ?

Thanks,

[Adam Tanner]

Hey John,

Vendoring should work with App Engine, but your app must be within a GOPATH. Vendoring is only supported within a GOPATH. If you have the package in both GOPATH and vendor you might try removing the one from GOPATH and seeing if that helps. If that doesn't fix your issue, we'll need a bit more information about the build errors you're seeing.

Adam

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Beckett]

I always use absolute import paths, so I don't believe it's the same issue.  I'll try looking into this more on Monday to see what's causing it.

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/Pv1M2oSTXAQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

[John Beckett]

Hi Adam,

Everything I do in Go is in the GOPATH, so that shouldn't be an issue.  I haven't tried removing the version from GOPATH though, but even if that works, it should indicate a bug.  I'll try this on Monday and get back to you.

~ John

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/Pv1M2oSTXAQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

[John Beckett]

Adam, I've tried removing the relevant package from GOPATH, but I still get the same errors.

So, further information:

- I'm trying to vendor 'google.golang.org/cloud/storage` as my app uses an older version of it, and I have yet to update the app to use the newer version.  I am partly doing this as I want to test whether I can have one version vendored for my app, and the more up to date version available for other apps.

- I have copied the code from the package in $GOPATH/src/google.golang.org/cloud/storage to $GOPATH/src/github.com/myname/myapp/vendor/google.golang.org/cloud/storage.

- When I try to build the app, I now get the following response:

package github.com/myname/myapp/gae
imports github.com/myname/myapp/ec
imports github.com/myname/myapp/auth
imports github.com/myname/myapp/user
imports github.com/myname/myapp/storage/upload
imports google.golang.org/cloud/storage
imports google.golang.org/cloud/internal/transport: use of internal package not allowed

Although the google.golang.org/cloud/storage package is the only one that I need to vendor right now, I tried vendoring another package, while having the original google.golang.org/cloud/storage package in my $GOPATH.  That built without an issue.  So I'm leaning towards the problem being the old google.golang.org/cloud/storage package using internals when it probably shouldn't.

 

On 3 July 2016 at 06:33, 'Adam Tanner' via google-appengine-go <google-ap...@googlegroups.com> wrote:

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/Pv1M2oSTXAQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

[Su-Au Hwang]

Hi John,
sorry for the late reply, could you detail/clarify the issues you are having ?

Is it google app engine classic/mvm ? What tools are you using for vendoring, project structure, gopath etc. and when are you running into that issue ?

In my project google.golang.org/cloud/storage is vendored and goapp build, test, serve and deploy are working fine as far as I know (1.9.36 and 1.9.40, I don't remember the exact steps, but I have working code and could reproduce if requested)

I have been busy with non-go projects lately, so I haven't touched my go code for quite a while and almost forgot about the pain I went through when switching to the official vendor directory.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/Pv1M2oSTXAQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-appengine-go+unsub...@googlegroups.com.

[joh...@joonix.se]

I've also had issues similar to this, please see https://github.com/joonix/appengine_vendor for my example.

And if anyone has a solution, an PR would be greatly appreciated!

[Su-Au Hwang]

A similar question came up on StackOverlflow: http://stackoverflow.com/questions/39948027/how-do-i-make-vendoring-work-with-google-app-engine/40118834#40118834

here is a copy of my answer:

Your project directory (where app.yaml is) is probably in the GOPATH/src. It shouldn't be. The go-app-builder will take everything in the app.yaml folder (and below) and additionally merge your GOPATH into it, meaning now you have it twice.

The solution is to move app.yaml out of the GOPATH/src folder. Additionally you'll find that goapp test works differently from goapp serve and goapp deploy when it comes to resolving dependencies.

So this is the solution I have been using (haven't used golang app engine in a while already) and it's the only setup I've found to work properly for all the goapp commands and for govendor to work properly (not sure about godep)

/GOPATH
├──/appengine
|   ├── app.yaml
|   └── aeloader.go
└──/src
    └── /MYPROJECT
         ├── main.go
         ├── /handler
         |    └── handler.go
         └── /vendor

details:

file: GOPATH/appengine/aeloader.go (NOTE the init function is necessary, probably a bug though)
package mypackage

import (
    _ "MYPROJECT"
)

func init() {
}

now run goapp serve and goapp deploy from ../GOPATH/appengine/ 

and goapp test ./... from ../GOPATH/src/MYPROJECT

P.S. I find the global GOPATH thing silly and simply set my GOPATH to current project folder (in the example above /GOPATH) and check the whole thing into version control.