go-app-builder: Failed parsing input: app file xxx.go conflicts with same file imported from GOPATH

[bsr]

Hello,

When I update my SDK to 1.9.1, I see lot of below error

 go_runtime.py:170] Failed to build Go application: 2014/03/29 23:09:22 go-app-builder: Failed parsing input: app file ui.go conflicts with same file imported from GOPATH

go_runtime.py:170] Failed to build Go application: 2014/03/29 23:09:22 go-app-builder: Failed parsing input: app file auth.go conflicts with same file imported from GOPATH

one per each module.

when put back to 1.9.0, everything is fine. What have changed?

I checked the release note and couldn't find anything specific

Thanks.

bsr

[Glenn Lewis]

1.9.1 is more strictly enforcing the structure of import paths to avoid ambiguities.

Please see the recent threads about structuring your apps.

-- Glenn

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bsr]

thanks Glenn. I used to have yaml files within the module.Now, I moved everything out to the top level, and works fine now. thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Ralf Rottmann]

Do you happen to have a link to that discussion? The update "broke" our application...

--

rottmann.net | Google+ | Twitter | Facebook | LinkedIn

[Glenn Lewis]

If you are on Windows (where 1.9.1 broke the builds only for Windows Go apps), you will want this thread:

https://groups.google.com/forum/#!topic/google-appengine-go/m3rH_UAeNG4

If you want info on structuring your app, you will want this thread:

https://groups.google.com/forum/#!searchin/google-appengine-go/trouble$20running$20goapp$20test/google-appengine-go/C_i5kQEi7-A/kY3pQR-rCHUJ

-- Glenn

--

You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

[David Byttow]

Ran into same problem. Embarrassingly it's not immediately obvious what's going on. Admittedly my project is somewhat complex, but it's setup pretty uniformly and seems to follow all conventions.

bsr, what did you mean by "yaml files within the module"?

[Glenn Lewis]

One simple way to avoid the issue is to keep all your yaml files and their associated source files outside of your GOPATH and then import any other packages you need that are in your GOPATH from those files.

[David Byttow]

Oh, I guess he meant "github" module.

Hm yeah, will require a bit more surgery. Current setup is

github.com/org/server

where in server:

common/

frontend/*.go

backend/*.go

frontend.yaml

backend.yaml

So starting up dev app server with frontend.yaml and backend.yaml together it results in this error complaining about conflicts with files in common/ because they get automatically compiled and then re-imported.

But, I like this setup because I can rely on init() within frontend/ and backend/ to do things like register handlers.

[Ralf Rottmann]

I understand why the more strict handling has been added. The one thing I think could be improved is to make this more clear in release notes and maybe even when running gcloud update components. Breaking potentially complex projects without crystal clear docs as to how things must be organised is somewhat... disturbing! 

On Sunday, March 30, 2014 5:13:58 AM UTC+2, bsr wrote:

[Glenn Lewis]

I apologize for the undocumented changes that are affecting existing projects and for the inconvenience.
-- Glenn

--

[Prateek Malhotra]

Just updated my SDK and I seem to be running into the same issue. I've looked over the threads but I don't think I'm understanding the change required:

Current setup:

GOPATH = ~/workspace/project-go

bitbucket.org/org/project

where in project:

models/

utils/

module1/

module2/

where in each moduleX:

app.yaml

I have designed my code so that each module acts a separate component, with URL rules and code that can be changed/modified with no effect on the other modules (similar to what David does I think). Each module can share code found in the models/ and utils/ packages by importing them (e.g. bitbucket.org/org/project/models).

Where do I have to move my code/yaml files for this to work correctly?

Thanks,

Prateek

[David Byttow]

The problem is actually kind of annoying.

Do you have any subpackages in moduleX/ that you include via proper paths?

e.g.,

module1/foo

module1/bar

and you ever do something like import "github.com/username/project/module1/foo"

you'll run into this issue. Basically, all .go files under module1 get compiled first, then any subsequent imports lead to this error.

It's a real pain. I reverted back to 1.9.0 for now before I refactor the world.

--

[David Byttow]

In other words, you can't import any packages within or under any directories that contain your module yaml

[Prateek Malhotra]

Well I just tried making a new module with a basic "hello world" example for the framework I use. There are 2 Go files:

new_module/main.go

new_module/controllers/default.go

main.go imports controllers via: bitbucket.org/org/project/new_module/controllers

Both main.go and default.go import a component or two from the framework I use but nothing points back to the module. For whatever reason, when I try to "serve" it tells me a component in my "utils" package has already been imported, but I never even point to it or use it in my bare-bones test.

I thought this was how you would design your code based on examples I've seen on gitub.com.

[David Byttow]

odd. and no modules in or above utils?

is main.go importing ../controllers/default?

--

[Prateek Malhotra]

I see. So by switching my main.go controllers import to

import "controllers"

vs

import "bitbucket.org/org/project/new_module/controllers"

it seems to compile and run fine.

[David Byttow]

yes but that just seems dirty to me...

--

[David Byttow]

somehow the builder should just handle this IMO, but I guess the (somewhat documented) gotcha is that all .go files within and underneath a module are compiled separate from the import process

[Prateek Malhotra]

I agree. I guess maybe my downfall was that I am a big "learn by example" kind of person, and when I looked at a few fairly large/complex Go projects I saw imports for subpackages being done relative to the GOPATH instead of the current package. One such example can be found here. The import for the subpackage "proto" is done using an import path relative to the GOPATH, not the current package.

I'm still not clear if this was an incorrect way for me to have been doing my imports or if the position of my YAML files relative to the module source is causing the issues (or just the appengine go compiler).

At the very least I can continue to code...

Thank you for the help.

[David Byttow]

I did the same thing. Admittedly, there was no clear documentation on structuring a go project with modules (IIRC). Now I'm going to have to pay the price and refactor it.

--

[David Symonds]

Sorry for the confusion folk. We realise this is a little surprising,
but the previous state was actually more surprising and potentially
dangerous.

Back in the days before GOPATH, the only logical way to decide on app
import paths was their directory path relative to the app base (where
the app.yaml is), but when GOPATH came along that became the standard
Go package mechanism, and import paths were relative to $GOPATH/src.
Now the question is what should be done in a Go App Engine app when
these two worlds collide? Previously we were just silently accepting
both, and compiling the same package under two import paths, which led
to lots of people getting confused why their app panicked at startup
due to duplicate HTTP handler registration, or did other much more
subtle things due to a package being initialised twice. With 1.9.1 we
chose to flag that situation as an outright error to avoid all that;
the rest of the Go world treats GOPATH as the gold standard, so we're
following that lead and making sure that silent misbehaviour doesn't
continue.

So again, sorry for the confusion (we under-communicated for sure),
but I hope everyone will appreciate that a small bit of pain to
reorganise code will be worth it.

The shortest path to a good app layout is to change your app to only
use the full import paths (i.e. relative to $GOPATH/src), and then put
your app.yaml, etc. in their own directory (with no descendent
directories), and add a single .go file that imports your app packages
by their full import paths.

[Ralf Rottmann]

Thanks, David! Could you possibly respond with a working & ideal sample project structure and stubs of the relevant go source files that would illustrate a setup with two modules cross-using packages etc.?

I guess, best practices would help greatly here.

Let's assume GOPATH equals ~/go and we're building an app which has one separate moduleA and a package shared between default and moduleA.

How should the folder structure look like? How the import statements? Where should the YAML files be placed?
--
--
rottmann.net | Google+ <https://plus.google.com/106658337982640166595> |
Twitter <http://twitter.com/ralf> | Facebook<http://www.facebook.com/ralfrottmann>|
LinkedIn <http://de.linkedin.com/in/rottmann>

[David Byttow]

I plan to do this soon, if someone else doesn't first, I can add my $0.02. My app is significantly complex... Uses several modules (frontend, workers, backend, some python and other modules)

[Prateek Malhotra]

I second the notion to have an example of how we should structure two modules that only share code from a common package (within the same project directory if possible). 

On another note, I am able to run my modules fine in the dev server but deploying my modules seems to try and compile my entire GOPATH which contains packages that are never used in my modules (and are not GAE compatible). Is this also a side-effect of the changes made since 1.9.1?

[Glenn Lewis]

No, deploying should not attempt to build everything in your GOPATH... only the packages that are imported.  If you see different behavior, please file an issue report or share an example with me to see if I can duplicate the problem.

-- Glenn

On Mon, Apr 7, 2014 at 4:10 PM, Prateek Malhotra <some...@gmail.com> wrote:

I second the notion to have an example of how we should structure two modules that only share code from a common package (within the same project directory if possible). 

On another note, I am able to run my modules fine in the dev server but deploying my modules seems to try and compile my entire GOPATH which contains packages that are never used in my modules (and are not GAE compatible). Is this also a side-effect of the changes made since 1.9.1?

--

[Prateek Malhotra]

Ah, that was a my bad. Found an import to an old package. Weird that it was working on the dev server but only threw an error when deploying.

[David Symonds]

I gave the rough sketch of the recommended layout in my last
paragraph, and it sounds like David Byttow will be able to give a real
example soon, but here's a sketch of a layout:

$GOPATH/src/github.com/foo/bar/app/app.yaml
$GOPATH/src/github.com/foo/bar/app/app.go
$GOPATH/src/github.com/foo/bar/moduleA/moduleA.yaml
$GOPATH/src/github.com/foo/bar/moduleA/moduleA.go
$GOPATH/src/github.com/foo/bar/pkg/quux.go

app.go and moduleA should import "github.com/foo/bar/pkg". Don't try
to import the stuff in app.go or moduleA.go from anywhere.

[Ralf Rottmann]

quux.go being in a folder named pkg is just to emphasize the fact that it's a separate place for shares code, correct? There is nothing special, so the folder could also have been named "shared" or anything else, is this correct?

[David Symonds]

On 8 April 2014 10:09, Ralf Rottmann <ra...@rottmann.net> wrote:

> quux.go being in a folder named pkg is just to emphasize the fact that it's a separate place for shares code, correct? There is nothing special, so the folder could also have been named "shared" or anything else, is this correct?

That's correct, though remember that the import path is
"github.com/foo/bar/pkg", not "pkg". The important part is that it
isn't in a subdirectory of the app dir.

[Ralf Rottmann]

Glenn,

I just wanted to report back, that I refactored a larger Go code base and this worked like a charm. Thanks a lot for your ongoing support.

On Sunday, March 30, 2014 5:13:58 AM UTC+2, bsr wrote:

Hello,

When I update my SDK to 1.9.1, I see lot of below error

 go_runtime.py:170] Failed to build Go application: 2014/03/29 23:09:22 go-app-builder: Failed parsing input: app file ui.go conflicts with same file imported from GOPATH

go_runtime.py:170] Failed to build Go application: 2014/03/29 23:09:22 go-app-builder: Failed parsing input: app file auth.go conflicts with same file imported from GOPATH

one per each module.

when put back to 1.9.0, everything is fine. What have changed?

I checked the release note and couldn't find anything specific

Thanks.

bsr

[Glenn Lewis]

I'm so glad to hear that, Ralf!
Thanks for the report.

--

[andras...@gmail.com]

Hey folks. I still can't figure out how to break down my app to smaller pieces.

The Go Runtime Environment doc says that you should keep your project files off the $GOPATH, otherwise they will be picked up twice. Is this outdated info? Isn't this what's happening when I put my go files on the GOPATH and try goapp serve? 

"app file account.go conflicts with same file imported from GOPATH"

Here is my project

https://github.com/andrascsibi/how-many-pullups/tree/gochopping

I checked it out under $GOPATH/src/github.com/andrascsibi/how-many-pullups

My tests run fine with

goapp test ./...

What am I missing?

Thanks

[Glenn Lewis]

No, that is not outdated information.

In your specific case, I tried this:

$ go get github.com/andrascsibi/how-many-pullups

$ cd $GOPATH/src/github.com/andrascsibi/how-many-pullups

$ goapp serve

$ curl localhost:8080  (from another terminal window)

(then changed app.yaml to my own project ID)

$ goapp deploy

and everything worked with no errors.  Can you please give me the commands that you ran (and from what directories) with the error messages?

Thanks.

-- Glenn

--

[Andras Csibi]

Thanks for the quick response. Well, sorry about that. I'm doing the go code chopping on a separate branch, go get github.com/andrascsibi/how-many-pullups gets the master branch, which is in production and is just working fine. Note that master only contains a single go compilation unit, accounts.go. My problems started when I wanted to extract some functionality to separate modules, eg. handler, account, challenge. account.go imports "github.com/andrascsibi/how-many-pullups/handler", challenge.go imports both others similarly.

Here's what happens on the gochopping branch.

$ cd $GOPATH/src/github.com/andrascsibi/

$ git clone -b gochopping g...@github.com:andrascsibi/how-many-pullups.git

$ cd how-many-pullups

$ goapp test ./...

ok   github.com/andrascsibi/how-many-pullups/account 4.426s

ok   github.com/andrascsibi/how-many-pullups/challenge 3.902s

ok   github.com/andrascsibi/how-many-pullups/handler 3.177s

ok   github.com/andrascsibi/how-many-pullups/stringset 0.012s

$ goapp serve

[...SOME INFO logs...]

ERROR    2014-07-27 00:08:17,551 go_runtime.py:170] Failed to build Go application: (Executed command: /Users/andras/Applications/go_appengine/goroot/bin/go-app-builder -app_base /Users/andras/gocode/src/github.com/andrascsibi/how-many-pullups -arch 6 -dynamic -goroot /Users/andras/Applications/go_appengine/goroot -nobuild_files ^^$ -unsafe -gopath /Users/andras/gocode -print_extras_hash challenge/challenge_test.go account/account.go challenge/challenge.go handler/handler.go account/account_test.go stringset/stringset.go handler/handler_test.go stringset/stringset_test.go)

2014/07/27 02:08:17 go-app-builder: Failed parsing input: app file account.go conflicts with same file imported from GOPATH

Thanks,

Andras

[Glenn Lewis]

OK, thanks.

Please try this:

$ cd $GOPATH/src/github.com/andrascsibi/

$ git clone -b gochopping g...@github.com:andrascsibi/how-many-pullups.git

$ cd how-many-pullups

$ mkdir website

$ mv app app.yaml bower.json bower_components design exports index.yaml karma.conf.js tmpl website

$ cd website

$ cat > app.go

package app

import (

_ "github.com/andrascsibi/how-many-pullups/account"

_ "github.com/andrascsibi/how-many-pullups/challenge"

_ "github.com/andrascsibi/how-many-pullups/handler"

_ "github.com/andrascsibi/how-many-pullups/stringset"

)

<Ctrl-d>

$ goapp serve

This works for me.

Bottom line: When your project lives within the GOPATH, arrange the directories so that source code doesn't get pulled in twice (once from the explicit "import" and once from the implicit scanning of subdirectories performed by the Go toolchain).  I hope that makes sense now.  Please let me know if it doesn't.

-- Glenn

[Andras Csibi]

Thanks that worked! 

Now I understand what David was talking about. The cloudy part for me was app.go importing all the other modules. I still don't get it though, app.go is still under GOPATH, why doesn't it get picked up twice?

Thanks again, I am very happy now.

[Glenn Lewis]

The search for Go source files starts in the app's root directory (where app.yaml is), not at the top of your GOPATH. Otherwise, you could only have a single app in that entire GOPATH directory tree.
-- Glenn

[ant...@indb.uk]

There are go-lang conventions in place. I don't get why GAE makes things messy. I have only absolute paths defined as imports and it complains that the app file conflicts with same file imported from GOPATH.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[David Symonds]

On 12 August 2014 17:13, <ant...@indb.uk> wrote:

> There are go-lang conventions in place. I don't get why GAE makes things
> messy. I have only absolute paths defined as imports and it complains that
> the app file conflicts with same file imported from GOPATH.

The App Engine model of defining the application files as everything
in and under the directory containing the app.yaml significantly
predates the existence of GOPATH (including for Go on App Engine). It
is unfortunate that it's consequently a little messy, true, but
there's not much we can do about it now without breaking lots of apps.
But the easy workaround (put app.yaml and any config files in a
separate directory, and a bootstrap main.go) is trivial and solves the
problem entirely.

[ri...@defend7.com]

Thanks. This is clear.

[Aaron Schlesinger]

Realize I'm a bit late to this thread. I want to get a little more background here - 

• is it necessary for goapp to compile all code in subdirectories first? if so, why?
• I asked the previous b/c it seems like goapp would need to pre-compile a subdir if it was a module. Is there a logical difference between a subdirectory w/o a yaml file and one with a yaml file (which I assume is a module?)

Apologies if these were already answered. I'm catching up on this & plenty other threads, and it's a lot to take in.

 

[Glenn Lewis]

goapp compiles all files in a module's directory (and subdirectories) all at once.  Each module has its own .yaml file in it at the top-level directory of the module.

You can think of each "module" as a separate executable on App Engine.  Each module runs its own completely separate Go binary executable.

I hope that helps.

-- Glenn

--

You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

[Aaron Schlesinger]

I see. For my knowledge, are cross module function calls implemented as RPC calls?

[Glenn Lewis]

Cross-module function calls are not possible (directly).

Please read: https://cloud.google.com/appengine/docs/go/modules/#Go_Communication_between_modules

Other ways to communicate are via standard HTTP requests (via urlfetch) or task queues, for example.

For more information about these topics, please see:

https://cloud.google.com/appengine/docs/go/urlfetch/

https://cloud.google.com/appengine/docs/go/taskqueue/

-- Glenn

[Aaron Schlesinger]

Thanks Glenn. I missed those docs somehow.

[da...@jewishinteractive.net]

Not sure if I'm a little too late to this party... but I had some errors when updating to newest SDK (now Go 1.6) so I refactored my project according to the recommendations here and everything's working fine

However, it's a bit strange since my static files and templates are now under the main app folder. Not exactly bad, but just wanted to make sure I'm doing things right. I'd appreciate confirmation this is good structure if you have a minute... compile times are a bit slow (not necessarily more than before, but I might have been doing something screwy all along, maybe that's why I needed to refactor...)

My project structure is essentially

$GOPATH/src/path/to/project/init/app.yaml

$GOPATH/src/path/to/project/init/queue.yaml

$GOPATH/src/path/to/project/init/main.go

$GOPATH/src/path/to/project/init/static/*.png,js,css,etc.

$GOPATH/src/path/to/project/api/auth/*.go

$GOPATH/src/path/to/project/datastore/*.go

$GOPATH/src/path/to/project/more-libs/*.go

and imports look like

import (

"path/to/project/datastore/user-datastore"

"path/to/project/api/auth"

)

and html templates are included via

template.ParseFiles("html-templates/template-name.html")

and app.yaml looks like:

application: project-name

version: 1

runtime: go

api_version: go1

handlers:

- url: /static

  static_dir: static

  secure: always

- url: /favicon\.ico

  static_files: static/images/favicon.ico

  upload: static/images/favicon\.ico

  secure: always

- url: /.*

  script: _go_app

  secure: always

thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Dave Day]

That structure looks perfectly reasonable to me. And yes, the status resources and templates etc. do need to be under the main app folder, since appcfg.py and dev_appserver.py (goapp deploy and goapp serve) both look for all the resources relative to the app.yaml that defines the app – packages from your gopath are about the only exception to that rule.

The compile times won't be affected at all by the change in layout. The compiler in Go 1.6 is known to be slower than the one in 1.4, and people are actively looking at improving performance for Go 1.7 and beyond. There's a good thread here (https://groups.google.com/forum/#!topic/golang-dev/DpMyPbclI3o) summarising some of the progress to date.

[David Komer]

Fantastic, thanks for the confirmation and explanation!

--

You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/dNhqV6PBqVc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

[David Komer]

You noticed and addressed something I left out, just for the record in case anyone else benefits from this, html-templates/ is $GOPATH/src/path/to/project/init/html-templates/

:)

To unsubscribe from this group and all its topics, send an email to google-appengine-go+unsub...@googlegroups.com.