Are imported packages cached, or local changes of packages not picked up?

1,015 views
Skip to first unread message

edunc...@gmail.com

unread,
Oct 29, 2014, 5:09:22 PM10/29/14
to golan...@googlegroups.com
Ok, I think I have a valid issue that raises to the rank of allowing me to post into this awesome forum for an awesome language.  :)

I have a constant problem across two Linux developer VMs over the last several months.  Both are running Debian (one stable, one testing).  They both experience this issue off and on, and today it has cost me too much time to the point that I just copied all files locally to get past this block.

It is an issue with developing my own packages - both for internal (private repos) and external uses.  

Problem: Once I import them into another executable/main() app, and then I go back and modify that package, the changes to the package are not detected by the main() app I imported it into, built and used before.  go build, go clean, go test, etc does not bring in the changes.

The process is:

* I create a package at ~/go/src/github.com/eduncan911/mypackage
* Write some code, tests, etc.  It all works.
* Commit code to remote repo.
* I create an app at ~/go/src/github.com/eduncan911/mymainapp
* I reference the first package, build mymainapp, go test, run it, etc.

Problem: Now, I go back and edit the original `mypackage` - the changes do not take affect in mymainapp.*
* most of the time.  sometimes, the package changes are picked up right away like they should be.

Is it a problem that I am referencing a full github.com/eduncan911/ url, after I committed code, but made local changes on my machine to that repo?  I would suspect that the go app would use my local files with local modifications, not the remote repo.

I've noticed almost exactly a 24 hour window before I can get those changes into mymainapp.  Almost like the imported package is cached.

I've rebooted countless times, deleted the package repo, etc.  

I've searched and searched for this, without resolve.

To complete the details:
Sublime w/GoSublime
Go 1.3.3 (built from source, w/cross-compiling)

Thanks in advance!
Eric

Dave Cheney

unread,
Oct 29, 2014, 6:50:28 PM10/29/14
to golan...@googlegroups.com, edunc...@gmail.com


On Thursday, 30 October 2014 08:09:22 UTC+11, edunc...@gmail.com wrote:
Ok, I think I have a valid issue that raises to the rank of allowing me to post into this awesome forum for an awesome language.  :)

I have a constant problem across two Linux developer VMs over the last several months.  Both are running Debian (one stable, one testing).  They both experience this issue off and on, and today it has cost me too much time to the point that I just copied all files locally to get past this block.

It is an issue with developing my own packages - both for internal (private repos) and external uses.  

Problem: Once I import them into another executable/main() app, and then I go back and modify that package, the changes to the package are not detected by the main() app I imported it into, built and used before.  go build, go clean, go test, etc does not bring in the changes.

You need to use go install to install those packages. http://dave.cheney.net/2014/06/04/what-does-go-build-build
 

The process is:

* I create a package at ~/go/src/github.com/eduncan911/mypackage
* Write some code, tests, etc.  It all works.
* Commit code to remote repo.
* I create an app at ~/go/src/github.com/eduncan911/mymainapp
* I reference the first package, build mymainapp, go test, run it, etc.

Problem: Now, I go back and edit the original `mypackage` - the changes do not take affect in mymainapp.*
* most of the time.  sometimes, the package changes are picked up right away like they should be.

This is likely to be a problem with your GOPATH configuration.

Can you post the entire output from

Eric Duncan

unread,
Oct 29, 2014, 11:23:16 PM10/29/14
to Dave Cheney, golan...@googlegroups.com
Thanks Dave!  Feel honored to get a reply from the guy who floods my tweeter feed.  :)  

See inlines below.

On Wed, Oct 29, 2014 at 6:50 PM, Dave Cheney <da...@cheney.net> wrote:

On Thursday, 30 October 2014 08:09:22 UTC+11, edunc...@gmail.com wrote:
Ok, I think I have a valid issue that raises to the rank of allowing me to post into this awesome forum for an awesome language.  :)

I have a constant problem across two Linux developer VMs over the last several months.  Both are running Debian (one stable, one testing).  They both experience this issue off and on, and today it has cost me too much time to the point that I just copied all files locally to get past this block.

It is an issue with developing my own packages - both for internal (private repos) and external uses.  

Problem: Once I import them into another executable/main() app, and then I go back and modify that package, the changes to the package are not detected by the main() app I imported it into, built and used before.  go build, go clean, go test, etc does not bring in the changes.

You need to use go install to install those packages. http://dave.cheney.net/2014/06/04/what-does-go-build-build
 

Humm.  I actually haven't been doing "go get" or "go install" for these packages I developed locally.  I reference them in the tooling, and it has been working (like I said, caching the original version I have locally for 24 hours, then I see changes I made to the package).  The tool I use them in, I just referenced as import and they worked.  But, I can see how that would work by moving them into the /pkg/ directory.  

And I see the other packages I developed in /pkg/ as well.  Now I am thinking that it does get installed somehow with "go get ." on the tooling, and that's why I have a cached version there.

This may be my lack of understanding of how to develop and test (test with external programs) packages locally.

How would I continue to develop/change the package code as I continued to develop the tool/runtime, if it is located in $GOPATH/pkg/ instead of $GOPATH/src/?  I ask because I often find myself adding features onto (or changing the API) of those packages I reference as I develop the tooling or what consumes them.  Must I, 1) edit package code, 2) push changes up to repo 3) run go install for each small tweak locally? I often times prototype changes in packages, which is something I don't want to commit up to a public repo (which I can get around with branches, sure).

(I often develop offline, during my 4 hour commute)

 

The process is:

* I create a package at ~/go/src/github.com/eduncan911/mypackage
* Write some code, tests, etc.  It all works.
* Commit code to remote repo.
* I create an app at ~/go/src/github.com/eduncan911/mymainapp
* I reference the first package, build mymainapp, go test, run it, etc.

Problem: Now, I go back and edit the original `mypackage` - the changes do not take affect in mymainapp.*
* most of the time.  sometimes, the package changes are picked up right away like they should be.

This is likely to be a problem with your GOPATH configuration.

Can you post the entire output from


I think you meant the package, not the /mymainapp?  For /mymainapp, it's located in a sub-sub-sub folder of other tools outside of my gopath.  So, if i run go install on it, it looks like this:


Which is true because it lives in a different directory along with other tools outside of my GOPATH.

As for the package itself, it looks like:


So that looks like it install the Github.com version into my /pkg/.  Great!  

Except, I have lots of changes locally for the /src/ version of that package in:


...that isn't being picked up.  

What would be the proper way to develop a package locally, and test it locally in other tools/apps/runtimes without pushing changes up to a remote repo and running "go install" for each change?  Or, must I commit all local prototyping/experimental changes up to the remote repo, run "go install" to get them back down into /pkg/, and then run the main app locally?

Thanks for your time Dave!
-Eric

Dave Cheney

unread,
Oct 29, 2014, 11:52:32 PM10/29/14
to edunc...@gmail.com, golang-nuts

Thanks for your reply. For me things are only getting more confusing so I can only offer generic advice.

On 30 Oct 2014 14:23, "Eric Duncan" <edunc...@gmail.com> wrote:
>
> Thanks Dave!  Feel honored to get a reply from the guy who floods my tweeter feed.  :)  
>
> See inlines below.
>
> On Wed, Oct 29, 2014 at 6:50 PM, Dave Cheney <da...@cheney.net> wrote:
>>
>>
>> On Thursday, 30 October 2014 08:09:22 UTC+11, edunc...@gmail.com wrote:
>>>
>>> Ok, I think I have a valid issue that raises to the rank of allowing me to post into this awesome forum for an awesome language.  :)
>>>
>>> I have a constant problem across two Linux developer VMs over the last several months.  Both are running Debian (one stable, one testing).  They both experience this issue off and on, and today it has cost me too much time to the point that I just copied all files locally to get past this block.
>>>
>>> It is an issue with developing my own packages - both for internal (private repos) and external uses.  
>>>
>>> Problem: Once I import them into another executable/main() app, and then I go back and modify that package, the changes to the package are not detected by the main() app I imported it into, built and used before.  go build, go clean, go test, etc does not bring in the changes.
>>
>>
>> You need to use go install to install those packages. http://dave.cheney.net/2014/06/04/what-does-go-build-build
>>  
>
>
> Humm.  I actually haven't been doing "go get" or "go install" for these packages I developed locally.  I reference them in the tooling, and it has been working (like I said, caching the original version I have locally for 24 hours, then I see changes I made to the package).  The tool I use them in, I just referenced as import and they worked.  But, I can see how that would work by moving them into the /pkg/ directory.  

I don't understand why 24 hours is important, unless the clock on your computer is running backwards.

The go tool, like most build tools compares the time of the source file with the compiled file (in GOPATH/pkg).

>
> And I see the other packages I developed in /pkg/ as well.  Now I am thinking that it does get installed somehow with "go get ." on the tooling, and that's why I have a cached version there.
>

Go get checks out the source using the bane of the package to figure out where the source repo lives then calls go install. There isn't any more magic than that, I'd you aren't trying to fetch someone else's changes, you don't need to be online.

> This may be my lack of understanding of how to develop and test (test with external programs) packages locally.
>
> How would I continue to develop/change the package code as I continued to develop the tool/runtime, if it is located in $GOPATH/pkg/

GOPATH/pkg is for use by the go tool, don't touch anything in there

instead of $GOPATH/src/?  I ask because I often find myself adding features onto (or changing the API) of those packages I reference as I develop the tooling or what consumes them.  Must I, 1) edit package code, 2) push changes up to repo 3) run go install for each small tweak locally?

Nope, just use go install, always and you'll have no problems.

I often times prototype changes in packages, which is something I don't want to commit up to a public repo (which I can get around with branches, sure).

The fact that the source lives in a directory that is under Git's control isn't important to the build process.

>
> (I often develop offline, during my 4 hour commute)
>
>  
>>>
>>>
>>> The process is:
>>>
>>> * I create a package at ~/go/src/github.com/eduncan911/mypackage
>>> * Write some code, tests, etc.  It all works.
>>> * Commit code to remote repo.
>>> * I create an app at ~/go/src/github.com/eduncan911/mymainapp
>>> * I reference the first package, build mymainapp, go test, run it, etc.
>>>
>>> Problem: Now, I go back and edit the original `mypackage` - the changes do not take affect in mymainapp.*
>>> * most of the time.  sometimes, the package changes are picked up right away like they should be.
>>
>>
>> This is likely to be a problem with your GOPATH configuration.
>>
>> Can you post the entire output from
>>
>> go install -x github.com/eduncan911/mymainapp
>
>
> I think you meant the package, not the /mymainapp?  For /mymainapp, it's located in a sub-sub-sub folder of other tools outside of my gopath.  So, if i run go install on it, it looks like this:
>

Ok, this is probably the problem,

1. Don't use symlinks to symlink source into your GOPATH, the go tool will ignore it
2. Move all your go code into your GOPATH
3. If your GOPATH contains multiple elements, collapse them into a single element.

Then everything will work properly.

Dave Cheney

unread,
Oct 29, 2014, 11:52:34 PM10/29/14
to edunc...@gmail.com, golang-nuts

Thanks for your reply. For me things are only getting more confusing so I can only offer generic advice.

On 30 Oct 2014 14:23, "Eric Duncan" <edunc...@gmail.com> wrote:
>

> Thanks Dave!  Feel honored to get a reply from the guy who floods my tweeter feed.  :)  
>
> See inlines below.
>
> On Wed, Oct 29, 2014 at 6:50 PM, Dave Cheney <da...@cheney.net> wrote:
>>
>>
>> On Thursday, 30 October 2014 08:09:22 UTC+11, edunc...@gmail.com wrote:
>>>
>>> Ok, I think I have a valid issue that raises to the rank of allowing me to post into this awesome forum for an awesome language.  :)
>>>
>>> I have a constant problem across two Linux developer VMs over the last several months.  Both are running Debian (one stable, one testing).  They both experience this issue off and on, and today it has cost me too much time to the point that I just copied all files locally to get past this block.
>>>
>>> It is an issue with developing my own packages - both for internal (private repos) and external uses.  
>>>
>>> Problem: Once I import them into another executable/main() app, and then I go back and modify that package, the changes to the package are not detected by the main() app I imported it into, built and used before.  go build, go clean, go test, etc does not bring in the changes.
>>
>>
>> You need to use go install to install those packages. http://dave.cheney.net/2014/06/04/what-does-go-build-build
>>  
>
>
> Humm.  I actually haven't been doing "go get" or "go install" for these packages I developed locally.  I reference them in the tooling, and it has been working (like I said, caching the original version I have locally for 24 hours, then I see changes I made to the package).  The tool I use them in, I just referenced as import and they worked.  But, I can see how that would work by moving them into the /pkg/ directory.  

I don't understand why 24 hours is important, unless the clock on your computer is running backwards.

The go tool, like most build tools compares the time of the source file with the compiled file (in GOPATH/pkg).

>


> And I see the other packages I developed in /pkg/ as well.  Now I am thinking that it does get installed somehow with "go get ." on the tooling, and that's why I have a cached version there.
>

Go get checks out the source using the bane of the package to figure out where the source repo lives then calls go install. There isn't any more magic than that, I'd you aren't trying to fetch someone else's changes, you don't need to be online.

> This may be my lack of understanding of how to develop and test (test with external programs) packages locally.


>
> How would I continue to develop/change the package code as I continued to develop the tool/runtime, if it is located in $GOPATH/pkg/

GOPATH/pkg is for use by the go tool, don't touch anything in there

instead of $GOPATH/src/?  I ask because I often find myself adding features onto (or changing the API) of those packages I reference as I develop the tooling or what consumes them.  Must I, 1) edit package code, 2) push changes up to repo 3) run go install for each small tweak locally?

Nope, just use go install, always and you'll have no problems.

I often times prototype changes in packages, which is something I don't want to commit up to a public repo (which I can get around with branches, sure).

The fact that the source lives in a directory that is under Git's control isn't important to the build process.

>


> (I often develop offline, during my 4 hour commute)
>
>  
>>>
>>>
>>> The process is:
>>>
>>> * I create a package at ~/go/src/github.com/eduncan911/mypackage
>>> * Write some code, tests, etc.  It all works.
>>> * Commit code to remote repo.
>>> * I create an app at ~/go/src/github.com/eduncan911/mymainapp
>>> * I reference the first package, build mymainapp, go test, run it, etc.
>>>
>>> Problem: Now, I go back and edit the original `mypackage` - the changes do not take affect in mymainapp.*
>>> * most of the time.  sometimes, the package changes are picked up right away like they should be.
>>
>>
>> This is likely to be a problem with your GOPATH configuration.
>>
>> Can you post the entire output from
>>
>> go install -x github.com/eduncan911/mymainapp
>
>
> I think you meant the package, not the /mymainapp?  For /mymainapp, it's located in a sub-sub-sub folder of other tools outside of my gopath.  So, if i run go install on it, it looks like this:
>

Ok, this is probably the problem,

1. Don't use symlinks to symlink source into your GOPATH, the go tool will ignore it
2. Move all your go code into your GOPATH
3. If your GOPATH contains multiple elements, collapse them into a single element.

Then everything will work properly.

Eric Duncan

unread,
Nov 18, 2014, 12:19:30 PM11/18/14
to Dave Cheney, golang-nuts
Thanks Dave!

I waited to spend intimate time with additional development before replying and I wanted to share my results with others that may run into the same.

Dave is correct on two counts: 

* That code, even one-off tooling for other projects, must live in your gopath.  This is an annoyance since there are dozens of repos here, mostly Python, and I am writing a tool in like the /tools/ folder for that specific project.  I have changed all projects to be in the gopath, and updated build scripts to just deploy the executable to those other repos (and commit them).  Learned to work with the tools, not fight it.  :)

* You need to run "go install" on those locally developed packages for their changes to show up immediately in your current page.  There is still a long day-long delay that happens though if you don't run go install that the changes are picked up in other tools/packages.  "go install" works to get around this every time so no big deal.

Nope, you don't have to be online for the "go install" (obviously) - if the package is local that is (e.g. in development).

Therefore, my development process when packages and tools are actively-being-developed side-by-side is to run "go install XX && go build" when I want to build my local tool.  For example:

(currently developing the package gomspec in parallel with gohyde)


As for Sublime, I have to close and reopen the file in the gohyde repo for it to pick up the changes in the gomspec repo I made.

Solved! 

The reason I do this is typically to share code with several other projects (or when developing a new package that I don't yet know how it will shape up).  Loved Dave's recent post on functional config options for APIs (which I am using for everything).

Thanks again!
Eric


Lars Seipel

unread,
Nov 18, 2014, 6:39:01 PM11/18/14
to Eric Duncan, Dave Cheney, golang-nuts
On Tue, Nov 18, 2014 at 12:19:17PM -0500, Eric Duncan wrote:
> Therefore, my development process when packages and tools are
> actively-being-developed side-by-side is to run "go install XX && go build"
> when I want to build my local tool. For example:
>
> (currently developing the package gomspec in parallel with gohyde)
>
> ~/go/src/github.com/eduncan911/gohyde $ go install
> github.com/eduncan911/gomspec && go build

Why not 'go install'? It will pick up changes automatically. No need to
explicitly 'go install' the dependencies first. Just 'go install' the
command. Add GOPATH/bin (or GOBIN, if set) to your path if not done
already, for added convenience.

Eric Duncan

unread,
Nov 18, 2014, 7:37:25 PM11/18/14
to Lars Seipel, golang-nuts, Dave Cheney

Wouldn't that install the tool as well? (which is not my intension)

Dave Cheney

unread,
Nov 18, 2014, 7:38:58 PM11/18/14
to Eric Duncan, Lars Seipel, golang-nuts
On Wed, Nov 19, 2014 at 11:37 AM, Eric Duncan <edunc...@gmail.com> wrote:
> Wouldn't that install the tool as well? (which is not my intension)

What's wrong with that ? You'll find them in $GOPATH/bin

Eric Duncan

unread,
Nov 18, 2014, 7:44:47 PM11/18/14
to Dave Cheney, golang-nuts, Lars Seipel

Good point.  It wouldn't hurt anything.  Though, for example, a tool I am working right now talks to (requires) 4 APIs + RabbitMQ - none of which I have installed locally.

go install && go run X -params...
^- yes, that is shorter now that I think of it. :-)

Lars Seipel

unread,
Nov 21, 2014, 11:22:23 AM11/21/14
to Eric Duncan, Dave Cheney, golang-nuts
On Tue, Nov 18, 2014 at 07:44:34PM -0500, Eric Duncan wrote:
> Good point. It wouldn't hurt anything. Though, for example, a tool I am
> working right now talks to (requires) 4 APIs + RabbitMQ - none of which I
> have installed locally.
>
> go install && go run X -params...

I don't understand the point of that go run there (if it doesn't work
locally, why run it?) but if your programs aren't intended to run on the
machine you're programming and building on you can also change the
output directory of go install by setting GOBIN in the environment.

You could mount the bin directory of your remote machine somewhere
(using sshfs or whatever is appropriate for your platform) and tell go
install to put the resulting binaries there. Now, when compiling, your
program ends up directly where it needs to be.

You could also not bother and just grab the command executable out of
GOPATH/bin and copy them to your remote machine. I still fail to see how
go build/run would be any more comfortable.
Reply all
Reply to author
Forward
0 new messages