Elementary Question About Calling Functions in Different Packages

[jlfo...@berkeley.edu]

I'm having trouble understanding what should be a trivial issue.

I have the following file structure:

.
├── go.mod
├── main.go
├── p1
│   └── p1.go
└── p2
    └── p2.go

The files are small and simple.

 -- go.mod

module wp

go 1.18

-- main.go

// main.go
package main

import (
        "wp/p1"
        "wp/p2"
)

func main() {
        p1.P()
        p2.P()
}

-- p1/p1.go

// p1.go
package p1

import (
        "fmt"
        "wp/p2"
)

func P() {
        fmt.Printf("p1\n")
        p2.P()
}

-- p2.go

// p2.go
package p2

import (
        "fmt"
        "wp/p1"
)

func P() {
        fmt.Printf("p2\n")
        p1.P()
}

In main.go I'm trying to call a function in packages p1 and p2. In package p1 p1.go is trying to call a function in package p2, and in package p2 p2.go is trying to call a function in package p1. Simple, right?

However, running "go build ." in the top level directory says:

package wp
        imports wp/p1
        imports wp/p2
        imports wp/p1: import cycle not allowed

Is this because p1 imports p2, which imports p1, which imports p2, ...?

How can this program be made to work? I've tried many things, which all lead back

to the situation I describe above.

On a separate but similar note, I originally had another file in the top level directory that

contained a function I wanted to call from one of the packages. (I know this isn't a good idea, but I'm rewriting existing non-Go code into Go). This also failed to build for similar

reasons. Researching the issue showed several postings claiming that this wasn't

possible. Is this true?

Cordially,

Jon Forrest

[Kurtis Rader]

Yes. From https://go.dev/ref/spec#Import_declarations:

> It is illegal for a package to import itself, directly or indirectly, or to directly import a package without referring to any of its exported identifiers.

 

How can this program be made to work? I've tried many things, which all lead back

to the situation I describe above.

You need to refactor the code to eliminate the import cycle. Exactly how you do that depends on specifics of the situation which you haven't shared. If you google "golang import cycle not allowed" you'll find many articles and stackoverflow questions that discuss this. For example:

https://jogendra.dev/import-cycles-in-golang-and-how-to-deal-with-them

https://quoeamaster.medium.com/golang-gotchas-2-the-curse-of-import-cycle-not-allowed-6abfa3523f57

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[jlfo...@berkeley.edu]

Thanks for the quick inciteful response.

From https://go.dev/ref/spec#Import_declarations:

> It is illegal for a package to import itself, directly or indirectly, or to directly import a package without referring to any of its exported identifiers.

 

I should have researched this. My apologies.

 

How can this program be made to work? I've tried many things, which all lead back

to the situation I describe above.

You need to refactor the code to eliminate the import cycle. 

https://jogendra.dev/import-cycles-in-golang-and-how-to-deal-with-them

As a learning exercise I'm converting a large app into Go. The app's source code

is located in many directories, and I've already learned that all files in a package have to be located

in the same directory so that rules out the suggestion in the posting above to put everything in

one package. (What's the reason for this requirement)?

So, I'll probably have to keep using multiple packages if I can somehow figure out how to solve

the package import cycle problem.

Cordially,

Jon Forrest

 

[Kurtis Rader]

On Thu, Jun 9, 2022 at 7:06 PM jlfo...@berkeley.edu <jlfo...@berkeley.edu> wrote: 

How can this program be made to work? I've tried many things, which all lead back

to the situation I describe above.

You need to refactor the code to eliminate the import cycle. 

https://jogendra.dev/import-cycles-in-golang-and-how-to-deal-with-them

As a learning exercise I'm converting a large app into Go.

That's going to be a painful way to learn Go. Worse, doing a straightforward, mechanical, translation of a program written in another language is likely to result in non-idiomatic Go code and reinforce bad habits (bad in the context of Go, not the other language).

 

The app's source code

is located in many directories, and I've already learned that all files in a package have to be located

in the same directory so that rules out the suggestion in the posting above to put everything in

one package. (What's the reason for this requirement)?

Technically, that is an implementation detail of a particular Go toolchain. From https://go.dev/ref/spec#Packages:

> An implementation may require that all source files for a package inhabit the same directory.

I don't know why the official (primary?) Go toolchain imposes that restriction. If you google "golang package structure multiple directories" you can find discussions of this question. Such as https://stackoverflow.com/questions/45899203/can-i-develop-a-go-package-in-multiple-source-directories.

I'm guessing the app you're translating is written in Java and you're comfortable with that language. I've only written a trivial amount of Java code but did start writing Go a few years ago after using Python as my primary language for several years (and, C, COBOL, FORTRAN, and many other languages before that). All I can say is that you really need to avoid imposing idioms from language J on language G. At least if you want to be successful at writing high quality code in language G.

So, I'll probably have to keep using multiple packages if I can somehow figure out how to solve

the package import cycle problem.

The most common solution is to factor out the aspects which cause the cycle into a third package that one or both of the current packages imports. In my four decades of programming I've only seen one import cycle (in Python) that was hard to remove. My team eventually used the "Python lazy import" hack. Refactoring the code to eliminate the import cycle, and therefore the hack to handle it, was high on my team's priority list when I left the project.

[jake...@gmail.com]

So, I'll probably have to keep using multiple packages if I can somehow figure out how to solve

the package import cycle problem.

The most common solution is to factor out the aspects which cause the cycle into a third package that one or both of the current packages imports. In my four decades of programming I've only seen one import cycle (in Python) that was hard to remove. My team eventually used the "Python lazy import" hack. Refactoring the code to eliminate the import cycle, and therefore the hack to handle it, was high on my team's priority list when I left the project.

In Go interfaces can also sometimes be helpful in eliminating import cycles

[ben...@gmail.com]

Kurtis has given some great answers about the general problem. I do have a bit of a counterpoint to this statement, however:

> > As a learning exercise I'm converting a large app into Go.

>

> That's going to be a painful way to learn Go. Worse, doing a straightforward, mechanical, translation of a program written in another language is likely to result in non-idiomatic Go code and reinforce bad habits (bad in the context of Go, not the other language).

I agree it might be painful, and might lead to non-idiomatic Go! But it sounds like a really good challenge, and will no doubt teach you a lot along the way. It looks like you've already learned something from this thread. :-)

Is the app you're converting open source, and if so, can you provide a link to the code that leads to the import cycle? With concrete code examples it might be easier to advise you how to refactor to avoid the cycle.

One other thing: I've seen a lot of projects in Java (but not just Java: C#, JavaScript, even Go) that break things up into very small pieces. One 5-line class per file, dozens of files and directories for something simple, that sort of thing. In Go (I think) it's more common to have larger packages.

-Ben

[jlfo...@berkeley.edu]

On Friday, June 10, 2022 at 6:48:58 PM UTC-7 ben...@gmail.com wrote:

I agree it might be painful, and might lead to non-idiomatic Go! But it sounds like a really good challenge, and will no doubt teach you a lot along the way. It looks like you've already learned something from this thread. :-)

I certainly have.

 

Is the app you're converting open source, and if so, can you provide a link to the code that leads to the import cycle? With concrete code examples it might be easier to advise you how to refactor to avoid the cycle.

Thanks but what I'm trying to do is so large, and probably infeasible, that I don't have any real code to show yet. The example I gave in my

original posting was the result of trying various experiments. I tried to construct the example to be a minimal illustration of the

problem I saw I was facing.

One hack solution I came up with to break cycles when Package A depends on Package B which depends on Package A is to

create a symbolic link in Package A to the file(s) in Package B that contain(s) the resources needed by Package A. Then Package A

wouldn't be dependent on Package B anymore, and the cycle will  be broken. (I just thought of this so I'm not 100% sure it will work

until I try it).

To tell the truth, the requirement that all files in a package be in the same directory will probably cause the most misery. The program I'm

trying to port has so many subdirectories containing so many files that this will be a big problem. I'm thinking now that I'll have to move all the files

in a package tree to the top level directory, and use a naming scheme so that I can see which subdirectory each file came from.

Cordially,

Jon Forrest

 

 

[jlfo...@berkeley.edu]

On Friday, June 10, 2022 at 8:47:54 PM UTC-7 jlfo...@berkeley.edu wrote:

One hack solution I came up with to break cycles when Package A depends on Package B which depends on Package A is to

create a symbolic link in Package A to the file(s) in Package B that contain(s) the resources needed by Package A. Then Package A

wouldn't be dependent on Package B anymore, and the cycle will  be broken. (I just thought of this so I'm not 100% sure it will work

until I try it).

To follow up to myself, I wasn't thinking clearly. This idea obviously won't work. It's not what directory a file is in that determines

what package it's is in. Rather, It's what's in its 'package' statement.

Sorry for the noise.

Jon

[Bakul Shah]

On Jun 10, 2022, at 8:47 PM, jlfo...@berkeley.edu <jlfo...@berkeley.edu> wrote:
>
> One hack solution I came up with to break cycles when Package A depends on Package B which depends on Package A is to
> create a symbolic link in Package A to the file(s) in Package B that contain(s) the resources needed by Package A. Then Package A
> wouldn't be dependent on Package B anymore, and the cycle will be broken. (I just thought of this so I'm not 100% sure it will work
> until I try it).

You can factor out the mutually dependent functions in a separate package. For example if p1.P calls p2.P and p2.P calls p1.P,
create package p3, move p1.P and p2.P into p3 and rename them to p3.P1P and p3.P2P respectively.

> To tell the truth, the requirement that all files in a package be in the same directory will probably cause the most misery. The program I'm
> trying to port has so many subdirectories containing so many files that this will be a big problem. I'm thinking now that I'll have to move all the files
> in a package tree to the top level directory, and use a naming scheme so that I can see which subdirectory each file came from.

You can start by simply treating each subdir as a package and then once things start working, refactor everything. The key is to either do refactoring up front or after you have a working, testable implementation, even if imperfect.

Another alternative is figure out major subsystems and data structures and create packages around them, adding many tests so that you can test each package in isolation.

Yet another alternative is to simply treat the existing app as a *reference* and nothing more. That is, rebuild your app in Go from scratch, simply based on the input/output behavior of the original app. But if the app is large and complex this may not be the best option. On the other hand I wouldn't want to convert a large and complex app to a different language, unless it can be broken up in smaller pieces relatively cleanly.