Circular Dependency workaround?

[si guy]

Hi all,

Currently I have a program which is nicely modular, with separate component packages to handle various tasks.

A problem that I've encountered (and I'm sure it's an old one) is how to make these components talk to each other. Specifically, because I can't have them importing each other (circular) I have to define go-interface types that satisfy the API of each neighbor component and pass objects in as go-interface references. This leads to effective duplication of interface API signatures. Is there a way to auto-generate the "interface" definition to satisfy the API of a packaged type without importing it? Or am I doomed to a pre-processor script to scan packages and insert "type myNeighborPackageType interface { MethodA(); MethodB() }" in all the relevant places? I am looking at using the go parser package to do this automatically... It looks like it would be messy though...

A naive way would be to have "header" files that are in a separate package that just define types and method signatures (via standardized renaming), these would import the real package and be importable by other packages without circular dependencies... also messy.

Somewhat lost here as to how to make this cleanly... please help?

Thanks.

-Simon Watt

[Andrew Gerrand]

On 31 January 2012 17:00, si guy <sjw...@gmail.com> wrote:
> A naive way would be to have "header" files that are in a separate package
> that just define types and method signatures (via standardized renaming),
> these would import the real package and be importable by other packages
> without circular dependencies... also messy.

It doesn't seem messy to me.

Define your interfaces in a high-level package that the other packages
in your project depend on, then they can just pass interface values
around. That's the idiomatic way of doing it.

It's also likely that you're over-compartmentalizing. You say it's
"nicely modular," but now you're having dependency problems? That
doesn't sound modular to me. A Go package typically contains a lot
more than a Java or C++ class would. If you put all your components in
the one package your dependency issues will go away.

Of course, it's hard to say not having seen any of the code.

Andrew

[si guy]

Nicely modular usually means for me at least, less than 10k lines of code in a package.

Putting the interfaces in a top level package wouldn't remove all duplication, but it does reduce it a lot. Hmm... thanks.

-Simon

[Timon ter Braak]

On 31/1/12 9:21 , si guy wrote:
> Nicely modular usually means for me at least, less than 10k lines of
> code in a package.

I would call that "maintainable", but having complex dependencies also
reduces that maintainability.

[André Moraes]

> On 31/1/12 9:21 , si guy wrote:
>>
>> Nicely modular usually means for me at least, less than 10k lines of
>> code in a package.

For me is:

Write programs that do one thing and do it well. :-)

--
André Moraes
http://andredevchannel.blogspot.com/

[John Asmuth]

Packages are units of code to import. If you cannot import A without B and vice versa (they have a circular dep), they need to be one package.

The fact that we get namespaces from packages too is, I think, a little unfortunate, but I don't see a better way to deal with it at the moment. 

[si guy]

On Tuesday, January 31, 2012 5:40:43 AM UTC-8, John Asmuth wrote:

Packages are units of code to import. If you cannot import A without B and vice versa (they have a circular dep), they need to be one package.

The fact that we get namespaces from packages too is, I think, a little unfortunate, but I don't see a better way to deal with it at the moment. 

 

The main problem faced here is that I'm trying to keep the functional units of my program separate from each other. Everything I learned in systems analysis classes way back urges me to do this. These units are all massively threaded and thus maintaining internal concurrency is already challenging. Having some of these units in the same package makes it far too easy (and tempting) to write "reach in and change things" code accidentally that makes for a mess of interdependence.

After spending the last 10 hours grinding the chaff away (and finding some dangerous timing related inter-dependencies creeping in :s ), I have decided to use a small number of Top level defined interfaces that I have to make sure match my new API sigs. I really can't see another way...

@ André Moraes

I _would_ write a program that does one thing well, except that I have client handlers, core game physics processors, a database request handler, and a whole mess of controllers, managers, inter-machine links, message parsers etc. and _everything_ is talking to _every_ central controller somehow. and the central controllers are all talking to each other.

I'm not saying that the program is large just cuz, I am saying that when everything communicates (via function calls on channel/netchan passed references) I need known interfaces and I have found no easy way to get them without circular dependence.

Maybe this is a _bad_ way to design a program, idk, but so far the circ dep problem is the only one not born of my own stupidity that I've faced (starting to feel wrong about this one too).

Thanks for the insight gonuts, you guys are great.

-Simon

[james...@gmail.com]

That's ridiculous, just because two packages are dependent on each other does not mean they should all be together.  How about if you have one package that has all the business logic in a program and another package that does complex parsing for program inputs.  They might share a single configuration object, but as go doesn't support forward declaration or circle imports this isn't possible.  I don't believe there is any other language that doesn't support at least one of these two typical options for solving this problem.

Having to have a single package with 30+ files in it is completely unmanageable for any realistic project.  

Just because some people are comfortable with over engineered complex code doesn't mean all developers like to work like that.

[Luna Duclos]

Typically, one makes a third package with all the things both need, this is fairly reasonable in my opinion and has hardly ever gotten in my way.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shawn Milochik]

I

On Jul 26, 2014 9:51 AM, <james...@gmail.com> wrote:
>
> Having to have a single package with 30+ files in it is completely unmanageable for any realistic project.  
>

net/httphas 47 files.

http://talks.golang.org/2013/bestpractices.slide#17

> Just because some people are comfortable with over engineered complex code doesn't mean all developers like to work like that.
>
> On Tuesday, 31 January 2012 13:40:43 UTC, John Asmuth wrote:
>>
>> Packages are units of code to import. If you cannot import A without B and vice versa (they have a circular dep), they need to be one package.
>>
>> The fact that we get namespaces from packages too is, I think, a little unfortunate, but I don't see a better way to deal with it at the moment. 
>

[Jan Mercl]

In the general case a dependency cycle can be unresolvable in many languages, it's not much Go specific. (Think a type T T class of problems).

In some other cases, there is a solution, but at the cost of incremental/separate compilation not being available anymore. That scales badly.

Go is simply honest: circular dependency is a design error.

-j

[james...@gmail.com]

and...

Just because a library has lots of class all bundle together in an unmanageable way doesn't mean that's the correct solution.

That's the kind of the response you hear in banking IT when people try to justify 100 line stored procedures by saying "but we already have hundreds already in production so it must be the correct way....".

[james...@gmail.com]

But that unfortunately doesn't work if you want to add functions to the shared config struct.  Particularly if you have a sensible function like a start that want to initiate the parsing of the configuration, but you want all the complex logic to parse configuration in a separate package.

This is particularly bad because you need to have all the tests in the same folders as production code so if your create a proper full set of tests at unit, component and integration level then there are far too many files in the same package to manage in a sensible way.

[james...@gmail.com]

Anyway the net/http package has a lot of untested code in it.  If there was full test coverage then the package would have twice as many files....

On Saturday, 26 July 2014 15:00:54 UTC+1, Shawn Milochik wrote:

[Henrik Johansson]

It sounds like you have a preconception that you don't want to let go of. I get that and I can relate but I really think you are overreacting in this particular case.

Can't you extract an interface that the various components that need to do parsing implements? Perhaps that is not enough but I usually do something along these lines.

1. Create a common package for a particular shared functionality

2. Create interface CommonAdaptor in said new poackage

3. Implement CommonAdaptor where it needs adaption

NB pseudo solutions like this tend to oversimplify but this type of  thing could work for you or?

As for circular deps I agree with many other posters, it is generally an error in the design although it can be very hard to avoid in some cases. Please don't blow up, I get your frustration.

[james...@gmail.com]

Well, I have already been trying to exactly what you stated as it is clear the only way that could potentially be solved however in this case it is definitely not possible.

In addition the strange thing is that in the 6 other languages I regularly code in I never have this problem because the linking is not so restrictive and simplistic.

[james...@gmail.com]

I don't believe it is always an error in design. There is nothing wrong with separating code into multiple packages to aid clarity and therefore each package could have some references to each other.  

This would only be an error in design if each package was a separate logical module.  

Clearly the previous commenters you refer to don't understand the distinction between a separate package for clarity and a separate logical component.

On Saturday, 26 July 2014 15:33:25 UTC+1, Henrik Johansson wrote:

[Ian Lance Taylor]

On Sat, Jul 26, 2014 at 7:49 AM, <james...@gmail.com> wrote:
>
> In addition the strange thing is that in the 6 other languages I regularly
> code in I never have this problem because the linking is not so restrictive
> and simplistic.

The restriction exists primarily so that initialization is
well-defined. Initialization in Go is very simple and hard to get
wrong.

Ian

[Andrew Gerrand]

On 27 July 2014 00:55, <james...@gmail.com> wrote:

I don't believe it is always an error in design. There is nothing wrong with separating code into multiple packages to aid clarity and therefore each package could have some references to each other.  

This would only be an error in design if each package was a separate logical module.  

Clearly the previous commenters you refer to don't understand the distinction between a separate package for clarity and a separate logical component.

I don't see the point of separating code into multiple packages if they are not logically separate pieces of code (as you put it, "separate logical modules"). Doing that will force you to define interfaces between the packages and resolve circular dependencies (as you are discovering).

Can you give an example where you are splitting code into pieces that are not logically separate?

How many types and functions do you have in each of your packages? It's possible that you're just not used to having many files in the same directory, and so you're trying to split things up when you don't need to. Sometimes packages need to be big, like net/http (which has 86% test coverage, btw), and sometimes they are small.

Andrew

[egon]

This looks like you do not have a common abstractions for the different pieces that you are putting together. It's hard to say whether there are nice abstraction boundaries, that would resolve this issue, without a proper description of the thing you are writing, which parts are involved, what is the end goal of the code... etc.

Also take a look at:

https://groups.google.com/d/topic/golang-nuts/EuZ-ClOtLzs/discussion

https://groups.google.com/d/topic/golang-nuts/tEykjBEQ1FQ/discussion

+ egon