Any reasonable way to organize source code without having separate packages?

[Toby Lawrence]

Hey y'all!

I've been tinkering with Go for a while now, and I'm finally working on a project where the inter-dependencies and number of types are testing the limits of Go's idiomatic packaging style.

Is there any way to be able to organize source code under folders without having to put them in a different package?  This feels... kind of stupid, to be honest, to not be able to do so.  I've poked around with godag, but honestly, the out-of-the-box toolchain is part of what drew me to Go.  Now, these constricting constraints are driving me to think of switching to another language.

Thoughts?  Ideas?  Suggestions.

[Dave Cheney]

In a word, no. 

Small print, if you wanted to build your own build tool to replace cmd/go, it is possible. But why make your life miserable, and your code incompatible with the rest of the ago ecosystem? 

Can you give some specific examples of the exact problems you have, there is probably a simple recommendation which will make things workable. 

Thoughts?  Ideas?  Suggestions.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Chris McGee]

Hi,

Can you give us a sense of how large your project is? How many Go files are you dealing with in your single package? How many types are there?

In one of my projects I have organized the code into about 15 source code files with names that make it fairly clear the responsibility of each in the application. So far it has been working reasonably well. If there are collisions between type names then I rely on the compiler to inform me about them. I have some editing tools that make it fairly easy to navigate across files to find references, declarations, callers, etc. The tools might help to keep things in order.

Chris

[Toby Lawrence]

The project is an MMORPG emulator.  It's a port of an existing project written in C#, so I expect parity on size of the codebase, but maybe less.  I should say that I haven't gotten everything written yet; I'm trying to set up a reasonable pattern for organization before spending any more time on a potential dead-end port.

There'll be a few hundred types at the bare minimum, and probably only 20 - 25% of those could be grouped together, file-wise i.e. instead of having a hundred of those types in a hundred files, it could be condensed to, say, 10 files with 10 types each.  That would be grouping by relationship.

[Jesse McNelis]

On Thu, Jan 30, 2014 at 2:16 PM, Toby Lawrence <tobias....@gmail.com> wrote:

The project is an MMORPG emulator.  It's a port of an existing project written in C#, so I expect parity on size of the codebase, but maybe less.  I should say that I haven't gotten everything written yet; I'm trying to set up a reasonable pattern for organization before spending any more time on a potential dead-end port.

There'll be a few hundred types at the bare minimum, and probably only 20 - 25% of those could be grouped together, file-wise i.e. instead of having a hundred of those types in a hundred files, it could be condensed to, say, 10 files with 10 types each.  That would be grouping by relationship.

Why do you need to put all the types in a single package?

If you've got hundreds of types they're probably not all similar enough in functionality to be in the same package.

 

--
=====================
http://jessta.id.au

[Dave Cheney]

Go does not limit you to one type per file. 

--

[Toby Lawrence]

I don't need to put every single one in the same package.  There's definitely a ton of stuff I could package up that doesn't have a circular dependency.  Here's an example of my current plight:

- there's a server struct and a client struct to handle the listening and handling of individual client connections

- there's a packet manager, which holds a list of functions to handle different packets

- there's N number of packet handlers, and in most cases, N number of packet responders (code to send a response to a request)

So, to me, it makes a lot of sense to put the server/client structs at the top level, because they aren't networking-specific.  The packets and packet manager are definitely related to networking, so I have them tentatively organized under a 'network' folder.  For the packet handlers and responders, they're down further in another directory to handle just those.

For one of the packet handlers, it's a little over thirty lines of code.  There's over a hundred packet handlers I'll need to implement, many of them that will be double or triple the amount of code.  I really don't fancy having to shove them all back into a single file.  This isn't the only place where there's a lot of units all related to each other.  It's an emulator for a game, so each player/character type will have tons of abilities, etc.  Is the only solution besides a custom toolchain to have tens to hundreds (because there's no feasible way to do it in less files) with thousands of lines per file?

It just seems asinine that basic folder organization isn't possible.

[Dave Cheney]

On 30 Jan 2014, at 14:30, Toby Lawrence <tobias....@gmail.com> wrote:

I don't need to put every single one in the same package.  There's definitely a ton of stuff I could package up that doesn't have a circular dependency.  Here's an example of my current plight:

- there's a server struct and a client struct to handle the listening and handling of individual client connections

- there's a packet manager, which holds a list of functions to handle different packets

- there's N number of packet handlers, and in most cases, N number of packet responders (code to send a response to a request)

So, to me, it makes a lot of sense to put the server/client structs at the top level, because they aren't networking-specific.  The packets and packet manager are definitely related to networking, so I have them tentatively organized under a 'network' folder.  For the packet handlers and responders, they're down further in another directory to handle just those.

For one of the packet handlers, it's a little over thirty lines of code.  There's over a hundred packet handlers I'll need to implement, many of them that will be double or triple the amount of code.  I really don't fancy having to shove them all back into a single file.  This isn't the only place where there's a lot of units all related to each other.  It's an emulator for a game, so each player/character type will have tons of abilities, etc.  Is the only solution besides a custom toolchain to have tens to hundreds (because there's no feasible way to do it in less files) with thousands of lines per file?

It just seems asinine that basic folder organization isn't possible.

Not really, I strongly recommend reading the net/http and crypto/tls packages for a good example of how to group types by function into files, not packages. 

On Wednesday, January 29, 2014 10:18:24 PM UTC-5, Jesse McNelis wrote:

On Thu, Jan 30, 2014 at 2:16 PM, Toby Lawrence <tobias....@gmail.com> wrote:

The project is an MMORPG emulator.  It's a port of an existing project written in C#, so I expect parity on size of the codebase, but maybe less.  I should say that I haven't gotten everything written yet; I'm trying to set up a reasonable pattern for organization before spending any more time on a potential dead-end port.

There'll be a few hundred types at the bare minimum, and probably only 20 - 25% of those could be grouped together, file-wise i.e. instead of having a hundred of those types in a hundred files, it could be condensed to, say, 10 files with 10 types each.  That would be grouping by relationship.

Why do you need to put all the types in a single package?

If you've got hundreds of types they're probably not all similar enough in functionality to be in the same package.

 

--
=====================
http://jessta.id.au

[Jan Mercl]

On Thu, Jan 30, 2014 at 4:16 AM, Toby Lawrence
<tobias....@gmail.com> wrote:
> The project is an MMORPG emulator. It's a port of an existing project
> written in C#, so I expect parity on size of the codebase, but maybe less.
> I should say that I haven't gotten everything written yet; I'm trying to set
> up a reasonable pattern for organization before spending any more time on a
> potential dead-end port.
>
> There'll be a few hundred types at the bare minimum, and probably only 20 -
> 25% of those could be grouped together, file-wise i.e. instead of having a
> hundred of those types in a hundred files, it could be condensed to, say, 10
> files with 10 types each. That would be grouping by relationship.

Throw the C# code away now and don't look at it anymore. The less you
will remember of it - the better your Go reimplementation will be.

-j

[Henrik Johansson]

I normally start simple. Everything in a file and when its too cumbersome I move to some functional grouping in separate files.

When/if a "functional grouping" distills and becomes nicely self contained I move it to a package since that's really where it belongs.

I find it very easy to grow a project structure like that in Go and it is tremendous plus in my book.

You have to resist the urge to design a project structure up front and let it grow naturally.

It is not the only way to do it and it is probably not the best way but it gets me going with real functionality while the meta stuff can come as is needed later. 

  

[chris dollin]

On 30 January 2014 03:30, Toby Lawrence <tobias....@gmail.com> wrote:

For one of the packet handlers, it's a little over thirty lines of code.  There's over a hundred packet handlers I'll need to implement, many of them that will be double or triple the amount of code.  I really don't fancy having to shove them all back into a single file.  

You don't need to. A package can be composed from multiple files

all in the same directory.

This isn't the only place where there's a lot of units all related to each other.  It's an emulator for a game, so each player/character type will have tons of abilities, etc.  Is the only solution besides a custom toolchain to have tens to hundreds (because there's no feasible way to do it in less files) with thousands of lines per file?

I don't know. I don't understand why you're problem can't be solved in
the usual go-tool way; put code in packages, break the packages into

multiple files.

 

It just seems asinine that basic folder organization isn't possible.

I don't understand why the usual Go layout doesn't count as "basic folder

organisation". What about it doesn't work for you?

Chris

--
Chris "allusive" Dollin

[Toby Lawrence]

I guess the biggest problem I see/have is that there are very few places where I could possibly avoid cyclic imports.  I can't abstract the packets via the packet manager because they need to know about the client, unless I made calls to the packet manager provider every little single value needed, which could be 15 - 25 things.

I see the value in deeper packages, so to speak, being uninterested/unconcerned with the things happening at a level above, or N levels above them.  I just don't see how to put it into practice here without having some sort of massive translation layer where any package boundary is satisfied only by interfaces.  It makes sense if I want to make it work under the Go toolchain, but it doesn't make any sense at all for the purpose of a project with tons and tons of interdependencies and tight coupling, at least not to me, at this moment, anyways.

[egon]

On Thursday, January 30, 2014 5:30:52 AM UTC+2, Toby Lawrence wrote:

I don't need to put every single one in the same package.  There's definitely a ton of stuff I could package up that doesn't have a circular dependency.  Here's an example of my current plight:

- there's a server struct and a client struct to handle the listening and handling of individual client connections

- there's a packet manager, which holds a list of functions to handle different packets

- there's N number of packet handlers, and in most cases, N number of packet responders (code to send a response to a request)

So, to me, it makes a lot of sense to put the server/client structs at the top level, because they aren't networking-specific.  The packets and packet manager are definitely related to networking, so I have them tentatively organized under a 'network' folder.  For the packet handlers and responders, they're down further in another directory to handle just those.

For one of the packet handlers, it's a little over thirty lines of code.  There's over a hundred packet handlers I'll need to implement, many of them that will be double or triple the amount of code.  I really don't fancy having to shove them all back into a single file.  This isn't the only place where there's a lot of units all related to each other.  It's an emulator for a game, so each player/character type will have tons of abilities, etc.  Is the only solution besides a custom toolchain to have tens to hundreds (because there's no feasible way to do it in less files) with thousands of lines per file?

It just seems asinine that basic folder organization isn't possible.

At some point I started to write a mud, but I due to lack of time I haven't had time to progress... but I did get ready a telnet communication part https://github.com/egonelbre/mud/tree/master/telnet... might give you some ideas... Not perfect, but I'm satisfied with it at the moment.

+ egon

[Vincent Callanan]

I sympathise entirely...

I once mulled the possibility of having some kind of "friend" relationship between sub-packages of a given package.

But of course that would open an entirely new can of worms.

For this and other reasons, I can respect the decision of the Go Authors to keep things as they are.

However, one "anomoly" stands out for me that could be rectified without minimal pain:

The ability to "privately" embed an exported struct from a third-party package, thereby exposing the struct's methods but not its "value". 

(As things stand, I have to merge packages or avoid embedding by laboriously write method wrappers).

Embedding is a powerful feature and if this anomaly were rectified, it would solve at least some of the issues you allude to.

Cheers,

Vincent

[Andy Balholm]

I wonder if some of your cyclic dependencies could be resolved by using interfaces.

[Toby Lawrence]

Here's a concrete example which I think illustrates the crux of my problem with trying to tease these things out.  My repo is here: https://github.com/tobz/phosphorus.  The code isn't entirely current to my explanation but I think it'll suffice nonetheless.

There's a Client type  - it holds the client's connection and some basic metadata.  These are things like the ID of their character object in the game world, their account name, etc.  This is a top-level type so it lives in the top level.  The Server type holds a list of Clients, etc etc... everything gels nicely with that simple model in mind.

Now, we move to packet handlers.  These are the aforementioned chunks of code for handling received packets and sending packets back to the client.  So, as an example, there's a request packet and response packet for checking if the name of a character is "invalid" - racial epithets, expletives, etc.  The client sends in the packet with simply the name to be checked.  So far, I have no need to actually know what the client is.  I could simply read the packet data, grab the string, access the global "name" manager to see if my name is OK, and that's that.  However, I have to send back a response.

Now, sending back the response could be abstracted out, too.  I could simply construct the packet and pass it back as a result of the handler function, which is called by the packet manager one "level" above the handlers/responders themselves.  The packet handler could then pass the packet to the client, maintaining that tight coupling and leaving the packet handlers/responders to be dumb, isolated chunks of code.  However, to respond to this particular request packet, you need to pass back the name that was being checked, whether it passes or not, AND the account name.  Now I have a required strong coupling to the client object.  The coupling might be at the request handler, where I could grab the account name and then pass it as data to the responder.  It could be at the responder where I use it directly.  Nonetheless, I need the account name, though, and a way to get it for the current client.

There's no other way, though, that I see, to decouple the client object from the packet handlers.  An interface, or interfaces, don't solve truly it... because while it may offer package-level isolation, I'd basically be making the interface a one-to-one copy of the methods available on the client object.  Is it *really* abstracting if you have to maintain two nearly-identical files per logical boundary?  Not to mention, that's only one of the boundaries.  I can think of another fifteen to twenty files that would be required in handling and constructing packets from/for the client.

So now I feel like I'm back at square one of throwing everything in the same folder... unless I want the... joy of maintaining an interface file for every single boundary that is expected.

[Ingo Oeser]

Try using interfaces to loosely couple components that "need to know each other". That way they don't need to know each other. Put those interfaces at top level, check interface implementation conformance in foo_test.go.

In Go you can implement an interface without even mentioning it at the implementation site. Leverage this to avoid circular dependencies.

[Andy Balholm]

Remember that you don't need to create an interface that covers all of a type's methods. You can make one that covers just the methods that another function or type needs access to.

[egon]

You can get rid of the circular dependency and excessive interfaces by registering the handlers in a separate package.

Currently you have handlers depending on client/server, which can be broken by adding interfaces between them, but is an annoyance. You have your server importing the packet handlers to have all the different behaviors; but this link can be broken.... i.e. introduce a main package that imports "Server" and all the different handlers, and let the handlers register themselves to the server. For example: https://gist.github.com/egonelbre/8758102 ... I used "_" for denoting subdirectories. But of course you don't have to put each handler to a separate directory, it just depends whether you need to for some configurations only import a few of them. Also you could skip doing the registration in "init" and do it in main package. Or if you want the server to be self contained you can make a "server" and separate "server/handlers" or "handlers" package that imports all the different handlers.

Essentially main -> handlers&server; handlers -> A&B; A -> server; B -> server... so no circular dependencies and no excessive interfaces.

+ egon

[tlaw...@bluestatedigital.com]

Interesting approach.  I'm pretty sure Go doesn't have it, but the particular project I'm porting leveraged C#'s attribute system and reflection to dynamically find packet handlers at run time and link them up,  Obviously I have to still solve problems like cyclic imports, and what not, but boy do I wish Go had that sort of thing.  Not the biggest deal in systems programming, I suppose.

For the time begin, I've been going the interface route.  It's a little more stubbing-out work, but, so far, so good.  I'm trying to at least get to the point where it supports basic functionality.  After reading everyone's responses and thinking about it, it was a bit of premature optimization to be so worried when the code was still a drawing on a napkin, so to speak.  I appreciate everyone's thoughts on the subject, though.  It was aggravating at first, but now it feels weirdly good to be using this interface approach.  I also realized that this lends itself very well to being able to mock objects for testing without complicated reflection or anything.  I imagine that will make designing tests much, much easier. 

[egon]

On Monday, February 3, 2014 4:50:38 AM UTC+2, tlaw...@bluestatedigital.com wrote:

Interesting approach.  I'm pretty sure Go doesn't have it, but the particular project I'm porting leveraged C#'s attribute system and reflection to dynamically find packet handlers at run time and link them up,  Obviously I have to still solve problems like cyclic imports, and what not, but boy do I wish Go had that sort of thing.  Not the biggest deal in systems programming, I suppose.

For the time begin, I've been going the interface route.  It's a little more stubbing-out work, but, so far, so good.  I'm trying to at least get to the point where it supports basic functionality.

Interfaces make sense when you are separating things from different implementations. Carlo Pescio has really good theory about structuring code http://www.physicsofsoftware.com/papers.html. I.e. Imagine that your code elements are planets that have gravitational pull, every link planet makes increases its mass. But you can make force-fields (abstractions) that protect against different planets.

Abstractions make sense when you don't where you don't want to see things collapsing into each other... e.g. I see your problem as this...

Which do you think will collapse into themselves and which are unnecessarily-complicated?

The client abstraction doesn't seem to be protecting anything... i.e. the client abstraction layer would be simply copying the client itself making it mostly useless... obviously it would be more useful if you had different client implementations.

The best place to put the abstraction is between Server/Handler... i.e. avoid importing handlers directly from server, make a separate "handlers" abstraction that puts server and the different handlers together.

After reading everyone's responses and thinking about it, it was a bit of premature optimization to be so worried when the code was still a drawing on a napkin, so to speak.

Usually that's the best place where to do large sweeping changes to the architecture... the more code/tests you have, the harder such changes become.

 

I appreciate everyone's thoughts on the subject, though.  It was aggravating at first, but now it feels weirdly good to be using this interface approach.  I also realized that this lends itself very well to being able to mock objects for testing without complicated reflection or anything.  I imagine that will make designing tests much, much easier. 

I made pull request on github :), look it through... you might be able to get some ideas from it.

 

+ egon