Interfaces, packages and naming

[hugo...@gmail.com]

Imagine you have a web app that (among other things) has some sort of CRUD-like functionality over user accounts. Since you can in theory use several data stores (persistent, in-memory, etc..) let's say you implement all users-related functionality in something called a "users service".

Imagine the service is under <project>/services/users in a file named service.go and its interface is like this:

package users

// ...

var ErrSomethingBad = errors.New("Something bad happened")

type Service interface {
    Create(name string) error // Ignore the function signature. I know it doesn't make sense, but it doesn't matter for this question (it's just a random example)
    // ...
}

Now imagine you want to create e.g. a MySQL implementation of this service. Under what package would you implement it? How would you name the functions?

package mysql // This package name kinda makes sense when looking at the full path, but calling mysql.NewUsersService feels weird

// ...

var errSomethingBad = errors.New("Something bad happened with MySQL") // I know we could place it under the users package, but errors like this could clash between implementations..

type UsersService struct { // UsersService? MySQLService? MySQLUsersService? Just Service? Everything feels weird
    // ...
}

func NewUsersService() *UsersService { // Same thing as above. Just New? NewMySQLService would probably look nice if this was under the users package
    // ...
}

func (us *UsersService) Create(name string) error {
    // ...
    return
}

I've been reading about this since it's been bugging me (client code always feels weird) but I can't find an idiomatic way of handling this.. Any suggestions?

[Benjamin Measures]

On Thursday, 1 October 2015 21:54:29 UTC+1, hugo...@gmail.com wrote:

Since you can in theory use several data stores (persistent, in-memory, etc..) let's say you implement all users-related functionality in something called a "users service".

I can't find an idiomatic way of handling this.. Any suggestions?

The standard library is a good place to look for idiomatic code.

Similar to your example of a "users service" with "several data stores" is a "database service" with "several database backends": https://golang.org/pkg/database/sql/#Register

[Konstantin Shaposhnikov]

I would:
- put mysql implementation of Service into package users in file mysql.go
- Provide facory method users.NewMySQLService
- Not export struct (make it private) that provides mysql implementation of users.Service from users package

[Giulio Iotti]

If you have a place to set the service to use:

mysql := mysql.New()

users.SetService(mysql)

Doesn't read too bad for me.

-- 

Giulio Iotti

[Egon]

On Thursday, 1 October 2015 23:54:29 UTC+3, hugo...@gmail.com wrote:

Imagine you have a web app that (among other things) has some sort of CRUD-like functionality over user accounts. Since you can in theory use several data stores (persistent, in-memory, etc..) let's say you implement all users-related functionality in something called a "users service".

What are you building? There are different ways of structuring it and the good solution depends on the importance of "users" and how it relates to other pieces of the software.

e.g. in "knowledgebase" the user is not the most important thing, hence there is no separate package for it. Instead there are different interfaces for it (https://github.com/raintreeinc/knowledgebase/blob/master/kb/database.go) and the implementation in a separate package, since it's not important detail in respect to the site functionality.

Of course if you are working on a social-network it might make sense to have a separate package called "user".

Side note, one of the important realizations during writing this is that separate your authentication from users management. It makes you avoid a lot of problems and makes testing nicer (e.g. easy to switch users), multiple login accounts for a single user, bots that need authentication, but do not have a username etc.

I've been seeing one pattern over-and-over-and-over again which relax some of the forces. (I.e. conflicting forces/needs are the reason you are feeling uncomfortable.)

I'm going to draft it here -- eventually I'll try to write it as a proper pattern.

Deconstruction by value

It's easy to get overwhelmed by the amount of things that need to get right. There are a lot of little pieces and implementation details that need to be worried about.

To avoid wasting time on implementing things that do not matter and implement things that provide most value you should concentrate on the value that things provide.

Therefore, structure folders, packages, files, types, functions based on how it provides the value.

For example, when you are writing an issue tracker, make the packages "issue", "tracker", "webclient" -- because those are the most important things in a issue tracker. "Issues" are the central concept, "tracker" provides functionality to track things, "client" provides a way for users to use the system.

Now you deconstruct each of those artifacts similarly. How does "tracker" provide it's value -- i.e. it must have an "User" and probably some "Service", hence you get... "tracker.User" and "tracker.Service".... and similarly, "issue.Info", "issue.Status", "webclient.Server".

And I must stress -- do not start your project with folders "services", "models", "views" ... etc. it's a brain dead way to bootstrap a project and only makes you feel good, because it looks like you have something of value; in reality you don't... It's like when you try to design a house and you start by creating 4 boxes, here go the sockets, here the walls, here windows, here electrical -- you lose the building as a whole, because you are mechanically separating things by form and similarity -- rather than figuring out how to create the rooms such that they support each other and function well together.

+ Egon

[hugo...@gmail.com]

Thanks for all the suggestions everyone, I appreciate it. I've been experimenting so here are some comments:
 

The standard library is a good place to look for idiomatic code.

Similar to your example of a "users service" with "several data stores" is a "database service" with "several database backends": https://golang.org/pkg/database/sql/#Register

That's where I looked first, but found nothing. I understand your database/sql suggestion, but don't really think it makes sense in this case (I can see how it does for DB drivers, though).

I like how this works for client code and think it's my preferred way for now, but it still has the clashing problem. If your mysql implementation declares an errCodeX with value Y, for example, and some other implementation wants to declare errCodeX with a different value, the design is broken. I guess I could always declare it as errCodeMySQLX or something, but that feels like a hack to get around the package system (vs file-private variables).

 

If you have a place to set the service to use:

mysql := mysql.New()

users.SetService(mysql)

Feels a bit weird considering the services have to follow an interface, and the "setter" would also have to follow it (+ have a new SetService method). Don't really like the mysql.New client code either (although it would look fine with the full import path).

 And I must stress -- do not start your project with folders "services", "models", "views" ... etc. it's a brain dead way to bootstrap a project and only makes you feel good, because it looks like you have something of value; in reality you don't... It's like when you try to design a house and you start by creating 4 boxes, here go the sockets, here the walls, here windows, here electrical -- you lose the building as a whole, because you are mechanically separating things by form and similarity -- rather than figuring out how to create the rooms such that they support each other and function well together.

Other than a handlers package and a server package (which make sense, since my application needs a server and handlers to work), the remaining packages are business-oriented, don't worry (e.g. models live with their respective service). I still need services though (you can call them something else) to handle e.g. authentication. My question was more related to organizing these packages that aren't necessarily static. Although my application needs an authentication package, the actual service that this package uses (i.e. what implements the package's public-facing interface) can be implemented in many different ways, and I'd like for client code to be able to use them "nicely" (hence the MySQL example).

[Egon]

On Sunday, 4 October 2015 11:22:07 UTC+3, hugo...@gmail.com wrote:

Thanks for all the suggestions everyone, I appreciate it. I've been experimenting so here are some comments:

 And I must stress -- do not start your project with folders "services", "models", "views" ... etc. it's a brain dead way to bootstrap a project and only makes you feel good, because it looks like you have something of value; in reality you don't... It's like when you try to design a house and you start by creating 4 boxes, here go the sockets, here the walls, here windows, here electrical -- you lose the building as a whole, because you are mechanically separating things by form and similarity -- rather than figuring out how to create the rooms such that they support each other and function well together.

Other than a handlers package and a server package (which make sense, since my application needs a server and handlers to work),

Your application doesn't need "handlers", your application needs some specific functionality. a way to manage users... "handler" part is an annoying implementation detail.

I know this looks like splitting hairs, but, the end-user won't care about the "handlers" -- hence you really shouldn't care that much about them either. Imagine going to a customer and saying... "Hey, I just implemented handlers"... the customer will just ask, "What, that's not what I asked for? Why am I paying you money?".

the remaining packages are business-oriented,

It's not about being "business-oriented" -- it's about understanding what is the value and what are the pieces that really matter in providing the value.

 

don't worry (e.g. models live with their respective service). I still need services though (you can call them something else) to handle e.g. authentication.

You might need an abstraction layer or a boundary, but not necessarily "services".

The way you think will highly correlate how you structure your code. I know these seem like minor points, but these things end up as accidental complexities in your code. I see that you feel that something is wrong with the code -- hence the question. And it seems the same feeling I had, things simply didn't want to fit together.

The major concern here is that you are flipping the "abstraction layer order" and hence end up mixing lower-level details with the higher level goals.

My question was more related to organizing these packages that aren't necessarily static. Although my application needs an authentication package, the actual service that this package uses (i.e. what implements the package's public-facing interface) can be implemented in many different ways, and I'd like for client code to be able to use them "nicely" (hence the MySQL example).

Anyways, to make the discussion more concrete - what exactly are you implementing and why does the end-user care about your web-app?

For simplicity let's call it "thingy", one possibility:

cmd/thingy-server --> pulls the 

cmd/thingy-server/user --> "user.Page", (implement an handler and uses thingy.DB)

cmd/thingy-server/home --> "home.Page"

cmd/thingy-server/admin --> "admin.Page"

db.go --> "thingy.DB"

user.go --> "thingy.User"

page.go --> "thingy.Page"

// implements DB

mysqldb/db.go

mysqldb/user.go

mysqldb/page.go

This is useful if you intend the "thingy" to be reusable and have multiple different servers. e.g. this is the approach I used for https://github.com/egonelbre/fedwiki, because the core server stays the same, but there are lots of different configurations that it can run.

Or...

user/mysql-userrepo/...

user/info.go

user/repo.go

page/mysql-pagerepo/... (package is named "pagerepo")

page/info.go

page/repo.go

pages/server.go

admin/server.go

admin/service.go

admin/page.go

home/page.go

main.go

If you intend it to contain lots of different functionalities and integrate them... (this I've used in another project, not public)

Here the main difference is caused by the difference in how important things are... in the 1st case, the important bit is how "User" and "Page" relate to each other, and the "server" becomes an implementation detail. In the second version the important pieces are the functionalities it has and "main.go" brings those pieces together.

There are tons of different ways of implementing those, but the main thing that controls the layout and where things go, is the "value" part and why it is "valued".

+ Egon

[hugo...@gmail.com]

Your application doesn't need "handlers", your application needs some specific functionality. a way to manage users... "handler" part is an annoying implementation detail.

My reasoning behind having handlers in a seperate package is that those are the "clients" of the entire project. Basically, if all packages work without handlers (e.g. by using their public interfaces from client-code, such as creating a cli-tool, a web server or a GUI app, etc..), why should they also include handlers? That doesn't really feel right to me.

Basically, my idea is to create the packages that cover your functionality and have each package provide a public interface (i.e. what I'm calling services) that allow client-code to access all functionality. Then, if your "client" decides to have e.g. a HTTP server to expose some (or all) of this functionality, he can build routes and handlers (using whatever framework he desires and not what the packages expose for him) and that's it. If he wants to expose via TCP or via a cli-tool, same thing (i.e. the "presentation layer" is split from the actual functionality).

You might need an abstraction layer or a boundary, but not necessarily "services".

In this sense, a "service" is just the name of the "layer" that can be used by client-code to interact with the package functionality.

Anyways, to make the discussion more concrete - what exactly are you implementing and why does the end-user care about your web-app?

This was literally just an example I made up (I'm not actually implementing anything that needs users or whatever), so the examples are appreciated. The project I'm looking at (it is already implemented but I'd like to polish it) is basically an admin tool that provides CRUD-like functionality over company-wide applications and domains (like a registry) and lets you interact with them in specific ways (so it needs authentication, for example). The packages are pretty much apps, authentication and a couple of other company-specific domains like countries. It's a Go web-app (because I wanted to see how Go development is like for web-apps) and interacts with a few things like MySQL and Memcached.

Here the main difference is caused by the difference in how important things are... in the 1st case, the important bit is how "User" and "Page" relate to each other, and the "server" becomes an implementation detail. In the second version the important pieces are the functionalities it has and "main.go" brings those pieces together.

I'll take a look at these and at your projects. Thanks for the discussion!

 

 

[Egon]

On Sunday, 4 October 2015 14:09:07 UTC+3, hugo...@gmail.com wrote:

Your application doesn't need "handlers", your application needs some specific functionality. a way to manage users... "handler" part is an annoying implementation detail.

My reasoning behind having handlers in a seperate package is that those are the "clients" of the entire project. Basically, if all packages work without handlers (e.g. by using their public interfaces from client-code, such as creating a cli-tool, a web server or a GUI app, etc..), why should they also include handlers? That doesn't really feel right to me.

In that case, yes, you probably would do something in between those two examples I showed. I.e. move the server things to a separate package, 

e.g. one potential:

cmd/thingy-server/main.go

cmd/thingy-server/admin/server.go

cmd/thingy-server/user/server.go

cmd/thingy-server/pages/server.go

// if these are really simple end-points, it might also make sense to have them just as:

cmd/thingy-server/admin_page.go

cmd/thingy-server/user_page.go

cmd/thingy-server/pages_page.go

user/mysql-userrepo/...

user/info.go

user/service.go

page/mysql-pagerepo/... (package is named "pagerepo")

page/info.go

page/service.go

admin/service.go

home/page.go

 

Basically, my idea is to create the packages that cover your functionality and have each package provide a public interface (i.e. what I'm calling services) that allow client-code to access all functionality. Then, if your "client" decides to have e.g. a HTTP server to expose some (or all) of this functionality, he can build routes and handlers (using whatever framework he desires and not what the packages expose for him) and that's it. If he wants to expose via TCP or via a cli-tool, same thing (i.e. the "presentation layer" is split from the actual functionality).

You might need an abstraction layer or a boundary, but not necessarily "services".

In this sense, a "service" is just the name of the "layer" that can be used by client-code to interact with the package functionality. 

Anyways, to make the discussion more concrete - what exactly are you implementing and why does the end-user care about your web-app?

This was literally just an example I made up (I'm not actually implementing anything that needs users or whatever), so the examples are appreciated. The project I'm looking at (it is already implemented but I'd like to polish it) is basically an admin tool that provides CRUD-like functionality over company-wide applications and domains (like a registry) and lets you interact with them in specific ways (so it needs authentication, for example).

Note had to invent lot of pieces here, to make it real enough... and called the server "dominator", because it sounded like a totally appropriate name.

In that case my initial packages layout would be:

cmd/dominator-server/main.go

cmd/dominator-server/login/page.go

cmd/dominator-server/domains/page.go

cmd/dominator-server/registry/page.go

cmd/dominator-server/app/page.go

// see note above, whether to create sub-packages or not...

auth/user.go

auth/provider.go

auth/repo.go

app/info.go

registry/service.go

registry/repo.go

domain/info.go

domain/service.go

registration/service.go

migration/service.go

While implementing this, I will start to get a better grasp how things should work together and adjust packages accordingly (e.g. merge some, split some).

the only interfaces here would be "auth.Repo" and "registry.Repo", everything else would be pretty much concrete implementations.

But, when "registration.Service" needs to depend on "registry.Service" and I want to mock-out registry.Service, then I would create an interface "registration.Registry", and implement the registration.Service using that. Essentially -- the contract how the "Registry" should look is inside "registration", but it's not a feature of registry.

I.e. the roles are local to the context that needs them and not -- i.e. not "it needs to be a registry.Service", but rather "I need a thing that does these things listed in registration.Registry".

Yes, you might get some duplication, but you can assure that the interfaces for a particular context are minimal to what they need.

Where to put the db implementations, it depends... I would either, create sub-package to their corresponding packages:

auth/authrepo-mysql/repo.go

Or, If I'm intending don't intend to vary the implementations by service. I would create a "mysqldb" package

// package is named "db"

db-mysql/auth.go

db-mysql/registry.go

db-mysql/app.go

db-mysql/domain.go

...

Here's one important point, the DB doesn't contain service logic, only DB logic, and the Service uses DB/Repo to implement it's logic.

The packages are pretty much apps, authentication and a couple of other company-specific domains like countries. It's a Go web-app (because I wanted to see how Go development is like for web-apps) and interacts with a few things like MySQL and Memcached.

Here the main difference is caused by the difference in how important things are... in the 1st case, the important bit is how "User" and "Page" relate to each other, and the "server" becomes an implementation detail. In the second version the important pieces are the functionalities it has and "main.go" brings those pieces together.

I'll take a look at these and at your projects. Thanks for the discussion!

Note, a lot of my code has been a discovery process, so they might not be the "best" version that is possible and sometimes it's better to leave things as they are, because the improvement might not have any real-world value other than making code look nicer. Basically, ask if you think something is wrong in my code --  it just might be, because I didn't notice or didn't have or didn't want to spend time on it.

 

+ Egon