How to group handlers in a large web app

[David C Cohen]

Hi,

What would a large web app written in idiomatic go look like? Should the handlers be grouped based, for instance in a RESTful fashion, or should they be listed flatly in a very long file?

It appears there are at least ways of doing this:

Method 1: Idiomatic go

In response to not using dependency injection, David Symonds says:

Just write the function/method/whatever such that it takes its
dependencies. There's no need for a fancy framework.

The gddo server seems to be following this approach. The main web app file contains over 800 lines, and possibly growing. One could break this file into smaller files, but there doesn't seem to be any rules of organization.

Following David's advice, we will end up with a bunch of handlers, in a flat manner. This is not the most readable code to me. Anyone new to the code would wonder where to start from. or, which handlers are responsible for which resources.

To make this even worse, when adding a layer of authorization, it would be difficult to trace, or easy to make a mistake about which user can have access to which resource.

David's advice seems to be perfect for a single page little web app, but it doesn't fit for large apps.


Method 2: The so called frameworks

Web frameworks (if  you can name them so) in go are growing like mushrooms. 99% of these frameworks use reflection.

Rob Pike says:

If you find yourself wanting reflection on day 1, ask us about what 
you're up to. Chances are there's another approach that will be 
simpler and faster.

So why do these frameworks use reflection? Because they try to implement an MVC pattern. In an MVC pattern, the http handlers are grouped as the methods of the controller struct. Grouping is good to keep the code organized, and helps with creating RESTful routing with less code.

The problem is that, this handlers grouping comes with the cost of losing compile time safety, and runtime performance. This makes the point of chosing go as the programming language almost void.


The final question is, is there a way of achieving http handlers grouping without the use of reflection? What are those 'other approaches' that Rob mentioned which could make this 'simpler and faster'?

[Robert Melton]

On Wed, Nov 27, 2013 at 1:50 PM, David C Cohen <buend...@gmail.com> wrote:
> What would a large web app written in idiomatic go look like? Should the
> handlers be grouped based, for instance in a RESTful fashion, or should they
> be listed flatly in a very long file?

I really think this varies based on the task at hand. On our current
project, we use Gorilla Mux (http://www.gorillatoolkit.org/pkg/mux) --
not a framework, but a toolkit. When we do the big namespacing at the
top level, but then allow our packages to control everything under
that namespace.

... in our webserver we have something like

baseRouter := mux.NewRouter()
v1Router := baseRouter.PathPrefix("/v1").Subrouter()
users.RegisterRoutes(v1Router.PathPrefix("/users").Subrouter(), ms, c)
// ms is our db handle, c is our config

Since users got its own subrouter to play with, and we don't let
anything else use the prefix, the person working on users can stomp
around without worry of bumping into anyone else. Then the
users.RegisterRoutes looks something like (sr is the subrouter)...

sr.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
createOne(w, r, ms, c) }).Methods("POST")
sr.HandleFunc("/{id}", func(w http.ResponseWriter, r *http.Request) {
readOne(w, r, ms, c) }).Methods("GET").Headers("SessionID", "")

I am sure there are better and far more clever ways, but it has been
working well enough for us so far.

--
Robert Melton

[David C Cohen]

On Wednesday, November 27, 2013 7:36:49 PM UTC, Islan Dberry wrote:

On Wednesday, November 27, 2013 10:50:59 AM UTC-8, David C Cohen wrote:

The problem is that, this handlers grouping comes with the cost of losing compile time safety, and runtime performance. This makes the point of chosing go as the programming language almost void.

Although I am not a fan of frameworks, I don't think these issues are a concern. 

I think Rob Pike would not agree with you. At least the author of go-rest thinks that way:

https://github.com/ungerik/go-rest#go-rest-a-small-and-evil-rest-framework-for-go
 

A framework that uses reflection can validate the registered handlers at startup and exit or panic with a meaningful error. This is not as good as a compile time error, but it's not bad.

The cost of invoking a a handler using reflection is small compared to the cost of handling the HTTP request, fetching data from a database and so on. 

[codegangsta]

It honestly depends on what problems you are solving. I firmly believe that reflection can lead to simpler code, but there is also a lot of nasty uses of reflection out there.

There will be tradeoffs on both sides. I wrote Martini to be productive and to give me a better way to separate concerns in the application. Some complain about reflection overhead, which can range anywhere between 500ns and 2000ns per request depending on how much middleware you have. We are talking nanoseconds here! I can confidently say that if you cannot give up the 2 microseconds of overhead that Martini reflection provides then it is not for you.

On Wednesday, November 27, 2013 10:50:59 AM UTC-8, David C Cohen wrote:

[David C Cohen]

On Wednesday, November 27, 2013 8:32:41 PM UTC, codegangsta wrote:

It honestly depends on what problems you are solving. I firmly believe that reflection can lead to simpler code, but there is also a lot of nasty uses of reflection out there.

There will be tradeoffs on both sides. I wrote Martini to be productive and to give me a better way to separate concerns in the application. Some complain about reflection overhead, which can range anywhere between 500ns and 2000ns per request depending on how much middleware you have. We are talking nanoseconds here! I can confidently say that if you cannot give up the 2 microseconds of overhead that Martini reflection provides then it is not for you.

How is it related to the subject of grouping handlers based on resources?

[Matt Silverlock]

> The problem is that, this handlers grouping comes with the cost of losing compile time safety, and runtime performance. This makes the point of chosing go as the programming language almost void.

You write this as if reflection is so slow that it turns Go into Ruby/PHP.

Reflection should always be used tastefully. Not everyone will have the same tastes. But in many cases, using a little bit of reflection imparts a negligible performance penalty.

[timothy holmes]

I really appreciate this discussion as I have chosen Go to learn programming. It may be an advantage to not have to unlearn stuff. My task is to think about programming in the way the Go language requires, and maybe using the language to do things that work well in other languages, OO, web frameworks with MVC patterns, as two examples, is not letting me think like a Go programmer as best I could. So perhaps the overhead in using reflection, in the context of this discussion, is not that big a deal, but is it the best way to learn how to think like a Go programmer?

Does that make sense? I have no time constraints impinging on my learning, and it is not either or, but it seems better to focus on things like

"The Laws of Reflection", or how to use a toolkit like Gorilla, than it is to get a web app up and running quickly. Any comments would be appreciated. Thanks again for the discussion.

[codegangsta]

I would say go ahead and start with basic net/http and graduate to other packages as you see fit. Once you find pain trying to solve your issue with just net/http you can then go searching for a solution. Some people pick up Gorilla, some Martini, some Revel, and some stick with net/http.

Pick the right tool for the job, that is your choice and up to your discretion.

[David C Cohen]

On Wednesday, November 27, 2013 11:57:36 PM UTC, timothy holmes wrote:

So perhaps the overhead in using reflection, in the context of this discussion, is not that big a deal, but is it the best way to learn how to think like a Go programmer?

That's exactly my point. Reflection-based frameworks make it easy for the framework user to continue thinking in dynamic language way. On the other hand, the go-way is not obvious, if not impossible.

 

[Peter S]

On Wednesday, November 27, 2013 7:50:59 PM UTC+1, David C Cohen wrote:

Hi,

What would a large web app written in idiomatic go look like? Should the handlers be grouped based, for instance in a RESTful fashion, or should they be listed flatly in a very long file?

I agree with Robert Melton. You are best served with considering the requirements you have now, and choosing a structure that serves your needs without introducing needless complexity. For small apps a single may be fine. As it grows you may find it is better to break them into separate files with a sensible naming convention. Large apps may have handlers grouped in structs or even packages, one for every (use case | menu item | resource | whatever works best).
 

So why do these frameworks use reflection? Because they try to implement an MVC pattern. In an MVC pattern, the http handlers are grouped as the methods of the controller struct. Grouping is good to keep the code organized, and helps with creating RESTful routing with less code.

The problem is that, this handlers grouping comes with the cost of losing compile time safety, and runtime performance. This makes the point of chosing go as the programming language almost void.

"Almost void" is kind of strongly put. Personally I think that polar opposite viewpoints seldom tend to be correct or useful. There are problems and pain points with 100% static languages. There are problems and pain points with 100% dynamic languages.  The best solutions are usually found somewhere in between.
 

 

The final question is, is there a way of achieving http handlers grouping without the use of reflection? What are those 'other approaches' that Rob mentioned which could make this 'simpler and faster'?

Yes. Go supports pointers to methods. Group your handlers in a struct. Create an instance of the struct with required dependencies (these should be stateless), and pass references to its handler funcs to net/http (or pat, or gorilla/mux etc). This is still using dependency injection, but you do it manually. I do this in bones (see server.go, where handlers are initialized and dependencies are configured).

[André Moraes]

> Yes. Go supports pointers to methods. Group your handlers in a struct. Create an instance of the struct with required dependencies
> (these should be stateless), and pass references to its handler funcs to net/http (or pat, or gorilla/mux etc). This is still using dependency
> injection, but you do it manually. I do this in bones (see server.go, where handlers are initialized and dependencies are configured).

I would suggest that instead of pointers to methods store instances of
http.Handler

type MethodMap struct {
MethodOne, MethodTwo http.Handler
}

And if you only want functions just call http.HandleFunc

func main() {
mm := &MethodMap{}
mm.MethodOne := http.HandlerFunc(func(w http.ResponseWriter, req
*http.Request) { /* ... */ })
mm.MethodTwo := &MyHandler{}
// ....
}

type MyHandler struct {
// some fields
}

func (m *MyHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
// ....
}

[Peter S]

Interesting. What would you say is the main advantage of your approach vs this?

package main

import (
    "io"
    "net/http"
)

type myHandler struct {
    message string
}

func (h myHandler) HelloWorld(res http.ResponseWriter, req *http.Request) {
    io.WriteString(res, h.message)
}

func main() {
    handler := myHandler{message: "hello world"}
    http.HandleFunc("/", handler.HelloWorld)
    http.ListenAndServe(":8080", nil)

[David C Cohen]

On Friday, November 29, 2013 2:05:34 PM UTC, Peter S wrote:

Interesting. What would you say is the main advantage of your approach vs this?

One problem is that you are creating a single myHandler for the whole app. In practice, we want a new myHandler per request, as the fields could be request specific. Example:

type myHandler struct {
    message string

    currentUser *User
}

In this case, some function has to create an instance of myHandler based on the request path, that's how people end up using reflection.
 

[Peter S]

That is true, there is a single instance of myHandler, and (most likely) single instances of the dependencies it is using.

However, I don't regard that as a problem. The dependencies need to be stateless. If you are injecting typical services, they should be stateless (i. e not request specific) anyway. For request specific dependencies, such as your currentUser example, you can inject a factory that returns a request specific value, or you can use a package like gorilla/context. Implement a wrapper/filter around the handler to set the context value if you prefer.

[André Moraes]

On Fri, Nov 29, 2013 at 12:05 PM, Peter S <peter...@gmail.com> wrote:
> Interesting. What would you say is the main advantage of your approach vs
> this?

You can use One single handler to dispatch to other handlers removing
the need to use reflection (yes, this is just a router like
gorilla/mux).

type handlerMap struct {
A, B, C http.Handler
}

func (h *handlerMap) ServeHTTP(w http.ResponseWriter, req *http.Request) {
if h.shouldCall(h.A, req) {
h.call(h.A, prepareA, w, req)
} else if h.shouldCall(h.B, req) {
h.callB(w, req)
}
}

func (h *handlerMap) call(h http.Handler, prepareRequest func(req
*http.Request), w http.ResponseWriter, req *http.Request) {
prepareRequest(req)
h.ServerHTTP(w, req)
}

Doing that remove the need for a reflection approach since
prepareRequest would be responsible for setting things up to the
actual handler. Personally I implement those things as "filters" (a
list of handlers called before/after a given handler) and use the req
variable as a data transport between the filters.

[David C Cohen]

On Friday, November 29, 2013 6:46:20 PM UTC, André Moraes wrote:

You can use One single handler to dispatch to other handlers removing
the need to use reflection (yes, this is just a router like
gorilla/mux).  

type handlerMap struct {
  A, B, C http.Handler
}

How does this group handlers per resource? Are you suggesting one handlerMap per resource?
 

[Peter S]

Nice approach.  Request specific dependencies are stored directly on the http.Request, as form values?

I use the Struct/MethodReference technique combined with a filter implementation and Gorilla/mux + context to achieve pretty much the same results.

There is another approach, and it is possibly the simplest one of all. Treat the HandlerFuncs/Handlers as a thin "glue" layer that creates and wires up services/actions in any manner required and sets them to work. Just a bridge/adapter between the web and your application. No fancy tricks, but it gets the job done. And it is very explicit.

[David C Cohen]

On Friday, November 29, 2013 7:49:04 PM UTC, Peter S wrote:

Nice approach.  Request specific dependencies are stored directly on the http.Request, as form values?

I use the Struct/MethodReference technique combined with a filter implementation and Gorilla/mux + context to achieve pretty much the same results.

There is another approach, and it is possibly the simplest one of all. Treat the HandlerFuncs/Handlers as a thin "glue" layer that creates and wires up services/actions in any manner required and sets them to work. Just a bridge/adapter between the web and your application. No fancy tricks, but it gets the job done. And it is very explicit.

Could you provide a simple code for this?
 

[Peter S]

Sure: http://play.golang.org/p/AHkp5acsJm

Now, this is very rough code, but hopefully it demonstates what I mean. I would possibly even move things like template rendering out of the "handler layer", to simplify unit testing. The main idea is that as much logic as possible is moved out of the handlers.

[artem....@gmail.com]

Method 2: The so called frameworks

Web frameworks (if  you can name them so) in go are growing like mushrooms. 99% of these frameworks use reflection.

Gin-Gonic framework doesn't use reflection. It's fast and easy to use https://github.com/gin-gonic/gin 

If you want to organize versions for your api   i.e.  /v1  /v2  /v3  - I prefer to have different executables(one for each version of my API) on different ports and then connect them together using Nginx. When new version is out - I'm just placing exe file and changing Nginx config for new route.