Multiple HTTP Clients best practice

[Hawari Rahman]

Hello everyone,

We're building an API Gateway in our company. Basically what this gateway does is that it processes request from external clients and when doing so, it communicates with our microservices. Some request can have as much as 5 microservices to communicate with in order to be processed.

Now, previously we're using code generated with swagger-codegen in order to make http requests to our microservices, but after facing a problem by doing so, we've decided to compose our own code using net/http. Then another problem arise, after initial stress test it seemed that by using the code we've composed, the throughput (how many requests it can process per set of time) is way worse than when we previously used swagger-codegen. After short investigation, the differences between each implementation that we've found is:

Swagger-codegen instantiate a new http.Client for each request, capable to be injected by only http.Transport, whereas our implementation only use single http.Client to be shared when communicating with every microservices.

I'm curious on how this would become a huge difference. My understanding with Go so far is that the one who pools connection is the http.Transport, not the http.Client. Has anyone encountered similar behavior or have/currently build this kind of system? If so, what kind of approach do you guys take? Do you:

• Share single http.Client across every request, but with some kind of tweaking
• Instantiate a separate http.Client for each host/microservice
• Instantiate a http.Client on every request
• or others I haven't got a clue about

If anyone asking, yes, we've set the max_idle_connections and max_idle_connections_per_host in the http.Transport, so it's not default.

Any kind of insights will be appreciated, since we're still early in adopting Go, thank you very much. :)

[Shawn Milochik]

One possibility I saw in a talk on YouTube a couple of years ago (sorry, couldn't find the link).

1. Have a buffered channel of clients.
   
2. Try to grab a client from the channel for each request, using a time.Timer to time out after a reasonable amount of time (one second, 100 ms, whatever)
3. If it times out, create a new client and use it.
4. Regardless of whether you re-used or created the client, put it into the channel when you're done with it (use a Timer again and just drop it if it times out).

You end up re-using a fixed-size set of clients. You're not using one, and you're not using one per request. Somewhere in between.

[Jakub Labath]

My first instinct is to simply use one http.Client per each micro service.

So when talking to microservice A - i would retrieve the client for microservice A from somewhere.
The immediate gain is that it presumably already has tcp connection (or several) open to the service A.

It is also possible to overwrite the dial methods to hook in your own code for tracing connections. To see when they are opened and closed.

At work we wrote this to abstract and share setting as they might be relevant to talking to each service.

https://github.com/gadventures/httpclient

Good luck

Jakub

[Hawari Rahman]

Hi Shawn,

That's a nice suggestion, but one question comes to mind though: I'm curious on where our previous implementation failed at. From what I learned at golang website, it states that an http.Client is concurrent safe and should be reused instead of created at will. So when should we consider using multiple http.Clients instead of one? Is it based on the load solely?

[Hawari Rahman]

Hi Jakub,

Yes, that's what we're finally doing now, we're using separate http.Client for each microservice. And I can say that the performance improvement is quite significant. Based on your comment, can you maybe give me an insight on the relationship between http.Client and http.Transport? Can I assume that the performance increase is now due to each client have its own connection pool (from its http.Transport) dedicated to one host only?

[Shawn Milochik]

Very good question. I don't know, and obviously it would be preferable to understand what's really going on. From a distance the best I can do is guess. 

If you did try it with multiple clients and it fixes the problem then that should point you in the right direction. If not, try Go's profiler, different machines, different operating systems, different LANs, etc.

It could even be a problem on the server side -- the app or Apache or the OS's network stack could have a buffer issue or some resource not being freed quickly enough -- who knows?

Maybe create a simple mock backend and run it locally and hammer it. If you can reproduce the problem that way, that would be very telling.

[Hawari Rahman]

Hi Shawn,

Based on what we've experienced after using separate http.Client per microservices (see my previous reply), my best guess that it was because of connection reuse. Since we now have each connection pool dedicated to single host only, it helps performance quite significantly.

Thank you for your suggestions, I really appreciate it :)