Detecting / Avoiding network bottleneck when sending concurrent HTTP requests to many hosts.

[nohehf]

Hi! I'm building a tool that needs to concurrently send request to multiple hosts (for fingerprinting purposes).
I'm using github.com/valyala/fasthttp for the client but it works the same with net/http.
The main program basically starts n workers and those will take endpoints to scan from a common channel.
It's working fine but as I try to increment the number of workers at some point i'll basically DOSing the machine it runs on which prevents some/all requests to be sent, and even impact any other service running in the machine (k8s cluster in fact).
Any Idea on how I can detect this kind of network bottleneck within my program ?
I'd like to be able to maximise speed (number of workers) without destroying the network. I can't really rely on the hosts responses to detect if the network is flooded as a DNS not found or request timeout can happen legitimately on some hosts (the program can scan non existent / very slow hosts).
Thank you!

PS: I'm pretty new to go and I'm a first time poster here, so please understand my lack of knowledge if any :)

[Robert Engels]

Have you used netstat to check how many open connections you have? You probably need to forcibly close them so they don’t end up in time wait states. 

Also, you don’t say what N is?

Lastly you will get different errors for a dns or host error than a resources exhausted. If you configure the maximum sockets/files allowed to be open by the process you won’t DOS the machine. 

On Mar 6, 2023, at 8:07 AM, nohehf <nohe.hinn...@gmail.com> wrote:

Hi! I'm building a tool that needs to concurrently send request to multiple hosts (for fingerprinting purposes).

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/60b4a477-2731-4609-a927-bb33ad3d409bn%40googlegroups.com.

[nohehf]

So I unfortunately answered to this via personal email so here's a quick summary of the exchange here (in case anybody else can help):

The thing that is weird is that N is pretty low (even with ~20/30 workers it will start to have trouble working, missing some fingerprints).
I'll try to netstat, but I limited the client to 1 concurrent connection per host and the workers only work on one host at a time, so my maximum number of connections should be around the number of workers, and I never had a file descriptor issue.
The thing I observe is that when increasing the charge I get more and more timeouts and dialing errors on my requests, so maybe I can try to back off when that happens ?
But in general how can you max out the number of workers / concurrent connections / speed without having issues ?
Thank you!
Btw I'm gonna open source the code, in a few days probably, i'll link it here :)

[Reply]

How many connection attempts per second are you doing?

Btw - pretty sure this has nothing to do with Go - but it is general resource management. 

[Answer]
The connection for each endpoint is 2sec and I drop it at the first timeout, so theoretically I should have n/2 connections per second, but it can be a bit more as I immediately skip hosts that I can't lookup on the dns, so the number of connection opened per second is included in [n/2; n] n being the number of workers.
   

I didn't mean it was the language's fault aha (it's almost never the case), but I wonder how I can detect the bottleneck in my program and improve its performance. The best solution would be that the program can detect network bottlenecks and backoff (with some exponential backoff time). I tried several approaches to do so but nothing was concluent. I asked the question as the amount of concurrent connections that starts to be problematic is fairly low and I believe it's not a resource issue, or it could be avoided in the code. Also I think i'm not the first guy trying to send a lot of requests to a lot of hosts so I believe there might be a solution.

PS: Thanks for helping !