Getting 502 errors in golang with heavy load

[Anand vishnu]

Hi, 

my code is running absolutely fine with a smaller load, when load is increased it is failing.

The changes i made in the newer release tag:

1. pushing data to aerospike at each and every chunk

when i turn off this line it is working fine , but when i turn this on it is failing

[Konstantin Khomoutov]

Please consider the following set of facts:

* While you did not explain that, "502 errors" appear to mean
some HTTP requests complete with the server answering with responses
containing the 502 HTTP status code.

Sadly, you did not tell us which HTTP requests actually fail this way.

* Aerospike does not use HTTP as its client protocol; it implements some
custom protocol over TCP sockets, so that 502

* HTTP status code 502 is defined to mean "Gateway Timeout".

Based on these facts I could do a guess: you have a setup like this:

[your clients] → [http proxy] → [your Go program] → [aerospike]

Depending on how your Go server is deployed, you can be easily unaware that an
HTTP proxy - such as nginx - is sitting between your server and your clients.

When the load from the client increases, the request handling code starts to
spend progressively more time in the calls which send data to Aerospike,
and eventually some of them spend so much time doing that they surpass a
maximum time span configured in that HTTP proxy which any request is allowed
to spend. Once a request trips that threshold, the proxy rips it and returns a
response to the client, telling it the service it mediates access to is not
keeping up.

[Konstantin Khomoutov]

On Fri, Aug 06, 2021 at 08:13:46PM +0300, Konstantin Khomoutov wrote:

[...]

> * Aerospike does not use HTTP as its client protocol; it implements some
> custom protocol over TCP sockets, so that 502

Sorry, fat fingers. I meant to say "... so that those '502 errors' could not
come from Aerospike". At least unless you're accessing it in some interesting
way, but then again - you did not tell us.