We are performing a migration to a new backend hosted on Google App Engine and based on Node.js.
Before performing the switch from the old to the new backend, we need to do tests and see if packets that the legacy server receives are correctly
processed by the new backend.
So, on the old backend, we have modified the handler to "duplicate the request" and both proceed with the regular processing and forward to out new backend.
In this way, we are able to keep the legacy system up and at the same time replicate the requests on our new backend.
We want to slow down the legacy server as little as possible, so what we do is we do not wait for a response on the legacy server.
Instead, on Google App Engine, we have deployed a second service (to
[our-proxy-service]-dot-[our-project].appspot.com) in out project that acts as a proxy and forwards the requests to the new backend.
We have use
node-http-proxy package to implement the proxy, and we made it redirect to out backend URL:
[our-backend-service]-dot-[our-project].appspot.com
The proxy also gets the responses so we are able to track the process.
The first question we have is whether this is a good solution with GAE,
and if it is a valid use case to redirect the requests from a service to another service in that way, or if there is a better way.
Here is a MSPaint version of our backend structure:
This brings us to the second problem: this structure actually seems to work, but every once in a while (no pattern discovered yet, seems random) the proxy gets a 502 Bad Gateway response from the backend.
We investigated a little on the Stackdriver Logs on the backend side, and we ended up finding this log filtering for nginx.error:
07:11:15.000 [error] 32#32: *84209 upstream prematurely closed connection while reading response header from upstream, client: 130.211.1.151, server: , request: "POST /collect HTTP/1.1", upstream: "http://172.17.0.1:8080/collect", host: "[ourprojectid].appspot.com"
We have any other ideas where to go from here.
Any help or hint or suggestion is very welcome.