Memory leak in gorouter?

[aa...@hubernet.net]

We've got a v147 environment deployed and are having problems with our routers stopping handling requests after a random period of time.  First symptom is that they stop logging anything to the access.log, but we still see entries going to gorouter.log about registrations.  Soon it will stop responding to direct HTTP requests (usually it will return a 404 if you hit it directly) but for a while it still proxies other requests, and soon after that the /routes and /varz queries to port 8080 will also stop responding.  Eventually it will stop handling any proxy requests but still accept TCP connections so haproxy keeps trying to send requests to it but they eventually time out.

 

Memory usage is increasing regularly on each of the services and the symptoms seems to start at around 200MB total used.  Is anyone else seeing this?  An easy way to look at the memory usage is by querying /varz.

 

Aaron Huber

Intel Corporation

[James Bayer]

we're not seeing this at pivotal aaron. we're now running v149 and higher in our systems.

perhaps you can send it a signal to get details of what's going on with a stack? here's a suggestion for how you do that. 

https://groups.google.com/d/msg/golang-nuts/jAdvY7kcIiY/7VqVwzVI_j8J

you may also want to try dumping the routing table at /routes to see if there are too many entries there? hard to say with the information provided thus far.

https://github.com/cloudfoundry/gorouter#instrumentation

let us know what you find out.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Thank you,

James Bayer