Postgres connection pool

[mahe...@thoughtworks.com]

Hi All,

We are trying to do 10K simultaneous reads from postgres. For this, we are spawning 10K goroutines. The code is https://github.com/mahendrakariya/postgis_spike/blob/master/src/read_write.go

Whenever we run the program, it throws an error read: connection reset by peer. We have increased the postgres max_connections to 10000. We even added pgbouncer, but it looks like (based on logs) the connections are not going thru pgbouncer, rather they are directly hitting the database.

Could someone guide us how to do this? Also, is there a better approach other than spawning 10K goroutines?

Thanks.

[James Bardin]

On Tuesday, December 15, 2015 at 8:27:15 AM UTC-5, mahe...@thoughtworks.com wrote:

Whenever we run the program, it throws an error read: connection reset by peer. We have increased the postgres max_connections to 10000. We even added pgbouncer, but it looks like (based on logs) the connections are not going thru pgbouncer, rather they are directly hitting the database.

"connection reset by peer"  means the remote host (the postgres server) reset the connection. There's nothing you can do in your Go code until the server is properly tuned to handle 10k connections. 

I don't know if postgres will accept that many simultaneous connections, but I can assure you if it does it's not going to handle it well. This is why you use a connection pool to limit the number of simultaneous connections to the appropriate number for your workload.  

[Nigel Vickers]

http://hans.io/blog/2014/02/19/postgresql_connection/

[Manlio Perillo]

Il giorno martedì 15 dicembre 2015 14:27:15 UTC+1, mahe...@thoughtworks.com ha scritto:

Hi All,

We are trying to do 10K simultaneous reads from postgres.

Why?

 

For this, we are spawning 10K goroutines. The code is https://github.com/mahendrakariya/postgis_spike/blob/master/src/read_write.go

Whenever we run the program, it throws an error read: connection reset by peer. We have increased the postgres max_connections to 10000. We even added pgbouncer, but it looks like (based on logs) the connections are not going thru pgbouncer, rather they are directly hitting the database.

Could someone guide us how to do this? Also, is there a better approach other than spawning 10K goroutines?

It depends on what you are trying to do.

Also note that pgbouncer it not stricly required, you can use Go concurrency support to limit the number of connections to the database.

Regards  Manlio