Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Postgres max_connections & multithreading
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  5 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Marco Sehrer [ninjaconcept.com]  
View profile  
 More options Jul 31 2012, 6:32 am
From: "Marco Sehrer [ninjaconcept.com]" <m...@ninjaconcept.com>
Date: Tue, 31 Jul 2012 12:32:31 +0200
Local: Tues, Jul 31 2012 6:32 am
Subject: Postgres max_connections & multithreading

Hi Group,

we have a lot of LocalParticipants which need to connect  Postgres (via ActiveRecord).
Trying to run them multithreaded leads to too many postgres connections quite fast:

postgres.log:
FATAL:  sorry, too many clients already

Finally we run all Participants connecting to postgres via:

def do_not_thread
    true
end

regardless of multithreaded or not I encountered the following warning:

postgres.log
LOG:  unexpected EOF on client connection

btw. our Postgres (version 9.1) max_connections is set to 100.
the scenario happens regardless of the number of ruote-workers (1-5)

Thanks a lot!

Marco


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
John Mettraux  
View profile  
 More options Jul 31 2012, 7:30 am
From: John Mettraux <jmettr...@gmail.com>
Date: Tue, 31 Jul 2012 20:30:09 +0900
Local: Tues, Jul 31 2012 7:30 am
Subject: Re: [ruote:3711] Postgres max_connections & multithreading

On Tue, Jul 31, 2012 at 12:32:31PM +0200, Marco Sehrer [ninjaconcept.com] wrote:
> Hi Group,

Hello Marco,

> we have a lot of LocalParticipants which need to connect  Postgres (via ActiveRecord).
> Trying to run them multithreaded leads to too many postgres connections quite fast:

> postgres.log:
> FATAL:  sorry, too many clients already

> Finally we run all Participants connecting to postgres via:

> def do_not_thread
>   true
> end

Well done, limiting the participants to use the same thread as the worker
should limit them to use the one connection bound to the thread (IIRC that's
how activerecord does its connection management).

Thanks for sharing the tip!

> regardless of multithreaded or not I encountered the following warning:

> postgres.log
> LOG:  unexpected EOF on client connection

Sorry, I don't know what your participant looks like, how db-intensive it is.

I'd google for that error message, maybe pitching in "activerecord".

> btw. our Postgres (version 9.1) max_connections is set to 100.
> the scenario happens regardless of the number of ruote-workers (1-5)

Well, Ruby runtimes won't share connections...

By scenario, do you mean the scenario with #do_not_thread set to false that
triggers the "FAIL: too many clients already" or the scenario where
#do_not_thread is set to true and triggers "LOG: unexpected EOF"?

I am sorry, I have trouble understanding the variant articulation in your
report.

Best regards,

--
John Mettraux - http://lambda.io/jmettraux


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Marco Sehrer [ninjaconcept.com]  
View profile  
 More options Jul 31 2012, 7:58 am
From: "Marco Sehrer [ninjaconcept.com]" <m...@ninjaconcept.com>
Date: Tue, 31 Jul 2012 13:58:11 +0200
Local: Tues, Jul 31 2012 7:58 am
Subject: Re: [ruote:3713] Postgres max_connections & multithreading

Hi John,

thanks for the fast response!
I was digging deeper, and found a workaround :-)

begin
   .. process the AR model ..
ensure    
   ActiveRecord::Base.clear_active_connections!
end

LOG:  unexpected EOF on client connection
was because of an resque-background job .. seams not to be the fault of ruote :-)

Now in my participant I use this workaround for the 'too many clients already' error.

begin
   .. process the AR model ..
ensure    
   ActiveRecord::Base.clear_active_connections!
end

Seams to solve the problem :-)

Thanks,
Marco

> Best regards,

> --
> John Mettraux - http://lambda.io/jmettraux

> --
> you received this message because you are subscribed to the "ruote users" group.
> to post : send email to openwferu-users@googlegroups.com
> to unsubscribe : send email to openwferu-users+unsubscribe@googlegroups.com
> more options : http://groups.google.com/group/openwferu-users?hl=en

Schöne Grüße Marco

--

NinjaConcept GmbH
Marco Sehrer
Geschäftsführung

Amalienstrasse. 44
76133 Karlsruhe

fon:    (+49) 0721 1803523-1
fax:    (+49) 721 961402-99
mobile: (+49) 151 20314416

email:  m...@ninjaconcept.com
www:    http://www.ninjaconcept.com/


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Hartog De Mik  
View profile  
 More options Jul 31 2012, 8:25 am
From: Hartog De Mik <hartog.de....@gmail.com>
Date: Tue, 31 Jul 2012 14:25:13 +0200
Local: Tues, Jul 31 2012 8:25 am
Subject: Re: [ruote:3715] Postgres max_connections & multithreading

Marco,

Don't know if you saw my comments on #ruote - but did you have a look at
pgpool (www.pgpool.net) or pgbouncer (
http://pgbouncer.projects.postgresql.org/)?

You would ''just'' have to beware of this issue:
https://github.com/rails/rails/issues/1627

Kind regards,
Hartog.

2012/7/31 Marco Sehrer [ninjaconcept.com] <m...@ninjaconcept.com>

  logo_240x60.gif
2K Download

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Marco Sehrer [ninjaconcept.com]  
View profile  
 More options Jul 31 2012, 10:57 am
From: "Marco Sehrer [ninjaconcept.com]" <m...@ninjaconcept.com>
Date: Tue, 31 Jul 2012 16:57:26 +0200
Local: Tues, Jul 31 2012 10:57 am
Subject: Re: [ruote:3716] Postgres max_connections & multithreading

Hi Hartog,
thanks a lot, I didn't know about pgpool + pgbouncer.

looking forward for an afterwork beer after holiday :-)
Bye

Am 31.07.2012 um 14:25 schrieb Hartog De Mik:

Schöne Grüße Marco

--

NinjaConcept GmbH
Marco Sehrer
Geschäftsführung

Amalienstrasse. 44
76133 Karlsruhe

fon:    (+49) 0721 1803523-1
fax:    (+49) 721 961402-99
mobile: (+49) 151 20314416

email:  m...@ninjaconcept.com
www:    http://www.ninjaconcept.com/


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »