Re: [BUGS] Patch to allow C extension modules to initialize/finish
Tom Lane
> Hence I propose the patch below (applies to PostgreSQL 8.1.4) which
> mimics the dlopen(3) and dlclose(3) behaviour of some Unix platforms
> and resolves and calls _PG_init and _PG_fini functions of an extension
> module right after/before the pg_dlopen/pg_dlclose calls in the FMGR.
This seems like a reasonably good idea, and we have got uses for at
least the "init" case in most or all of our PLs. It's nominally too
late for 8.2 feature freeze, but I said just a couple days ago that
we shouldn't take a very hard line on that. Does anyone object to
considering this for 8.2?
One question I have is whether it really works as expected in all cases.
In particular what if the library is "preloaded" into the postmaster?
Both plpgsql and plperl seem to think they might need to make a
distinction between things to do at library load time and things to do
per-backend ... and yet, neither of them *actually* have anything they
need to do per-backend.
Also, what about Windows? I assume that DSOs don't remain attached
across the pseudo-fork/exec, will that mess anything up?
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings
David Fetter
> "Ralf S. Engelschall" <r...@engelschall.com> writes:
> > Hence I propose the patch below (applies to PostgreSQL 8.1.4)
> > which mimics the dlopen(3) and dlclose(3) behaviour of some Unix
> > platforms and resolves and calls _PG_init and _PG_fini functions
> > of an extension module right after/before the pg_dlopen/pg_dlclose
> > calls in the FMGR.
>
> This seems like a reasonably good idea, and we have got uses for at
> least the "init" case in most or all of our PLs. It's nominally too
> late for 8.2 feature freeze, but I said just a couple days ago that
> we shouldn't take a very hard line on that. Does anyone object to
> considering this for 8.2?
Nope :)
> One question I have is whether it really works as expected in all
> cases. In particular what if the library is "preloaded" into the
> postmaster? Both plpgsql and plperl seem to think they might need
> to make a distinction between things to do at library load time and
> things to do per-backend ... and yet, neither of them *actually*
> have anything they need to do per-backend.
I'm not sure quite what you mean here, but PL/PerlU functions can
use() modules, and those are called per-backend, i.e. when the
function is invoked. There's also some possibility that something
might go into %_SHARED.
Cheers,
D
--
David Fetter <da...@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter
Remember to vote!
Tom Lane
> On Thu, Aug 03, 2006 at 05:30:48PM -0400, Tom Lane wrote:
>> One question I have is whether it really works as expected in all
>> cases. In particular what if the library is "preloaded" into the
>> postmaster?
> I'm not sure quite what you mean here, but PL/PerlU functions can
> use() modules, and those are called per-backend, i.e. when the
> function is invoked. There's also some possibility that something
> might go into %_SHARED.
Well, the point is that you could have a scenario where the PG_init
function is executed in the postmaster, the process image is duplicated
via fork(), and then in a specific backend a LOAD command is executed
causing the PG_fini function to be called. Is it likely that anything
would get confused by PG_init and PG_fini getting called by different
processes?
Also, if we do this we probably ought to remove the special-purpose
hack for preload_libraries to specify an init function --- it should
just happen by default. Any objections to simplifying that?
regards, tom lane
Andrew Dunstan
Tom Lane wrote:
>"Ralf S. Engelschall" <r...@engelschall.com> writes:
>
>
>>Hence I propose the patch below (applies to PostgreSQL 8.1.4) which
>>mimics the dlopen(3) and dlclose(3) behaviour of some Unix platforms
>>and resolves and calls _PG_init and _PG_fini functions of an extension
>>module right after/before the pg_dlopen/pg_dlclose calls in the FMGR.
>>
>>
>
>This seems like a reasonably good idea, and we have got uses for at
>least the "init" case in most or all of our PLs. It's nominally too
>late for 8.2 feature freeze, but I said just a couple days ago that
>we shouldn't take a very hard line on that. Does anyone object to
>considering this for 8.2?
>
>
I don't. We've been porous in the past and I think we should be prepared
to be a bit lenient again, especially since this release is not hugely
feature rich.
>One question I have is whether it really works as expected in all cases.
>In particular what if the library is "preloaded" into the postmaster?
>Both plpgsql and plperl seem to think they might need to make a
>distinction between things to do at library load time and things to do
>per-backend ... and yet, neither of them *actually* have anything they
>need to do per-backend.
>
>
I have longterm plans for plperl concerning preloading perl modules,
which might involve the preloaded lib. At the moment it's just a thought
in my head, though.
>Also, what about Windows? I assume that DSOs don't remain attached
>across the pseudo-fork/exec, will that mess anything up?
>
>
>
>
Good question.
cheers
andrew
---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
Joe Conway
>
> Also, if we do this we probably ought to remove the special-purpose
> hack for preload_libraries to specify an init function --- it should
> just happen by default. Any objections to simplifying that?
>
The original idea of using the init function with preload_libraries was
to eliminate library startup that was expensive and only needed once.
Specifically in the case of libR (and presumably other libraries as
well), the init time was much greater than the actual library load time.
If it is removed from preload_libraries, then we'll pay that price for
every backend startup, no?
Joe
Tom Lane
> Tom Lane wrote:
>> Also, if we do this we probably ought to remove the special-purpose
>> hack for preload_libraries to specify an init function --- it should
>> just happen by default. Any objections to simplifying that?
> The original idea of using the init function with preload_libraries was
> to eliminate library startup that was expensive and only needed once.
> Specifically in the case of libR (and presumably other libraries as
> well), the init time was much greater than the actual library load time.
> If it is removed from preload_libraries, then we'll pay that price for
> every backend startup, no?
No, my thought is that you'd rename PL/R's init function to PG_init, and
then it'd get called automagically without needing to assume that the DBA
remembers to specify it in preload_libraries. If there's a reason *not*
to do that then it'd be a strike against this whole proposal, methinks.
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?
Joe Conway
> Joe Conway <ma...@joeconway.com> writes:
>>Tom Lane wrote:
>>
>>>Also, if we do this we probably ought to remove the special-purpose
>>>hack for preload_libraries to specify an init function --- it should
>>>just happen by default. Any objections to simplifying that?
>
>>The original idea of using the init function with preload_libraries was
>>to eliminate library startup that was expensive and only needed once.
>>Specifically in the case of libR (and presumably other libraries as
>>well), the init time was much greater than the actual library load time.
>>If it is removed from preload_libraries, then we'll pay that price for
>>every backend startup, no?
>
> No, my thought is that you'd rename PL/R's init function to PG_init, and
> then it'd get called automagically without needing to assume that the DBA
> remembers to specify it in preload_libraries. If there's a reason *not*
> to do that then it'd be a strike against this whole proposal, methinks.
Oh, well that sounds perfect to me. At least in the case of a procedural
language handler you can easily initialize and cache anything that must
be done per-backend anyway. It's the "expensive but must be done at
least once stuff" that was a problem. As long as that happens, I'm
happy. And if we eliminate a dba dependency, so much the better.
Joe
Gregory Stark
> No, my thought is that you'd rename PL/R's init function to PG_init, and
> then it'd get called automagically without needing to assume that the DBA
> remembers to specify it in preload_libraries. If there's a reason *not*
> to do that then it'd be a strike against this whole proposal, methinks.
If I understand the question correctly it hinges on whether you want to do all
the initialization pre-fork or post-fork? I'm pretty sure you have to allow
for both possibilities.
I know when I was using mod_perl heavily we wanted to load as many perl
modules and code pre-fork as possible. The more we loaded pre-fork the more
memory was shared across processes and the more processes we could run on a
box without suffering from memory pressure.
On the other side the classic case of something that cannot be set up pre-fork
is actually database connections :) So, for example if for someone wanted to
have a persistent Oracle connection they could not open it pre-fork at library
load time but they might want to open it immediately after the fork rather
than when it's first used.
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?
Tom Lane
> So, for example if for someone wanted to
> have a persistent Oracle connection they could not open it pre-fork at library
> load time but they might want to open it immediately after the fork rather
> than when it's first used.
Uh ... why? Seems like all you're accomplishing there is to expend
cycles that might be wasted, if the particular session never uses
the feature.
In any case, the PG_init proposal neither adds nor takes away ability
to do stuff immediately post-fork, so I think that's an orthogonal
consideration.
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend
Greg Stark
> Gregory Stark <gss...@mit.edu> writes:
> > So, for example if for someone wanted to
> > have a persistent Oracle connection they could not open it pre-fork at library
> > load time but they might want to open it immediately after the fork rather
> > than when it's first used.
>
> Uh ... why? Seems like all you're accomplishing there is to expend
> cycles that might be wasted, if the particular session never uses
> the feature.
Well as long as we're talking hypothetically I could come up with plenty.
Maybe you know every session will be using the connection and you can't deal
with the latency creating the connection may create in your critical path. Or
maybe you don't want to deal with connection errors from the depths of your
call tree where it'll cause a user-visible error. Or perhaps you're using some
library that expects to be handed a connection from some initialization
routine and it's just not convenient to or possible to invoke that on an
on-demand basis.
> In any case, the PG_init proposal neither adds nor takes away ability
> to do stuff immediately post-fork, so I think that's an orthogonal
> consideration.
So is the only question whether there's a need to do stuff pre-fork?
--
greg
Tom Lane
> Tom Lane <t...@sss.pgh.pa.us> writes:
>> In any case, the PG_init proposal neither adds nor takes away ability
>> to do stuff immediately post-fork, so I think that's an orthogonal
>> consideration.
> So is the only question whether there's a need to do stuff pre-fork?
That's not a question, that's a well-established fact --- pl/R certainly
needs it, and any other library that has expensive setup work that can
propagate through a fork does too.
I think adding a hook to allow a postmaster-preloaded library to execute
some work immediately post-fork is a separate consideration. Feel free
to propose it if you want, but I don't see what it's got to do with the
patch on the table.
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster
Tom Lane
> Hence I propose the patch below (applies to PostgreSQL 8.1.4) which
> mimics the dlopen(3) and dlclose(3) behaviour of some Unix platforms
> and resolves and calls _PG_init and _PG_fini functions of an extension
> module right after/before the pg_dlopen/pg_dlclose calls in the FMGR.
Patch applied, with consequent changes to simplify preload_libraries
feature in favor of using _PG_init().
regards, tom lane
---------------------------(end of broadcast)---------------------------
Ralf S. Engelschall
> "Ralf S. Engelschall" <r...@engelschall.com> writes:
> > Hence I propose the patch below (applies to PostgreSQL 8.1.4) which
> > mimics the dlopen(3) and dlclose(3) behaviour of some Unix platforms
> > and resolves and calls _PG_init and _PG_fini functions of an extension
> > module right after/before the pg_dlopen/pg_dlclose calls in the FMGR.
>
> Patch applied, with consequent changes to simplify preload_libraries
> feature in favor of using _PG_init().
Thanks.
Ralf S. Engelschall
r...@engelschall.com
www.engelschall.com
---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majo...@postgresql.org so that your
message can get through to the mailing list cleanly