SelfLoader and 'open(FH, "-|")'

[mcn...@nin.iats.missouri.edu]

Hey all,

I have a weird problem. I maintain an application that I wrote that uses
the Net-SNMP SNMP.pm module (version 4.1.2). The app has a lot of
different functions, so I make heavy use of 'use SelfLoader' to keep the
runtime short.

Example:

jscan bs450custom 10.1.1.0/22 sysDescr

That will query all Nortel BayStack 450 switches in the subnet specified
for 'sysDescr'.

One of the more commonly-used functions of the script is 'ipfind'. This
function, given an IP address, will search through the switches in our
network and return the IP address and port number of the switch where the
given IP host is found.

Example:

jscan ipfind 10.1.5.3

Our security people make great use of this. Since they do, they recently
asked me if I could make the script accept multiple IP targets at once.
The command-line parsing part was pretty simple (comma-separated), but due
to the internals of the top-level package, it became simplest to do
something like this:

while ( $#{$args{iplist}} ) {
open (README, "-|");
if ( $pid ) {
while ( <README> ) {
print $_;
}
close(README) || warn "child exited $?";
shift @{$args{iplist}};
} else {
last;
}
}

The parent itself then processes the last item in the list.

Now here's the problem. I have a bunch of different functions that are
called from the top-level package. Each function is in it's own module,
since I use these same functions in other apps. So above the while loop
somewhere will be, say:

use jscanmacfind;

...and then later on will be:

macfind(args, moreargs, otherargs);

The jscanmacfind package has "use SelfLoader;" at the top. The macfind()
forward declaration is the only thing between the BEGIN and END blocks, so
the package winds up looking like this:

package jscanmacfind;

use (some packages);
use SelfLoader;

BEGIN { .... }

sub macfind($$$);

END { }

1;

__DATA__

sub macfind($$$) {
...code...
}

Now when I run my script with multiple IP addresses (thus children are
created), the first child executes successfully, but each subsequent
child, and the parent (the last iteration) all fail like this:

Missing right curly or square bracketsyntax error at (eval 26) line 113,
at EOF
at /usr/local/bin/jscan line 390
child exited 65280 at /usr/local/bin/jscan line 153.

Line 390 of /usr/local/bin/jscan is where macfind() is called. Line 153
is the "close(README) || warn..." line in the parent.

Here's the kicker: If I move the END block, the 1; statement, and the
__DATA__ token to the bottom of the module, the script executes
successfully, even if 'use SelfLoader;' is still invoked.

I have 'use strict;' in every module, including the top-level package.

I'd really rather not remove all of the SelfLoader stuff from my modules,
since they do speed things up quite a bit for this and other scripts, and
using the named pipe saves me from having to rewrite a bunch of other code
in the top-level package just to get a foreach loop.

So what does the SelfLoader do that breaks subsequent calls to a
self-loaded function by child processes?

--J

[mcn...@nin.iats.missouri.edu]

More information:

1) I am running perl version 5.6.1 on Linux (Slackware 8.0).

2) I found the following in 'man perldelta':

> Spurious subroutine stubs after failed subroutine calls

> Perl could sometimes create empty subroutine stubs when a subroutine was
> not found in the package. Such cases stopped later method lookups from
> progressing into base packages. This has been corrected.

Could it be that this problem was not truly solved in all cases?

3) This problem - AFAIK - does not occer with all SelfLoad-ed modules.
It does happen with other modules besides 'jscanmacfind.pm', but not every
module referenced from the top-level module.

Anyway, if someone knows the answer to this, great! If not, what's the
best place to look next? I'm completely confused by the particular syntax
error that I'm getting.

--J

[mcn...@nin.iats.missouri.edu]

A guess: The SelfLoader module has a hash named (surprise) %Cache, in
which it appears to store the previously-SelfLoad-ed modules. Is it
possible that this cache is the source of the problem?

If that's the case it would be awfully strange. Consider:

In the case of the macfind() function, the first process to call that
subroutine is the first child fork()-ed. That child runs successfully.
However, the next child fork()-ed does not run successfully. In theory,
each child is a copy of the parent. Fortunately, the first child has
already shut down before the second child is spawned, so there shouldn't
be any weird cross-communication there.

So what is the first child doing to the parent or the Perl interpreter that
is causing the next child to fail? If this were C, it would smell like
stale pointers to me.

Suggestions on where to look next? This is the kind of problem that I'm not
exactly sure how to diagnose. I more or less know what's going on, but I
haven't been doing this quite long enough to know how to debug it when it
gets whacked.

--J

mcn...@nin.iats.missouri.edu wrote: > More information:

[mcn...@nin.iats.missouri.edu]

Was just wondering if anyone had thought about this any and come up with
any ideas...

For now, I'm probably going to have to remove all of the SelfLoader
references and interpret everything at runtime. :-(

--J

[Andrew Gierth]

[bogus group comp.lang.perl removed]

>>>>> "mcnuttj" == <mcn...@nin.iats.missouri.edu> writes:

mcnuttj> Was just wondering if anyone had thought about this any and
mcnuttj> come up with any ideas...

(didn't see this before, I only read the perl groups intermittently)

I think your problem is that the child processes are messing up each
other's reading of the DATA filehandle. Without looking at how DATA is
implemented internally I can't be certain, but my guess is that it
actually leaves the original source FH open at that point. All the
child processes will end up sharing that FH, which means that only one
of them will be able to read it successfully.

The fix is actually very simple; force the data to be read immediately
(but not compiled, it's the compilation that takes the time) rather
than waiting for the first function call:

package jscanmacfind;

use (some packages);
use SelfLoader;

BEGIN { .... }

SelfLoader->load_stubs();

END { }

1;

__DATA__

sub macfind($$$) {
...code...
}

By calling load_stubs, you make SelfLoader read from DATA and locate
the subroutines in it (and declare them for you, saving you the bother
of doing it for yourself).

--
Andrew.

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Andrew Gierth
<and...@erlenstar.demon.co.uk>], who wrote in article <87vghqp...@erlenstar.demon.co.uk>:

> implemented internally I can't be certain, but my guess is that it
> actually leaves the original source FH open at that point.

Correct.

> All the
> child processes will end up sharing that FH, which means that only one
> of them will be able to read it successfully.

Hmm, should not fork() take care of this? Oups, the docs I have state:

o The child process has its own copy of the parent's
file descriptors and directory streams. Each of the
child's file descriptors shares a common file pointer
with the corresponding file descriptor of the parent.

I have been always saying than fork() is a broken solution for a non-problem...

Ilya

[mcn...@dnps-linux1.telecom.missouri.edu]

This solved the problem. My script now gets along with fork(). Thanks!

Now I just need to re-arrange my source modules and my Makefile to install
with *and* without SelfLoader...

--J