Specifying Worker Process IDs in Output File?

[Brian Brandes]

Hello,

I'd appreciate some help here, as I've spent the better part of my
work day slamming my head against the wall with this one.

To give some background, I'm trying to benchmark a perl web
application running on Apache, Perl 5.8.9. Naturally, I'm using the
Devel::NYTProf::Apache package for this. In the simplest of setups,
when I only put "PerlModule Devel::NYTProf::Apache" in httpd.conf. I
then run httdperf to slam my webserver, and I see plenty of files that
match the default format of /tmp/nytprof.$parent.out.$$. This is fine
and dandy, since each of my apache processes spawns many workers, but
the problem I'm running into is that when I run nytperfhtml on any of
these, I'm experiencing a couple problems, one the line by line
profiling isn't working out, numerous lines are skipped and the
statement count is very off. And secondly, I'm unable to see any of
the files that are being run via perl -e.

In an attempt to solve the above problem, I wanted to try out using
the settings optimize=0:use_db_sub=1. So, I added "PerlSetEnv NYTPROF
optimize=0:use_db_sub=1" to httpd.conf. I restart apache, slam my
webserver, but now, despite not supplying any output file location,
the files now are output to my current working directory (not /tmp/).
Furthermore, they no longer have their worker process IDs. This is
bad, because the vast majority of the work is being done in these
worker threads, and these files seem to only contain the
initialization sequences.

Essentially what I'm trying to figure out is if there a way to specify
the worker process id within the file portion of the NYTPROF
environment variable. I was hoping specifying the file such as
"PerlSetEnv NYTPROF optimize=0:use_db_sub=1:file=/nytprof/nytprof.
$parent.out.$$" would work, but that only literally places the dollar
signs in the file name.

So, if anyone knows how to solve the specifying the worker ID problem,
I'd most appreciate it. If you have another solution for getting the
nytperfhtml report generation to include the perl -e files (outside of
upgrading perl, though I may wind up doing that), I'd much appreciate
it.

Thanks,

Brian

[Tim Bunce]

On Wed, Aug 17, 2011 at 01:49:51PM -0700, Brian Brandes wrote:
>
> To give some background, I'm trying to benchmark a perl web
> application running on Apache, Perl 5.8.9. Naturally, I'm using the
> Devel::NYTProf::Apache package for this. In the simplest of setups,
> when I only put "PerlModule Devel::NYTProf::Apache" in httpd.conf. I
> then run httdperf to slam my webserver, and I see plenty of files that
> match the default format of /tmp/nytprof.$parent.out.$$. This is fine
> and dandy, since each of my apache processes spawns many workers, but
> the problem I'm running into is that when I run nytperfhtml on any of
> these, I'm experiencing a couple problems, one the line by line
> profiling isn't working out, numerous lines are skipped and the
> statement count is very off.

I don't know what that would be.

> And secondly, I'm unable to see any of
> the files that are being run via perl -e.

> In an attempt to solve the above problem, I wanted to try out using
> the settings optimize=0:use_db_sub=1. So, I added "PerlSetEnv NYTPROF
> optimize=0:use_db_sub=1" to httpd.conf. I restart apache, slam my
> webserver, but now, despite not supplying any output file location,
> the files now are output to my current working directory (not /tmp/).

The docs say:

If the NYTPROF environment variable isn't set at the time
Devel::NYTProf::Apache is loaded then Devel::NYTProf::Apache will issue
a warning and default it to:

file=/tmp/nytprof.$$.out

(The docs ought to mention that the addpid an endatexit options are also
enabled, but directly, not via the NYTPROF environment variable.)

So you want to use

file=/tmp/nytprof.$$.out:optimize=0:use_db_sub=1

> Furthermore, they no longer have their worker process IDs.

Try

file=/tmp/nytprof.$$.out:optimize=0:use_db_sub=1:addpid=1

> Essentially what I'm trying to figure out is if there a way to specify
> the worker process id within the file portion of the NYTPROF
> environment variable. I was hoping specifying the file such as
> "PerlSetEnv NYTPROF optimize=0:use_db_sub=1:file=/nytprof/nytprof.
> $parent.out.$$" would work, but that only literally places the dollar
> signs in the file name.

You could just use PerlPassEnv NYTPROF and set the NYTPROF env var
yourself before starting apache.

> So, if anyone knows how to solve the specifying the worker ID problem,
> I'd most appreciate it. If you have another solution for getting the
> nytperfhtml report generation to include the perl -e files (outside of
> upgrading perl, though I may wind up doing that), I'd much appreciate
> it.

See http://search.cpan.org/~timb/Devel-NYTProf-4.06/lib/Devel/NYTProf.pm#PROFILING
specifically the PERL5OPT env var. Maybe best to set it within the
apache worker process so as not to interfere with the parent.

Tim.

[Paul Silevitch]

I figured out the solution to the missing line by line profiling.  We preload our modules in a startup script.  If I skip the preload when running Devel::NYTProf, line by line profiling appears.  I'm curious if someone knows why.  I'm using apache with prefork MPM so I imagine the issue is fork related.

Thanks,

Paul

[Paul Silevitch]

Turns out this is easy to explain.  We were trying to load the profiler conditionally based on an env variable and were 'requiring' it before we preloaded the modules with 'use'.  Use wraps the calls in a BEGIN block which means it actually got called before the require which isn't good for the profiler.  Wrapping the require in a BEGIN block solved the problem.

Thanks,

Paul

[Tim Bunce]

Thanks for the update Paul.

Could you take a look at the docs and suggest how they could be improved
so others don't hit the same problem?

A description of changes, like an extra paragraph, would be great.
A patch would be awesome! :)

Tim.

> See [1]http://search.cpan.org/~timb/Devel-NYTProf-4.06/lib/Devel/NYTProf.pm#PROFILING

> specifically the PERL5OPT env var. Maybe best to set it within the
> apache worker process so as not to interfere with the parent.
>
> Tim.
>

> --
> --
> You've received this message because you are subscribed to
> the Devel::NYTProf Development User group.
>
> Group hosted at: [2]http://groups.google.com/group/develnytprof-dev
> Project hosted at: [3]http://perl-devel-nytprof.googlecode.com
> CPAN distribution: [4]http://search.cpan.org/dist/Devel-NYTProf
>
> To post, email: develnyt...@googlegroups.com
> To unsubscribe, email: develnytprof-d...@googlegroups.com
>
>
>
> References
>
> Visible links
> 1. http://search.cpan.org/~timb/Devel-NYTProf-4.06/lib/Devel/NYTProf.pm#PROFILING
> 2. http://groups.google.com/group/develnytprof-dev
> 3. http://perl-devel-nytprof.googlecode.com/
> 4. http://search.cpan.org/dist/Devel-NYTProf