Enabling/Disabling mod_pagespeed programatically on per request basis

[Marcus Hietala]

I know that one can turn mod_pagespeed on/off and control which
filters are applied for each request by adding ModPagespeed*
parameters to the URL.

Is there a list of some kind which filters are allowed to be
controlled in such a fashion?

As controlling mod_pagespeed thru URL parameters potentially allows
users to enable filters which I might not want to be run, how do I
disable the URL parameter parsing completely? Is there a config option
of some kind or do I have to change the source code for
ScanQueryParamsForRewriterOptions in net/instaweb/apache/
mod_instaweb.cc?

The application I'm working on uses mod_proxy to fetch the html pages
and a custom mod_perl module which makes some modifications to page
contents before it gets fed to mod_pagespeed.
What is the best way to enable/disable mod_pagespeed filters on per
request basis from my mod_perl module without having to rewrite the
original URL? Any suggestions will be greatly appreciated.

Are there any performance or other issues which I should be aware if I
change the which filters are ran on each request?

Thanks,
Marcus

[Joshua Marantz]

Hi Marcus,

Sorry it took a while to respond; you've asked some great questions.

On Tue, Apr 5, 2011 at 4:29 PM, Marcus Hietala <marcus....@gmail.com> wrote:

I know that one can turn mod_pagespeed on/off and control which
filters are applied for each request by adding ModPagespeed*
parameters to the URL.

Is there a list of some kind which filters are allowed to be
controlled in such a fashion?

Currently the complete list of filters can be specified as query params, as well as enabling/disabling the module.  No other options can currently be specified via query-params, although I've been tempted to add the inlining/outlining thresholds in order to facilitate iteratively tuning those parameters without requiring a server restart.

We would never allow query-param configuration of any of the server-scope parameters like file-cache paths or the maximum byte sizes of the cache, or the domain-mapping ones.

As controlling mod_pagespeed thru URL parameters potentially allows
users to enable filters which I might not want to be run, how do I
disable the URL parameter parsing completely? Is there a config option
of some kind or do I have to change the source code for
ScanQueryParamsForRewriterOptions in net/instaweb/apache/
mod_instaweb.cc?

There is currently no such option in mod_pagespeed but there probably should be.  But you've found the correct place if you want to hack source and rebuild.

The application I'm working on uses mod_proxy to fetch the html pages
and a custom mod_perl module which makes some modifications to page
contents before it gets fed to mod_pagespeed.
What is the best way to enable/disable mod_pagespeed filters on per
request basis from my mod_perl module without having to rewrite the
original URL? Any suggestions will be greatly appreciated.

Your request is an interesting one.  I don't know much about mod_perl but it might not be too hard for us to support setting filter-sets from an upstream filter via the C API.  Do you know if it's possible to attach a programatically generated string to the request_rec object via the 'notes' field or something, via mod_perl?

Are there any performance or other issues which I should be aware if I
change the which filters are ran on each request?

There is some extra overhead associated with setting up distinct filter-chains to handle one request, but we've tried to make this reasonably fast.  We could improve this if we uncovered evidence that this impacted performance (e.g. by profiling during a load test).  At one point I measured the time consumed by this setup and it was very small. 

-Josh

[Joshua Marantz]

On Thu, Apr 7, 2011 at 8:13 AM, Joshua Marantz <jmar...@google.com> wrote:

As controlling mod_pagespeed thru URL parameters potentially allows
users to enable filters which I might not want to be run, how do I
disable the URL parameter parsing completely? Is there a config option
of some kind or do I have to change the source code for
ScanQueryParamsForRewriterOptions in net/instaweb/apache/
mod_instaweb.cc?

There is currently no such option in mod_pagespeed but there probably should be.  But you've found the correct place if you want to hack source and rebuild.

I've added http://code.google.com/p/modpagespeed/issues/detail?id=259 to track this. 

-Josh

[Marcus Hietala]

Hi Joshua,

On Apr 7, 8:13 am, Joshua Marantz <jmara...@google.com> wrote:
> Hi Marcus,
>
> Sorry it took a while to respond; you've asked some great questions.
>

Thank you for answering my questions.

> On Tue, Apr 5, 2011 at 4:29 PM, Marcus Hietala <marcus.hiet...@gmail.com>wrote:
>
> > I know that one can turn mod_pagespeed on/off and control which
> > filters are applied for each request by adding ModPagespeed*
> > parameters to the URL.
>
> > Is there a list of some kind which filters are allowed to be
> > controlled in such a fashion?
>
> Currently the complete list of filters can be specified as query params, as
> well as enabling/disabling the module.  No other options can currently be
> specified via query-params, although I've been tempted to add the
> inlining/outlining thresholds in order to facilitate iteratively tuning
> those parameters without requiring a server restart.
>

Good to know. Is the modification of filter parameters not available
thru URL parameters because it is not advisable to change them or just
because nobody asked for this feature, yet? Or maybe it was not needed
due to the relatively few filters parameters which can be changed at
the moment?

> We would never allow query-param configuration of any of the server-scope
> parameters like file-cache paths or the maximum byte sizes of the cache, or
> the domain-mapping ones.
>
> > As controlling mod_pagespeed thru URL parameters potentially allows
> > users to enable filters which I might not want to be run, how do I
> > disable the URL parameter parsing completely? Is there a config option
> > of some kind or do I have to change the source code for
> > ScanQueryParamsForRewriterOptions in net/instaweb/apache/
> > mod_instaweb.cc?
>
> There is currently no such option in mod_pagespeed but there probably should
> be.  But you've found the correct place if you want to hack source and
> rebuild.
>

Thanks for creating a ticket to track this. One more question I have
is related to
RewriteOptions class. If mod_pagespeed is configured in PassThru mode
and
there no filters enabled in the pagespeed.conf but instead a
programatically set
by doing something like this:

MessageHandler* handler = factory->message_handler();
RewriteOptions custom_options;
custom_options.EnableFiltersByCommaSeparatedList(filters, handler);
custom_options.DisableAllFiltersNotExplicitlyEnabled(); // Do I need
this call?

do I need the call to DisableAllFiltersNotExplicitlyEnabled() ?

>>  The application I'm working on uses mod_proxy to fetch the html pages
> > and a custom mod_perl module which makes some modifications to page
> > contents before it gets fed to mod_pagespeed.
> > What is the best way to enable/disable mod_pagespeed filters on per
> > request basis from my mod_perl module without having to rewrite the
> > original URL? Any suggestions will be greatly appreciated.
>
> Your request is an interesting one.  I don't know much about mod_perl but it
> might not be too hard for us to support setting filter-sets from an upstream
> filter via the C API.  Do you know if it's possible to attach a
> programatically generated string to the request_rec object via the 'notes'
> field or something, via mod_perl?

Yes, I can easily set a value or multiple values in the request_rec
object notes.
mod_perl exposes to perl most of the existing Apache C API.

> Are there any performance or other issues which I should be aware if I
>
> > change the which filters are ran on each request?
> There is some extra overhead associated with setting up distinct
> filter-chains to handle one request, but we've tried to make this reasonably
> fast.  We could improve this if we uncovered evidence that this impacted
> performance (e.g. by profiling during a load test).  At one point I measured
> the time consumed by this setup and it was very small.

Great. That was useful information to have. I have no easy way to
determine
what the overhead from per request filter chain changes will be.

Marcus

[Shawn Ligocki]

On Thu, Apr 7, 2011 at 4:59 PM, Marcus Hietala <marcus....@gmail.com> wrote:

Hi Joshua,

On Apr 7, 8:13 am, Joshua Marantz <jmara...@google.com> wrote:
> Hi Marcus,
>
> Sorry it took a while to respond; you've asked some great questions.
>

Thank you for answering my questions.

> On Tue, Apr 5, 2011 at 4:29 PM, Marcus Hietala <marcus.hiet...@gmail.com>wrote:

>
> > I know that one can turn mod_pagespeed on/off and control which
> > filters are applied for each request by adding ModPagespeed*
> > parameters to the URL.
>
> > Is there a list of some kind which filters are allowed to be
> > controlled in such a fashion?
>
> Currently the complete list of filters can be specified as query params, as
> well as enabling/disabling the module.  No other options can currently be
> specified via query-params, although I've been tempted to add the
> inlining/outlining thresholds in order to facilitate iteratively tuning
> those parameters without requiring a server restart.
>

Good to know. Is the modification of filter parameters not available
thru URL parameters because it is not advisable to change them or just
because nobody asked for this feature, yet? Or maybe it was not needed
due to the relatively few filters parameters which can be changed at
the moment?

Mostly, we just haven't seen much need to implement them. Of course, there are certain options that could be security risks to change, like MapRewriteDomain or MapOriginDomain, we would not allow those from query parameters.

The reason for this function is that there may be many levels of options (system, .htaccess, query param, request_rec->notes soon) and we need to merge them. If you set EnableFilter foo in your system and EnableFilter bar in your .htaccess, you get both foo and bar. For specificity of debugging, we want to set the exact filters to be enabled for query params, so we run DisableAllFiltersNotExplicitlyEnabled() so that if we put ?ModPagespeedFilters=baz we get exactly baz turned on, not foo, bar and baz.

But, if you have all system and .htaccess files set with no filters turned on, then DisableAllFiltersNotExplicitlyEnabled() is unnecessary (although it might make the code a little more robust in case someone changes the configs).

I'm curious, could you explain in what context you are programatically dealing with options objects? Are you patching our code? Writing your own that interacts with our objects?

 

>>  The application I'm working on uses mod_proxy to fetch the html pages
> > and a custom mod_perl module which makes some modifications to page
> > contents before it gets fed to mod_pagespeed.
> > What is the best way to enable/disable mod_pagespeed filters on per
> > request basis from my mod_perl module without having to rewrite the
> > original URL? Any suggestions will be greatly appreciated.
>
> Your request is an interesting one.  I don't know much about mod_perl but it
> might not be too hard for us to support setting filter-sets from an upstream
> filter via the C API.  Do you know if it's possible to attach a
> programatically generated string to the request_rec object via the 'notes'
> field or something, via mod_perl?

Yes, I can easily set a value or multiple values in the request_rec
object notes.
mod_perl exposes to perl most of the existing Apache C API.

Great, I've opened another feature request: http://code.google.com/p/modpagespeed/issues/detail?id=260. Lets follow up there with appropriate format for such a thing.