MPS is utilizing CPU every hour

[Ivan Vlk]

Hello,

We have recurring (alarm) issue with high CPU load, which is triggered exactly every hour. I would like to check, if that's not something caused by MPS, so please:

- is there any job (cleanup, etc), started by MPS periodically ?
- which kind of timer is MPS using ? I supposed it's at, but our admin told me, that it's not

Thank you.
Ivan

[Jeff Kaufman]

PageSpeed's timers are internal, not through at or cron.

The only timer I can think of which defaults to hourly is the cache
cleaner. https://developers.google.com/speed/pagespeed/module/system

The default is:

ModPagespeedFileCacheCleanIntervalMs 3600000

Every hour it scans through the file cache on disk and removes the
least recently used files if it's over the cache cleaning limit.

Is this level of CPU load causing problems, or just setting off an alarm?

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mod-pagespeed-di...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mod-pagespeed-discuss/e9f0bcc7-6f34-4f1e-bcfa-6c71b3e42c64%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Joshua Marantz]

Good catch Jeff.  Related to the file-cache cleaning: is your cache on a physical disk on the machine?  Or is on something network mounted?

And in any case if the cache-cleaning is a lot of work, that could be an indication that your file-cache limits are too low and you are forcing PageSpeed to do a bunch of work, such as re-optimizing images.  Maybe you could bump up the file-cache size?

-Josh

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUuaqONY_GtK8r8YqTGy37g0pH0PkP9uU-8zkQE0_sv7Vg%40mail.gmail.com.

[Ivan Vlk]

Hi, thank you to point me, however extending cache by 50% did not solved the issue.

I think in case of re-optimizing images, alarm is not triggered every hour periodically :)

Is there anything else, what can I adjust to avoid low cache limits, please ?

I have actually 3GB of File cache on local server's HDD (same machine as webserver ) + 100MB of Shared cache in RAM (shm, ModPagespeedCreateSharedMemoryMetadataCache)

Thank you.
Ivan


Dňa streda, 5. novembra 2014 19:10:11 UTC+1 jmarantz napísal(-a):

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mod-pagespeed-discuss/e9f0bcc7-6f34-4f1e-bcfa-6c71b3e42c64%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Jeff Kaufman]

One option would be to switch to memcache, which requires almost no
cache cleaning [1] but this would require more memory.

[1] The file cache is used for resources too large to fit in memcache,
and it still needs to be checked to make sure it's not too large. But
this shouldn't come up much.

>>> > email to mod-pagespeed-di...@googlegroups.com.

>>> > To view this discussion on the web visit
>>> >
>>> > https://groups.google.com/d/msgid/mod-pagespeed-discuss/e9f0bcc7-6f34-4f1e-bcfa-6c71b3e42c64%40googlegroups.com.
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "mod-pagespeed-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to mod-pagespeed-di...@googlegroups.com.

>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUuaqONY_GtK8r8YqTGy37g0pH0PkP9uU-8zkQE0_sv7Vg%40mail.gmail.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mod-pagespeed-di...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/67ad58c5-bdb8-47b9-bc1f-faa0e7efdd98%40googlegroups.com.

[Ivan Vlk]

Jeff, thanks for explanation. It's simply seems, that we are just facing lack of HW resources. Anyway, it's not a big issue, but it is good for me to understabd why is this happening and what can I set in config / scale in HW.

Ivan

Dňa štvrtok, 6. novembra 2014 16:04:39 UTC+1 Jeff Kaufman napísal(-a):

>>> > email to mod-pagespeed-discuss+unsub...@googlegroups.com.

>>> > To view this discussion on the web visit
>>> >
>>> > https://groups.google.com/d/msgid/mod-pagespeed-discuss/e9f0bcc7-6f34-4f1e-bcfa-6c71b3e42c64%40googlegroups.com.
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "mod-pagespeed-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUuaqONY_GtK8r8YqTGy37g0pH0PkP9uU-8zkQE0_sv7Vg%40mail.gmail.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Jeff Kaufman]

Sounds good. We'll think more about how we can reduce the spikiness
of the job that keeps our cache size under control.

>> >>> > email to mod-pagespeed-di...@googlegroups.com.

>> >>> > To view this discussion on the web visit
>> >>> >
>> >>> >
>> >>> > https://groups.google.com/d/msgid/mod-pagespeed-discuss/e9f0bcc7-6f34-4f1e-bcfa-6c71b3e42c64%40googlegroups.com.
>> >>> > For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "mod-pagespeed-discuss" group.
>> >>> To unsubscribe from this group and stop receiving emails from it, send
>> >>> an

>> >>> email to mod-pagespeed-di...@googlegroups.com.

>> >>> To view this discussion on the web visit
>> >>>
>> >>> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUuaqONY_GtK8r8YqTGy37g0pH0PkP9uU-8zkQE0_sv7Vg%40mail.gmail.com.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mod-pagespeed-discuss" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to mod-pagespeed-di...@googlegroups.com.

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/mod-pagespeed-discuss/67ad58c5-bdb8-47b9-bc1f-faa0e7efdd98%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mod-pagespeed-di...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/cd22afd9-1e6a-4d92-b96a-a33e1a44c706%40googlegroups.com.

[Ivan Vlk]

Hi Jeff,

After some time, I must re-open this issue. Currently, as cache is growing, we are facing web timeouts during the MPS cache cleanup periods. Simply utilising CPU at 100% is cutting all resources needed by webserver. Seems, that this issue is basicaly linked to the VPS usage - as HDDs are simply SW discs.

Meanwhile, we got MPS cache cleanup hung, so I have strace (included) - maybe it can help somehow.

Curent version we are using is 1.8.31.6-r4412 from the Debian repo (latest stable as far as I know).

Is there any possibility to limit CPU resources used by MPS ?

Thank you.
Ivan



Dňa pondelok, 10. novembra 2014 17:18:49 UTC+1 Jeff Kaufman napísal(-a):

>> >>> > email to mod-pagespeed-discuss+unsub...@googlegroups.com.

>> >>> > To view this discussion on the web visit
>> >>> >
>> >>> >
>> >>> > https://groups.google.com/d/msgid/mod-pagespeed-discuss/e9f0bcc7-6f34-4f1e-bcfa-6c71b3e42c64%40googlegroups.com.
>> >>> > For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "mod-pagespeed-discuss" group.
>> >>> To unsubscribe from this group and stop receiving emails from it, send
>> >>> an

>> >>> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

>> >>> To view this discussion on the web visit
>> >>>
>> >>> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUuaqONY_GtK8r8YqTGy37g0pH0PkP9uU-8zkQE0_sv7Vg%40mail.gmail.com.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mod-pagespeed-discuss" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to mod-pagespeed-discuss+unsub...@googlegroups.com.

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/mod-pagespeed-discuss/67ad58c5-bdb8-47b9-bc1f-faa0e7efdd98%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Ivan Vlk]

We have upgraded to latest v 1.9 stable, but unfortunately the issue still persists. Please, is there any official bug tracker or feature request to track this ?

Thank you.
Ivan

Dňa streda, 5. novembra 2014 16:33:26 UTC+1 Ivan Vlk napísal(-a):

[Joshua Marantz]

Ivan -- two questions still remain from the earlier discussion:

1. Is your cache small enough that you are periodically evicting and then re-inserting entries?  You can use stats found on /pagespeed_admin/statistics to find out.  It is less work to clean the cache if it's just verifying you are within your limits every hour.  When you actually start deleting (and then subsequently repopulating) needed images and other files, that will definitely add CPU, network, memory, and disk pressure.

The file cache definitely works best on a physical disk but you have to work with what you have.

2. Have you considered putting your cache on memcached?  That would eliminate this problem for the most part.

-Josh

--

You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/0af427de-e76b-47f4-bc08-366892aa6d75%40googlegroups.com.

[Ivan Vlk]

Hi Josh,

well, I'm not sure which part of mentionend stats is relevant to me :) Anyway I'm including them.

So, by the memcached, I suppose you mean local process, which will store things in the memory, right ?

Just if I understand correctly: to get the rid of this issue, we simply need to have cache space on FS bigger than sum size of stored files, right ? Then cleanup wont run anymore ?

Thank you.
Ivan

Dňa pondelok, 19. januára 2015 16:43:26 UTC+1 jmarantz napísal(-a):

Ivan -- two questions still remain from the earlier discussion:

1. Is your cache small enough that you are periodically evicting and then re-inserting entries?  You can use stats found on /pagespeed_admin/statistics to find out.  It is less work to clean the cache if it's just verifying you are within your limits every hour.  When you actually start deleting (and then subsequently repopulating) needed images and other files, that will definitely add CPU, network, memory, and disk pressure.

The file cache definitely works best on a physical disk but you have to work with what you have.

2. Have you considered putting your cache on memcached?  That would eliminate this problem for the most part.

-Josh

On Fri, Jan 16, 2015 at 11:31 AM, Ivan Vlk <ari...@gmail.com> wrote:

We have upgraded to latest v 1.9 stable, but unfortunately the issue still persists. Please, is there any official bug tracker or feature request to track this ?

Thank you.
Ivan

Dňa streda, 5. novembra 2014 16:33:26 UTC+1 Ivan Vlk napísal(-a):

Hello,

We have recurring (alarm) issue with high CPU load, which is triggered exactly every hour. I would like to check, if that's not something caused by MPS, so please:

- is there any job (cleanup, etc), started by MPS periodically ?
- which kind of timer is MPS using ? I supposed it's at, but our admin told me, that it's not

Thank you.
Ivan

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

memcached is a server than can run on your machine (or on any machine(s) in your LAN) to store the content in RAM.  See https://developers.google.com/speed/pagespeed/module/system#memcached for details.

If you simply increase the size of your file-cache to accommodate all your content (both rewritten and non-rewritten), you will reduce the cost of the cleanup runs dramatically, but the cleanup runs will still occur.  They will be much less impactful to your system load if they don't have to actually delete anything, but are just summing disk-space.

Your stats file looks like it was taken 5 hours after the last server restart.  Is that right?

file_cache_bytes_freed_in_cleanup: 0 file_cache_cleanups: 0 file_cache_disk_checks: 5 file_cache_evictions: 0

You probably need to take another one after you observe the CPU spike to see if anything got evicted.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/24b291be-2d7c-432b-b057-1f4de6d23a42%40googlegroups.com.

[Ivan Vlk]

Actually webserver (Apache) is restarted every night, because of proper log rotation (server is owner by 3rd party, which is maintaining all the stuff). Do you think, this can be an issue ? We are getting this alarm approx. every half hour this time (I'm getting CPU alarms). Last time, when cleanup worker got stuck, we stopped to get all the alarms :-)

Btw, I can see this in the logs:

shm_cache_hits:                                    290918
shm_cache_inserts:                                  47051
shm_cache_misses:                                   58316

 
We have setup both shm and HDD cache, maybe it's not a good idea ?

We can tryr to add more space to the HDD cache, but from logs it seems, that it's not really full, as nothing is deleted. Can be, that only checksum is utilising CPU up to 100% ?

Thank you.
Ivan


Dňa štvrtok, 22. januára 2015 17:59:54 UTC+1 jmarantz napísal(-a):

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsubscri...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/0af427de-e76b-47f4-bc08-366892aa6d75%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

Having the shared-memory cache is very important to avoid hitting the disk all the time for small objects.  It looks like that's performing fine.

If nothing is being evicted (even after day) then making the cache bigger won't help.  But making it faster might, hence the recommendation to use memcached.  Another alternative is to use a tmpfs cache: https://developers.google.com/speed/pagespeed/module/faq#tmpfs

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/8a418a39-90b2-4cee-b123-165d54c8289c%40googlegroups.com.

[Ivan Vlk]

OK, so we can't simply move from VPS to real HW right now, I see only one possibility - to schedule cleanup during the night, just once. Is there any possibility to setup schedule for specific time, please ?

Thank you.
Ivan

Dňa štvrtok, 22. januára 2015 18:21:51 UTC+1 jmarantz napísal(-a):

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/24b291be-2d7c-432b-b057-1f4de6d23a42%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Ivan Vlk]

Actually I found only

ModPagespeedFileCacheCleanIntervalMs

..maybe this will help us, as server is restarting overnight..

Dňa štvrtok, 22. januára 2015 18:21:51 UTC+1 jmarantz napísal(-a):

Having the shared-memory cache is very important to avoid hitting the disk all the time for small objects.  It looks like that's performing fine.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/24b291be-2d7c-432b-b057-1f4de6d23a42%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

Right; if you know when your server restarts are scheduled, I guess you can set your interval to (say) 23 hours (expressed in milliseconds; sorry!) and that will achieve what you want.

RE real HW: I don't think that's a fundamental requirement, but we do need HW-class disk access.  It's hard for me to say what kind of performance you will get in your VM, and if that's the real issue causing your load spikes.

Plus I gave you two other options to ease the disk pressure: tmpfs and memcached, which might be possible in your VM.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/f43d90b6-6877-4f9d-87b8-1b351c93108c%40googlegroups.com.