Re: Too many "Waiting for property cache fetch to complete" errors

[Jeff Kaufman]

This log message happens when pagespeed has been waiting too long for
a asynchronous response from the cache. [1] In your case it looks
like there are three simultaneous broken cache lookups:

process 31335, stuck for about 4686s (1hr)
process 24479, stuck for about 6028s (1.5hr)
process 23208, stuck for about 6331s (1.5hr)
process 18618, stuck for about 7184s (2hr)

Could you paste your memcached configuration and the part of the
pagespeed configuration that deals with memcached?

If you restart apache do you get new "Waiting for property cache fetch
to complete" messages?

We should probably add a timeout; after even a minute of waiting for a
response a visitor has probably given up.

[1] PropertyCallback::BlockUntilDone()

On Fri, Apr 12, 2013 at 6:48 AM, Joyce Babu <jo...@joycebabu.com> wrote:
> My apache error log is full of 'Waiting for property cache fetch to
> complete' errors after enabling memcached support.
>
> [Fri Apr 12 16:14:33 2013] [error] [mod_pagespeed 1.4.26.2-2759 @31335]
> Waiting for property cache fetch to complete. Elapsed time: 4682s
> [Fri Apr 12 16:14:33 2013] [error] [mod_pagespeed 1.4.26.2-2759 @18618]
> Waiting for property cache fetch to complete. Elapsed time: 7179s
> [Fri Apr 12 16:14:33 2013] [error] [mod_pagespeed 1.4.26.2-2759 @24479]
> Waiting for property cache fetch to complete. Elapsed time: 6022s
> [Fri Apr 12 16:14:33 2013] [error] [mod_pagespeed 1.4.26.2-2759 @23208]
> Waiting for property cache fetch to complete. Elapsed time: 6325s
> [Fri Apr 12 16:14:34 2013] [error] [mod_pagespeed 1.4.26.2-2759 @31335]
> Waiting for property cache fetch to complete. Elapsed time: 4683s
> [Fri Apr 12 16:14:34 2013] [error] [mod_pagespeed 1.4.26.2-2759 @18618]
> Waiting for property cache fetch to complete. Elapsed time: 7180s
> [Fri Apr 12 16:14:34 2013] [error] [mod_pagespeed 1.4.26.2-2759 @24479]
> Waiting for property cache fetch to complete. Elapsed time: 6023s
> [Fri Apr 12 16:14:34 2013] [error] [mod_pagespeed 1.4.26.2-2759 @23208]
> Waiting for property cache fetch to complete. Elapsed time: 6326s
> [Fri Apr 12 16:14:35 2013] [error] [mod_pagespeed 1.4.26.2-2759 @31335]
> Waiting for property cache fetch to complete. Elapsed time: 4684s
> [Fri Apr 12 16:14:35 2013] [error] [mod_pagespeed 1.4.26.2-2759 @18618]
> Waiting for property cache fetch to complete. Elapsed time: 7181s
> [Fri Apr 12 16:14:35 2013] [error] [mod_pagespeed 1.4.26.2-2759 @23208]
> Waiting for property cache fetch to complete. Elapsed time: 6327s
> [Fri Apr 12 16:14:35 2013] [error] [mod_pagespeed 1.4.26.2-2759 @24479]
> Waiting for property cache fetch to complete. Elapsed time: 6024s
> [Fri Apr 12 16:14:36 2013] [error] [mod_pagespeed 1.4.26.2-2759 @31335]
> Waiting for property cache fetch to complete. Elapsed time: 4685s
> [Fri Apr 12 16:14:36 2013] [error] [mod_pagespeed 1.4.26.2-2759 @18618]
> Waiting for property cache fetch to complete. Elapsed time: 7182s
> [Fri Apr 12 16:14:36 2013] [error] [mod_pagespeed 1.4.26.2-2759 @24479]
> Waiting for property cache fetch to complete. Elapsed time: 6025s
> [Fri Apr 12 16:14:36 2013] [error] [mod_pagespeed 1.4.26.2-2759 @23208]
> Waiting for property cache fetch to complete. Elapsed time: 6328s
> [Fri Apr 12 16:14:37 2013] [error] [mod_pagespeed 1.4.26.2-2759 @31335]
> Waiting for property cache fetch to complete. Elapsed time: 4686s
> [Fri Apr 12 16:14:37 2013] [error] [mod_pagespeed 1.4.26.2-2759 @18618]
> Waiting for property cache fetch to complete. Elapsed time: 7183s
> [Fri Apr 12 16:14:37 2013] [error] [mod_pagespeed 1.4.26.2-2759 @24479]
> Waiting for property cache fetch to complete. Elapsed time: 6026s
> [Fri Apr 12 16:14:37 2013] [error] [mod_pagespeed 1.4.26.2-2759 @23208]
> Waiting for property cache fetch to complete. Elapsed time: 6329s
> [Fri Apr 12 16:14:38 2013] [error] [mod_pagespeed 1.4.26.2-2759 @31335]
> Waiting for property cache fetch to complete. Elapsed time: 4687s
> [Fri Apr 12 16:14:38 2013] [error] [mod_pagespeed 1.4.26.2-2759 @18618]
> Waiting for property cache fetch to complete. Elapsed time: 7184s
> [Fri Apr 12 16:14:38 2013] [error] [mod_pagespeed 1.4.26.2-2759 @24479]
> Waiting for property cache fetch to complete. Elapsed time: 6027s
> [Fri Apr 12 16:14:38 2013] [error] [mod_pagespeed 1.4.26.2-2759 @23208]
> Waiting for property cache fetch to complete. Elapsed time: 6330s
> [Fri Apr 12 16:14:39 2013] [error] [mod_pagespeed 1.4.26.2-2759 @31335]
> Waiting for property cache fetch to complete. Elapsed time: 4688s
> [Fri Apr 12 16:14:39 2013] [error] [mod_pagespeed 1.4.26.2-2759 @18618]
> Waiting for property cache fetch to complete. Elapsed time: 7185s
> [Fri Apr 12 16:14:39 2013] [error] [mod_pagespeed 1.4.26.2-2759 @24479]
> Waiting for property cache fetch to complete. Elapsed time: 6028s
> [Fri Apr 12 16:14:39 2013] [error] [mod_pagespeed 1.4.26.2-2759 @23208]
> Waiting for property cache fetch to complete. Elapsed time: 6331s
> [Fri Apr 12 16:14:40 2013] [error] [mod_pagespeed 1.4.26.2-2759 @31335]
> Waiting for property cache fetch to complete. Elapsed time: 4689s
>
> How can I fix this?
>
> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mod-pagespeed-di...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Joshua Marantz]

Hi -- I just want to clarify something: We already do have timeouts for memcached reads.  So something must be going wrong with that.  It would be really great to understand all the details of your memcached configuration in your .conf file, and also whether you are using event, prefork, or worker MPMs.  We definitely need to fix this.

Also, are you using the 'real' memcached, or are you using some other implementation or proxy that supports the memcached protocol?

We probably don't want to avoid add a higher-level timeout due to the complexity & risk of handling when that goes wrong; testing it, etc.  We want our lower-level cache & fetch implementations to reliably implement timeouts.

-Josh

[Joshua Marantz]

Can you try setting MemcachedThreads to 1?

I think we haven't done enough testing with threads>1, and should temporarily make increasing the thread-count a no-op until we've tested it more.  Sorry about that.

-Josh

On Fri, Apr 12, 2013 at 3:05 PM, Joyce Babu <jo...@joycebabu.com> wrote:

Today I installed three versions of pagespeed.

1. The stable version

2. 1.4.26.2 beta release

3. 1.5.x beta release

I had the above error with all three versions. I performed graceful restart after installing the 1.4.x beta release. And the error log was showing the above error message for both the stable version and the 1.4.x beta release. The error message for the stable version stopped appearing only after I performed a normal restart (service httpd restart). Hence I think the HTTP request is not timing out.

PageSpeed configuration related to memcached

    ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 102400

    ModPagespeedMemcachedServers "127.0.0.1:11211"

    ModPagespeedMemcachedThreads 3

I tested it with and without ModPagespeedCreateSharedMemoryMetadataCache enabled.

# cat /etc/sysconfig/memcached 

PORT="11211"

USER="memcached"

MAXCONN="1024"

CACHESIZE="64"

OPTIONS=""

I had increased the cache size to 1GB in /etc/init.d/memcached. But it was overridden by the value from /etc/sysconfig/memcached.

[Jud Porter]

From that error log, it looks like it's the same pid (9206) that is stuck before and after restarting httpd. I think graceful will not restart a process until it finishes processing it's current request. Since this one is stuck, it won't finish the request and isn't getting restarted. Can you try either just killing that pid (apache should then restart a new one), or running 'sudo service httpd restart'?

On Fri, Apr 12, 2013 at 3:07 PM, Joyce Babu <jo...@joycebabu.com> wrote:

If you restart apache do you get new "Waiting for property cache fetch 
to complete" messages? 

Yes, for another lookup.

Also, are you using the 'real' memcached, or are you using some other implementation or proxy that supports the memcached protocol?

I'm using the real memcached

[Joshua Marantz]

Good catch, Jud on the PID.  In fact you will probably need to 'sudo killall httpd' to get Apache to stop, since it's stuck in a loop waiting for memcached to finish.  Make sure there are no live httpd processes before you restart.

Would you mind forwarding us also the entire log file?  I'm surprised also by the cache-key collision you reported; we have generated 20-byte md5-sums and it seems very unlikely we should get a cache-key collision.

-Josh

[Joyce Babu]

Yes, that is true. Graceful restart does not kill the error, only a full restart using service httpd restart works. As I mentioned before, when I used graceful restart I had error from two different mod_pagespeed version at the same time.

[Joyce Babu]

The error log contains a total of 319049 entries. Do you want the whole file or just mod_pagespeed related entries?

There are 19 key collision entries in the error log.

[Joyce Babu]

74968 entries related to mod_pagespeed.

[Joshua Marantz]

Yes, sorry :)

Just the mod_pagespeed-related entries from the log.  Feel free to email me directly as well rather than the entire group.

-Josh

On Wed, Apr 17, 2013 at 11:22 PM, Joyce Babu <jo...@joycebabu.com> wrote:

Josh,

I'm still awaiting your reply. Shall I mail just the mod_pagespeed related entries from the log?

Joyce

[Joshua Marantz]

There are a couple of errors discussed in this message-thread.  Are you talking about "Waiting for property cache fetch to complete" or "AprMemCache::Get key collision file:/" or something else?

For the key-collision data can you email me some colliding keys?  We were hoping we'd avoid collisions by using lots of hash bits so I'm curious if something is going wrong.

-Josh

On Tue, May 21, 2013 at 12:58 PM, Joyce Babu <jo...@joycebabu.com> wrote:

I am still seeing the same error with mod_pagespeed 1.5.27.2-2912

ModPagespeedMemcachedThreads is set to 1

[Joshua Marantz]

I am not seeing the 'waiting for property cache to complete' error in the log you just pasted, which is good.  I'm guessing also the key collisions were related to the threads being >1.

We are no longer blocking for an hour on a pcache read, but all these timeouts are not good.

Can you try increasing the memcached size?  64M might not be enough for your site, which means memcached will evict our rewritten images and then mod_pagespeed will consume CPU re-optimizing them, making memcached less responsive.  But that's just a guess.

What's the load average on the machine running MPS & mod_pagespeed?

Can you take a few snapshots of your statistics to see how much image rewriting we are doing, and check the memcached hit rate?

-Josh

On Tue, May 21, 2013 at 1:53 PM, Joyce Babu <jo...@joycebabu.com> wrote:

I was referring to the "waiting for property cache fetch to complete error". I can still see a large number of that in my error log.

Collision errors are not appearing anymore. But there are other AprMemCache errors

[Tue May 21 22:09:29.321624 2013] [pagespeed:error] [pid 13433] [mod_pagespeed 1.5.27.2-2912 @13433] AprMemCache::Get error: Internal error (20014) on key rname/cf_RNkys8ZMaROrutMbO6Pf/j/A8TLcxrLgF4UKutRBeG@_@7whAZDX_GDIt44hYoDabA.@w__@7whAZDX_GDIt44hYoDab

[Tue May 21 22:09:29.321626 2013] [pagespeed:error] [pid 13433] [mod_pagespeed 1.5.27.2-2912 @13433] AprMemCache::Get error: Internal error (20014) on key file://hostname.domain.com/var/www/vhosts/username/public_html/m/assets/js/astrology.js

[Tue May 21 22:11:49.546618 2013] [pagespeed:error] [pid 19303] [mod_pagespeed 1.5.27.2-2912 @19303] AprMemCache::Get error: Internal error (20014) on key file://hostname.domain.com/var/www/vhosts/username/public_html/general/dictionary/images/contribute-bottom.png

[Tue May 21 22:13:35.741713 2013] [pagespeed:error] [pid 19446] [mod_pagespeed 1.5.27.2-2912 @19446] AprMemCache::Get error: Internal error (20014) on key file://hostname.domain.com/var/www/vhosts/username/public_html/astrology/imgs/manisha_koushik1.jpg

[Tue May 21 22:13:35.791999 2013] [pagespeed:error] [pid 19446] [mod_pagespeed 1.5.27.2-2912 @19446] AprMemCache::MultiGet error: The timeout specified has expired (70007) on key http://cdn.domain.com/astrology/imgs/manisha_koushik1.jpg

[Tue May 21 22:13:35.792080 2013] [pagespeed:error] [pid 19446] [mod_pagespeed 1.5.27.2-2912 @19446] AprMemCache::MultiGet error: The timeout specified has expired (70007) on key http://cdn.domain.com/travel/tourism/images/kerala-ayurveda-tour-package.jpg

[Tue May 21 22:14:29.550825 2013] [pagespeed:error] [pid 19500] [mod_pagespeed 1.5.27.2-2912 @19500] AprMemCache::MultiGet error: The timeout specified has expired (70007) on key rname/ci_X0qZACt0nvVM2tHxX44I/http://s1.nxstatic.com/assets/css/A.common.css.pagespeed.cf.X7iTRlarlp.css@@_

[Tue May 21 22:15:03.078330 2013] [pagespeed:error] [pid 19862] [mod_pagespeed 1.5.27.2-2912 @19862] AprMemCache::MultiGet error: The timeout specified has expired (70007) on key http://cdn.domain.com/js/nyromodal/jquery.nyroModal.custom.js

[Tue May 21 22:15:09.216386 2013] [pagespeed:error] [pid 13469] [mod_pagespeed 1.5.27.2-2912 @13469] AprMemCache::Get error: Internal error (20014) on key rname/cf_RNkys8ZMaROrutMbO6Pf/v/n4KUHdfN0dCfafdf1Lm@_@jxyaSdGQA4HNNrTUUiI9A.@__@jxyaSdGQA4HNNrTUUiI9

[Tue May 21 22:15:09.216392 2013] [pagespeed:error] [pid 13469] [mod_pagespeed 1.5.27.2-2912 @13469] AprMemCache::Get error: Internal error (20014) on key file://hostname.domain.com/var/www/vhosts/username/public_html/m/assets/js/autocomplete.js

[Tue May 21 22:16:56.796350 2013] [pagespeed:error] [pid 19810] [mod_pagespeed 1.5.27.2-2912 @19810] AprMemCache::MultiGet error: The timeout specified has expired (70007) on key rname/cf_X0qZACt0nvVM2tHxX44I/2/52uAysB_8z3EWb9DTLw@_@UkH_8v-T69hmRFwebr_aA.@w__@UkH_8v-T69hmRFwebr_a

[Tue May 21 22:16:56.796402 2013] [pagespeed:error] [pid 19810] [mod_pagespeed 1.5.27.2-2912 @19810] AprMemCache::MultiGet error: The timeout specified has expired (70007) on key rname/jc_X0qZACt0nvVM2tHxX44I/http://www.domain.com/js/jquery-1.6.js@@_

[Tue May 21 22:16:56.796423 2013] [pagespeed:error] [pid 19810] [mod_pagespeed 1.5.27.2-2912 @19810] AprMemCache::MultiGet error: The timeout specified has expired (70007) on key rname/jc_X0qZACt0nvVM2tHxX44I/6/Rd0KgR5wBVBiQuPV6dy@

[Tue May 21 22:19:56.367619 2013] [pagespeed:error] [pid 20895] [mod_pagespeed 1.5.27.2-2912 @20895] AprMemCache::Get error: Internal error (20014) on key file://hostname.domain.com/var/www/vhosts/username/public_html/favicon.ico

[Tue May 21 22:19:56.417752 2013] [pagespeed:error] [pid 20895] [mod_pagespeed 1.5.27.2-2912 @20895] AprMemCache::MultiGet error: The timeout specified has expired (70007) on key rname/jc_szHOAK3-Rj_fiC4Nyn4K/b/gX5PKyoYQeEelSSMp9g@

[Tue May 21 22:20:29.633966 2013] [pagespeed:error] [pid 21495] [mod_pagespeed 1.5.27.2-2912 @21495] AprMemCache::Get error: Internal error (20014) on key file://hostname.domain.com/var/www/vhosts/username/public_html/astrology/love-horoscope/imgs/capricorn.gif

[Tue May 21 22:20:29.633975 2013] [pagespeed:error] [pid 21495] [mod_pagespeed 1.5.27.2-2912 @21495] AprMemCache::Put error: Internal error (20014) on key rname/ic_szHOAK3-Rj_fiC4Nyn4K/http://www.domain.com/astrology/love-horoscope/imgs/capricorn.gif@x@_, value-size 1005

[Joshua Marantz]

Yes you can see the stats on /mod_pagespeed_statistics including some histograms as well as name/value pairs of various sorts.  Look for 'memcache' on that page.

RE the seg-fault: I am very interested in finding & fixing this, but I don't think I can decode the stack-trace.  Would it be possible for you to build from source & install a debug version of mod_pagespeed?  Then we'd get a more meaningful stack trace.

That might also give us some clues about why the pcache lookup gets wedged.

On Tue, May 21, 2013 at 2:37 PM, Joyce Babu <jo...@joycebabu.com> wrote:

The memcached caching was enabled only for a few minutes. ModPagespeed is the only program on the server accessing memcached

root@mars [/usr/local/apache/logs]# memcached-tool 127.0.0.1:11211

  #  Item_Size  Max_age   Pages   Count   Full?  Evicted Evict_Time OOM

  1      96B         0s       1       0     yes        0        0    0

  2     120B      1405s       1     190     yes        0        0    0

  3     152B      6727s       1      67     yes        0        0    0

  4     192B      6731s       1     544     yes        0        0    0

  5     240B      6733s       1    1249     yes        0        0    0

  6     304B      6733s       1    1924     yes        0        0    0

  7     384B      6733s      18   49131     yes        0        0    0

  8     480B      6733s       1    1255     yes        0        0    0

  9     600B      6728s       1     320     yes        0        0    0

 10     752B      6730s       1     260     yes        0        0    0

 11     944B      6731s       1     752     yes        0        0    0

 12     1.2K      6731s       1     344     yes        0        0    0

 13     1.4K      6733s       1     368     yes        0        0    0

 14     1.8K      6733s       1     187     yes        0        0    0

 15     2.3K      6733s       1     359     yes        0        0    0

 16     2.8K      6721s       1     186     yes        0        0    0

 17     3.5K      6706s       1      63     yes        0        0    0

 18     4.4K      6732s       1      71     yes        0        0    0

 19     5.5K      6724s       1      25     yes        0        0    0

 20     6.9K      6731s       1      11     yes        0        0    0

 21     8.7K      6732s       1      21     yes        0        0    0

 22    10.8K      6548s       1      14     yes        0        0    0

 23    13.6K      6733s       1      20     yes        0        0    0

 24    16.9K      6435s       1       5     yes        0        0    0

 25    21.2K       490s       1       1     yes        0        0    0

 26    26.5K      6548s       1       2     yes        0        0    0

 27    33.1K      6548s       1       2     yes        0        0    0

 28    41.4K      6637s       1       5     yes        0        0    0

 29    51.7K      6638s       1       4     yes        0        0    0

 30    64.7K       582s       1       1     yes        0        0    0

 31    80.9K      6727s       1       1     yes        0        0    0

 32   101.1K      6638s       1       3     yes        0        0    0

 33   126.3K      6733s       2      10     yes        0        0    0

 34   157.9K      1077s       1       1     yes        0        0    0

 36   246.8K      6062s       1       1     yes        0        0    0

root@mars [/usr/local/apache/logs]# memcached-tool 127.0.0.1:11211 stats

#127.0.0.1:11211   Field       Value

         accepting_conns           1

               auth_cmds           0

             auth_errors           0

                   bytes    24003623

              bytes_read  2300110290

           bytes_written   994127956

              cas_badval           0

                cas_hits           0

              cas_misses           0

               cmd_flush           0

                 cmd_get     5371939

                 cmd_set    11481807

               cmd_touch           0

             conn_yields           0

   connection_structures         165

        curr_connections          17

              curr_items       57592

               decr_hits           0

             decr_misses           0

             delete_hits     4517886

           delete_misses       23375

       evicted_unfetched           0

               evictions           0

       expired_unfetched     2474982

                get_hits     2805051

              get_misses     2566888

              hash_bytes      524288

       hash_is_expanding           0

        hash_power_level          16

               incr_hits           0

             incr_misses           0

                libevent 1.4.13-stable

          limit_maxbytes  1073741824

     listen_disabled_num           0

                     pid        6560

            pointer_size          64

               reclaimed     2475111

            reserved_fds          20

           rusage_system  946.773068

             rusage_user  311.972572

                 threads           4

                    time  1369161118

       total_connections     4547525

             total_items     9101340

              touch_hits           0

            touch_misses           0

                  uptime      343328

                 version      1.4.15

Do I need to enable the memcached integration, before getting the pagespeed stats?

Joyce

On Tuesday, May 21, 2013 11:32:32 PM UTC+5:30, jmarantz wrote:

Can you take a few snapshots of your statistics to see how much image rewriting we are doing, and check the memcached hit rate?

-Josh

[Joshua Marantz]

Yes, debug version of mod_pagespeed into your normal Apache is what I meant.

A coredump can be used if we have the binary that made it, and that binary has symbols (which our product build doesn't).

-Josh

On Tue, May 21, 2013 at 3:12 PM, Joyce Babu <jo...@joycebabu.com> wrote:

This is a live server, hence I cannot install the debug version of apache. Can I install debug version of mod_pagespeed with normal apache? Will coredump suffice?

[Joyce Babu]

I tried running the debug version of mod_pagespeed on my server. Within a few minutes I saw a message that apache has reached the MaxClients limit. I was not sure whether this was due to the module, hence I disabled it.

I could find

1. Segmentation Faults

2. Key Collision Errors

3. AprMemCache::Get Error

4. property cache fetch error

in the error log during this time. I do not want to run this on the live server for a long time, hence can you tell me what exactly I need to look for? Also, which version should I test 1.5.x beta or 1.4.x stable?

[Joshua Marantz]

Beta is up to 1.6 now.  I thought you had avoided the Key Collision errors once you set MemcachedThreads to 1.  Did they re-appear?  If MemcachedThreads is not set to 1 please reset it to 1.  I'm pretty sure that in 1.6 we force it to 1 independent of the conf setting.

If you can get stack traces now for the segmentation faults we should be able to see what's going on.

Actually there's been an independent report of seg-faults when both memcached and shared-mem cache are enabled, and above you mention:

    ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 102400

Turning either memcached or shared-mem cache off will work around the problem.  We would still like to get to the bottom of that, since we think the best practice in most cases would be to use both, once it is working properly.

-Josh

--

[Joyce Babu]

On Monday, July 29, 2013 7:45:42 AM UTC+5:30, jmarantz wrote:

Beta is up to 1.6 now.  I thought you had avoided the Key Collision errors once you set MemcachedThreads to 1.  Did they re-appear?  If MemcachedThreads is not set to 1 please reset it to 1.  I'm pretty sure that in 1.6 we force it to 1 independent of the conf setting.

So, should I test with 1.6 beta or 1.4 stable?

Key collision error was never solved. The errors appears when enable memcached integration. MemcachedThreads had no effect. I have it currently set to 1.

 

If you can get stack traces now for the segmentation faults we should be able to see what's going on.

Actually there's been an independent report of seg-faults when both memcached and shared-mem cache are enabled, and above you mention:

    ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 102400

Turning either memcached or shared-mem cache off will work around the problem.  We would still like to get to the bottom of that, since we think the best practice in most cases would be to use both, once it is working properly.

Until the issue is fixed, which will provide better performance? Enabling memcached or shm metadata caching? 

[Joshua Marantz]

I'd recommend 1.6 beta.

Can you (privately if you wish) give me the full text of a few pairs of colliding keys?

RE memcached vs shm-metadata: it depends on your situation.  If you have multiple servers then memcached may have more of an impact.  On a single server, given that we have to iron out  some conflict, your best bet is probably to use shm-cache or a tmpfs-based file-cache.

-Josh

--

[Joyce Babu]

On Monday, July 29, 2013 7:16:57 PM UTC+5:30, jmarantz wrote:

I'd recommend 1.6 beta.

Thanks. I'll install the debug version and revert with the error log entries. Do you need coredump too?

 

Can you (privately if you wish) give me the full text of a few pairs of colliding keys?

I have sent the file to your email address directly.

 

RE memcached vs shm-metadata: it depends on your situation.  If you have multiple servers then memcached may have more of an impact.  On a single server, given that we have to iron out  some conflict, your best bet is probably to use shm-cache or a tmpfs-based file-cache.

-Josh

On Mon, Jul 29, 2013 at 9:12 AM, Joyce Babu <jo...@joycebabu.com> wrote:

On Monday, July 29, 2013 7:45:42 AM UTC+5:30, jmarantz wrote:

Beta is up to 1.6 now.  I thought you had avoided the Key Collision errors once you set MemcachedThreads to 1.  Did they re-appear?  If MemcachedThreads is not set to 1 please reset it to 1.  I'm pretty sure that in 1.6 we force it to 1 independent of the conf setting.

So, should I test with 1.6 beta or 1.4 stable?

Key collision error was never solved. The errors appears when enable memcached integration. MemcachedThreads had no effect. I have it currently set to 1.

 

If you can get stack traces now for the segmentation faults we should be able to see what's going on.

Actually there's been an independent report of seg-faults when both memcached and shared-mem cache are enabled, and above you mention:

    ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 102400

Turning either memcached or shared-mem cache off will work around the problem.  We would still like to get to the bottom of that, since we think the best practice in most cases would be to use both, once it is working properly.

Until the issue is fixed, which will provide better performance? Enabling memcached or shm metadata caching? 

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

I received the collision detail; thanks.  I think I could use the core-dump file if you also sent with it your mod_pagespeed.so library.  But first emailing a stack-trace might be really useful.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

[Joyce Babu]

According to https://developers.google.com/speed/pagespeed/module/build_mod_pagespeed_from_source

Use it to get either the latest stable version:

  gclient config http://modpagespeed.googlecode.com/svn/branches/latest-beta/src
  gclient sync --force --jobs=1

Is that a documentation error? Shouldn't it be latest-stable for stable?

You received this message because you are subscribed to a topic in the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mod-pagespeed-discuss/idDVL0-ZxKE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mod-pagespeed-di...@googlegroups.com.

[Joyce Babu]

I've send the error log with segfaults to your email address. I forgot to enable coredump before the run. But if you require the coredump, let me know, I'll run it again.

[Joshua Marantz]

Hi Joyce.  Sorry for the long silence on this issue.  I think we finally understand the cause of this.  Please see this thread:

https://groups.google.com/forum/#!msg/mod-pagespeed-discuss/fZ3qVPTRlQg/Tb3U-5m3ragJ

I believe you have an old version of memcached.  Can you upgrade to memcached 1.4.15 and see if the problem disappears?  The above thread also has details on how to do that if your OS doesn't make it easy.

Thanks

On Mon, Jul 29, 2013 at 5:28 PM, Joyce Babu <jo...@joycebabu.com> wrote:

I've send the error log with segfaults to your email address. I forgot to enable coredump before the run. But if you require the coredump, let me know, I'll run it again.

[Joyce Babu]

Hi Josh,

The installed memcached version is 1.4.15. There is a newer version 1.4.16, did you mean that?

root@mars [~]# memcached -h

memcached 1.4.15

Joyce

On Tuesday, January 14, 2014 3:33:14 AM UTC+5:30, jmarantz wrote:

Hi Joyce.  Sorry for the long silence on this issue.  I think we finally understand the cause of this.  Please see this thread:

https://groups.google.com/forum/#!msg/mod-pagespeed-discuss/fZ3qVPTRlQg/Tb3U-5m3ragJ

I believe you have an old version of memcached.  Can you upgrade to memcached 1.4.15 and see if the problem disappears?  The above thread also has details on how to do that if your OS doesn't make it easy.

Thanks

On Mon, Jul 29, 2013 at 5:28 PM, Joyce Babu <jo...@joycebabu.com> wrote:

I've send the error log with segfaults to your email address. I forgot to enable coredump before the run. But if you require the coredump, let me know, I'll run it again.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

Unfortunately, as discussed in the other thread, upgrading from 1.4.5 to 1.4.15 did not in fact eliminate the symptom.  I don't have any reason to believe that upgrading to 1.4.16 would fix it either.  I have not been able to repro the problem either, so this is still a mystery that we need to solve.

In the meantime, have you been getting by by disabling memcached?

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/be22ef77-7b3c-4b99-aecb-08c04869b3e6%40googlegroups.com.

[Joyce Babu]

Unfortunately, as discussed in the other thread, upgrading from 1.4.5 to 1.4.15 did not in fact eliminate the symptom.  I don't have any reason to believe that upgrading to 1.4.16 would fix it either.  I have not been able to repro the problem either, so this is still a mystery that we need to solve.

In the meantime, have you been getting by by disabling memcached?

Memcached integration is disabled in my configuration because, on enabling it for half an hour, the error log will be flooded with the error message.

 

[Joshua Marantz]

Thanks very much for your continued investigation into this.  We really need to fix this problem but we can't reproduce it.  Joyce, can you share with me more details of your setup?

Which Apache version?  Which Linux distribution?  Which MPM?  Which other modules are enabled in Apache?

-Josh

--

You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CACQdstNBPtKw%3DBzzZCYWktnYKpU%3D8o-wn2FEnp8fkYozcGibcg%40mail.gmail.com.

[Joyce Babu]

Hi Josh,

Sorry for the late reply. I had recently upgraded httpd to 2.4 and wanted to make sure that the problem is still present.

I have sent a mail with all the required details. The operating system is CentOS 6.5 (Final) 64 bit.

--
You received this message because you are subscribed to a topic in the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mod-pagespeed-discuss/idDVL0-ZxKE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAGKR%2BEDBJjMJSkHQzqq-28vkqf2kf4bvOhSZTXNCWncwvYxdsg%40mail.gmail.com.

[Joshua Marantz]

We believe we have found and fixed the issue in trunk:

https://code.google.com/p/modpagespeed/issues/detail?id=885

https://code.google.com/p/modpagespeed/source/detail?r=3759

[Joyce Babu]

Hi Josh,

I don't see any more of those errors. I forgot to change ModPagespeedMemcachedThreads back to 1. I changed it only now, so I will continue monitoring for another day.

Multiple LoadFromFile entries are present in my configuration.

--
You received this message because you are subscribed to a topic in the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mod-pagespeed-discuss/idDVL0-ZxKE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAGKR%2BED72XLdDfkH-TCo6kBSrcTyAE-rZL%2Beegx8J3n2crdw5Q%40mail.gmail.com.