[Plone-developers] Please help make plone.org fast again
Alexander Limi
After upgrading, plone.org is ridiculously slow. Since Plone 2.5 is faster
than 2.1, it has to be related to something else, like the caching setup,
something about the server setup, or something else.
I have exhausted my limited performance-tuning skills, and none of the
"usual suspects" seem to have the time to help out at the moment.
It's very detrimental to the image of Plone that our site is so slow,
especially as we are ramping up PR around the upcoming Plone 3.0 release.
Any assistance in verifying the caching setup or otherwise troubleshooting
the setup will be appreciated.
Interestingly, plone.net is fast — and it's running on the exact same
version of Plone, on the exact same server.
--
Alexander Limi · http://limi.net
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Plone-developers mailing list
Plone-de...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plone-developers
Justizin
> Hi guys,
>
> After upgrading, plone.org is ridiculously slow. Since Plone 2.5 is faster
> than 2.1, it has to be related to something else, like the caching setup,
> something about the server setup, or something else.
>
Squid 2.6 could give some advantage, but it has a security issue I am
not 100% wrapped around yet. Word on the street is that the latest
squid is fine, but it's not clear to me if it may be related to the
way CacheFu's generated configs look.
Another advantage can be to split the catalogs onto a separate
physical device or cached volume, e.g. SAN.
> I have exhausted my limited performance-tuning skills, and none of the
> "usual suspects" seem to have the time to help out at the moment.
>
> It's very detrimental to the image of Plone that our site is so slow,
> especially as we are ramping up PR around the upcoming Plone 3.0 release.
> Any assistance in verifying the caching setup or otherwise troubleshooting
> the setup will be appreciated.
>
I can take a look, afaik wiggy set squid up per CacheFu, and that
hasn't changed much for 2.5. When I spoke with him last about it, he
was not in favor of a move to Squid 2.6, so I wouldn't want to promote
such a thing without at least hearing his opinion as of now.
> Interestingly, plone.net is fast — and it's running on the exact same
> version of Plone, on the exact same server.
>
Have you examined the main site with livehttpheaders? Also, if
CacheFu was reinstalled, the custom types for PSC, Poi, etc.. will not
be cached properly.
--
Justin Alan Ryan, Director, VonGogo Foundation
Independent Interactivity Architect, ACM SIGGRAPH
http://www.bitmonk.net/
* : +1-415-226-1199
Alexander Limi
<just...@siggraph.org> wrote:
> Squid 2.6 could give some advantage, but it has a security issue I am
> not 100% wrapped around yet. Word on the street is that the latest
> squid is fine, but it's not clear to me if it may be related to the
> way CacheFu's generated configs look.
In any case, this isn't a some-minor-percentage-slower case — it's quite
noticably slower, and it used to be much faster with same Squid.
> Another advantage can be to split the catalogs onto a separate
> physical device or cached volume, e.g. SAN.
This is not where the bottleneck is either.
> Have you examined the main site with livehttpheaders?
I did, and from what I could see, it does actually do its job. I'm not
very experienced with what to look for, though — so a second pair of eyes
would be appreciated.
> Also, if
> CacheFu was reinstalled, the custom types for PSC, Poi, etc.. will not
> be cached properly.
This is likely part of the problem. What are good settings for these
content types?
--
Alexander Limi · http://limi.net
Anton Stonor
> Have you examined the main site with livehttpheaders? Also, if
> CacheFu was reinstalled, the custom types for PSC, Poi, etc.. will not
> be cached properly.
Haven taken a quit look at a couple of http headers at the site, that
seems to be the case: E.g. the front page and product pages don't cache.
/Anton
Martijn Pieters
Hash: SHA1
On 6/15/07, Alexander Limi wrote:
> > Have you examined the main site with livehttpheaders?
>
> I did, and from what I could see, it does actually do its job. I'm not
> very experienced with what to look for, though — so a second pair of eyes
> would be appreciated.
http://www.ircache.net/cgi-bin/cacheability.py?query=http%3A%2F%2Fplone.org&descend=on
The homepage is explicitly *not* cacheable, it appears.
Another problem is that the server cock is off by two weeks, this can
cause problems with cache validations.
Use http://www.ircache.net/cgi-bin/cacheability.py to test other URLs.
- --
Martijn Pieters
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
Comment: http://firegpg.tuxfamily.org
iD8DBQFGcjeq3xaj2GOvgP0RAk3CAJ9DWK05yqD+6yca1oO045uiTCvLLgCghuGS
WIdAOd+DkgcC0eN6o5Kg+5s=
=BFEL
-----END PGP SIGNATURE-----
Wichert Akkerman
> Squid 2.6 could give some advantage, but it has a security issue I am
> not 100% wrapped around yet. Word on the street is that the latest
> squid is fine, but it's not clear to me if it may be related to the
> way CacheFu's generated configs look.
The problem is that the Plone site got slower. Squid has not changed so
it does not make much sense to look there.
> Another advantage can be to split the catalogs onto a separate
> physical device or cached volume, e.g. SAN.
IO is not a bottleneck here.
> > I have exhausted my limited performance-tuning skills, and none of the
> > "usual suspects" seem to have the time to help out at the moment.
> >
> > It's very detrimental to the image of Plone that our site is so slow,
> > especially as we are ramping up PR around the upcoming Plone 3.0 release.
> > Any assistance in verifying the caching setup or otherwise troubleshooting
> > the setup will be appreciated.
> >
>
> I can take a look, afaik wiggy set squid up per CacheFu, and that
> hasn't changed much for 2.5. When I spoke with him last about it, he
> was not in favor of a move to Squid 2.6, so I wouldn't want to promote
> such a thing without at least hearing his opinion as of now.
The CacheFu configuration is sane and the cache miss ratios have not
changed. The cache miss service time has gone up though.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
Wichert Akkerman
> Another problem is that the server cock is off by two weeks, this can
> cause problems with cache validations.
That is not true. This is what NTP says:
offset 0.003551, delay 0.16156, error bound 0.12248, filter error 0.01868
0.003551 seconds is significantly less than two weeks.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
-------------------------------------------------------------------------
Martijn Pieters
> Previously Martijn Pieters wrote:
>> Another problem is that the server cock is off by two weeks, this can
>> cause problems with cache validations.
>
> That is not true. This is what NTP says:
>
> offset 0.003551, delay 0.16156, error bound 0.12248, filter error
> 0.01868
>
> 0.003551 seconds is significantly less than two weeks.
Heh, I think the server the cacheability script runs at is in need of
a NTP update then.. :)
Martijn
Wichert Akkerman
> The CacheFu configuration is sane and the cache miss ratios have not
> changed. The cache miss service time has gone up though.
Apparently sane but it lost a number of items and was not reconfigured
for the custom frontpage we now use. I tweaked things a bit and I now
get the frontpage in 239ms.
Emyr Thomas
message news:op.ttx5kxfwvhba00@-emo...
>> Also, if
>> CacheFu was reinstalled, the custom types for PSC, Poi, etc.. will not
>> be cached properly.
>
> This is likely part of the problem. What are good settings for these
> content types?
"CacheFu settings for popular Plone add-on products" would make a very good
How-to/Tutorial for the documentation section on plone.org. Instructions for
things like PHC/PSC/Poi/Ploneboard/Poi/CompositePack would be really useful.
--Emyr
Wichert Akkerman
> "Alexander Limi" <li...@plone.org> wrote in
> message news:op.ttx5kxfwvhba00@-emo...
>
> >> Also, if
> >> CacheFu was reinstalled, the custom types for PSC, Poi, etc.. will not
> >> be cached properly.
> >
> > This is likely part of the problem. What are good settings for these
> > content types?
>
> "CacheFu settings for popular Plone add-on products" would make a very good
> How-to/Tutorial for the documentation section on plone.org. Instructions for
> things like PHC/PSC/Poi/Ploneboard/Poi/CompositePack would be really useful.
Some of those require extra work unfortunately. The settings for PHC are
configured correctly in CacheSetup but no caching headers are being
generated for it. Someone is going to have to look into that.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
-------------------------------------------------------------------------
Justizin
> Previously Emyr Thomas wrote:
> > "Alexander Limi" <li...@plone.org> wrote in
> >
> > "CacheFu settings for popular Plone add-on products" would make a very good
> > How-to/Tutorial for the documentation section on plone.org. Instructions for
> > things like PHC/PSC/Poi/Ploneboard/Poi/CompositePack would be really useful.
>
> Some of those require extra work unfortunately. The settings for PHC are
> configured correctly in CacheSetup but no caching headers are being
> generated for it. Someone is going to have to look into that.
>
Maybe PHC has moved to Five views?
--
Justin Alan Ryan, Director, VonGogo Foundation
Independent Interactivity Architect, ACM SIGGRAPH
http://www.bitmonk.net/
* : +1-415-226-1199
-------------------------------------------------------------------------
Justizin
> > Squid 2.6 could give some advantage, but it has a security issue I am
> > not 100% wrapped around yet. Word on the street is that the latest
> > squid is fine, but it's not clear to me if it may be related to the
> > way CacheFu's generated configs look.
>
> The problem is that the Plone site got slower. Squid has not changed so
> it does not make much sense to look there.
>
Sure..
> > Another advantage can be to split the catalogs onto a separate
> > physical device or cached volume, e.g. SAN.
>
> IO is not a bottleneck here.
>
Are you sure? Plone 2.1 performance on Plone.org was never stellar.
> > > I have exhausted my limited performance-tuning skills, and none of the
> > > "usual suspects" seem to have the time to help out at the moment.
> > >
> > > It's very detrimental to the image of Plone that our site is so slow,
> > > especially as we are ramping up PR around the upcoming Plone 3.0 release.
> > > Any assistance in verifying the caching setup or otherwise troubleshooting
> > > the setup will be appreciated.
> > >
> >
> > I can take a look, afaik wiggy set squid up per CacheFu, and that
> > hasn't changed much for 2.5. When I spoke with him last about it, he
> > was not in favor of a move to Squid 2.6, so I wouldn't want to promote
> > such a thing without at least hearing his opinion as of now.
>
> The CacheFu configuration is sane and the cache miss ratios have not
> changed. The cache miss service time has gone up though.
>
Odd..
--
Justin Alan Ryan, Director, VonGogo Foundation
Independent Interactivity Architect, ACM SIGGRAPH
http://www.bitmonk.net/
* : +1-415-226-1199
-------------------------------------------------------------------------
Gilles Lenfant
>Hi guys,
>After upgrading, plone.org is ridiculously slow. Since Plone 2.5 is faster
>than 2.1, it has to be related to something else, like the caching setup,
>something about the server setup, or something else.
>I have exhausted my limited performance-tuning skills, and none of the
>"usual suspects" seem to have the time to help out at the moment.
Hint: Associating the PlonePAS (acl_users) and plugins with a dedicated Zope
RAM cache speeds up Plone for authenticated users.
...
>Interestingly, plone.net is fast — and it's running on the exact same
>version of Plone, on the exact same server.
Plone.net has less content, thus less catalog entries.
Cheers
--
Gilles Lenfant
--
View this message in context: http://www.nabble.com/Please-help-make-plone.org-fast-again-tf3925120s6745.html#a11140744
Sent from the Core Developers mailing list archive at Nabble.com.
Wichert Akkerman
>
> >Hi guys,
>
> >After upgrading, plone.org is ridiculously slow. Since Plone 2.5 is faster
> >than 2.1, it has to be related to something else, like the caching setup,
> >something about the server setup, or something else.
>
> >I have exhausted my limited performance-tuning skills, and none of the
> >"usual suspects" seem to have the time to help out at the moment.
>
> Hint: Associating the PlonePAS (acl_users) and plugins with a dedicated Zope
> RAM cache speeds up Plone for authenticated users.
Which is why plone.org has done that pretty much since it started using
LDAP.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
-------------------------------------------------------------------------
Justizin
> On 6/15/07, Wichert Akkerman <wic...@wiggy.net> wrote:
> >
> > The CacheFu configuration is sane and the cache miss ratios have not
> > changed. The cache miss service time has gone up though.
> >
>
> Odd..
>
Here's a question: did plone.org use the macro cache on Plone 2.1? I
note that this is removed from trunk as of a couple weeks ago, but it
does in fact work on Plone 2.5.
I'm really not sure the 2.[15] / 3.0 oriented split of CacheFu has
been entirely beneficial..
Justizin
>
> ...
>
> >Interestingly, plone.net is fast — and it's running on the exact same
> >version of Plone, on the exact same server.
>
> Plone.net has less content, thus less catalog entries.
>
Quite likely a factor..
--
Justin Alan Ryan, Director, VonGogo Foundation
Independent Interactivity Architect, ACM SIGGRAPH
http://www.bitmonk.net/
* : +1-415-226-1199
-------------------------------------------------------------------------
Alexander Limi
<just...@siggraph.org> wrote:
> Here's a question: did plone.org use the macro cache on Plone 2.1? I
> note that this is removed from trunk as of a couple weeks ago, but it
> does in fact work on Plone 2.5.
No, we removed that a while back, so plone.org wasn't using macro cache in
the last months before we switched to 2.5.
--
Alexander Limi · http://limi.net
Stefan Eletzhofer
wichert wrote:
>
> Previously Justizin wrote:
>> Squid 2.6 could give some advantage, but it has a security issue I am
>> not 100% wrapped around yet. Word on the street is that the latest
>> squid is fine, but it's not clear to me if it may be related to the
>> way CacheFu's generated configs look.
>
> The problem is that the Plone site got slower. Squid has not changed so
> it does not make much sense to look there.
>
>> Another advantage can be to split the catalogs onto a separate
>> physical device or cached volume, e.g. SAN.
>
> IO is not a bottleneck here.
>
Moving the catalog to a different database gives the opportunity to
cache more objects for this database.
Apparently the cache setting "max number of objects in cache" is
a per-database setting.
Catalog objects are small compared to AT content objects, thus
giving the catalog its own database file will allow you to ramp up the
number of cached objects.
So primarily this is not a fix for a IO bottleneck, but for a cache/memory
bottleneck.
Stefan.
--
View this message in context: http://www.nabble.com/Please-help-make-plone.org-fast-again-tf3925120s6745.html#a11162103
Sent from the Core Developers mailing list archive at Nabble.com.
Wichert Akkerman
> Moving the catalog to a different database gives the opportunity to
> cache more objects for this database.
We've had a seperate database for the catalog for over a year as well.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
-------------------------------------------------------------------------
Stefan Eletzhofer
wichert wrote:
>
> Previously Stefan Eletzhofer wrote:
>> Moving the catalog to a different database gives the opportunity to
>> cache more objects for this database.
>
> We've had a seperate database for the catalog for over a year as well.
>
>
Doh.
BTW, is there some sort of documentation where people can read about the
server setup?
It seems to me that many people post some "hey, have you tried _THAT_",
just for getting
"yeah, I know, been there, done that. We do that since _AGES_ now" :) :)
Cheers,
Stefan.
--
View this message in context: http://www.nabble.com/Please-help-make-plone.org-fast-again-tf3925120s6745.html#a11164178
Sent from the Core Developers mailing list archive at Nabble.com.
Wichert Akkerman
> BTW, is there some sort of documentation where people can read about the
> server setup?
> It seems to me that many people post some "hey, have you tried _THAT_",
> just for getting
> "yeah, I know, been there, done that. We do that since _AGES_ now" :) :)
Embarrasingly enough there is no real documentation for the setup. All
the common best practices have been used though. The remaining issues
that I can see are: figure out why PHC does not produce proper caching
headers and upgrade the LDAP config to add a patch I have laying around
which speeds up user property access.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
-------------------------------------------------------------------------
Justizin
Previously Stefan Eletzhofer wrote:
> Moving the catalog to a different database gives the opportunity to
> cache more objects for this database.
We've had a seperate database for the catalog for over a year as well.
What sort of physical storage is on plone.org? You'd be amazed at how much faster siggraph.org was when I stopped zeo, moved each Data.fs to its' own dedicated volume on Verizon's SAN, mounted in place, and fired back up. Those aren't even on separate physical volumes, they're just benefitting from the SAN's monster cache.
But, that doesn't mean that has anything to do with the recent problem. You can always get faster.
Wichert - Speaking of the recent problem, plone.org seems to be rather performant the past couple of days, I recall you said you optimized something.. Was this related to PSC / PHC entirely, or is there something the rest of us can learn from your experience?
Ruda Filgueiras
> > Moving the catalog to a different database gives the opportunity to
> > cache more objects for this database.
>
> We've had a seperate database for the catalog for over a year as well.
>
> Wichert.
I recently put the portal_catalog on a separated ZODB in a big site
with 80K indexed objects.
On the zope.conf the cache for the ZODB catalog in number objects was
configured to 90K.
The results was so good, from 2.5 seconds to do a query using path
expression to 0.12 seconds.
The most interesting thing are that after start ZOPE the ZODB cache
for portal_catalog jumps to 65K objects on the first access.
I can't say that it is the bottleneck but it can boost the plone.org
performance, since everything uses the portal_catalog to query
information.
--
Rudá Porto Filgueiras
Weimar Consultoria
http://python-blog.blogspot.com
Hospedagem Plone, Zope e Python?
http://www.pytown.com
David Convent
> On 6/17/07, Wichert Akkerman <wic...@wiggy.net> wrote:
>
>>Previously Stefan Eletzhofer wrote:
>>
>>>Moving the catalog to a different database gives the opportunity to
>>>cache more objects for this database.
>>
>>We've had a seperate database for the catalog for over a year as well.
>>
>>Wichert.
>
>
> I recently put the portal_catalog on a separated ZODB in a big site
> with 80K indexed objects.
> On the zope.conf the cache for the ZODB catalog in number objects was
> configured to 90K.
> The results was so good, from 2.5 seconds to do a query using path
> expression to 0.12 seconds.
>
> The most interesting thing are that after start ZOPE the ZODB cache
> for portal_catalog jumps to 65K objects on the first access.
>
> I can't say that it is the bottleneck but it can boost the plone.org
> performance, since everything uses the portal_catalog to query
> information.
From reading the quoted part of your email, you should know that
plone.org already has a separate ZODB for its portal catalog, since over
a year ;-)
Jon Baldivieso
Previously Stefan Eletzhofer wrote:
BTW, is there some sort of documentation where people can read about the
server setup?
It seems to me that many people post some "hey, have you tried _THAT_",
just for getting
"yeah, I know, been there, done that. We do that since _AGES_ now" :) :)
Embarrasingly enough there is no real documentation for the setup. All
the common best practices have been used though.
Manuel Spuhler
we had a performance issue a few month ago, and what did the trick
was to change the default settings for the cache-size in zope.conf
on each instance. I think this is due to the size of BTreeFolder
Now it's set to 130Mb and 35000 object cached, and we plan to set
that parameter up to 300Mb. A page was displaying in 40s before, it
only takes 4sec max now.
<zodb_db main>
..
# # ZODB cache, in number of objects
cache-size 35000
...
cache-size 130MB
hope that helps
Manuel Spuhler
www.rsr.ch
PS
don't look at our site to see how fast it is now, our network is
having big troubles at this very momemt (I swear its true ;-) and
also we use akamaï in frontend, so you won't see anything relevant
Le 15 juin 07 à 02:17, Alexander Limi a écrit :
> Hi guys,
>
> After upgrading, plone.org is ridiculously slow. Since Plone 2.5 is
> faster
> than 2.1, it has to be related to something else, like the caching
> setup,
> something about the server setup, or something else.
>
> I have exhausted my limited performance-tuning skills, and none of the
> "usual suspects" seem to have the time to help out at the moment.
>
> It's very detrimental to the image of Plone that our site is so slow,
> especially as we are ramping up PR around the upcoming Plone 3.0
> release.
> Any assistance in verifying the caching setup or otherwise
> troubleshooting
> the setup will be appreciated.
>
> Interestingly, plone.net is fast — and it's running on the exact same
> version of Plone, on the exact same server.
>
> --
> Alexander Limi · http://limi.net
>
>
> ----------------------------------------------------------------------
> ---