Clustering
22 views
Skip to first unread message
Dan Petzen
Jun 2, 2009, 11:58:30 AM6/2/09
to Confluence in the (real) Enterprise
I know I'm the new kid on the block, but I nonetheless wanted to get a
discussing going about clustering.
I'm currently gearing up to perform the attachment migration back into
the database. It blew up spectacularly when we migrated them out of
the database, so I expect it to be quite exciting in this direction as
well.
This is part of the problem that created this discussion in the first
place: the requirements to have all attachments in the DB when running
a cluster. I've now had to double the DB size, which isn't very
desirable.
I've chosen to run it without multicast and to create a smaller batch
system that sits beside a larger cluster node that handles all
interactive requests. I did some tests with share indices (NFS/RSync),
but unfortunately it didn't work.
The next steps are to test node fail-over and general resilience. Any
experiences here would also be very welcome.
discussing going about clustering.
I'm currently gearing up to perform the attachment migration back into
the database. It blew up spectacularly when we migrated them out of
the database, so I expect it to be quite exciting in this direction as
well.
This is part of the problem that created this discussion in the first
place: the requirements to have all attachments in the DB when running
a cluster. I've now had to double the DB size, which isn't very
desirable.
I've chosen to run it without multicast and to create a smaller batch
system that sits beside a larger cluster node that handles all
interactive requests. I did some tests with share indices (NFS/RSync),
but unfortunately it didn't work.
The next steps are to test node fail-over and general resilience. Any
experiences here would also be very welcome.
Per Fragemann
Jun 5, 2009, 6:08:33 PM6/5/09
to Confluence in the (real) Enterprise
Hi Dan,
I'd like to encourage you to have a look at Confluence 3.0. It
features some massive performance improvements when clustering under
high load (and of course various other great features and
improvements). If performance is a concern, then please check out the
performance release notes over here:
http://confluence.atlassian.com/display/DOC/Confluence+3+Performance+improvements
We can't claim to have solved all possible performance issues yet, and
attachments are still required to be in the database, but this is a
large step into the right direction, so please do have a look. It may
be more work for you right now to do the extra step, but you won't
regret it. Your future workload will be much more enjoyable. :-) I'd
be interested to learn about the other performance issues too.
Confluence clustering is not a High Availability solution yet (it was
designed with performance and scale in mind first of all), but we are
considering improving resilience for a medium-term release. If you
have performance issues that are not yet in our JIRA issue tracker, we
would love to hear about them, especially how you overcame them.
Cheers,
Per
I'd like to encourage you to have a look at Confluence 3.0. It
features some massive performance improvements when clustering under
high load (and of course various other great features and
improvements). If performance is a concern, then please check out the
performance release notes over here:
http://confluence.atlassian.com/display/DOC/Confluence+3+Performance+improvements
We can't claim to have solved all possible performance issues yet, and
attachments are still required to be in the database, but this is a
large step into the right direction, so please do have a look. It may
be more work for you right now to do the extra step, but you won't
regret it. Your future workload will be much more enjoyable. :-) I'd
be interested to learn about the other performance issues too.
Confluence clustering is not a High Availability solution yet (it was
designed with performance and scale in mind first of all), but we are
considering improving resilience for a medium-term release. If you
have performance issues that are not yet in our JIRA issue tracker, we
would love to hear about them, especially how you overcame them.
Cheers,
Per
Dan Petzen
Jun 12, 2009, 12:14:54 PM6/12/09
to Confluence in the (real) Enterprise
Hi Per.
Sorry for the delay in answering.
> I'd like to encourage you to have a look at Confluence 3.0.
I've had a look at it and I appreciate the performance improvements.
We can't really afford bleeding edge here, so we would probably hold
off until 3.1. Unfortunately, my contract is coming to an end here, so
I won't be around to create a cluster based on 3.0. I'll have to focus
on getting a 2.10.2 cluster working.
I am, however, passing on the view that 3.0 should be pursued as it's
likely to give huge performance gains.
> Confluence clustering is not a High Availability solution yet (it was
I'm aware of this, but there are still some features that will
increase availability, even though it can't be called HA, no matter
what definition is used. It's especially the ability to perform some
changes without downtime and failover when an error occurs that we're
after.
> [..] If you
Atlassian. Admittedly, there have been a long range of small snags
we've sorted out as well. I think the only major performance
improvement that isn't in there is the Apache tuning we did. Some of
it is, as we got some advice on static contents etc.
Regards,
Dan
P.S.
Per is very Swedish. Any connections?
Sorry for the delay in answering.
> I'd like to encourage you to have a look at Confluence 3.0.
We can't really afford bleeding edge here, so we would probably hold
off until 3.1. Unfortunately, my contract is coming to an end here, so
I won't be around to create a cluster based on 3.0. I'll have to focus
on getting a 2.10.2 cluster working.
I am, however, passing on the view that 3.0 should be pursued as it's
likely to give huge performance gains.
> Confluence clustering is not a High Availability solution yet (it was
increase availability, even though it can't be called HA, no matter
what definition is used. It's especially the ability to perform some
changes without downtime and failover when an error occurs that we're
after.
> [..] If you
> have performance issues that are not yet in our JIRA issue tracker, we
> would love to hear about them, especially how you overcame them.
I think I've registered most of the serious performance problems with
> would love to hear about them, especially how you overcame them.
Atlassian. Admittedly, there have been a long range of small snags
we've sorted out as well. I think the only major performance
improvement that isn't in there is the Apache tuning we did. Some of
it is, as we got some advice on static contents etc.
Regards,
Dan
P.S.
Per is very Swedish. Any connections?
Igor Minar
Jun 12, 2009, 1:51:29 PM6/12/09
to enterprise...@googlegroups.com
Hi Dan,
On Jun 12, 2009, at 9:14 AM, Dan Petzen wrote:
>
> Hi Per.
>
> Sorry for the delay in answering.
>
>> I'd like to encourage you to have a look at Confluence 3.0.
>
> I've had a look at it and I appreciate the performance improvements.
> We can't really afford bleeding edge here, so we would probably hold
> off until 3.1. Unfortunately, my contract is coming to an end here, so
> I won't be around to create a cluster based on 3.0. I'll have to focus
> on getting a 2.10.2 cluster working.
I suggest that you go with 2.10.3 + patch it with the latest security
patches released recently.
http://confluence.atlassian.com/display/DOC/Confluence+Security+Advisory+2009-06-01
Based on our previous experience we are eager to see Confluence 3.0.x
or 3.1 out too. :)
>
> I am, however, passing on the view that 3.0 should be pursued as it's
> likely to give huge performance gains.
>
>> Confluence clustering is not a High Availability solution yet (it was
>
> I'm aware of this, but there are still some features that will
> increase availability, even though it can't be called HA, no matter
> what definition is used. It's especially the ability to perform some
> changes without downtime and failover when an error occurs that we're
> after.
Unfortunately the clustering in 2.x will most likely do the exact
opposite for you.
We were running cluster for a long time, but then finally gave up on
it as it was causing more problems and outages than giving us benefits
when it comes to performance, scalability or availability.
Depending on your authentication mechanism you might be interested in
looking at http://jira.atlassian.com/browse/CONF-12319 which makes
even confluence 3.0 clustering a no-go for us.
>
>> [..] If you
>> have performance issues that are not yet in our JIRA issue tracker,
>> we
>> would love to hear about them, especially how you overcame them.
>
> I think I've registered most of the serious performance problems with
> Atlassian. Admittedly, there have been a long range of small snags
> we've sorted out as well. I think the only major performance
> improvement that isn't in there is the Apache tuning we did. Some of
> it is, as we got some advice on static contents etc.
Something you want to share?
cheers,
Igor
On Jun 12, 2009, at 9:14 AM, Dan Petzen wrote:
>
> Hi Per.
>
> Sorry for the delay in answering.
>
>> I'd like to encourage you to have a look at Confluence 3.0.
>
> I've had a look at it and I appreciate the performance improvements.
> We can't really afford bleeding edge here, so we would probably hold
> off until 3.1. Unfortunately, my contract is coming to an end here, so
> I won't be around to create a cluster based on 3.0. I'll have to focus
> on getting a 2.10.2 cluster working.
patches released recently.
http://confluence.atlassian.com/display/DOC/Confluence+Security+Advisory+2009-06-01
Based on our previous experience we are eager to see Confluence 3.0.x
or 3.1 out too. :)
>
> I am, however, passing on the view that 3.0 should be pursued as it's
> likely to give huge performance gains.
>
>> Confluence clustering is not a High Availability solution yet (it was
>
> I'm aware of this, but there are still some features that will
> increase availability, even though it can't be called HA, no matter
> what definition is used. It's especially the ability to perform some
> changes without downtime and failover when an error occurs that we're
> after.
opposite for you.
We were running cluster for a long time, but then finally gave up on
it as it was causing more problems and outages than giving us benefits
when it comes to performance, scalability or availability.
Depending on your authentication mechanism you might be interested in
looking at http://jira.atlassian.com/browse/CONF-12319 which makes
even confluence 3.0 clustering a no-go for us.
>
>> [..] If you
>> have performance issues that are not yet in our JIRA issue tracker,
>> we
>> would love to hear about them, especially how you overcame them.
>
> I think I've registered most of the serious performance problems with
> Atlassian. Admittedly, there have been a long range of small snags
> we've sorted out as well. I think the only major performance
> improvement that isn't in there is the Apache tuning we did. Some of
> it is, as we got some advice on static contents etc.
cheers,
Igor
Dan Petzen
Jun 15, 2009, 9:42:58 AM6/15/09
to Confluence in the (real) Enterprise
Hi Igor.
Our site is internal, and even though it may sound ignorant to not
patch for XSS vulnerabilities, that is still how I have to prioritise
the fairly strained engineering resources we have.
>
> Based on our previous experience we are eager to see Confluence 3.0.x
> or 3.1 out too. :)
Me too. Unfortunately, I won't have enough time to see that through.
Btw, the chap that is responsible for the Sun JVM/JRE here pointed out
that 1.6.0_14 has a lot of performance improvements that may be very
beneficial for Confluence. I've not had a chance to try it myself yet,
though.
> Unfortunately the clustering in 2.x will most likely do the exact
> opposite for you.
>
> We were running cluster for a long time, but then finally gave up on
> it as it was causing more problems and outages than giving us benefits
> when it comes to performance, scalability or availability.
Hmmm... Very worrying indeed. Do you have any examples of situations
where the cluster caused (un)expected problems?
>
> Depending on your authentication mechanism you might be interested in
> looking athttp://jira.atlassian.com/browse/CONF-12319which makes
groups. I've not checked the code, but I think it uses the API a bit
better as we haven't had any performance problems.
> Something you want to share?
The performance tweaks or Apache in particular?
Let me elaborate on the Apache tuning, just in case that was what you
were interested in.
We did some standard tuning, by removing costly log details, cleaning
up the module includes and so on. We then disabled SSO on static
contents, such as /images/ and enabled caching. I used the memory
based cache module for /images/ and the disk based for /s/1518/ and /
download/. Atlassian confirmed that the dynamic content that came
from /s/<build_number>/ could be cached, as it was generated once and
then didn't change.
I can dig up some config examples if anyone is interested.
As for the other performance tweaks, I think removing the usage plugin
resulted in the biggest performance gain so far. Also adjusting the
internal object cache helped a lot. The SQL traffic dropped with over
90%. We've also done a lot of system changes, especially moving the
different storage areas around, e.g. index, temp attachments and so
on. I've also tried quite a few Java options that has improved
performance.
I've not put the latest tweaks into production yet, but I hope to load
test them shortly. They are (on a 4 core box):
-Xms128m -Xmx256m -XX:MaxPermSize=128m -Xms1024m -Xmx1024m -XX:
+UseParNewGC -XX:ParallelGCThreads=3 -XX:+DisableExplicitGC -
XX:MaxPermSize=128m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -
Xloggc:/usr/local/confluence/logs/tomcat/gc.log
...and the Confluence specific are:
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -
Djava.awt.headless=true -Djira.jelly.on=true -
Dconfluence.optimize.index.modulo=30 -
Datlassian.indexing.contentbody.maxsize=131072 -
Djava.awt.headless=true
Regards,
Dan
>
> I suggest that you go with 2.10.3 + patch it with the latest security
> patches released recently.http://confluence.atlassian.com/display/DOC/Confluence+Security+Advis...
> I suggest that you go with 2.10.3 + patch it with the latest security
Our site is internal, and even though it may sound ignorant to not
patch for XSS vulnerabilities, that is still how I have to prioritise
the fairly strained engineering resources we have.
>
> Based on our previous experience we are eager to see Confluence 3.0.x
> or 3.1 out too. :)
Btw, the chap that is responsible for the Sun JVM/JRE here pointed out
that 1.6.0_14 has a lot of performance improvements that may be very
beneficial for Confluence. I've not had a chance to try it myself yet,
though.
> Unfortunately the clustering in 2.x will most likely do the exact
> opposite for you.
>
> We were running cluster for a long time, but then finally gave up on
> it as it was causing more problems and outages than giving us benefits
> when it comes to performance, scalability or availability.
where the cluster caused (un)expected problems?
>
> Depending on your authentication mechanism you might be interested in
> even confluence 3.0 clustering a no-go for us.
Interesting. We do, however, run a separate application to manage
groups. I've not checked the code, but I think it uses the API a bit
better as we haven't had any performance problems.
> Something you want to share?
Let me elaborate on the Apache tuning, just in case that was what you
were interested in.
We did some standard tuning, by removing costly log details, cleaning
up the module includes and so on. We then disabled SSO on static
contents, such as /images/ and enabled caching. I used the memory
based cache module for /images/ and the disk based for /s/1518/ and /
download/. Atlassian confirmed that the dynamic content that came
from /s/<build_number>/ could be cached, as it was generated once and
then didn't change.
I can dig up some config examples if anyone is interested.
As for the other performance tweaks, I think removing the usage plugin
resulted in the biggest performance gain so far. Also adjusting the
internal object cache helped a lot. The SQL traffic dropped with over
90%. We've also done a lot of system changes, especially moving the
different storage areas around, e.g. index, temp attachments and so
on. I've also tried quite a few Java options that has improved
performance.
I've not put the latest tweaks into production yet, but I hope to load
test them shortly. They are (on a 4 core box):
-Xms128m -Xmx256m -XX:MaxPermSize=128m -Xms1024m -Xmx1024m -XX:
+UseParNewGC -XX:ParallelGCThreads=3 -XX:+DisableExplicitGC -
XX:MaxPermSize=128m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -
Xloggc:/usr/local/confluence/logs/tomcat/gc.log
...and the Confluence specific are:
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -
Djava.awt.headless=true -Djira.jelly.on=true -
Dconfluence.optimize.index.modulo=30 -
Datlassian.indexing.contentbody.maxsize=131072 -
Djava.awt.headless=true
Regards,
Dan
Igor Minar
Jun 15, 2009, 12:40:21 PM6/15/09
to enterprise...@googlegroups.com
Hi Dan,
On Jun 15, 2009, at 6:42 AM, Dan Petzen wrote:
> Btw, the chap that is responsible for the Sun JVM/JRE here pointed out
> that 1.6.0_14 has a lot of performance improvements that may be very
> beneficial for Confluence. I've not had a chance to try it myself yet,
> though.
I'm not sure which improvement you have in mind, but I'm specifically
eagerly waiting for G1 gc.
Though from what I know its totally experimental at the moment and
shouldn't be used in production without support contract or assuming
the risk of major problems. I know of some applications that currently
crash when run with G1.
>
>> Unfortunately the clustering in 2.x will most likely do the exact
>> opposite for you.
>>
>> We were running cluster for a long time, but then finally gave up on
>> it as it was causing more problems and outages than giving us
>> benefits
>> when it comes to performance, scalability or availability.
>
> Hmmm... Very worrying indeed. Do you have any examples of situations
> where the cluster caused (un)expected problems?
http://www.google.com/search?q=confluence+"cluster+panic"
For us the main blocker is http://jira.atlassian.com/browse/CONF-12319
When we have confluence running in a cluster and there is a spike in
the logging in activity, confluence goes down because it's tries to
pull of our users from the db and synchronize this data between the
nodes over and over again.
Then there are some random gc or network related delays that make
confluence unhappy.
The worst of all is that if there is something wrong, confluence will
shut down *all* of its nodes, rather than only those that are
suspected to be failing. This caused us completely unnecessary
outages, when a restart of the affected node would be good enough.
Another potential issue is http://jira.atlassian.com/browse/
CONF-15233, which we haven't experienced in cluster because we were
not running cluster at that time any more, but I can imagine
possibility of cluster going down in this case too.
>
>>
>> Depending on your authentication mechanism you might be interested in
>> looking athttp://jira.atlassian.com/browse/CONF-12319which makes
>> even confluence 3.0 clustering a no-go for us.
>
> Interesting. We do, however, run a separate application to manage
> groups. I've not checked the code, but I think it uses the API a bit
> better as we haven't had any performance problems.
We hit problems only after we had certain amount of users in our db.
With us, we do SSO and can't do user management asynchronously because
we have millions of users in our main identity system + user roles
there change dynamicaly. For this reason only synchronous user
management during login time is the only way we can do user
provisioning for confluence securely and effectively.
>
>> Something you want to share?
>
> The performance tweaks or Apache in particular?
>
> Let me elaborate on the Apache tuning, just in case that was what you
> were interested in.
>
> We did some standard tuning, by removing costly log details, cleaning
> up the module includes and so on. We then disabled SSO on static
> contents, such as /images/ and enabled caching. I used the memory
> based cache module for /images/ and the disk based for /s/1518/ and /
> download/. Atlassian confirmed that the dynamic content that came
> from /s/<build_number>/ could be cached, as it was generated once and
> then didn't change.
>
> I can dig up some config examples if anyone is interested.
Interesting, sounds similar to what I did for confluence using
Varnish. Except that I took it one step further and enabled caching of
all content distributed to anonymous users with expiration set to a
few minutes. I also fixed some broken caching instructions that
confluence sends to clients, which improved the efficiency of static
assets files in browsers.
We haven't deployed this yet because of some internal issues, but I
can tell you that I've never seen Confluence on fire like this. It's
totally slashdot ready.
>
> As for the other performance tweaks, I think removing the usage plugin
> resulted in the biggest performance gain so far. Also adjusting the
> internal object cache helped a lot. The SQL traffic dropped with over
> 90%.
Yup, that is a must.
> We've also done a lot of system changes, especially moving the
> different storage areas around, e.g. index, temp attachments and so
> on.
Can you elaborate? I haven't heard of anyone doing this.
> I've also tried quite a few Java options that has improved
> performance.
>
> I've not put the latest tweaks into production yet, but I hope to load
> test them shortly. They are (on a 4 core box):
> -Xms128m -Xmx256m -XX:MaxPermSize=128m -Xms1024m -Xmx1024m -XX:
> +UseParNewGC -XX:ParallelGCThreads=3 -XX:+DisableExplicitGC -
> XX:MaxPermSize=128m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -
> Xloggc:/usr/local/confluence/logs/tomcat/gc.log
You have Xms and Xmx there twice, I hope it's just a typo in the email.
I also found that increasing the young generation makes things better
as it prevents some short-lived object from being promoted to the old
generation.
>
> ...and the Confluence specific are:
> -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -
> Djava.awt.headless=true -Djira.jelly.on=true -
> Dconfluence.optimize.index.modulo=30 -
> Datlassian.indexing.contentbody.maxsize=131072 -
any docs/comments on these two?
cheers,
Igor
> Djava.awt.headless=true
>
> Regards,
> Dan
> >
On Jun 15, 2009, at 6:42 AM, Dan Petzen wrote:
> Btw, the chap that is responsible for the Sun JVM/JRE here pointed out
> that 1.6.0_14 has a lot of performance improvements that may be very
> beneficial for Confluence. I've not had a chance to try it myself yet,
> though.
eagerly waiting for G1 gc.
Though from what I know its totally experimental at the moment and
shouldn't be used in production without support contract or assuming
the risk of major problems. I know of some applications that currently
crash when run with G1.
>
>> Unfortunately the clustering in 2.x will most likely do the exact
>> opposite for you.
>>
>> We were running cluster for a long time, but then finally gave up on
>> it as it was causing more problems and outages than giving us
>> benefits
>> when it comes to performance, scalability or availability.
>
> Hmmm... Very worrying indeed. Do you have any examples of situations
> where the cluster caused (un)expected problems?
For us the main blocker is http://jira.atlassian.com/browse/CONF-12319
When we have confluence running in a cluster and there is a spike in
the logging in activity, confluence goes down because it's tries to
pull of our users from the db and synchronize this data between the
nodes over and over again.
Then there are some random gc or network related delays that make
confluence unhappy.
The worst of all is that if there is something wrong, confluence will
shut down *all* of its nodes, rather than only those that are
suspected to be failing. This caused us completely unnecessary
outages, when a restart of the affected node would be good enough.
Another potential issue is http://jira.atlassian.com/browse/
CONF-15233, which we haven't experienced in cluster because we were
not running cluster at that time any more, but I can imagine
possibility of cluster going down in this case too.
>
>>
>> Depending on your authentication mechanism you might be interested in
>> looking athttp://jira.atlassian.com/browse/CONF-12319which makes
>> even confluence 3.0 clustering a no-go for us.
>
> Interesting. We do, however, run a separate application to manage
> groups. I've not checked the code, but I think it uses the API a bit
> better as we haven't had any performance problems.
With us, we do SSO and can't do user management asynchronously because
we have millions of users in our main identity system + user roles
there change dynamicaly. For this reason only synchronous user
management during login time is the only way we can do user
provisioning for confluence securely and effectively.
>
>> Something you want to share?
>
> The performance tweaks or Apache in particular?
>
> Let me elaborate on the Apache tuning, just in case that was what you
> were interested in.
>
> We did some standard tuning, by removing costly log details, cleaning
> up the module includes and so on. We then disabled SSO on static
> contents, such as /images/ and enabled caching. I used the memory
> based cache module for /images/ and the disk based for /s/1518/ and /
> download/. Atlassian confirmed that the dynamic content that came
> from /s/<build_number>/ could be cached, as it was generated once and
> then didn't change.
>
> I can dig up some config examples if anyone is interested.
Varnish. Except that I took it one step further and enabled caching of
all content distributed to anonymous users with expiration set to a
few minutes. I also fixed some broken caching instructions that
confluence sends to clients, which improved the efficiency of static
assets files in browsers.
We haven't deployed this yet because of some internal issues, but I
can tell you that I've never seen Confluence on fire like this. It's
totally slashdot ready.
>
> As for the other performance tweaks, I think removing the usage plugin
> resulted in the biggest performance gain so far. Also adjusting the
> internal object cache helped a lot. The SQL traffic dropped with over
> 90%.
> We've also done a lot of system changes, especially moving the
> different storage areas around, e.g. index, temp attachments and so
> on.
> I've also tried quite a few Java options that has improved
> performance.
>
> I've not put the latest tweaks into production yet, but I hope to load
> test them shortly. They are (on a 4 core box):
> -Xms128m -Xmx256m -XX:MaxPermSize=128m -Xms1024m -Xmx1024m -XX:
> +UseParNewGC -XX:ParallelGCThreads=3 -XX:+DisableExplicitGC -
> XX:MaxPermSize=128m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -
> Xloggc:/usr/local/confluence/logs/tomcat/gc.log
I also found that increasing the young generation makes things better
as it prevents some short-lived object from being promoted to the old
generation.
>
> ...and the Confluence specific are:
> -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -
> Djava.awt.headless=true -Djira.jelly.on=true -
> Dconfluence.optimize.index.modulo=30 -
> Datlassian.indexing.contentbody.maxsize=131072 -
cheers,
Igor
> Djava.awt.headless=true
>
> Regards,
> Dan
> >
Dan Petzen
Jun 25, 2009, 11:22:03 AM6/25/09
to Confluence in the (real) Enterprise
[Note: I'm sorry for the delayed reply. The company I work has been
blocking Google Groups since last week. Very frustrating. Fortunately
enough, I saved my reply and I can now finally post it.]
Hi Igor.
I really appreciate youre reply. it's been very interesting to read.
> When we have confluence running in a cluster and there is a spike in
> the logging in activity, confluence goes down because it's tries to
> pull of our users from the db and synchronize this data between the
> nodes over and over again.
I read the links and the first one is a bit worrying. They've designed
the
cluster in a very awkward way.
>
> Then there are some random gc or network related delays that make
> confluence unhappy.
The first link clearly indicated that. Did you have problem with GC
stop-the-world and network latency?
>
> The worst of all is that if there is something wrong, confluence will
> shut down *all* of its nodes, rather than only those that are
> suspected to be failing. This caused us completely unnecessary
> outages, when a restart of the affected node would be good enough.
I was thinking about a way of put it more eloquently, but the only
thing
that sprang to mind was "that is exceptionally poor".
>
> Another potential issue ishttp://jira.atlassian.com/browse/
by
the minute.
Any examples or additional cautions would be very much appreciated.
I'd
like to pass on these concerns to people here before we go live with
the
cluster.
> We hit problems only after we had certain amount of users in our db.
>
> With us, we do SSO and can't do user management asynchronously because
> we have millions of users in our main identity system + user roles
> there change dynamicaly. For this reason only synchronous user
> management during login time is the only way we can do user
> provisioning for confluence securely and effectively.
We may only have 22,000, but they're all in the internal database, so
I
wouldn't be too surprised if we run into performance issues with users
and
groups as well.
> Interesting, sounds similar to what I did for confluence using
> Varnish. Except that I took it one step further and enabled caching of
> all content distributed to anonymous users with expiration set to a
> few minutes. I also fixed some broken caching instructions that
> confluence sends to clients, which improved the efficiency of static
> assets files in browsers.
I did meddle with some headers:
CacheIgnoreCacheControl On
CacheIgnoreNoLastMod On
...but nothing more than that, really. I'll do some reading up on that
and
Varnish.
> > We've also done a lot of system changes, especially moving the
> > different storage areas around, e.g. index, temp attachments and so
> > on.
>
> Can you elaborate? I haven't heard of anyone doing this.
It's probably mostly because I couldn't get hold of proper hardware. I
ran
the system on a blade with 2.5" disks. They were so slow they made the
system grind to a halt. Before we identified the problem with the
usage
stat plugin, the write to the index was horrific, so we mapped it to a
tmpfs (memory mapped) filesystem. We then, amazingly enough, gained
further performance by moving all variable data directories to NFS(!).
> You have Xms and Xmx there twice, I hope it's just a typo in the email.
Ha! Brilliant! I had forgotten to update the setenv.sh. Thanks for
spotting a bug for me. I owe you a beer.
>
> I also found that increasing the young generation makes things better
> as it prevents some short-lived object from being promoted to the old
> generation.
Ah, that is very interesting I'll talk to one of the JVM Teflon heads
here. That makes sense.
>
>
>
> > ...and the Confluence specific are:
> > -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -
> > Djava.awt.headless=true -Djira.jelly.on=true -
> > Dconfluence.optimize.index.modulo=30 -
> > Datlassian.indexing.contentbody.maxsize=131072 -
>
> any docs/comments on these two?
Sorry, but which ones? I'm not being difficult, but it'll take some
time
to dig up the details about all of them. From the top of my head:
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
Limiting of Tomcat (JSP) caching (stability precaution)
-Djava.awt.headless=true -Djira.jelly.on=true
Recommended by Atlassian. AWT optimisation.
-Djira.jelly.on=true
Can't remember!
-Dconfluence.optimize.index.modulo=30
Gah! I know this one. I can't remember it now and I have to dash
shortly.
Datlassian.indexing.contentbody.maxsize=131072
The maximum size a document that is indexed can have. If it's above
this
value, then it won't be indexed.
I'll elaborate on any of them if you want me to, but it'll have to be
next
week.
Regards,
Dan
blocking Google Groups since last week. Very frustrating. Fortunately
enough, I saved my reply and I can now finally post it.]
Hi Igor.
I really appreciate youre reply. it's been very interesting to read.
> When we have confluence running in a cluster and there is a spike in
> the logging in activity, confluence goes down because it's tries to
> pull of our users from the db and synchronize this data between the
> nodes over and over again.
the
cluster in a very awkward way.
>
> Then there are some random gc or network related delays that make
> confluence unhappy.
stop-the-world and network latency?
>
> The worst of all is that if there is something wrong, confluence will
> shut down *all* of its nodes, rather than only those that are
> suspected to be failing. This caused us completely unnecessary
> outages, when a restart of the affected node would be good enough.
thing
that sprang to mind was "that is exceptionally poor".
>
> Another potential issue ishttp://jira.atlassian.com/browse/
> CONF-15233, which we haven't experienced in cluster because we were
> not running cluster at that time any more, but I can imagine
> possibility of cluster going down in this case too.
Oh, my. That could bring down the cluster. I'm getting less impressed
> not running cluster at that time any more, but I can imagine
> possibility of cluster going down in this case too.
by
the minute.
Any examples or additional cautions would be very much appreciated.
I'd
like to pass on these concerns to people here before we go live with
the
cluster.
> We hit problems only after we had certain amount of users in our db.
>
> With us, we do SSO and can't do user management asynchronously because
> we have millions of users in our main identity system + user roles
> there change dynamicaly. For this reason only synchronous user
> management during login time is the only way we can do user
> provisioning for confluence securely and effectively.
I
wouldn't be too surprised if we run into performance issues with users
and
groups as well.
> Interesting, sounds similar to what I did for confluence using
> Varnish. Except that I took it one step further and enabled caching of
> all content distributed to anonymous users with expiration set to a
> few minutes. I also fixed some broken caching instructions that
> confluence sends to clients, which improved the efficiency of static
> assets files in browsers.
CacheIgnoreCacheControl On
CacheIgnoreNoLastMod On
...but nothing more than that, really. I'll do some reading up on that
and
Varnish.
> > We've also done a lot of system changes, especially moving the
> > different storage areas around, e.g. index, temp attachments and so
> > on.
>
> Can you elaborate? I haven't heard of anyone doing this.
ran
the system on a blade with 2.5" disks. They were so slow they made the
system grind to a halt. Before we identified the problem with the
usage
stat plugin, the write to the index was horrific, so we mapped it to a
tmpfs (memory mapped) filesystem. We then, amazingly enough, gained
further performance by moving all variable data directories to NFS(!).
> You have Xms and Xmx there twice, I hope it's just a typo in the email.
spotting a bug for me. I owe you a beer.
>
> I also found that increasing the young generation makes things better
> as it prevents some short-lived object from being promoted to the old
> generation.
here. That makes sense.
>
>
>
> > ...and the Confluence specific are:
> > -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -
> > Djava.awt.headless=true -Djira.jelly.on=true -
> > Dconfluence.optimize.index.modulo=30 -
> > Datlassian.indexing.contentbody.maxsize=131072 -
>
> any docs/comments on these two?
time
to dig up the details about all of them. From the top of my head:
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
Limiting of Tomcat (JSP) caching (stability precaution)
-Djava.awt.headless=true -Djira.jelly.on=true
Recommended by Atlassian. AWT optimisation.
-Djira.jelly.on=true
Can't remember!
-Dconfluence.optimize.index.modulo=30
Gah! I know this one. I can't remember it now and I have to dash
shortly.
Datlassian.indexing.contentbody.maxsize=131072
The maximum size a document that is indexed can have. If it's above
this
value, then it won't be indexed.
I'll elaborate on any of them if you want me to, but it'll have to be
next
week.
Regards,
Dan
Dan Hardiker
Jun 25, 2009, 4:10:49 PM6/25/09
to enterprise...@googlegroups.com
2009/6/25 Dan Petzen <dan...@petzen.co.uk>
This is because the thumbnailing in Confluence requires a running X11 system, unless you push it into a headless mode.
See also: http://confluence.atlassian.com/display/CONFKB/Confluence+doesn%27t+generate+thumbnails
Jelly is an XML based scripting language used in JIRA. This has no effect in Confluence (to the best of my knowledge).
http://confluence.atlassian.com/display/DOC/Recognised+System+Properties will probably be more succinct than me rattling off info out of my head for you :)
--
Dan Hardiker
Adaptavist.com Ltd
-Djava.awt.headless=true
Recommended by Atlassian. AWT optimisation.
This is because the thumbnailing in Confluence requires a running X11 system, unless you push it into a headless mode.
See also: http://confluence.atlassian.com/display/CONFKB/Confluence+doesn%27t+generate+thumbnails
-Djira.jelly.on=true
Can't remember!
Jelly is an XML based scripting language used in JIRA. This has no effect in Confluence (to the best of my knowledge).
-Dconfluence.optimize.index.modulo=30
-Datlassian.indexing.contentbody.maxsize=131072
http://confluence.atlassian.com/display/DOC/Recognised+System+Properties will probably be more succinct than me rattling off info out of my head for you :)
--
Dan Hardiker
Adaptavist.com Ltd
Igor Minar
Jun 25, 2009, 5:14:28 PM6/25/09
to enterprise...@googlegroups.com
/i
Igor Minar
Jun 25, 2009, 5:57:34 PM6/25/09
to enterprise...@googlegroups.com
On Jun 25, 2009, at 8:22 AM, Dan Petzen wrote:
>
> [Note: I'm sorry for the delayed reply. The company I work has been
> blocking Google Groups since last week. Very frustrating. Fortunately
> enough, I saved my reply and I can now finally post it.]
>
> Hi Igor.
>
> I really appreciate youre reply. it's been very interesting to read.
because together we can do more than each of us alone.
>
>> When we have confluence running in a cluster and there is a spike in
>> the logging in activity, confluence goes down because it's tries to
>> pull of our users from the db and synchronize this data between the
>> nodes over and over again.
>
> I read the links and the first one is a bit worrying. They've designed
> the
> cluster in a very awkward way.
>
>>
>> Then there are some random gc or network related delays that make
>> confluence unhappy.
>
> The first link clearly indicated that. Did you have problem with GC
> stop-the-world and network latency?
to a full GC taking too long.
>
>>
>> The worst of all is that if there is something wrong, confluence will
>> shut down *all* of its nodes, rather than only those that are
>> suspected to be failing. This caused us completely unnecessary
>> outages, when a restart of the affected node would be good enough.
>
> I was thinking about a way of put it more eloquently, but the only
> thing
> that sprang to mind was "that is exceptionally poor".
>
>>
>> Another potential issue ishttp://jira.atlassian.com/browse/
>> CONF-15233, which we haven't experienced in cluster because we were
>> not running cluster at that time any more, but I can imagine
>> possibility of cluster going down in this case too.
>
> Oh, my. That could bring down the cluster. I'm getting less impressed
> by
> the minute.
>
> Any examples or additional cautions would be very much appreciated.
> I'd
> like to pass on these concerns to people here before we go live with
> the
> cluster.
in cluster they are always stored in the db, which will cause problems
if you don't limit the size and have wiki users who are eager to
upload and download attachments.
>
>> We hit problems only after we had certain amount of users in our db.
>>
>> With us, we do SSO and can't do user management asynchronously
>> because
>> we have millions of users in our main identity system + user roles
>> there change dynamicaly. For this reason only synchronous user
>> management during login time is the only way we can do user
>> provisioning for confluence securely and effectively.
>
> We may only have 22,000, but they're all in the internal database, so
> I
> wouldn't be too surprised if we run into performance issues with users
> and
> groups as well.
if you are using MySQL, I suggest you patch confluence or in case of
Oracle or postgresql create extra indexes. See: http://jira.atlassian.com/browse/CONF-10030
cluster. In general it is too slow for any serious production use. You
should use google analytics or similar web analytics instead.
>
>
>> You have Xms and Xmx there twice, I hope it's just a typo in the
>> email.
>
> Ha! Brilliant! I had forgotten to update the setenv.sh. Thanks for
> spotting a bug for me. I owe you a beer.
I think that the default (1MB) is reasonable, but maybe you found it
too high for your environment.
>
> I'll elaborate on any of them if you want me to, but it'll have to be
> next
> week.
cheers,
Igor
>
> Regards,
> Dan
>
> >
Eric Wells
Aug 18, 2009, 5:33:43 PM8/18/09
to Confluence in the (real) Enterprise
Hi,
We've been running Confluence for 3 years as a standalone app, no
clustering. I'm a techy guy by nature but know little to nothing of
Java, Websphere, and clustering. So without getting into the nitty
gritty details, I'm looking for feedback on whether we should apply
clustering now.
A few details on the current environment:
==============================
- Currently we're on Java 1.4, WAS 6, and UDB 9
- We run Confluence 2.5.1
- Users are in the low thousands (as in 1 or 2K)
- Aside from "confluence-users" most people are not in any groups
- We do a call to Siteminder for authentication, but then users (1 or
2K) and groups (just a few small ones) are maintained internal to
confluence (no link to LDAP groups)
- Most applications at our company are clustered
- We have a lot of Java expertise around, it's just not me :)
We have a project to upgrade to Confluence 3, and are considering
applying clustering as part of this upgrade. I think when we first did
a POC with Confluence 2.3 clustering was not an option. By the time we
deployed with 2.5.1 clustering was available but it was too close to
deployment to go that route without adding more risk than we wanted to
take on. So we didn't cluster at the time.
We are not experiencing performance issues. But we may scale up our
users to possibly 4 or 5K (still not very big). And we'd like to align
ourselves as a more standard application (most are clustered here).
But I don't want to do clustering "just because" and shoot ourselves
in the foot.
Having read over the issues described above, I'm not sure they apply
to us.
- One dealt with large numbers of users in groups, but we don't have
that many users and groups are very tiny (admins).
- Purging large amounts of trash (looks like an issue unique to one
situation, not applicable to us)
- File storage has to be in the DB? Ok, we can switch, is this a
problem? (Attachments are currently on unix file system)
Do you have any advice for us? Will clustering add problems and
headaches above and beyond any benefits we might find? Or should we go
ahead with this?
Eric
> Watch out for attachmentshttp://jira.atlassian.com/browse/CONF-9335
> > Dan- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -
We've been running Confluence for 3 years as a standalone app, no
clustering. I'm a techy guy by nature but know little to nothing of
Java, Websphere, and clustering. So without getting into the nitty
gritty details, I'm looking for feedback on whether we should apply
clustering now.
A few details on the current environment:
==============================
- Currently we're on Java 1.4, WAS 6, and UDB 9
- We run Confluence 2.5.1
- Users are in the low thousands (as in 1 or 2K)
- Aside from "confluence-users" most people are not in any groups
- We do a call to Siteminder for authentication, but then users (1 or
2K) and groups (just a few small ones) are maintained internal to
confluence (no link to LDAP groups)
- Most applications at our company are clustered
- We have a lot of Java expertise around, it's just not me :)
We have a project to upgrade to Confluence 3, and are considering
applying clustering as part of this upgrade. I think when we first did
a POC with Confluence 2.3 clustering was not an option. By the time we
deployed with 2.5.1 clustering was available but it was too close to
deployment to go that route without adding more risk than we wanted to
take on. So we didn't cluster at the time.
We are not experiencing performance issues. But we may scale up our
users to possibly 4 or 5K (still not very big). And we'd like to align
ourselves as a more standard application (most are clustered here).
But I don't want to do clustering "just because" and shoot ourselves
in the foot.
Having read over the issues described above, I'm not sure they apply
to us.
- One dealt with large numbers of users in groups, but we don't have
that many users and groups are very tiny (admins).
- Purging large amounts of trash (looks like an issue unique to one
situation, not applicable to us)
- File storage has to be in the DB? Ok, we can switch, is this a
problem? (Attachments are currently on unix file system)
Do you have any advice for us? Will clustering add problems and
headaches above and beyond any benefits we might find? Or should we go
ahead with this?
Eric
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -
Igor Minar
Aug 18, 2009, 5:57:28 PM8/18/09
to enterprise...@googlegroups.com
If I were you I wouldn't worry about "fixing something that isn't
broken".
It sounds like your instance is relatively tiny, so clustering would
not give you any benefits that would be worth the trouble.
That's just my opinion.
cheers,
Igor
broken".
It sounds like your instance is relatively tiny, so clustering would
not give you any benefits that would be worth the trouble.
That's just my opinion.
cheers,
Igor
unixg...@gmail.com
Aug 19, 2009, 4:53:45 PM8/19/09
to Confluence in the (real) Enterprise
No experience with clustered confluence, and we are still working on
our 2.8 to 3.0 migration. And we are a site with some bad performance
problems.
However, past experience and good engineering principles would say
that don't make two major changes at once. Much harder to isolate a
problem, rollback, or identify which one produced any improvement.
our 2.8 to 3.0 migration. And we are a site with some bad performance
problems.
However, past experience and good engineering principles would say
that don't make two major changes at once. Much harder to isolate a
problem, rollback, or identify which one produced any improvement.
Rodney McDuff
Mar 4, 2010, 2:40:53 AM3/4/10
to enterprise...@googlegroups.com
Hi All
Does anyone know of a clustered confluence deployment using the
confluence shibboleth authenticator?
Does anyone know of a clustered confluence deployment using the
confluence shibboleth authenticator?
--
Dr. Rodney G. McDuff |Ex ignorantia ad sapientiam
Manager, Strategic Technologies Group| Ex luce ad tenebras
Information Technology Services |
The University of Queensland |
EMAIL: mcd...@its.uq.edu.au |
TELEPHONE: +61 7 3365 8220 |
Reply all
Reply to author
Forward
0 new messages