Optimizing IdP's performance

[Manuel Haim]

Hi list,

I'd like to collect and hear some of your thoughts on optimizing the
overall performance of your IdP cluster - especially those ideas that
still seem to be missing on the Shibboleth wiki.

Here is what we got:
1) Enable debug mode (or even trace mode) in your IdP's logging.xml and
perform a login. Then look at the logs. Which actions do consume most of
the time? Anything above 50ms could be worth optimizing.
2) Disable debug mode before going productive or performing load tests
with the Grinder!
3) Have enough physical RAM available. If a node starts to use swap
space, it will become incredibly slow. Under Debian GNU/Linux, 2GB seem
to be good for a Tomcat setup without Terracotta, 3-4GB if you run
Tomcat and a Terracotta server on the same machine.
4) Give enough RAM to Tomcat. In our case, the options -Xms256m
-Xmx1536m gave good results, even after 150.000 logins.
5) Spend enough CPU power for your processes. Check the load on your
servers - how busy is the CPU when performing a load test? If running on
a virtual server, do some load testing with different numbers of CPUs,
and compare the results.
6) Optimize your software stack. We use a light-weight http(s) load
balancer (pound) to connect to the Tomcat instances directly (no Apache
needed). For back-channel requests to work correctly, the SSL Client
Cert must be forwarded (and as we use http instead of an ajp connection,
we needed to add or implement an SSLValve to Tomcat's server.xml which
moves the certificate from the http header into the httpRequest object).
7) Optimize Terracotta. What are your ideas here? Can the persistent
disk cache be turned off, so the Terracotta server runs in RAM only? The
Shibboleth wiki gives pretty less hints on this...
8) If using LDAP: Optimize your LDAP queries. Do not use
":caseExactmatch:" if an attribute is not indexed case-sensitive. For
your LDAP connectors in attribute-resolver.xml, try to define connection
pools (however, when under load, the IdP already does connection pooling
by default). And for LDAP authentication / JAAS Configuration
(login.config), try to implement a custom dnResolver (which statically
builds the entryDn, for example; by default, the IdP performs two costly
LDAP binds for each login: one by the SearchDnResolver which looks for
the user's entryDn but does not use connection pooling whatsoever - this
one can be optimized as described - and one for the LDAP authentication
request which must bind by the user's entryDn and thus cannot be pooled).

How do you optimize your IdP? (except for the points already mentioned
on the Productionalization pages in the Shibboleth wiki...)

-Manuel

--
To unsubscribe from this list send an email to users-un...@shibboleth.net

[Chad La Joie]

Manual, thanks for bring up this topic. I've included a bunch of
information below. However, as I note every time this conversation
comes up; before anyone goes on an optimizing binge it's important to
understand:
- You must understand what your needs are. Spending 6 months
customizing your IdP to perform 100 simultaneous SSOs when you only
ever see 10 in your real world use is a waste of time and likely
leaves you with an IdP setup that is harder to upgrade.
- The measure that you really care about, when doing this testing, is
# of simultaneous requests which perform under a given latency
threshold. As I mention below, I've already done quite a bit of
per-request optimization in the IdP so people almost never need to
worry about the per-request latency except whatever is added by
querying attribute sources.
- If people ever feel like they've discovered a memory leak or a
specialized request that seems to take longer than it should, I can
provide instructions for capturing the data necessary for me to
analyze the IdP and either tell you why you're not seeing what you
think your seeing or fix any issue in the IdP.

On Mon, Aug 1, 2011 at 08:11, Manuel Haim <ha...@hrz.uni-marburg.de> wrote:
> Hi list,

> 1) Enable debug mode (or even trace mode) in your IdP's logging.xml and
> perform a login. Then look at the logs. Which actions do consume most of
> the time? Anything above 50ms could be worth optimizing.

I've already done pretty extensive profiling on the IdP. On an
average request, sans waiting for user input, approximately 50% of the
time is spent doing cryptography and approximately 50% of the time is
spent waiting for attribute data sources to return information.
Optimizing the crypto is possible but would be quite complicated for
deployers. Optimizing attribute access is possible but different
solutions will apply for different deployments. One option, if user
data doesn't change much, is to turn on caching in the data
connectors. Another option is to host a "shadow" copy of the data
source (or at least the part that you care about) on the IdP machine
itself.

Note though, that optimizing any of this does not necessarily increase
throughput, it just decreases latency.

> 4) Give enough RAM to Tomcat. In our case, the options -Xms256m
> -Xmx1536m gave good results, even after 150.000 logins.

The amount of memory needed is directly proportional to the amount of
metadata that you are loading in the IdP. Unfortunately I can't give
a good formula for computing the amount of memory used based on the
file size as it changes, sometimes quite drastically, with changes in
the XML parser itself.

> 5) Spend enough CPU power for your processes. Check the load on your
> servers - how busy is the CPU when performing a load test? If running on
> a virtual server, do some load testing with different numbers of CPUs,
> and compare the results.

Oracles and IBM's JVMs don't scale very well beyond 16 CPUs, so
throwing any more at the process quickly demonstrates the concept of
diminishing returns. I *believe* that the Oracle Java 7 CPU can now
take advantage of the AES instructions showing up on various CPUs. As
crypto takes up ~50% of all the time for a given request, having
access to these instructions would be very nice however most
virtualization technologies today do not allow access to them. So
there would certainly be a trade off.

> 6) Optimize your software stack. We use a light-weight http(s) load
> balancer (pound) to connect to the Tomcat instances directly (no Apache
> needed). For back-channel requests to work correctly, the SSL Client
> Cert must be forwarded (and as we use http instead of an ajp connection,
> we needed to add or implement an SSLValve to Tomcat's server.xml which
> moves the certificate from the http header into the httpRequest object).

You can also try changing out Tomcat if you really want to optimize at
this layer. Tomcat's connection handling is pretty poor in my areas
and has a couple of high-latency sections (due to global or
semi-global locks). It also does not tend to handle high loads very
well.

> 8) If using LDAP: Optimize your LDAP queries. Do not use
> ":caseExactmatch:" if an attribute is not indexed case-sensitive. For
> your LDAP connectors in attribute-resolver.xml, try to define connection
> pools (however, when under load, the IdP already does connection pooling
> by default). And for LDAP authentication / JAAS Configuration
> (login.config), try to implement a custom dnResolver (which statically
> builds the entryDn, for example; by default, the IdP performs two costly
> LDAP binds for each login: one by the SearchDnResolver which looks for
> the user's entryDn but does not use connection pooling whatsoever - this
> one can be optimized as described - and one for the LDAP authentication
> request which must bind by the user's entryDn and thus cannot be pooled).

A potential big win for the LDAP data connector (total impact depends
on the LDAP server) is explicitly listing the attributes you want to
get back.

I *strongly* discourage people from simply creating the DN from given
input from the user. It's a very brittle setup and I would remove any
such suggestion added to the wiki. If that extra search, done via
pooled connections, is such a onerous amount of work then the problem
is with your LDAP.

--
Chad La Joie
www.itumi.biz
trusted identities, delivered

[Manuel Haim]

Chad,

thank you for your response and helpful recommendations.

> I *strongly* discourage people from simply creating the DN from given
> input from the user. It's a very brittle setup and I would remove any
> such suggestion added to the wiki. If that extra search, done via
> pooled connections, is such a onerous amount of work then the problem
> is with your LDAP.

Well, yes. However, it seems that the SearchDnResolver does not make use
of pooled connections yet. In our setup, this not only impaired the
latency, but also resulted in less logins per second (compared to using
a static DN). Is connection pooling planned here for a future release?

-Manuel

[Chad La Joie]

I'll have to ask Dan, the developer of the LDAP library, I thought he
had already moved to using a pool for the DN searching.

--

Chad La Joie
www.itumi.biz
trusted identities, delivered

[Marvin Addison]

> Tomcat's connection handling is pretty poor in my areas
> and has a couple of high-latency sections (due to global or
> semi-global locks).

I'm curious about this comment. Are you saying this is generally true
and applies to all connectors (Coyote, NIO, APR) in your experience,
or only certain ones?

M

[Chad La Joie]

I have not looked closely at the APR connectors, that has had issues
of its own in the past (and they pop up again from time to time) and
it's a lot of JNI which, to me, is just voodoo of the most evil sort.

The main problem with the pure-Java connectors, when I looked at them,
was the lack of a dispatch queue. Initially the blocking connector
spun up a thread per-reques, which pretty much caused the whole server
to die under load. The non-blocking, and more recently the blocking,
connectors have had a thread pool that are used to answer requests.
However, they don't have a dispatch queue, which means that once all
the threads in the pool are in use you start getting 500 errors. So
it doesn't degrade very gracefully.

I'll note, I haven't looked at the code in the last year or so (since
deciding we'll be using Jetty for the IdP), so v7 or perhaps more
recent v6 releases, might have addressed this issue.

--

Chad La Joie
www.itumi.biz
trusted identities, delivered

[Daniel Fisher]

On Mon, Aug 1, 2011 at 10:46 AM, Manuel Haim <ha...@hrz.uni-marburg.de> wrote:

Chad,

thank you for your response and helpful recommendations.

> I *strongly* discourage people from simply creating the DN from given
> input from the user.  It's a very brittle setup and I would remove any
> such suggestion added to the wiki.  If that extra search, done via
> pooled connections, is such a onerous amount of work then the problem
> is with your LDAP.

Well, yes. However, it seems that the SearchDnResolver does not make use
of pooled connections yet. In our setup, this not only impaired the
latency, but also resulted in less logins per second (compared to using
a static DN). Is connection pooling planned here for a future release?

Early versions of the 2.x IDP attempted to use connection pooling for DN resolution, with mixed results due to some bugs. Ultimately we decided it simply violated the stateless nature of JAAS and moved to the current implementation which opens and closes a connection for every DN lookup. Open a feature request for this, it's a good time for me to review this again.

--Daniel Fisher

[Chad La Joie]

Ah, that's right, I forgot about the whole JAAS stateless LoginModule thing.

> --
> To unsubscribe from this list send an email to
> users-un...@shibboleth.net
>

--

Chad La Joie
www.itumi.biz
trusted identities, delivered

[Manuel Haim]

Daniel,

>
> Early versions of the 2.x IDP attempted to use connection pooling for
> DN resolution, with mixed results due to some bugs. Ultimately we
> decided it simply violated the stateless nature of JAAS and moved to
> the current implementation which opens and closes a connection for
> every DN lookup. Open a feature request for this, it's a good time for
> me to review this again.
>
>

thanks for your comment, I've just added an issue under
http://code.google.com/p/vt-middleware/issues/detail?id=118

If no connection pool was explicitly defined in the IdP's attribute
resolver, the IdP by default seems to keep an connection pool open only
if multiple LDAP queries are performed in a short timeframe. Maybe this
could be an option for vt-ldap, too.

[Daniel Fisher]

On Tue, Aug 2, 2011 at 5:33 AM, Manuel Haim <ha...@hrz.uni-marburg.de> wrote:

Daniel,

>
> Early versions of the 2.x IDP attempted to use connection pooling for
> DN resolution, with mixed results due to some bugs. Ultimately we
> decided it simply violated the stateless nature of JAAS and moved to
> the current implementation which opens and closes a connection for
> every DN lookup. Open a feature request for this, it's a good time for
> me to review this again.
>
>

thanks for your comment, I've just added an issue under
http://code.google.com/p/vt-middleware/issues/detail?id=118

Thanks. This comment in the issue is disconcerting:

It is indeed the LDAP bind operations reducing performance, each taking about 50ms and somehow blocking the whole Shibboleth IdP. 

Do you have any logs demonstrating that the IDP is blocking on DN resolution specifically or JAAS authentication generally?

--Daniel Fisher

[Manuel Haim]

Daniel,

It is indeed the LDAP bind operations reducing performance, each taking about 50ms and somehow blocking the whole Shibboleth IdP. 

Do you have any logs demonstrating that the IDP is blocking on DN resolution specifically or JAAS authentication generally?

I did a quick check and must excuse: The Shibboleth IdP is not blocked (according to the logs, the login requests seem to run in parallel threads), but just processing about 30-50% less logins per second.

-Manuel