Performance recommendations/APC?

[Mike Grady]

I found the performance results for SimpleSAML that UNINETT did in 2008. Simple searching didn't turn up much else. But would using APC, which appeared to make a big difference in that 2008 testing, be a general recommendation? Any other general recommendations/tips to maximize performance?

[Jaime Perez Crespo]

Hi Mike,

> On 03 Apr 2015, at 19:34 pm, Mike Grady <mapg...@gmail.com> wrote:
> I found the performance results for SimpleSAML that UNINETT did in 2008. Simple searching didn't turn up much else. But would using APC, which appeared to make a big difference in that 2008 testing, be a general recommendation? Any other general recommendations/tips to maximize performance?

Well, that is ancient and quite outdated now. But you already know that, of course.

I don’t know if there’s any other assessment on SSP’s performance more up to date than that. What I can do is to give you a hint on what we do in Feide, just in case that helps you.

We use SimpleSAMLphp with a custom authentication source as the unique identity provider in the federation, and that’s connected to the different LDAP backends of the institutions in Norway. Last year we got more than 75M authentications. This year, we’ve seen already days with more than .5M authentications. We’ve also observed peaks of 80 logins per second. It’s definitely not something very big, but quite good for a single identity provider, I think.

The infrastructure running this is relatively simple. We have four servers in production, each of them running SSP on their own. Two load balancers in front of them, and another two load-balanced memcache backend servers behind, holding session data and consent. Requests can be directed to any of the production servers, which will then gather session information from the backend. We also have a couple more servers as backup, with their own load balancers in front, which we use when we detect a problem in the production infrastructure or when we do some maintenance work. Switchover is performed by means of DNS.

We never had performance problems with this infrastructure and our current volume of use. We’ve done some performance tests before, stopping, if I recall correctly, at 10K authentications per minute. By coincidence, this saturday we’ll be running some more performance tests, this time looking for the real bottleneck of the infrastructure and trying to choke it. I can share with you the conclusions, if you are interested.

So my advice is:

- Prefork helps. There’s people who have a good experience with nginx, but I'm also aware of some problems with it.
- Avoid external connections as much as possible. If you really need to use a backend for sessions or something else, go for something really fast like memcache or others.
- If you have external sources of metadata, do caching. Aggressively. If possible, cache directly in memory. You could use tmpfs for that.
- If possible, distribute load across different servers.
- Use load balancers.
- Reduce the amount of things running in the servers. If you have a box for SSP on its own (either physical or virtual), that’s much better.
- Do not run expensive, periodic operations on the servers, as metadata refreshing. Specially if you manage huge metadata feeds as eduGAIN’s or UK fed’s. Use a dedicated machine for that, then synchronize metadata by other means (git, svn, rsync, your favourite choice here).
- Set reasonable timeouts for keep alive connections. If you are using load balancers, it might be wise to have connections always opened with your load balancers, so that you can avoid the maximum connections limit and share the resources optimally.
- Set a reasonable limit to the amount of processes apache is using. It doesn’t really help to allow more processes if you run out of memory. As soon as your server starts caching to disk, performance will degrade significantly (unless you are using ultrafast, SSD disks).

Hope that helps :-)

--
Jaime Pérez
UNINETT / Feide
mail: jaime...@uninett.no
xmpp: ja...@jabber.uninett.no

"Two roads diverged in a wood, and I, I took the one less traveled by, and that has made all the difference."
- Robert Frost

[Mike Grady]

Hope that helps :-)

--

Thanks, that is quite useful information, and yes, I would be interested in your conclusions after your planned performance tests.

A bit of further research on PHP opcode caching seems to indicate the now preferred tool is Zend OpCache versus APC, and that with PHP 5.5. that will be "built-in". Are you using an opcode cache?

Lastly, what would you recommend in terms of memory? Any rules of thumb you operate under, memory versus number of Apache processes? Of course, the cryptographic stuff can also, presumably, put a load on the CPUs.

[Jaime Perez Crespo]

Hi,

> On 09 Apr 2015, at 21:33 pm, Mike Grady <mapg...@gmail.com> wrote:
> Thanks, that is quite useful information, and yes, I would be interested in your conclusions after your planned performance tests.

Noted :-)

> A bit of further research on PHP opcode caching seems to indicate the now preferred tool is Zend OpCache versus APC, and that with PHP 5.5. that will be "built-in". Are you using an opcode cache?

I can’t tell, since we’re not running the servers directly. But in any case, it definitely makes sense that an OpCache helps with performance. Normal flows of SimpleSAMLphp usually involve very few scripts, so if you can “compile” and cache those in memory for faster access, then that's a good idea.

> Lastly, what would you recommend in terms of memory? Any rules of thumb you operate under, memory versus number of Apache processes? Of course, the cryptographic stuff can also, presumably, put a load on the CPUs.

In general we don’t worry much about it since memory is cheap these days and we never had a problem in there. I suspect, though, that it could be a potential bottleneck. So if you really want to optimize that, you’ll need to fine tune your apache processes limit to fit in your memory available. Trial an error...

Regarding crypto, that’s resource consuming, yes. The first and most intuitive step would be to offload TLS to the load balancers, so that the servers don’t need to worry about that. There’s signature validations and SAML response signing then. Unfortunately, we don’t have support for HSM modules in SSP, though there’s people who have been successful on implementing it (the guys from wayf.dk may have a word on this). If you can move your cryptographic operations out to an external, dedicated module, that should help improve your performance. But then you need to be careful because you are introducing a new module and, potentially, a new bottleneck.

There’s also some other considerations. XML canonicalization is very time expensive the way it is implemented in PHP, and that you can clearly see when consuming big metadata files. We included a workaround in the xmlseclibs that ships with the latest SSP version, but that workaround is yet to be confirmed as secure (as in, it doesn’t break any type of canonicalization). For our purposes it works great and we are unaware of any glitches, though. Now, if you follow my advice to consume metadata outside of the main servers running SSP, this is not a big deal as SAML requests and responses are usually quite small. However, you need to keep in mind that if you validate signatures, someone could perform a DoS attack against your SSP deployment by sending huge, fake, signed responses or requests to your SSP. Then SSP would take a long time to validate the signatures, which happens before analyzing the messages themselves. So it doesn’t matter if the messages are legit or not, because SSP will take a long time to discover. To avoid that, make sure you set a limit to the maximum size of the parameters you can receive in PHP, reasonable enough to allow all legitimate traffic, while keeping such an attack away.

--