EngineBlock Key Rollover

53 views
Skip to first unread message

andrew...@cybera.ca

unread,
Feb 16, 2017, 5:55:11 PM2/16/17
to OpenConext Community
Hi all,

We're currently looking into SAML key-rollover for EngineBlock in case we ever need to switch to a new SAML key. Looking briefly at the EB codebase, I found these lines referring to a Key Rollover: https://github.com/OpenConext/OpenConext-engineblock/blob/5.x-dev/application/configs/application.ini#L44-L46  , however, they don't seem to be referenced anywhere in the OpenConext-deploy playbooks to use this functionality.

Has this been something that you've ever needed to do? Is it just a matter of updating the config file from the shell with the new cert+key file and then running an EngineBlock reload? 

Thanks,

Andrew

Pieter van der Meulen

unread,
Feb 17, 2017, 10:53:08 AM2/17/17
to OpenConext Community
Hi Andrew,

The functonality you are refering to was used by SURFconext to switch to a new signing key in response to the heartbleed vulnerability. We were using a different LB method and were not using php-fpm at the time and we determined that the SAML signing key could have been exposed, so we wanted to do a key rollover.

There are two methods to switch to a new signing key using OpenConext:


1. The "classic rollover" method

- You generate a new key and SAML signing certificate
- You add the new certificate as a second certificate to the metadata of EB, you do this by adding the new certificate in SR to the metadata of the IdP with the EntityID (connection ID) of EB. You can use the medata field "certData2" file for this. You should see the 2nd certificate appearing in the metadata's published by eb (i.e. .../idp/metadata )
- You wait until all your SPs have refreshed their metadata, you will want to communicate this change to them beforehand. Your SPs now trust both your new and your old certificate
- You update the configuration of "encryption.keys.default" so EB will now sign using your new key
- You update the EB IdP Metadata in serviceregistry so that it contains your new certificate only, in certData
- After all your SPs have refreshed their metadata again, they now only trust your new key
- You are done

I call this the "classic rollover" method because this is what a rollover of a key of a typical SAML IdP or SP looks like.

Of you can be reasonably certain that all you SP will update your metadata, and you can wait for them to do so, this method will be fine. It should also be readily understood by SPs, because this is the common approach.

Note that "encryption.keys.KEYID.publicFile" and "encryption.keys.KEYID.privateFile" options are not used a all.


2. Using multiple keys at the same time - using the "KEYID" option

The "KEYID" in "encryption.keys.KEYID.publicFile" and "encryption.keys.KEYID.privateFile" are just placeholders. You are to replace these with the name of your new key. E.g. "encryption.keys.20170217.publicFile" and "encryption.keys.20170217.privateFile" for a key generated at this date, but you can use any name. This explains why you could not find this option mentioned in the EB code.

Using this method EB can sign using multiple keys at the same time. When you add a key like this an additional endpoints are added for the metadata of the IdP side of EB:

The following metadata that uses the default key:
authentication/idp/certificate
authentication/idp/certificate/key:default
authentication/idp/metadata
authentication/idp/metadata/key:default
authentication/proxy/idps-metadata
authentication/proxy/idps-metadata/key:default

But you will now also have metadata that uses the new "20170217" key you defined:
authentication/idp/certificate/key:20170217
authentication/idp/metadata/key:20170217
authentication/proxy/idps-metadata/key:20170217

When you look into the metadata you will notice that the SSO location hase the "key:KEYID" part added. I.e. "authentication/idp/single-sign-on/key:20170217". This part selects the key that EB will use to sign the assertion back to the SP and allows multiple keys to be active at the same time.

A keyrollover works like this:
- You generate a new key and SAML signing certificate
- You add the second key to EB config
- (optional) You add the new certificate as a second certificate to the metadata of EB, you do this by adding the new certificate in SR to the metadata of the IdP with the EntityID (connection ID) of EB. You can use the medata field "certData2" file for this. You should see the 2nd certificate appearing in the metadata's published by eb (i.e. .../idp/metadata ). So, yes, you can combine this method with the "classic rollover" method above
- You notify your SPs of the changed metadata location by pointing them to the key:20170217 versions of the metadata
- You can monitor uptake of the new key
- At some point you decide to update the configuration of "encryption.keys.default" so EB will now always sign using your new key
- You update the EB IdP Metadata in serviceregistry so that it contains your new certificate only, in certData.
- You are done

So with this method, after adding a new key, as soon as an SP has switched to the new metadata, they use the new key, and do no longer trust the old key. Note that the SP MUST update the metadata location to get this benefit. SPs that are slow to update will continue to work using the old key. By monitoring the SSO location that SPs use to login you can see wether an SP has switch or not, so you can focus your communication efforts on those SPs.

Maybe we should call this method of key rollover "Dutch Keyrollover" because it uses the same trick as "Dutch Scoping" - it makes the SP to send some extra info in the SSO location URL to EB.

Op donderdag 16 februari 2017 23:55:11 UTC+1 schreef andrew...@cybera.ca:

Pieter.

Andrew Klaus

unread,
Feb 17, 2017, 12:39:40 PM2/17/17
to OpenConext Community
Awesome!  Thanks so much for that detailed answer Pieter. It also helps explain the "key:default" in the different EB URLs, which I didn't know about before either.

Thanks Pieter

Andrew

--
OpenConext - Open For Collaboration
---
You received this message because you are subscribed to a topic in the Google Groups "OpenConext Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openconext/1oexBnx5KHA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openconext+...@googlegroups.com.
To post to this group, send email to openc...@googlegroups.com.
Visit this group at https://groups.google.com/group/openconext.
To view this discussion on the web visit https://groups.google.com/d/msgid/openconext/f880807d-0dec-4f25-85d2-367cd7f25da9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tim van Dijen

unread,
Aug 29, 2018, 2:54:24 AM8/29/18
to OpenConext Community
Hi Pieter,


Op vrijdag 17 februari 2017 16:53:08 UTC+1 schreef Pieter van der Meulen:

- You add the new certificate as a second certificate to the metadata of EB, you do this by adding the new certificate in SR to the metadata of the IdP with the EntityID (connection ID) of EB. You can use the medata field "certData2" file for this. You should see the 2nd certificate appearing in the metadata's published by eb (i.e. .../idp/metadata )


This mechanism appears to be not working anymore on the latest EB-release... certData2 is never added to the published metadata..
Was this functionality removed over the last year or did I just walk into a bug?

- Tim

Thijs Kinkhorst

unread,
Aug 30, 2018, 4:03:32 AM8/30/18
to openc...@googlegroups.com
Hi Tim,

Op 29-08-18 om 08:54 schreef Tim van Dijen:
I'm not so sure that what Pieter says is the correct way to do it.

I think you need to configure this in engineblock.ini instead, like so:


; The Signing / Encryption keys used for the SAML2 authentication and
metadata
; When EngineBlock signs responses (when it acts as an Idp)
; or requests (when it acts as an SP) it uses these X.509 certs.
encryption.keys.default.publicFile = "/etc/openconext/engineblock.crt"
encryption.keys.default.privateFile = "/etc/openconext/engineblock.key"

; Additional keys for easy key rollover
; encryption.keys.KEYID.publicFile = "/etc/openconext/engineblock.KEYID.crt"
; encryption.keys.KEYID.privateFile =
"/etc/openconext/engineblock.KEYID.key"


You add a new key with a new keyid (eg 20180830).

This is all from the top of my head, maybe Pieter can correct me if I'm
wrong. In any case this should probably be better documented. Since we
plan to do a key rollover early next year, that might be a good time, if
no-one beats us to it.


Cheers,
Thijs


Cheers,
Thijs

Tim van Dijen

unread,
Aug 30, 2018, 4:16:22 AM8/30/18
to OpenConext Community
Hi Thijs,

That's the second mechanism Pieter described earlier in this thread and I have this in place already..
It creates additional endpoints that allow for seamless rollover for SP's..

Unless I'm missing something, this can't be used for IDP's.. They need the old+new certificate for signature verification.
The way Pieter described this, adding the new certificate to certData2 should add it to the published metadata, additional to the old one, but that's not happening..
Hopefully Pieter can help out here and I can beat you when it comes to documentation.. My roll-over has to happen before early november.

- Tim

Op donderdag 30 augustus 2018 10:03:32 UTC+2 schreef Thijs Kinkhorst:

Pieter van der Meulen

unread,
Sep 10, 2018, 3:48:04 AM9/10/18
to openc...@googlegroups.com
Hi All,

Just to be sure, we’re talking about rolling the EB signing certificate, i.e the certificate that is used for signing the SAML Assertions generated by EB and metadata published by EB.

I just checked setting cert2 on the EB IdP entry in manage with EB 5.8.2, and can confirm that the certificate does not show up in the IdP metadata of EB (/authentication/idp/metadata). So that’s a feature that went missing. The other information from the EB IdP entry in manage is sill being used in the metadata.

To recap there are TWO key rollover methods in EB:
1. The classic one, where you publish the new certificate along the old one in the metadata for a while and then start signing using it. No changes in metadata urls or sso locations.
2. The alternate one, where EB can use two signing keys concurrently. It publishes two sets of metadata with different urls and uses the SSO location to determine what key it should sign the assertion with.

The two methods can be combined by publishing the new certificate in the old metadata only.
The downside of the 2nd method is that SPs have to change the metadata URL and the signing certificate they trust manually. The advantages are that as soon as an SP is using the new metadata, it will be trusting the new key only. Also the migration can be monitored by looking at the SSO locations used by SPs during authentication.

Thijs describes the 2nd method. This is indeed configured in engineblock.ini. To use the first method, you need a way to add a second (i.e. the new) certificate to the metadata. This is what the cert2 key in the IdP metadata is used for. As Tim reported, this is not working. This should be fixed.

We have a key rollover planned for SURFconext next year, so we are looking at the rollover procedure with renewed interest.

Regarding the use of the EB IdP and SP entries in manage for configuring some of the information that goes into the EB metadata the consensus here is that we should move this to engineblock.ini, and stop using manage for this. But this should be a reason for fixing this cert2 bug.

Pieter.
> --
> OpenConext - Open For Collaboration
> ---
> You received this message because you are subscribed to the Google Groups "OpenConext Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to openconext+...@googlegroups.com.
> To post to this group, send email to openc...@googlegroups.com.
> Visit this group at https://groups.google.com/group/openconext.
> To view this discussion on the web visit https://groups.google.com/d/msgid/openconext/e3bdf4ee-3c5d-48f2-b9fb-a8aa247adaff%40googlegroups.com.

Tim van Dijen

unread,
Sep 10, 2018, 6:51:08 AM9/10/18
to openc...@googlegroups.com
Thanks Pieter, for confirming this bug...
I can use method 2 for our SP's and I'm not worried about changing endpoints.. They are all SP-proxies and the admins know what they're doing.
My issue here is with the IdP's that need to verify AuthnRequest-signatures.. Method 2 does not work for us in this situation as it would require a big-bang migration with too many parties involved.
That's why method 1 would be perfect in this case..

I've been trying to go through the code and find the part where certificates are added so I can perhaps do a quickfix and continue the rollover, but I'm strugling to locate it..
Any pointers?

- Tim


> To unsubscribe from this group and stop receiving emails from it, send an email to openconext+unsubscribe@googlegroups.com.
> To post to this group, send email to openc...@googlegroups.com.
> Visit this group at https://groups.google.com/group/openconext.
> To view this discussion on the web visit https://groups.google.com/d/msgid/openconext/e3bdf4ee-3c5d-48f2-b9fb-a8aa247adaff%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
OpenConext - Open For Collaboration
---
You received this message because you are subscribed to a topic in the Google Groups "OpenConext Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openconext/1oexBnx5KHA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openconext+unsubscribe@googlegroups.com.

To post to this group, send email to openc...@googlegroups.com.
Visit this group at https://groups.google.com/group/openconext.

Pieter van der Meulen

unread,
Sep 10, 2018, 7:28:35 AM9/10/18
to openc...@googlegroups.com
Hi Tim,

> On 10 Sep 2018, at 12:51, Tim van Dijen <tvd...@gmail.com> wrote:
>
> Thanks Pieter, for confirming this bug...
> I can use method 2 for our SP's and I'm not worried about changing endpoints.. They are all SP-proxies and the admins know what they're doing.
> My issue here is with the IdP's that need to verify AuthnRequest-signatures.. Method 2 does not work for us in this situation as it would require a big-bang migration with too many parties involved.
> That's why method 1 would be perfect in this case..

Good point. That is something that we did not have to deal with at the last rollover because we did not sign AuthnRequest at that time.

In this case you want to add a certificate to the EB SP metadata (and not the IdP metadata). I don't know whether we implemented that at the time. So, technically, this could be a feature request. But also for adding the certData2 to the SP metadata the plumbing should already mostly be there because other information from the EB SP entity appears in the EB SP metadata.

> I've been trying to go through the code and find the part where certificates are added so I can perhaps do a quickfix and continue the rollover, but I'm strugling to locate it..
> Any pointers?

That should be here, for the IdP metadata: https://github.com/OpenConext/OpenConext-engineblock/blob/master/library/EngineBlock/Corto/Mapper/Metadata/Entity/SsoDescriptor/Certificates.php#L52

> - Tim

Pieter.

Tim van Dijen

unread,
Sep 10, 2018, 11:02:39 AM9/10/18
to OpenConext Community
Thanks Pieter,

I found out I only had to comment this line to make work again. This will help me through my rollover for now!

By the way, you said earlier:
"Regarding the use of the EB IdP and SP entries in manage for configuring some of the information that goes into the EB metadata the consensus here is that we should move this to engineblock.ini, and stop using manage for this."
Why would you move metadata-stuff outside of the metadata repository? Makes absolutely no sense to me...

- Tim

Michiel Kodde

unread,
Sep 10, 2018, 11:21:28 AM9/10/18
to openc...@googlegroups.com

--
OpenConext - Open For Collaboration
---
You received this message because you are subscribed to the Google Groups "OpenConext Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openconext+...@googlegroups.com.

To post to this group, send email to openc...@googlegroups.com.
Visit this group at https://groups.google.com/group/openconext.

For more options, visit https://groups.google.com/d/optout.
--
ibuildings
WEB & MOBILE APP DEVELOPMENT

Software Engineer

e-mail: mko...@ibuildings.nl 
phone: +31 88 00 24 000
www.ibuildings.nl

To all agreements with Ibuildings.nl BV, our general terms and conditions apply.

Michiel Kodde

unread,
Sep 11, 2018, 2:51:21 AM9/11/18
to openc...@googlegroups.com
Hello again Tim,

Yesterday, I was pressed for time and responded quickly. In the meantime, I've thought about a possible solution to the problem. Thijs & Pieter please feel free to chime in or correct me if I'm proposing something out of the obvious!

Currently we only use the configured (ini settings) signing certificate for the EngineBlock IdP instance. The EBMetaddataVisitor ensures this cert is used (see reference in the previous reply).

We could venture into a new situation where the visitor can be configured to either use the certificates provided by Manage (service registry) or fall back on the configured certificates.

We could make this an 'implicit' feature toggle, where we would automatically use the EB IdP cert data from Manage when it is provided.

This would entail some extending of the Visitor, which shouldn't be that difficult as it can be configured using DI.

Please be aware that the same visitor is used to decorate the EB SP and IdP role! So adding some type checks might be a requirement here.

Whether or not the actual key rollover logic is functioning at this moment remains questionable to me at this point. I would have to experiment further to be sure of that.

Creating a Behat feature for this logic seems a no-brainer to me.

Cheers,
Michiel

Pieter van der Meulen

unread,
Sep 11, 2018, 3:41:16 AM9/11/18
to openc...@googlegroups.com
Hi Tim,

> On 10 Sep 2018, at 17:02, Tim van Dijen <tvd...@gmail.com> wrote:

<Snip>

> By the way, you said earlier:
> "Regarding the use of the EB IdP and SP entries in manage for configuring some of the information that goes into the EB metadata the consensus here is that we should move this to engineblock.ini, and stop using manage for this."
> Why would you move metadata-stuff outside of the metadata repository? Makes absolutely no sense to me…

Because what you are actually doing in the EB SP and IdP entities is configuring aspects of the metadata that engineblock publishes, you are not telling it about remote entities. EB already knows what entities it hosts and does not learn that from the metadata in manage. The EB SPs / IdP entities are unlike any of the other entities in the metadata in that respect. Things like consent, ACL, ARP, manipulations and many of the SAML “metadata” options like request.sign have no effect for the EB entities. I’ve seen people confused by this many times.

The idea was to make things simpler and clearer by moving this configuration out of manage and into engineblock.ini. So I want it to make more sense, not less :)

Pieter.

Pieter van der Meulen

unread,
Sep 12, 2018, 4:35:28 AM9/12/18
to openc...@googlegroups.com
Hi,

I’d like add some experience and thoughts from when we implemented this the last time:

- You don’t want to override the certificate configured in engineblock.ini because this is the certificate that corresponds to the key that is actually used by EB for signing. EB can have multiple signing keys, and publish different metadata’s for them, with different certificates, but they will be for the same entity. So they will all use the same certData(2) configuration from manage.
- You don’t want to add the old certificates to the metadata’s that sign with the new key.
- A signing certificate should appear only once per entity in the metadata

What we did:
- certData has no effect (the certificate from engineblock.ini is used).
- certData2 has the additional certificate (i.e. the new certificate). This is only added if this certificate is not already in the metadata to prevent duplicate certificates in the old metadata.

So the configuration during rollover would be:

engineblock.ini:
Key 1 (default): current key + current certificate
Key 2: new key + new certificate

In manage for IdP and SP entities set certData2 to new certificate

Pieter.
> To view this discussion on the web visit https://groups.google.com/d/msgid/openconext/CAMR1wNMZTy%2BCvcERbOYZBYVzUTbRZmVh%3DFH-vQPax%2B5YE1prXA%40mail.gmail.com.

Thijs Kinkhorst

unread,
Nov 30, 2018, 10:38:14 AM11/30/18
to openc...@googlegroups.com
Op 12-09-18 om 10:35 schreef Pieter van der Meulen:
> I’d like add some experience and thoughts from when we implemented this the last time:
>
> - You don’t want to override the certificate configured in engineblock.ini because this is the certificate that corresponds to the key that is actually used by EB for signing. EB can have multiple signing keys, and publish different metadata’s for them, with different certificates, but they will be for the same entity. So they will all use the same certData(2) configuration from manage.
> - You don’t want to add the old certificates to the metadata’s that sign with the new key.
> - A signing certificate should appear only once per entity in the metadata
>
> What we did:
> - certData has no effect (the certificate from engineblock.ini is used).
> - certData2 has the additional certificate (i.e. the new certificate). This is only added if this certificate is not already in the metadata to prevent duplicate certificates in the old metadata.
>
> So the configuration during rollover would be:
>
> engineblock.ini:
> Key 1 (default): current key + current certificate
> Key 2: new key + new certificate
>
> In manage for IdP and SP entities set certData2 to new certificate

Might be useful for people willing to use this to know that in recent
versions of EB this approach seems to be buggy (we've used it most
recently over 4 years ago). I'll be working on getting it back into
(some) shape, but be warned that you might run into issues right now
when using it.


Cheers,
Thijs

Tim van Dijen

unread,
Nov 30, 2018, 10:42:27 AM11/30/18
to openc...@googlegroups.com
I'd like to add that the workaround I've been posting earlier in this
thread got me through a key-rollover flawlessly!

Kind regards,

Tim

Op 30-11-2018 om 16:38 schreef Thijs Kinkhorst:
Reply all
Reply to author
Forward
0 new messages