SAML with Okta is very slow

[Mark Schroering]

We have noticed it taking a very long time (up to 60s) to complete the SAML auth flow.  Here are some logs showing the bigger time gaps.  We are on version 1.1.7 of the SAML plugin and running Jenkins version 2.257. 


Sep 24, 2020 7:52:17 AM FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:35 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:45 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

Looking at the browser tools on page load:

GET /securityRealm/commenceLogin  <-- 57s
GET /securityRealm/finishLogin <--- 38s

the Okta SSO parts in between seem to be quick as expected. 

Any tips on how to further debug or troubleshoot would be appreciated. 

Thanks for the help. 

[Mark Schroering]

Here are the logs in a better format.  

[Ivan Fernandez Calvo]

Is your Jenkins home in a NFS or other network storage? I think so for the mount point, when a user enter a few files are written, because your IO is slow the IO operations are blocked waiting to finish that make the login slower than expected. You probably has more performance issues, I usually recommend to not use NFS file systems for the Jenkins home, take a look to this KB https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide

[Mark Schroering]

We are using AWS EFS for the Jenkins Home mount.   It was configured for burst throughput, and after reading https://aws.amazon.com/blogs/storage/best-practices-for-using-amazon-efs-for-container-storage/ we just changed it to provisioned throughput of 150 MiB/s.   The change did not help with the slow login times.  We are still digging through the logs, but are not sure what is causing the big time gaps. 

[kuisathaverat]

as I said your problem is the IO, if you enter en the Instance by ssh and check the iostats you will see more than 5-10% of your operations waiting for IO. NFS, EFS, and in general network filesystems works well with big files but with small files and write concurrence is where the problems start

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/WgAwcT0OGvk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1caf58e5-020d-44ab-a682-00a974054fbcn%40googlegroups.com.

--

Un Saludo

Iván Fernández Calvo

https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033

[Mark Schroering]

It ended up not being an IO issue.  We use https://www.jenkins.io/projects/jcasc/ and the official Jenkins docker image to deploy it to AWS ECS.  We have a startup script that does some cleanup in the mounted jenkins home directory to make sure that updated plugins are installed properly.  We noticed some saml*.xml files in the home directory.  We updated the startup script to remove these and now the auth flow with Okta works as expected.