Auth Request Signature - Enable signature of the Redirect Binding Auth Request, If you enable it the encryption and signing key would available in the SP metadata file and URL (JENKINS_URL/securityRealm/metadata).
Encryption - If your provider requires encryption or signing, you can specify the keystore details here that should be used. If you do not specify a keystore, the plugin would create one with a key that is valid for a year, this key would be recreate when it expires, by default the key is not exposed in the SP metadata if you do not enable signing.
https://github.com/jenkinsci/saml-plugin/blob/master/src/main/webapp/help/encryption.html