That was the point I tried to make. The public/private key pair is used for signing or encrypting data. The X.509 certificate, a convenient way to distribute the public key, has
an expiration date that is used to limit the time the key strength is deemed sufficient. That means that whatever is encrypted with the private key can be considered as good/secure within the validity of the certificate. People who use a very long expiration
date for they self-signed certificate do so because they don't want to deal with the complication of certificates management. But that doesn't mean it's a desired security practise.
Now, if the private key is compromised, the owner of that key would issue a certificate revocation to tell holders of the corresponding public key to no longer trust encryption or signing done with that private key. In that case, even if the SAML response is
valid and its signing can be verified, I still cannot trust it because someone else might have stolen the private key and issues/signs the SAML messages in place of the IdP.
It's fine if pac4j doesn't do certificate validation, but that still doesn't mean it's a non-issue. It just means that those who care about checking the validity of the IdP certificate must do it through other means.