CAT_INVALID_AUTHENTICATION for service and native users after restart

[J.P. Mc Farland]

Hi Everybody!

After updating rules (security and other) on one of our test zone servers (4.3.4) and restarting, the service user and all "native" users cannot authenticate (CAT_INVALID_AUTHENTICATION), but "pam_password" users can.  The configuration did not change and the rules were effectively "read-only" in nature (only checks with *_pre PEPs).  Reverting the rules and restarting has not helped.  Running iinit on native users (including the service user) has not helped.  Restarting the server process and entire server has not helped.  Using an appropriately reconfigured server recently cloned from the first has not helped (ICAT issue?).  There is nothing of note in the logs except the CAT_INVALID_AUTHENTICATION errors.  Neither iinit nor iexit was run on any of the native users prior to this occurrence.  Relevant documentation (https://docs.irods.org/4.3.4/system_overview/troubleshooting/#service-account-ran-iexit-or-otherwise-is-not-authenticated) indicates simply rerunning iinit should do the trick, but it does not.  We even tried running iexit prior to iinit and still no change.

Has anybody else seen such behavior?  Are there any other areas where we should check or perform tests?

Cheers,

--John