Hello Leopold,
Restarting the middleware will not solve your issue, it is just a component in between your browser and smart card (reader) that facilitate the communication, but it does not cache anything from previous connections.
What might be worth looking at is how you log out the previous user from the https connection. (make sure that the TLS session gets terminated, if the browser tries to re-use/continue the previous client session, you run into the situation you describe)
I assume that the initial user is capable of logging in again? (probably continuing/refreshing the tls session).
Wkr,
Frederik