I'd like to configure shared state, cluster wide, for an instance of an object.
One master process will create the state path (using a shared atomic long), then that ID will be distributed to other components in the cluster which will read/write to the appropriate path with that ID.
My challenge is that there might be multiple instances of this object; and sometimes that master process might fail and not clean up after itself.
Is there a good way to manage housekeeping in this case? E.g. ensuring that the old directories that were orphaned get cleaned up?
Or should I just roll my own mechanism, such as storing a last access datetime, and doing cleanup after a period of non-activity is identified?