How to manage internal state of the system

[mandeep gandhi]

Hi,

I am building a system where I am receiving a request from the client and putting it into messaging queue. I only send the response to the client once I have an acknowledgment from the messaging queue that the message was placed into. 

The way I am doing it is using Akka-HTTP by passing the request to an internal actor form a directive and then the internal actor ( say MyLordActor) puts it in a queue.  MyLordActor, being a publisher,  has to maintain certain internal state intermittently ( until acknowledgment comes from the queue). 

Similarly, there is a queue consumer actor ( QConsumerActor) which is a queue consumer which de-queues the message and does not acknowledge the message (that the message is consumed) to the queue until the processing of the message is done. Processing of the message takes a sequence of non-trivial steps and passing ( and processing) messages between a bunch of actors.   This means that would have message processing would need the message to be processed by other services which may be either REST based or queue based. Finally, after the message is processed by the actors, a message is sent to QConsumerActor, which acknowledges the queue that the message is consumed. Hence the consumer actor, QConsumerActor,  also maintains some state to track the message in the system. 

Addtional fact -  this is deployed as a container on top of Apache Mesos ( DC/OS).  If I want to scale system - I scale containers. 

Here comes the bunch of question I wanted to ask - 

a. If I scale the system by scaling containers (say I want multiple consumers), in my case at least, I have the problem that my message can end up with the wrong consumer ( not the one who started consuming the message)

Now if the actor does not have any information that it consumed the message, it even won't be able to acknowledge it to the queue. 

How do people deal with this internal state nuisance? Can I avoid Akka clustering and remoting and still deal with the problem?

b.  How do people deal with concurrency ( DB writes) which you have multiple containers of the same actor system? In other words, if I have a DB actor which updates the DB and if I have multiple such containers ( I am horizontally scaling the system), my DB actor has to be extra careful while updating message with same message id because the other DB actor also might be updating it.

Do their DB actor have to deal with comparing custom logic such as comparing last modified time of the message etc. or people generally follow Akka remoting for this case - maintaining a single actor which writes to the DB. 

Thanks,

Mandeep Gandhi