I've been looking at the Erlang ecosystem, and coming from a Java world, I am wondering what is the "Erlang way" to implement the following.
The use case is very simple: I want to receive requests from users, and later process them asynchronously. However I want to be sure that if a request is received, it will be safe from single-node crashes, in other words, I want requests to be replicated across my cluster.
In Java I would implement it using a replicated message queue. However RabbitMQ doesn't have replicated queues, only the meta-data is clustered (as far as I know). Mnesia has replication, but then it is a database, not a queue (although you could probably implement a mq ontop od that, but I didn't see anybody doing that).
If I understood correctly your question, there are two points there:
1. how you can replicate a message;
2. how you can be sure that the message is received.
If that is the case, here are few simple answers (hopefully, more skilled
Erlang users will come with better ideas/solutions):
1. Take a look at gproc or any equivalent implementation.
2. That depends on how the message is sent, but the general solution is to
have a reply from the receiver. Here are two simple solutions for doing
that:
a) Message sent via threads messaging (Pid ! MyMessage):
-> sender: ReceiverPid ! {self(), MyMessage}
-> receiver listener (basics only):
receive
Message -> {SenderPid, SenderMessage} = Message, SenderPid ! received
...
end
or you can add SenderPid in the group of receivers.
b) Message sent via TCP connection (for the receiver only; the sender
should have also a listener to be able to receive the answer):
listen_socket(Socket) ->
case gen_tcp:recv(Socket, 0) of
{ok,Data} ->
gen_tcp:send(Socket,<<"received">>),
listen_socket(Socket);
...
end.
These are only few basic options. Of course, there are many other options
(e.g., monitoring your receiver). I resume my answer to a simple one. Take
care of the length of the queue (if you need a personalized queue, I
suggest gen_fsm).
I hope this answer helps and if I misunderstood your problem, I kindly ask
you to ignore my answer.
On Fri, Jul 27, 2012 at 12:47 PM, Adam Warski <a...@warski.org> wrote:
> Hello,
> I've been looking at the Erlang ecosystem, and coming from a Java world, I
> am wondering what is the "Erlang way" to implement the following.
> The use case is very simple: I want to receive requests from users, and
> later process them asynchronously. However I want to be sure that if a
> request is received, it will be safe from single-node crashes, in other
> words, I want requests to be replicated across my cluster.
> In Java I would implement it using a replicated message queue. However
> RabbitMQ doesn't have replicated queues, only the meta-data is clustered
> (as far as I know). Mnesia has replication, but then it is a database, not
> a queue (although you could probably implement a mq ontop od that, but I
> didn't see anybody doing that).
On 27 lip 2012, at 13:24, CGS <cgsmcml...@gmail.com> wrote:
> Hi Adam,
> If I understood correctly your question, there are two points there:
> 1. how you can replicate a message;
> 2. how you can be sure that the message is received.
> If that is the case, here are few simple answers (hopefully, more skilled Erlang users will come with better ideas/solutions):
> 1. Take a look at gproc or any equivalent implementation.
Thanks, looks quite big, but I'll try to dig through it. Though it's not only replicating the message, it's also consuming the message across the cluster once it is consumed on the master node, etc.
Adam, please take some time to walk through the rabbitmq documentation - there's quite a lot of it! Rabbit supports replication using HA (master/slave mirror) queues, or federation for replicating over the WAN/Internet. This does exactly what you've asked for.
If you'd like to understand how it works, take a lot at gm.erl (guaranteed multicast) which is a fully asynchronous ring protocol developed independently at rabbit to support HA.
Cheers.
Tim
On 27 Jul 2012, at 19:27, Adam Warski <a...@warski.org> wrote:
> On 27 lip 2012, at 13:24, CGS <cgsmcml...@gmail.com> wrote:
>> Hi Adam,
>> If I understood correctly your question, there are two points there:
>> 1. how you can replicate a message;
>> 2. how you can be sure that the message is received.
>> If that is the case, here are few simple answers (hopefully, more skilled Erlang users will come with better ideas/solutions):
>> 1. Take a look at gproc or any equivalent implementation.
> Thanks, looks quite big, but I'll try to dig through it. Though it's not only replicating the message, it's also consuming the message across the cluster once it is consumed on the master node, etc.
> Adam, please take some time to walk through the rabbitmq documentation - there's quite a lot of it! Rabbit supports replication using HA (master/slave mirror) queues, or federation for replicating over the WAN/Internet. This does exactly what you've asked for.
I did some time ago, but either the docs got improved, or I just skipped a whole paragraph after reading "An exception to this are message queues, which by default reside on the node that created them, though they are visible and reachable from all nodes.". Anyway, all is clear now :).
> If you'd like to understand how it works, take a lot at gm.erl (guaranteed multicast) which is a fully asynchronous ring protocol developed independently at rabbit to support HA.
