publish acknowledgement

[Andrew Longley]

Hi. I'm modifying the faye-redis engine for business purposes and I'm finding the need to tell a client that publishes a message whether the message was delivered to any subscribers. One could extrapolate this to, on publish, give the client information relevant to the outcome of the publish.

My understanding of the current faye-redis engine is that it basically fires the engine.publish and turns around and immediately sends a publish acknowledgement to the client (with a success or error). There is no asynchronous waiting on the publish.

The engines/proxy.publish function appears to accept a return from engine.publish, however, if I tweak the engine code to return a promise and have protocol.server._handle add a .then and send it's response in the .then, the published message is apparently not marked as "resolved" and the message is republished every approximately 45 seconds.

If I instead tweak the protocol/server._handle method to pass a callback, tweak engines.proxy to pass that callback to engine.publish, and in engine.publish call the callback, things continue to work properly, and no duplicate messages are sent after the timeout. I can then insert whatever data I want by invoking the callback with whatever I want, such as a boolean about whether the message was delivered.

I hesitate to make tweaks to the faye library code and am curious if others have solved this issue. I am also considering, instead of tweaking proxy.js and server.js, sending an independent asynchronous "/meta/status" message to the client, and using, say, an incoming hook on the client to pull these types of meta messages off and do something with them. 

Thanks.

[Justin Karneges]

Hi Andrew,

Putting code aside, pub/sub acknowledgements are tricky business.

For example, if you want to know if other Bayeux clients received a published message, it's not enough to confirm that the message was received by Redis. You'd need acknowledgements at every hop (Redis, other Faye instances, clients connected to those other instances). And then you have flow control issues to deal with as well (does one slow receiver block acknowledgements?).

Since this kind of reverse communication adds complexity, many pub/sub architectures avoid it entirely, including Faye and Redis. Redis clients and Bayeux clients have no way to acknowledge received messages. Waiting on a publish call to Redis simply tells you the message was received by Redis (not a bad thing to know), but it doesn't imply any remote Faye instance received the message or anything after that.

If you aren't sending to too many subscribers, one solution could be to build an end-to-end acknowledgement system, where senders include a return address channel, and each receiver publishes an ack message back to it. Then all the complex logic goes into the sending client code, to decide how to track the acks.

If you are publishing to potentially lots of subscribers (e.g. >100) and it doesn't make sense to flood senders with a bunch of individual acks, you might try building a feedback system, where anytime a client receives a message on a channel, it reports this to a central place in a durable manner, which the senders can inspect.

Justin

--
You received this message because you are subscribed to the Google Groups "Faye users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to faye-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Longley]

Hi Justin,

Thanks for your insight. I think I'm going to go down the path of using the overly simplistic redis client result (when messages are published in the Faye-Redis Engine it does a redis lookup and returns an array of subscribed clients and then attempts to publish to them, understanding that any number of factors could cause the number of subscribed clients and the number of clients which actually get the message to be different), and publish back to the publishing client on a private channel a pubStatus message, and pull that off using an incoming hook.

I understand that it's not bullet proof. We'll have to do some testing to understand under what circumstances it is insufficient enough to warrant further development time.

Thanks again.

Andrew

--
You received this message because you are subscribed to a topic in the Google Groups "Faye users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/faye-users/VHVTn4aQ2GE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to faye-users+unsubscribe@googlegroups.com.

[Justin Karneges]

The pubStatus seems reasonable to me, if you want a rough idea of what is going on with clients, like a diagnostic metric. It can also be used to confirm the data successfully passed through Redis and was received by the other Faye servers.

Justin