NSQ as a chunked binary transfer on ephemeral channels

[Howard Lince III]

Hello,

I have NSQ servicing several disparate pieces of code from different areas of concern in architecture where the NSQ queuing service sits at the middle of the entire platform. Generally these services are unaware of each other but by using NSQ plus a central coordinator may use data from each other as inputs.

As a general rule, these services don't really access a database or any service directly other than the one they're designed for.

This leaves me in a strange situation where I may want to access a blob available from another service. I have a few options for this based on what I've read and considered but would like to understand more on the general consensus from the community.

Option 1 is that I introduce a centralized service all agents can access and store/read using keys, ideally with the ability to put limits and auto expire data. This changes the behavior of the platform in an important way however and would require further considerations.

Option 2 is to build off of ephemeral topics and allow temporary topics to be setup on demand where NSQ acts as a broker for small chunks (512kb) of data to be read off and transmitted through the queue. Once the reader is done then, the topic closes and it's as if it never existed. This doesn't change the infrastructural requirements and more or less just adds a bit of protocol on top of the way the services interact with NSQ.

Option 2 is probably my preferred approach however as the files could be variable in size I could see that while most files will be transferred in a single chunk I would anticipate eventually a user will do something which will maybe ask the system to transfer 1GB, this would be at least 2048 messages through the Queue, but more importantly - 1GB of transfer over a short period of time.

My understanding is generally the community favors sending binary data via referential links, which is option 1 however I want to understand if there would be any major issues by going option 2, and more importantly if anybody has had success in going this route or similar historically.

Thanks in Advance.

[Matt Reiferson]

Why can't the service that wants the blob request it directly from the source (e.g. via RPC)?

--
You received this message because you are subscribed to the Google Groups "nsq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nsq-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nsq-users/6cf9ad5d-542c-45e9-a8ac-666ac848582cn%40googlegroups.com.