Send files via nats. Does it make sense?

[Thomas]

Hello everyone.

I'd like to discuss sending files via nats concerning the following points:

1. How could it be achieved?
2. Is it possible to be notified with progress?
3. What could be implications of sending files?

Please forgive me if there's an obvious answer to this.

I'm currently looking for a messaging system for files and have not used nats before.

[derek]

While this is possible, I would advise using NATS to discover the proper endpoints and transferring via HTTP(S) or another method directly.

Note that if you do move forward, NATS enforces a maximum payload as configured by the server. You will need to take this into account and

break up large files.

[Tyler Treat]

To add to what Derek said, the pattern I would suggest is using NATS to pass around file "pointers", e.g. an S3 bucket URL, and transferring the actual files out-of-band. NATS is better suited to smaller messages, so I would advise against sending large BLOBs through it since you'd potentially be sequencing lots of 1MB messages (the default max payload size) and reassembling them.

[Larry McQueary]

+1 

This (reference passing) is the recommended pattern in general. Brokered messaging systems and NATS in particular are not designed for large file transmission and are usually neither suitable nor efficient for that purpose.

--
You received this message because you are subscribed to the Google Groups "nats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to natsio+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Larry McQueary | Director, Messaging Technology

la...@apcera.com | @mcqueary | github.com/mcqueary

[Thomas]

Thanks for your replies.

What I want to avoid is completely uploading the file first and then downloading it again as it would be the case with file pointers.

I wanted to stream it in chunks, thats why I thought about using nats, but based on your replies I will look for another solution.

[Larry McQueary]

Thomas, let me explain the "why", as this question comes up a lot. It's certainly possible to do what you're asking, but as you've already heard, nobody here will tell you that "yes, this is a great solution for file distribution". If you did create such a strategy around NATS, you'd need to manage every aspect of the process including the "chunking" on the sender end, ordered reassembly on the receiver end, checksum verification, progress monitoring, etc. You'd have to do all that work yourself. 

One of the key reasons that no one (including myself) will recommend this as a solution for your problem is scaling and performance efficiency. The performance of a system like NATS is largely bounded by I/O constraints. It's very good at distributing lots of small messages efficiently between lots of clients. In the type of solution you're envisioning, the NATS server would be spending a lot of time reading/writing large message chunks across socket connections and therefore its ability to do other things (service other clients efficiently) would be limited. If NATS was going to be dedicated to this use case, and you were able to bound the number of files being sent simultaneously through a given NATS server, and the number of simultaneous receivers was small, this might be manageable. But once you start putting these constraints on the solution, you have to wonder if there's not a better way...

[ged...@gmail.com]

I still agree that Pointers is best. Minio also has Notifications into NATS, so that if a file is changed, NATS is told, and so then you can tell everything else using S3, that the file changed.

Nats now has Objects which allows sending and receiving files with NATS. Its still experimental.  

I have no idea how efficient it is.

I also have no idea of the Use Cases that Derek has in mind for it yet.  Wish i did, cause that would shorten the investigation aspects.

[derek]

It's pretty efficient, end to end flow control over any NATS topology. We may make some updates to the API as we learn more, and we have had some folks ask about JetStream and Min.io as well. 

The Object store today allows multiple entries, so think of it like a bucket. Those entries can be links to other bucket/entries as well. And since its JetStream you can get notifications, do watchers etc.

I would suggest playing with the nats cli to get a sense of things. nats obj -h