Transferring huge files (>1GB) over distributed environment

[Rohit Pathak]

Hi guys, I have been preparing a prototype for my organization in Scala/Lift. The project involves:

 

- One web server (main server)

- Other worker servers (50+)

 

The use-case scenario involves allowing user to upload a file of huge size, through web application (which I have made in Lift/Scala). These large files (each file > 1GB) are to be distributed over the worker nodes where processing will be performed.

 

What I have originally though is to have communication between Master node (web server) and worker nodes using Akka. But currently I am not sure how will I transfer the large files over the network. Does anyone have a good solution to this? A scala/lift related method/framework is preffered, and more preferrable if it can be done using Akka. So summing up there are 2 main issues:

 

- Best way to allow user to upload file using web-application (I am thinking of uploading using Ajax and showing progress to user)

- A way to distribute and transfer these files from web-server to worker nodes.

 

Thanks in advance :)

[Tobias Pfeiffer]

Hi,

Am 22.05.2013 16:01, schrieb Rohit Pathak:
> - A way to distribute and transfer these files from web-server to
> worker nodes.

I would do this via some traditional network file system (NFS?) or maybe
MongoDB's GridFS, mounted on all the processing machines, and then only
send small messages between the hosts involved, not the whole file.

Tobias

[Rohit Pathak]

I would do this via some traditional network file system (NFS?) or maybe
MongoDB's GridFS, mounted on all the processing machines, and then only
send small messages between the hosts involved, not the whole file.

Tobias

But I would want to add and remove nodes on demand at runtime, so I would prefer my own services to communicate for file transfer and not rely on NFS. But I don't quiet know about MongoDB. Will it be able to provide me such dynamism.

[Austen Holmes]

Hadoop has HDFS as well.  In amazon, I would probably fire up worker nodes on-demand using spot instances, let them process, and then turn them off.  That would be pretty cheap.

[Tobias Pfeiffer]

Hi,

Am 22.05.2013 18:33, schrieb Rohit Pathak:
>> I would do this via some traditional network file system (NFS?) or
>> maybe MongoDB's GridFS, mounted on all the processing machines,
>> and then only send small messages between the hosts involved, not
>> the whole file.
>

> But I would want to add and remove nodes on demand at runtime, so I
> would prefer my own services to communicate for file transfer and not
> rely on NFS. But I don't quiet know about MongoDB. Will it be able to
> provide me such dynamism.

I guess I don't understand the limitation of NFS in that case. If you
have one NFS server plus your web server accepting the files and storing
them in the NFS mount. Then your actual processing machines could also
just boot, mount the NFS share, do the processing, shut down. Where is
the problem with adding and removing nodes then?

Tobias

[David Pollak]

Personally, I'd use scp to move 1GB files around. Yeah, it's shell-land rather than JVM land, but it's going to get the job done.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Telegram, Simply Beautiful CMS https://telegr.am

Lift, the simply functional web framework http://liftweb.net

Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im

[Rohit Pathak]

Hmm, but the case can be that worker is a WINDOWS machine, and we don't have scp already installed, or maybe SCP port is blocked or so.

Maybe I better to use MongoDB as Tobias suggested. Does lift provide an abstraction over MongoDB to store and manage files?

[David Pollak]

I don't know if Mongo is going to work. It's not optimized for very large file blobs. I could be wrong.

If you're going to use Mongo, I'd go with the Mongo Scala drivers. You want fine control over what's going on rather than an abstraction.

You may also want to see if RabbitMQ deals with 1GB+ messages. This task seems more "message queue-ish" than database-ish.

If the target machine is Windows, then you might want to use SAMBA where the web server publishes certain directories as SAMBA shares and then the windows box just mounts them.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code
 
---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Rohit Pathak]

Hmm, I did a POC on MongoDB and that seem to be somewhat fine. But the next thing to worry is the time consumed in upload and transfer.

 

1. User will upload the file to webserver.

2. Then it will be saved in MongoDB.

3. Then when a worker is assigned to process file. That worker will fetch the file from Mongo DB.

 

So effectively 3 transfers are involved. So can anyone please guide me whether we can stream the upload content directly into MongoDB.

 

What I mean is that when Lift is handling the upload request, I want to stream the content directly to MongoDB server.

 

So I wan't to do something like:

 

val fileSys = new GridFS(database)
val inputFile = fileSys.createFile(uploadInputStream)

 

Also for clear view on what I am doing: I am uploading the file using Lift wizard and a custom file field in it.

[Maarten Koopmans]

 You might also want to hook Finagle from Twitter into this. http://twitter.github.io/finagle/

Fine control, pretty "proven". Making a fast fileserver/client should be not too hard.

--Maarten

Op donderdag 23 mei 2013 schreef Rohit Pathak (insolen...@gmail.com) het volgende:

--

[Rohit Pathak]

But again, what about storing those large files? I need to stream the files directly to storage server.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

[Maarten Koopmans]

 Xfs, or openstack if you want to go (private) cloud?

Op maandag 27 mei 2013 schreef Rohit Pathak (insolen...@gmail.com) het volgende:

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.