Distributed public file system technology over WAN with HTTP gateway

[zsigmond....@gmail.com]

Hello,

I am after a home friendly solution where the cluster would be across the internet, where a group of min. 5 people would provide their 5 nodes to the cluster, their laptop/desktop with 15-30 Gb to shared storage with internet access. XtreemFS matches these as I read up the site. The problem I see I might get into with XtreemFS is that most of these nodes would have a Windows OS, and there are only clients available for that OS. Another thing that's unclear for me as that clients are there only for reading while servers are having the replicated storage? What I would like to have is clients putting out their storages and having a single server that can grab/cache the files from these clients without taking up too much space on that server (max. 5-10 GB). 

An example scenario to better understand what I'm after:

There is an amateur music band that wants to share all their weekly 2-3 recordings of 10M. They would share these recordings on the web, so there has to be http access and also (s)ftp access so on their web interface they could upload the new recordings. Let's say I'm the geek from this group of 5 people who wants to try out this distributed fs approach instead of paying a cheap storage somewhere.

Maybe you know other technologies that address this scenario better? 

Or you know how to do this with some more general purpose tools? Then answer with describing the important steps/configurations to get there. If available/applicable also link to tutorials to make your approach better treated. My general approach would be to build spring boot applications for the nodes to access / verify the files replication and integrity and transmit to the other nodes and if one of these node is set to be master, then it serves as a gateway to the other nodes, serving the file form the closest/fastest node that is available to the requester. But I'm not that overbold to just start build such an application myself :-).

Thanks for your consideration,

Laci

[Robert Schmidtke]

Hi Laci,

you're right, we are not explicitly targeting Windows machines as servers. There is probably a way to configure the build environment using make, cmake, maven etc. on a Windows machine too, such that it builds the necessary jar files for running the servers.

However, since we're not testing on Windows, I cannot promise that the servers will work as intended. You may want to consider using VMs or Docker, however this may put additional load on the machines. Also note that these machines will always have to be online for files to be accessible through FTP.

The way XtreemFS works is that you have a central DIR and MRC service, responsible for discovery and metadata, and a number of OSD services that actually hold the data. So you would need to have an OSD on each machine in order to make their disk space available to the network. Everyone can then mount the volume and use it as shared device.

Alternatively you could have the DIR, MRC and OSD on one (Linux-) machine, and only use that machine to store files. Each client can still mount the volume and use the remote space. However in this case you're probably better of with Windows network shares (I'm not experienced with these though).

I'm not aware of alternatives for Windows, you may want to try HDFS thought. As for general purpose methods, I'm not going to elaborate on these, as I would also have to do research beyond the scope of XtreemFS, which is not the purpose of this mailing list.

Cheers

Robert

[László Zsigmond]

Thanks Robert!

--

---
You received this message because you are subscribed to a topic in the Google Groups "XtreemFS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xtreemfs/e89z6dxSzRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xtreemfs+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.