Creating a backup server with node.js

[Tom]

Hi there,

This is a rather general topic and I assume not all is related
directly to node.js.

I'm wondering how to best implement a backup server with node.js. The
server would have to retrieve files from different clients and store
them in the correct places. I have a couple of questions:

1. What kind of transfer method should I use? These files may not get
corrupted. Should I use UDP with somekind of layer on top or TCP?

2. How should I store the incoming files? As normal files on the
filesystem or as binary data in a database? If in a database, what
kind of database is suggested for file storage?

3. Is Node.js a good platform for streaming huge files?

Thanks for your help.

- Tom

[AJ ONeal]

Any particular reason not to use rsync?

I've thought about this before and to me it seems that the simplest solution would probably be to use rsync and your backup backend and use node for storing configuration in a database.

1. I can't see any reason to not use TCP. You may be able to come up with some more efficient algorithm on top of UDP... but it's probably not worth it even if you could. It would just be more code that ends up under your maintenance.

(You might go a step further and use HTTP)

2. I vote you store the files as files. If you were trying to build some sort of enterprise solution like what Mozy has you could investigate more fancy mechanisms once you have the basics in place.

However, you may want a database to be able to search contents of files.

MediaTags is a library I'm developing for the purpose of extracting meta-data from common data types.

https://github.com/coolaj86/mtags

(I'd really appreciate funding to support more types if you're headed in that direction.)

3. Probably. I've seem claims of streaming 500mb/s using node-formidable over http.

Keep me updated on your progress. I'm very interested in this concept.

AJ ONeal

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

[Tom]

Hi AJ,

I am fairly sure that rsync is not what I need:

1) The service is made for consumers, every consumer will get an auth
ID which they set in their software
2) The software can then upload files within the restrictions that
match this auth ID (eg. storage limit, bandwidth limit etc.)
3) The software has to run on Windows, Mac and Linux

What I need is a node.js server that keeps track of all clients
(consumers) and accepts incoming connections if they are
authenticated, then stores the files and offers files for download on
request etc.

Regards,
Tom

On 29 dec, 23:33, AJ ONeal <coola...@gmail.com> wrote:
> Any particular reason not to use rsync?
>
> I've thought about this before and to me it seems that the simplest solution
> would probably be to use rsync and your backup backend and use node for
> storing configuration in a database.
>
> 1. I can't see any reason to not use TCP. You may be able to come up with
> some more efficient algorithm on top of UDP... but it's probably not worth
> it even if you could. It would just be more code that ends up under your
> maintenance.
> (You might go a step further and use HTTP)
>
> 2. I vote you store the files as files. If you were trying to build some
> sort of enterprise solution like what Mozy has you could investigate more
> fancy mechanisms once you have the basics in place.
>
> However, you may want a database to be able to search contents of files.
> MediaTags is a library I'm developing for the purpose of extracting

> meta-data from common data types.https://github.com/coolaj86/mtags

> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>
> > .

[Liam]

Have you seen boxbackup.org?

[Ram Viswanadha]

Why not use git based backup system like https://github.com/apenwarr/bup?

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.

To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

--
Best Regards,

Ram Viswanadha