[erlang-questions] write appending a file on a replicated file store

[Joe Armstrong]

Is write appending a file on a replicated file store
like dropbox or gdrive an inefficient operation?

I want to make a write-append only changelog on a replicated
store (like dropbox or gdrive)

Does anybody know if these stores handle write appends in an efficient manner.
The worse case would be that each new write append treats the new version of
the file as a totally new file and copies everything.

I have no idea if this is the case or not - does anybody know
if the entire file gets copied every time or just the increment?

If the entire file gets copied then I'd need some kind of incremental
system, create many small files, then occasionally merge them
has anybody done this?

Cheers

/Joe
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Jesper Louis Andersen]

On Sep 4, 2012, at 8:36 AM, Joe Armstrong <erl...@gmail.com> wrote:

> Does anybody know if these stores handle write appends in an efficient manner.
> The worse case would be that each new write append treats the new version of
> the file as a totally new file and copies everything.
>

There are two methods which come to mind: Rsync-like functionality and merkle-trees.

>
> If the entire file gets copied then I'd need some kind of incremental
> system, create many small files, then occasionally merge them
> has anybody done this?

Merkle trees is the tool you want I think. It will quickly find out that there are new data appended and only transfer the data there. Coincidentally, there is some merkle tree stuff inside riak :)