Datacenter replication, resync

[Stefan Hagström]

Hi, just started to look into this project, looks really good!

I just have one question before i'll dive in and start some tests.

If i have two datacenters one primary and one secondary, the secondary is basically only used for backup of data/files.
The two datacenters are connected through VPN.

If a specified a datacenter replication strategy when storing a file and the VPN-connection is down, will it let me store the file anyway
and then replicate the file when the VPN comes up. Or will i get an error when storing the file?

BR
Stefan

[Stefan Hagström]

Sorry, "If a specified" should be "If i specified".

[Chris Lu]

The replicated writing is currently synchronous. All copies must be written before return successfully.

Your backup data center is a good use case for asynchronous replication. We can build a tool to replicate data across data centers instead of adding features to existing simple (and robust) volume servers.

Chris

--
You received this message because you are subscribed to the Google Groups "Weed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weed-file-syst...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shahar Fleischman]

hi,

just read your reply above and have two questions:

1. does this mean that async replication is not on your roadmap ?

2. if so, can i ask why, this can hurt performance and also make administering the cluster less easy ?

we reviewed a few options for our new file storage service and although weedfs was the fastest, the lack of async replication was one of the requirements that we can't live without.

thanks.

The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.  If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. Thanks.

[Chris Lu]

Hi, Shahar,

Can you make your requirement more clear? Just the phrase "async replication" can mean many different things. For example, do you just want a copy of the files, or want to fail over transparently or manually, etc. Or you can let me know your current use case.

Chris

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.

[Shahar Fleischmam]

Hi Chris,

I meant when uploading a file to weedfs that has a replication policy that require it to copy  the file to other hosts, that some of them are in a different geo location, waiting for a file to sync with other servers can create a slow response time towards the client.

Another question, on this matter, when as server gets a file from the client, what process handles the replication, what happens if the host goes down before it managed to copy the file to it's replicate hosts, is this process persistent ?

thanks,

shahar.

[Shahar Fleischman]

hi Chris,

can you comment on this ?

thanks.

[Chris Lu]

1. Current implementation is synchronous copy. The file write time is the slowest server write time.

2. When the server gets the file, it will be concurrently sent to the replicating servers, and only return success if all of the replicating servers are successful. So if the host goes down, the write will fail.

You actually are talking about 2 opposite sides of the replicated writes. One for the write performance, the other for the write consistency. Current implementation is for simplicity and read performance. Write response time is not optimal yet.

Chris

[Shahar Fleischman]

hi Chris,

i understand why copying data to all all replica is good for consistency, but it is very limiting in cross-dc situations.

also what would happen if one of the replica servers is down, will the write fail ?

[Chris Lu]

If one server is down, the write will fail. And the master server will notice the server as dead later if not receiving heartbeat from it.

In this case, the client should restart a write request, starting from master server again to assign to a different volume, which will go to another set of servers.​

For write, we choose CP relating to CAP theorem.

Chris

[Tim D]

(Replication) : If there are 3 masters and 1 master goes down.. Does that mean the whole system goes down since the "write" will fail if a master is down ?

Thanks

[Chris Lu]

The masters runs RAFT protocol to elect a leader. The leader keeps track of all volume servers. 

If non-leader master goes down, nothing really breaks unless the client did not know other masters.

If the leader master goes down, the leader election will be held and the new leader will handle all traffic assignments.

Chris

--

[Shahar Fleischmam]

Hi Chris,

just a few questions to clarify for me:

1. If a volume server dies, the master will notice and will direct writes to a diff volume server ?

2. So the writes that will fail are the writes destined to that volume server that happened between the time if stopped responding, and the time the master marked it as down ?

3. If the client tries again to write, it will get redirected to a diff volume server ?

4. Will the process of writing to all replicas continue to be sync, and not async ?

Thanks,

Shahar.

[Chris Lu]

Yes to 1,2,3.

Working on 4 for async options now. Maybe ready in a few months.

--

You received this message because you are subscribed to the Google Groups "Seaweed File System" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.