Re: [sqlite] Introducing Bedrock: SQLite for the Enterprise

[David Barrett]

(Including the mailing list again -- it's new, let's clutter it!)

Correct -- Bedrock takes care of all the details about detecting when a node is down, and repairing the cluster when it comes back up.  You don't need to worry about any of that.

Additionally, Bedrock already has a backup feature: just send SIGHUP and it'll disconnect from peers, copy the database aside, and then reconnect.  So, just run a three node cluster, and send SIGHUP to one of the nodes to trigger its backup, and then upload that backup somewhere.  

As for two versus three nodes, the "split brain" problem is only a risk if all nodes are receiving traffic.  In your case, it sounds like you are fine sending traffic only to the master, you just want the slave to occasionally backup without taking the master offline.  Accordingly, I think a 2-node cluster with the 2nd node set as a "permaslave" will work perfect for you.

So, I'd recommend you do a 2 node Bedrock cluster, one "real" (which will be MASTER), and one "permaslave".  Send all your writes to the master, and they will be replicated to the permaslave.  Send SIGHUP to the permaslave whenever you want to back up: it'll go offline, copy the database aside, come back online, and "catch up" with what it missed while offline -- with no downtime or impact on the master.

Does this sound like it'd work for you?

-david

On Thu, Oct 20, 2016 at 6:09 AM, Stephen Chrzanowski <pont...@gmail.com> wrote:

Taking this to private emails, since I don't want to clutter the mailing list. ;)

So I'd have to take the 3rd node offline, then flat-file copy to tape/backup site, then bring it online and it'd catch up.  That is feasible to do, even if the 3rd node runs on the primary system.  I could even setup a 2x2, so two machines running two instances, but then you'd run into the chance for split brain unless you set the priorities.  So many thoughts and methods for sanity checks.... :]

How do you alert the other nodes that the 3rd node is going down, or is being reintroduced?  Does just killing the one Bedrock process take it out of the loop, then restarting the script (As per your example on your site) to get them all connected again is sufficient?

On Thu, Oct 20, 2016 at 2:17 AM, David Barrett <dbar...@expensify.com> wrote:

Hm, that's an interesting case that I don't quite follow.  But you could
definitely use Bedrock on 2 nodes, and Bedrock would handle realtime
replication from one to the other.  This would provide an instant offsite
backup.  Granted, if there were only 2 nodes, you couldn't take one
offline, as the other one wouldn't have quorum.  If your goal is to not
just have a backup, but also copy that backup to tape, I actually recommend
a 3 node deployment.  This way you can take one node offline and the
remaining two still have quorum.

Now, another crazy idea would be to have a 2 node cluster, where one of the
nodes is a "permaslave".  Permaslaves don't participate in quorum, so
taking it offline wouldn't affect the master -- which would go forward
exactly as if it were alone (because from a quorum perspective, it is, and
thus it always represents its own quorum).  This *might* "just work" out of
the box, I haven't tried it.  But it's an interesting case!

-david