data replication

[じょいすじょん]

I need to run instances of ArangoDB on several different Macs.

They should all have the exact same data.

To accomplish this I need to understand how to set up replication.

Are there any tutorials?

All of these machines have static IPs on a Class A IPv4 network.

If feasible, I would like to also understand if clustering is a better solution?

It would be nice if they should all be reachable at the same address, but not sure how that's a feasible thing without some proxy or something.

Any blog posts or tutorials beyond what I've found at ArangoDB.com would totally help me out.

Thanks,

John Joyce

[Frank Celler]

A description of the components of the cluster can be found here: https://www.arangodb.com/why-arangodb/cluster/

The ArangoDB starter provides an easy way to start a cluster: https://www.arangodb.com/2016/12/starting-arangodb-cluster-easy-way/

In your case, you would start e.g. three instances and choose a replication factor of 3. This will distribute the data to all three servers. What kind of queries do you expect? Is the data updated continuously or is it static?

An alternative is to use asynchronous replication of a master and three followers. In that case you must ensure that all write requests are directed to the master. Therefore I recommend going cluster. However, if you need more information about this setup, please let me know.

[Frank Celler]

PS: If you already found the above links on arangodb.com and there were not detailed enough, please let me know.

[じょいすじょん]

Hi Frank,

I have a few follow up questions here.

Does the arangodb cluster tool require a  Go language setup?

Would you all consider including the cluster tool with the standard app?

Otherwise, would you all consider adding it to various package managers like ArangoDB itself is?

In any case, it might be good to consider renaming the cluster tool to avoid confusion.

My suggested name would be arangocluster or arangofarm or arangoparty or something similarly implying the cluster.

Is it possible to make each node of the cluster independently reachable?

By this I mean, each node having its own IP/domain in a DNS and then each DNS entry being a way to reach the cluster?

Or each node being directly reachable while being part of the cluster?

(I will find out soon as I am working on this today)

[Max Neunhöffer]

Hi,
The starter tool arangodb is now contained in the standard distribution and we publish precompiled binaries on github. No go language setup is needed.
Best regards,
Max

[じょいすじょん]

Hi Max,

I do not see it when installing from homebrew on macOS.

Thanks,
John Joyce

> --
> You received this message because you are subscribed to a topic in the Google Groups "ArangoDB" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/arangodb/HGbsnKK63i0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to arangodb+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[じょいすじょん]

Hi Max
Are you referring to using the arangod options that start with --cluster.* ?
Seems like the article on setting up a cluster still refers to using a CLI tool named arangodb
There isn't one of those in the homebrew version as far as I can see.
It IS in the ArangoDB3-CLI.app distribution.
Is that version recommended instead of the homebrew one?
Using homebrew is a little more convenient for building out.

Thanks!
John Joyce

[じょいすじょん]

I tried the unstable version as well.
brew install --HEAD arangodb
and it also contains no arangodb item.

[じょいすじょん]

hmm. It also seems some issues with the arangodb tool at 

/Applications/ArangoDB3-CLI.app/Contents/MacOS/arangodb

Every command seems to fail with:

Error: unknown shorthand flag: 'c' in -c

followed by the usage notes.

I was able to reproduce this on multiple machines.

Am I doing something wrong there?

On Oct 27, 2017, at 15:34, Max Neunhöffer <max.neu...@gmail.com> wrote:

[じょいすじょん]

ah, so I see now that is a shell script at

/Applications/ArangoDB3-CLI.app/Contents/MacOS/arangodb

doing a few things, but ultimately calling

/Applications/ArangoDB3-CLI.app/Contents/MacOS/usr/bin/arangodb

I think I will have some suggestions to put in pull request if you could point me to what files generate the .app bundle in the github repository.

One of them being that it is unreliable to do this:

export ROOTDIR="/Applications/ArangoDB3-CLI.app/Contents/MacOS/"

Mac apps are drag and drop installs and .app bundles should normally be runnable from most locations.

(Safari is an unusual exception and you cannot run apps from the Trash folder, among others)

Especially, since the install is drag and drop from a mounted .dmg disk image, users could expect to be able to relocate the bundle to wherever.

I don't know if this is bash specific, but perhaps

export ROOTDIR="$(dirname $0)"

could work more reliably.

[じょいすじょん]

Well, now that I figured out which arangodb was the proper one and used the .app bundled version, I have to say, that is quite nice.

Really easy to stand up. This is a hallmark of ArangoDB to me.

This worked more brilliantly than you might know.

Every node was reachable by its internal DNS name. That was fabulous.

As I will use this on test automation nodes, it will mean that most or all nodes can ping a series of domains to read and write data, starting with localhost. If no localhost, they can go through a known list of peers.

All will have strong perf and strong syncing.

There is a little latency potential when some nodes are across an ocean, but it works so far.

Very happy, finally have it working.

Nice touch is, it's also a pretty good example of how to use the .app bundle on macOS creatively!

The more I thought about it, the more I realized it makes a lot of sense for not only easy deployment, but pretty easy drop in replacement/updation.

Now, I just need to tackle the ssl portion.

(It wasn't ever as easy as RethinkDB)

[じょいすじょん]

Hi Frank,

I have a few follow up questions here.

Does the arangodb cluster tool require a  Go language setup?

Would you all consider including the cluster tool with the standard app?

Otherwise, would you all consider adding it to various package managers like ArangoDB itself is?

In any case, it might be good to consider renaming the cluster tool to avoid confusion.

My suggested name would be arangocluster or arangofarm or arangoparty or something similarly implying the cluster.

Is it possible to make each node of the cluster independently reachable?

By this I mean, each node having its own IP/domain in a DNS and then each DNS entry being a way to reach the cluster?

Or each node being directly reachable while being part of the cluster?

(I will find out soon as I am working on this today)

Thanks!

John Joyce

[Max Neunhöffer]

Hi John,

No golang setup is needed to run the ArangoDB starter `arangodb`. You can simply use the supplied executable.

By now, all binary packages we release should contain the starter executable. If this is not the case, this is a bug. I am no expert in Mac packaging and have no clue how the situation is with homebrew or the app bundle we ship. We need the Mac experts to look into this, I will alert Frank. In any case, you can always just download the latest starter executable from https://github.com/arangodb-helper/arangodb just see the releases thete.

As to the name: our idea is that the arangodb starter tool is the primary tool to start ArangoDB in all configurations: single server, cluster and the upcoming single server resilience config. The user should either use a package in which ArangoDB is started automatically, or use the starter (or some orchestration solution like mesos or kubernetes).

In the standard configuration the starter starts a cluster with a coordinator on each machine, by default on port 8529. You can then contact any of these endpoints to talk to the cluster. This at first concerns endpoints. You can use ip addresses or host names as you like.

However, your DNS configuration is completely outside the scope of the starter and ArangoDB. You have to manage this by other means.

I hope this answers your questions. I will alert the Macexperts about your other questions regarding homebrew.

Cheers, Max

[じょいすじょん]

Hi Max,
Thanks for getting back to me always.

Comments inline!

> On Nov 9, 2017, at 16:58, Max Neunhöffer <max.neu...@gmail.com> wrote:
>
> Hi John,
>
> No golang setup is needed to run the ArangoDB starter `arangodb`. You can simply use the supplied executable.

I did discover this! I think I did see some old information somewhere on Github.

>
> By now, all binary packages we release should contain the starter executable. If this is not the case, this is a bug. I am no expert in Mac packaging and have no clue how the situation is with homebrew or the app bundle we ship. We need the Mac experts to look into this, I will alert Frank. In any case, you can always just download the latest starter executable from https://github.com/arangodb-helper/arangodb just see the releases thete.

It is missing from the homebrew version but definitely in the direct download ArangoDB-CLI.app
After inspecting the app bundle contents, I could see quickly how everything was laid out there, and it makes great sense for easy upgrades and deployments.
Quite nice in fact!

>
> As to the name: our idea is that the arangodb starter tool is the primary tool to start ArangoDB in all configurations: single server, cluster and the upcoming single server resilience config. The user should either use a package in which ArangoDB is started automatically, or use the starter (or some orchestration solution like mesos or kubernetes).

That makes sense!
Especially after using it successfully.

>
> In the standard configuration the starter starts a cluster with a coordinator on each machine, by default on port 8529. You can then contact any of these endpoints to talk to the cluster. This at first concerns endpoints. You can use ip addresses or host names as you like.
>
> However, your DNS configuration is completely outside the scope of the starter and ArangoDB. You have to manage this by other means.

It turns out, so simple and impressive.
I could start the cluster, join any node at any DNS entry for it, and raise the cluster.
I honestly do not know how chatty this will make my network, but I intend to put an instance up on each node in a Jenkins CI.
Then things that run on each Jenkins CI node can try to reach the first available ArangoDB cluster node.
They start with localhost, then query all the IPs/URLs of the nodes in Jenkins, until one is reached to do database things with.
In theory, this is simple, reachable and fast, and easy to keep up and available.

One of my use cases is a simple pool of resources in-use.
Each Jenkins CI node checks the pool first.
If resource is not in the pool, the Jenkins CI node claims the resource by adding to the pool.
If resource is in the pool already, Jenkins CI node must choose another resource.

>
> I hope this answers your questions. I will alert the Macexperts about your other questions regarding homebrew.

It does!

>
> Cheers, Max

[Frank Celler]

Hi John,

the homebrew package is indeed missing the ArangoDB starter. I have opened an issue: https://github.com/arangodb/arangodb/issues/3633

best Frank