Problems with Backup of a production cluster in 1.9.RC2

[Lin Daiyu]

Hello, 

I downloaded Neo4j-Enterprise-1.9.RC2 to try out the HA Cluster.

 

Everything works so far. Now I would like to use the backup-tool but when I try to do this:

 

./neo4j-backup -from ha://10.10.30.31:5001 -to /mnt/backup/neo4j-backup

I get just an error:

 

Could not find backup server in cluster neo4j.ha at 10.10.30.32:5001, operation timed out

 

I followed this tutorial: http://docs.neo4j.org/chunked/milestone/ha-haproxy.html.

 

I have 3 different machines for the Neo4j-Servers with:

Debian 6.0

Apache Maven 3.0.5
Java version: 1.7.0_17

Neo4j-Enterprise 1.9.RC2

 

I've attached the config-file of one machine here as reference. The other config-files are similar.

Do I miss something? Is the the online_backup_server correct?  Any suggestions are welcome.

Thank you!

Lin

[Lasse Westh-Nielsen]

Lin,

I have had a play to try and figure this out, here is what I found:

Firstly, you have this config:

online_backup_server=10.10.30.31:6362 //this is fine as long as that port doesn't conflict with anything else
ha.server_id=1
ha.initial_hosts=10.10.30.31:5001,10.10.30.32:5001,10.10.30.33:5001
ha.server=10.10.30.31:6363 // here we normally use the range 6001-..., but that shouldn't matter
ha.cluster_server=10.10.30.31:5001

Secondly, the error you are getting ("Could not find backup server in cluster neo4j.ha at 10.10.30.32:5001, operation timed out"): I get the same error in two cases, namely when 1) nothing is listening on port 5001, or 2) port 5001 is firewalled. I have been playing around on AWS, but I am sure the same holds for your environment. So the things you need to check are:

 - is the cluster server running on 5001? (run `lsof | grep 5001` for example)

 - is there are firewall blocking between the instance and the machine from where you run the backup client? AWS security rules in my case, and note, you need to open for both 5001 AND 6263 so you can first get the cluster server to get the backup service port, and then secondly reach the backup service from the backup client machine

Hope it helps!

 - Lasse

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Lin Daiyu]

Lasse,

 

Thanks a lot! It works fine! I think it was the ha.server configuration that was incorrect.

 

Maybe the documation for the production cluster at http://docs.neo4j.org/chunked/milestone/ha-haproxy.html could be a little bit more detailed...

 

But for now I have a different problem: The HAProxy doesn't work fine with Neo4j Spatial. If I configure the HAProxy just to talk to slaves and create a layer from my C#-Client other a HTTP-command, the spatial index is created on the slaves but not on the master and I got errors when I try to add some nodes to the layer.

Now I thought about a little workaround - adding the spatial index on the master and anything else over the slaves. I manged this over a context switching in the HAProxy. But when I try to add some nodes the scenario repeat again: no problems if I transfer all request to the master but quite slow and Bad Requests if the HAProxy just talks to slaves.

 

Is it generally possible to use Neo4j Spatial in a HA cluster with a HAProxy? Or is the scenario other the REST-API simply not supported?

 

Best,

Lin

[Lasse Westh-Nielsen]

On Thu, May 16, 2013 at 2:39 PM, Lin Daiyu <daiy...@googlemail.com> wrote:

Maybe the documation for the production cluster at http://docs.neo4j.org/chunked/milestone/ha-haproxy.html could be a little bit more detailed...

You mean, more like this? http://docs.neo4j.org/chunked/milestone/ha-setup-tutorial.html ;)

 

But for now I have a different problem: The HAProxy doesn't work fine with Neo4j Spatial. If I configure the HAProxy just to talk to slaves and create a layer from my C#-Client other a HTTP-command, the spatial index is created on the slaves but not on the master and I got errors when I try to add some nodes to the layer.

Now I thought about a little workaround - adding the spatial index on the master and anything else over the slaves. I manged this over a context switching in the HAProxy. But when I try to add some nodes the scenario repeat again: no problems if I transfer all request to the master but quite slow and Bad Requests if the HAProxy just talks to slaves.

 

Is it generally possible to use Neo4j Spatial in a HA cluster with a HAProxy? Or is the scenario other the REST-API simply not supported?

That's a good question, my colleague Peter might have an answer - Peter? Or anyone else for that matter?

[Peter Neubauer]

Mmh,
the in-graph index configuration that Spatial uses, might actually
have a problem propagating through the cluster! I will raise a bug and
see if we can investigate!

/peter

Cheers,

/peter neubauer

G: neubauer.peter
S: peter.neubauer
P: +46 704 106975
L: http://www.linkedin.com/in/neubauer
T: @peterneubauer

The authoritative book on graph databases - http://graphdatabases.com
Neo4j questions? Please use SO - http://stackoverflow.com/search?q=neo4j

[Lin Daiyu]

@Lasse: My fault - I have copied the wrong link... But while the configuration of a local cluster is described very detailed, the production cluster part is kept very short and briefly. Maybe this part could be written in the same manner as the section of the locale cluster.

 

@Peter: Thanks a lot for the fast answer! I hope the bug will be fixed soon.

 

Best,

Lin

 

 

[Lasse Westh-Nielsen]

Lin,

We really appreciate your feedback, as we don't actually use the database "in production", we "just" develop it :)

So, the intention is that http://docs.neo4j.org/chunked/1.9.RC2/ha-setup-tutorial.html discusses the HA setup up stuff, ie clustering and machine config, whereas http://docs.neo4j.org/chunked/1.9.RC2/ha-haproxy.html gives an example of how to configure a load balancer in front of a cluster. Separate but complimentary sections in the manual.

That's the intention anyway, but you seem to have found a gap. So I'm going to do the cheeky thing and ask if you would consider contributing an update to the manual, or give a more detailed outline of what's missing/ what should be restructured?

For example, were you using HAProxy or some other load balancer? And for the clustering setup, did you test that separately before you picked up the load balancer work? The cluster should be functional without an LB, queries are routed from slaves to master automatically.

Anyway, in short, your contribution can make a difference!

 - Lasse

--

[Lin Daiyu]

Lasse,

 

The Intention of the different sections in the chapter High Availability is clear to me - I think you missunderstood me.

The thing is, that I started with a locale cluster and wanted to test the production cluster in the next step. At that point I noticed that the descriptions of configuration the locale and production cluster differ in question of detail. This was quite irritating, so maybe this part of production cluster could be written as detailed as the part of the locale cluster, e.g. attach the missing ha.server configuration. Just that the one who used the locale cluster before could easily see, which configurations should be changed to use the existing cluster as a production cluster ;)

 

I have tested it as a locale cluster without a load balancer and as a production cluster with and without the HAProxy. The locale cluster and the  production cluster without the HAProxy worked fine but a little bit slow. The only thing that don't work properly is the production cluster with the HAProxy.

 

Hope that I could make it clear.

 

Best,

Lin