Slaves not starting properly using spring when master db is huge

[social...@gmail.com]

Hi,

I am having an application where my graph db is around 20GB in size. I am initializing my application using spring. Below are the steps followed by me in order to have a clean start.

Environment : HA with 1 master and 2 slaves.

1. Stopped my application and cleaned the graph.db folder from slaves.

2. Deleted the messages log from master.

3. Started the master. (it was waiting for some other instance to join the cluster)

4. Started first slave. I am loading my application using Spring. 

    The slave started to copy graph.db files from the master, but by this time my application starts to load all the beans which are dependent on "underlyingDBService". And it fails throwing the exception.

Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'indexManager' defined in class path resource [MET

A-INF/spring/graphdb-bean-context.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [graphdb.utils.IndexManager]: Constructor threw exception; nested exception is org.neo4j.graphdb.TransactionFailureException: Index creation failed for GLOBAL_SETTINGS_INDEX, {provider=lucene, type=exact}

        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:283)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:984)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:886)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:479)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:290)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:287)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:189)

        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:557)

        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:842)

        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:416)

        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)

        at graphdb.bootstrap.GraphServiceBootstrap.main(GraphServiceBootstrap.java:25)

Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [graphdb.utils.IndexManager]: Constructor threw exception; nested exception is org.neo4j.graphdb.TransactionFailureException: Index creation failed for GLOBAL_SETTINGS_INDEX, {provider=lucene, type=exact}

        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141)

        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:107)

        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:275)

        ... 14 more

Caused by: org.neo4j.graphdb.TransactionFailureException: Index creation failed for GLOBAL_SETTINGS_INDEX, {provider=lucene, type=exact}

        at org.neo4j.kernel.IndexManagerImpl.getOrCreateIndexConfig(IndexManagerImpl.java:235)

        at org.neo4j.kernel.IndexManagerImpl.getOrCreateNodeIndex(IndexManagerImpl.java:317)

        at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:305)

        at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:298)

        at graphdb.utils.IndexManager.<init>(IndexManager.java:78)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)

        ... 16 more

Caused by: java.lang.IllegalArgumentException: XaResourceManager is null

        at org.neo4j.kernel.impl.transaction.xaframework.XaConnectionHelpImpl.<init>(XaConnectionHelpImpl.java:88)

        at org.neo4j.index.impl.lucene.LuceneXaConnection.<init>(LuceneXaConnection.java:44)

        at org.neo4j.index.impl.lucene.LuceneDataSource.getXaConnection(LuceneDataSource.java:378)

After this exception , my application is not started in slave, but in graph.db/messages.log, it shows that copying files from master is in progress and it stops after copying all the files from master.

I have to restart slave again a few times to make it properly up.(same is the case with 2nd slave) Below is my spring context file.

<bean id="indexManager" class="graphdb.utils.IndexManager"

init-method="init" depends-on="underlyingDBService">

<constructor-arg ref="underlyingDBService"></constructor-arg>

</bean>

<bean id="underlyingDBService" class="org.neo4j.kernel.HighlyAvailableGraphDatabase">

<constructor-arg value="${GRAPHDB_PATH}"></constructor-arg>

<constructor-arg>

<map>

<entry key="ha.server_id" value-ref="neo4jServerId" /> <!-- generating server id based on hostname -->

<entry key="ha.initial_hosts" value="${GRAPH_CLUSTER_URLS}" />

<entry key="ha.tx_push_factor" value="2" />

<entry key="ha.pull_interval" value="120s" />

<entry key="ha.max_concurrent_channels_per_slave" value="100" />

<entry key="ha.join_timeout" value="300s" />

<entry key="ha.state_switch_timeout" value="30s" />

<entry key="ha.heartbeat_interval" value="10s" />

<entry key="ha.heartbeat_timeout" value="21s" />

<entry key="ha.default_timeout" value="10s" />

<entry key="ha.election_timeout" value="10s" />

<entry key="allow_store_upgrade" value="true" />

<entry key="online_backup_enabled" value="true" />

</map>

</constructor-arg>

</bean>

Note : in the init method I am trying to begin a transaction and create an index. 

Please suggest on this.

[social...@gmail.com]

Doesn't <bean id="underlyingDBService" class="org.neo4j.kernel.HighlyAvailableGraphDatabase"> wait until all the store is copied from master to slave and HighlyAvailableGraphDatabase is initialized??

[social...@gmail.com]

Hello... anyone.. any suggestions on this please,,,,????

[Peter Neubauer]

Hi there,

is this only happening with Spring? Could you bring up a cluster with just e.g. Neo4j Server? It might, as you suspect, be that Spring is not waiting long enough ...

/peter

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Kids in Malmö this summer?        - http://www.kidscraft.se
Neo4j questions? Use GraphGist. - http://gist.neo4j.org

[Peter Neubauer]

Hi there,

is this only happening with Spring? Could you bring up a cluster with just e.g. Neo4j Server? It might, as you suspect, be that Spring is not waiting long enough ...

/peter

On Wednesday, August 28, 2013, wrote:

Caused by: org.springframework.beans.

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Peter Neubauer]

Hi there,

just talked to Chris Gioran, who advised as follows:

The timeout for the instance access guard must be increased. The configuration value for this is ha.state_switch_timeout and is by default 20s. Setting that to something like 300 should be safe, as when the store copy is done it flips the state automatically, not relying on backup.

Does that help?

/peter

[Rickard Öberg]

For future reference for others with similar problems, we have now introduced the following method on GraphDatabaseService:

http://api.neo4j.org/1.9.5/org/neo4j/graphdb/GraphDatabaseService.html#isAvailable(long)

During startup you can call this with a suitably long timeout to ensure that once you're app is up the db is synced and available for use.

/Rickard