Ubuntu ERROR in ch.qos.logback.core.joran.action.IncludeAction

[Robert Crowe]

I've installed Neo4j 1.8.1 on Ubuntu 12.04, and when I start the service I get:

Starting Neo4j Server...02:18:12,273 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
02:18:12,273 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
02:18:12,273 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/usr/share/neo4j/system/lib/neo4j-server-1.8.1.jar!/logback.xml]
02:18:12,293 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@464c4975 - URL [jar:file:/usr/share/neo4j/system/lib/neo4j-server-1.8.1.jar!/logback.xml] is not of type file
02:18:12,349 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
02:18:12,361 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
02:18:12,366 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
02:18:12,380 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
02:18:12,430 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
02:18:12,430 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
02:18:12,432 |-ERROR in ch.qos.logback.core.joran.action.IncludeAction - Could not find resource corresponding to [custom-logback.xml]
02:18:12,435 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@3a6ac461 - Registering current configuration as safe fallback point

At first the server seems to work, and I can run the web admin console on 7474, but after creating about 50k nodes it crashes and will not restart.  My only clue as to why is in these messages, as far as I know.  Most of them are info, but that one error worries me.  I've seen posts that discuss this on MacOS, but not Ubuntu.  I tried this fix, but it didn't fix it:

copy the: ./system/lib/concurrentlinkedhashmap-lru-1.3.1.jar
to ./lib/concurrentlinkedhashmap-lru-1.3.1.jar

Any ideas?

Robert

[Robert Crowe]

No love for Ubuntu?

[Michael Hunger]

Can you share your logs?

server/data/logs

server/data/graph.db/messages.log

Sent from mobile device

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Robert Crowe]

Happy to, and thanks for any help that you can offer!  Logs are attached.  In the meantime I did a full purge and reinstall of Neo4j, but it didn't seem to change anything as far as the messages are concerned.  I haven't tried creating a large set of nodes since the reinstall.  I don't think that I was getting these messages after the original install at first.  At one point I tried to move the files from /var/lib/neo4j to a different folder to manage space on the disk, but I've reverted that back.  Not sure if it's related to the error messages.

[Michael Hunger]

Robert,

you missed the messages.log

The other two don't contain any hints :)

Michael

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

<logs.zip>

[Robert Crowe]

Sorry, see attached.

[Michael Hunger]

Robert,

according to the logs there is no indication that the server has actually crashed,

how did that manifest itself?

Am 09.02.2013 um 01:56 schrieb Robert Crowe <valeri...@gmail.com>:

Sorry, see attached.

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

<messages.zip>

[Robert Crowe]

Sorry, I cleaned out the logs after that.  Probably a bad idea, but I've recovered the messages.log from before that, see attached.  The crash happened when I was loading a large number of nodes from a process running under cron and reading from a MySQL database.  The service crashed, and would not start again.

[Robert Crowe]

The console and neo4j.0.0 logs from before cleaning are attached.

[Michael Hunger]

Robert,

it seems that

# I saw that you are importing with cypher, please make sure to use parameters if possible: http://docs.neo4j.org/chunked/milestone/cypher-parameters.html

http://docs.neo4j.org/chunked/milestone/rest-api-cypher.html#rest-api-send-queries-with-parameters

# it seems to have issues acessing the user home directory, if you are starting it as user neo4j make sure he has a writable home dir

# you seem to have two servers running at some point "BindException, cannot assign the requested port", make sure to stop or kill the previous server

# on re-startup it tries to recover the logs but takes too long, please start with bin/neo4j console and wait until it has finished

Cheers

Michael

(neo4j.org) <-[:WORKS_ON]- (@mesirii) -[:TAKES_CARE_OF]-> (you) -[:WORKS_WITH]->(@Neo4j)

Am 09.02.2013 um 02:41 schrieb Robert Crowe:

The console and neo4j.0.0 logs from before cleaning are attached.

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

<console.zip>

[Robert Crowe]

Thanks Michael!  I'm actually using neo4jphp, so I don't think that I can use parameters, can I?  I'll fix the home directory problem.

I'm running Neo4j as a service, so I start it with:

sudo service neo4j-service start

how would I "start with bin/neo4j console"?

[Michael Hunger]

Did you install it with the debian installer? Then it is in the path.

Otherwise when you downloaded the tgz.

Check where you extracted it, there is a bin directory

bin/neo4j console

Cheers

Michael

(neo4j.org) <-[:WORKS_ON]- (@mesirii) -[:TAKES_CARE_OF]-> (you) -[:WORKS_WITH]->(@Neo4j)

[Robert Crowe]

I don't know if this might be a bug or not, but after starting as a service (sudo service neo4j-service start) when I run:

/var/lib/neo4j/bin/neo4j info

it says "Neo4j Server is not running".  When I then run:

sudo service neo4j-service status

it says "Neo4j Server is running at pid 2004"

Could that be causing problems when I use neo4jphp?

[Robert Crowe]

It crashed again, this time with 64,529 nodes.  Just before it crashed it threw a few hundred of these to messages.log (attached):

2013-02-10 00:49:11,818 INFO  [neo4j]: GC Monitor: Application threads blocked for an additional 1107ms [total block time: 5950.342s]

I saw several php processes running just before it crashed, maybe a dozen or so.  They could easily have been trying to update the same nodes, or querying the same nodes.  But I would expect that to make things slow, not crash.

[Michael Hunger]

How does the crash manifest? Is the running process gone? It should never crash.

Can you note the steps you take to make this happen?

The messages are from excessive garbage collection. How much memory do you have on the machine? Did you configure the memory used by neo4j manually?

How exactly are you querying / adding the nodes? Where do the several php processes come from? Are they running in parallel?

I'm not sure if neo4jphp uses streaming by default. If not the complete result is built up in memory before sent out that can cause effects like this for large results.

Cheers

Michael

(neo4j.org) <-[:WORKS_ON]- (@mesirii) -[:TAKES_CARE_OF]-> (you) -[:WORKS_WITH]->(@Neo4j)

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

<messages.zip>

[Robert Crowe]

The crash manifests itself by the process dying, which I notice when the web admin console goes gray and says it's trying to reconnect.  I then check with "service neo4j-service status", and with ps, and the server process is indeed gone.

What I was doing was running a PHP process under cron, using neo4jphp and reading from a MySQL database to build the graph.  The process was checking time in a loop, and was supposed to stop itself after 2 minutes, so I was running it every 5 minutes to be safe.  When I saw several, maybe a dozen PHP processes I realized that they weren't ending soon enough and were overlapping, so now I set a lock file to prevent a new process from starting before the old one has finished.  That seems to have stopped the crash (so far anyway) and I'm now up to 78,278 nodes.

I'm running on an Amazon EC2 medium instance under Ubuntu 12.04 LTS 64 bit:

M1 Medium Instance

3.75 GiB memory
2 EC2 Compute Unit (1 virtual core with 2 EC2 Compute Unit)
410 GB instance storage
32-bit or 64-bit platform
I/O Performance: Moderate
EBS-Optimized Available: No
API name: m1.medium

I haven't done any tuning really, other than the basic stuff at install like open file limits, and mounting the volume with noatime,nodiratime,  Neo4j has a 200GB volume and is currently using only 1% of that.

cheers,

Robert