Unable to start neo4j server on linux

[Jithin Emmanuel]

Hi,
I am using latest version neo4j version (1.6) and java version 1.6. This is what happens when I attempt to start neo4j.

[jithin1987@old neo4j]$ sudo -u neo4j ./bin/neo4j start
Starting Neo4j Server...WARNING: not changing user
 process [16177]... waiting for server to be ready......................................................................................................................... BAD.
 another server-process is running with []
[jithin1987@old neo4j]$ echo $?
2
[jithin1987@old neo4j]$ sudo -u neo4j ./bin/neo4j stop
ERROR: Neo4j Server not running

I am not able to find anything on console.log other than these

2/2/12 6:04:31 PM org.neo4j.server.NeoServerWithEmbeddedWebServer INFO: Starting Neo Server on port [7474] with [10] threads available
2/2/12 6:04:31 PM org.neo4j.server.database.Database INFO: Using database at /usr/local/neo4j/data/graph.db

Kindly help me out in figuring out whats going wrong.

[Peter Neubauer]

can you try

./bin/neo4j console and see what is happening?

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[Jithin Emmanuel]

Looks like console mode worked.
[jithin1987@old neo4j]$ sudo -u neo4j ./bin/neo4j console
Starting Neo4j Server console-mode...
2/2/12 6:36:52 PM org.neo4j.server.NeoServerWithEmbeddedWebServer INFO: Starting Neo Server on port [7474] with [10] threads available
2/2/12 6:36:52 PM org.neo4j.server.database.Database INFO: Using database at /usr/local/neo4j/data/graph.db







2/2/12 6:40:02 PM org.neo4j.server.modules.DiscoveryModule INFO: Mounted discovery module at [/]
2/2/12 6:40:02 PM org.neo4j.server.plugins.PluginManager INFO: Loaded server plugin "CypherPlugin"
2/2/12 6:40:02 PM org.neo4j.server.plugins.PluginManager INFO: Loaded server plugin "GremlinPlugin"
2/2/12 6:40:02 PM org.neo4j.server.modules.RESTApiModule INFO: Mounted REST API at [/db/data/]
2/2/12 6:40:02 PM org.neo4j.server.modules.ManagementApiModule INFO: Mounted management API at [/db/manage/]
2/2/12 6:40:02 PM org.neo4j.server.modules.WebAdminModule INFO: Mounted webadmin at [/webadmin]
2/2/12 6:40:02 PM org.neo4j.server.web.Jetty6WebServer INFO: Mounting static content at [/webadmin] from [webadmin-html]
2/2/12 6:40:06 PM org.neo4j.server.statistic.StatisticStartupListener INFO: adding statistic-filter to /webadmin
2/2/12 6:40:06 PM org.neo4j.server.statistic.StatisticStartupListener INFO: adding statistic-filter to /db/manage
2/2/12 6:40:06 PM org.neo4j.server.statistic.StatisticStartupListener INFO: adding statistic-filter to /db/data
2/2/12 6:40:06 PM org.neo4j.server.statistic.StatisticStartupListener INFO: adding statistic-filter to /
2/2/12 6:40:06 PM org.neo4j.server.NeoServerWithEmbeddedWebServer INFO: Server started on [http://localhost:7474/]

[thoma...@atns.de]

hej jithin,

dont sudo to neo4j, like every init-script it will change the user by it self.

starting "sudo ./bin/neo4j start" should be fine.

when you have started the server in console-mode, it is possible that the database-directories are now owned by root,

cheers, thomas

[Jithin Emmanuel]

Hi Thomas,

Without sudo -u neo4j I get the warning for max open files. I have relaxed it for neo4j user.

But still I am unable to start server.  It still gives 

Starting Neo4j Server... process [19037]... waiting for server to be ready......................................................................................................................... BAD.

 another server-process is running with []

btw I find this i console.log

2/2/12 5:38:34 PM org.neo4j.server.database.Database INFO: Using database at /var/apps/stage/neo4j-community-1.6/data/graph.db

Exception in thread "main" java.lang.ExceptionInInitializerError

        at org.neo4j.server.web.Jetty6WebServer.addJAXRSPackages(Jetty6WebServer.java:207)

        at org.neo4j.server.modules.DiscoveryModule.start(DiscoveryModule.java:36)

        at org.neo4j.server.NeoServerWithEmbeddedWebServer.startModules(NeoServerWithEmbeddedWebServer.java:139)

        at org.neo4j.server.NeoServerWithEmbeddedWebServer.start(NeoServerWithEmbeddedWebServer.java:102)

        at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:87)

        at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:52)

Caused by: java.util.MissingResourceException: Can't find bundle for base name javax.servlet.LocalStrings, locale en_US

        at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1539)

        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1278)

        at java.util.ResourceBundle.getBundle(ResourceBundle.java:733)

        at javax.servlet.GenericServlet.<clinit>(GenericServlet.java:64)

        ... 6 more

But this was not logged when I was attempting to start server.

--
Thanks
Jithin Emmanuel

[Jithin Emmanuel]

I am still not able to start the server. Kindly let me know if there is anything I need to try out. (console mode works though)

[James Thornton]

Two questions:

1. Did you build neo4j from the GitHub source or did you download the tarball?

If you built from source, try downloading the pre-built version from http://neo4j.org .

2. Is it possible the files/permissions got hosed on the false starts?

If so, try starting fresh or at the least deleting the contents of the data/graph.db dir:

First, make sure there isn't a neo4j process running:

$ ps -ef | grep neo4j

If there is, kill it.

Then remove the PID file and delete the contents of the data/graph.db dir:

$ rm  /var/apps/stage/neo4j-community-1.6/neo4j-service.pid

$ rm -rf  /var/apps/stage/neo4j-community-1.6/data/graph.db/*

Now try starting it again:

$  /var/apps/stage/neo4j-community-1.6/bin/neo4j start

- James

[Michael Hunger]

Emanuel,

could you share your log-files (in data/logs ).

And perhaps also if there is any system classpath set
echo $CLASSPATH

which exact version of linux are you running and how did you install the jdk/jre ?

So that we can try to reproduce that on an AWS instance (or perhaps you can do it there).

Otherwise, would it be possible to access the linux system you're trying to run the server on?

Thanks

Michael

[Jithin Emmanuel]

Hi Michael,

Command ran - 

[deploy@store-1 ~]$ sudo -u neo4j  /usr/local/neo4j/bin/neo4j start

WARNING: Detected a limit of 1024 for maximum open files, while a minimum value of 40000 is recommended.

WARNING: Problems with the operation of the server may occur. Please refer to the Neo4j manual regarding lifting this limitation.

Starting Neo4j Server...WARNING: not changing user

 process [27198]... waiting for server to be ready......................................................................................................................... BAD.

 another server-process is running with []

Thiese are logs,

==> data/log/neo4j.0.0.log.lck <==

==> data/log/neo4j.0.0.log <==

Feb 3, 2012 5:11:13 PM org.neo4j.server.logging.Logger log

INFO: Store files missing, or not in suitable state for upgrade. Leaving this problem for main server process to resolve.

Feb 3, 2012 5:11:13 PM org.neo4j.server.logging.Logger log

INFO: Starting Neo Server on port [7474] with [10] threads available

==> data/log/console.log <==

2/3/12 5:11:13 PM org.neo4j.server.NeoServerWithEmbeddedWebServer INFO: Starting Neo Server on port [7474] with [10] threads available

==> data/log/neo4j.0.0.log <==

Feb 3, 2012 5:11:13 PM org.mortbay.log.Slf4jLog info

INFO: Logging to org.slf4j.impl.JDK14LoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

Feb 3, 2012 5:11:13 PM org.neo4j.server.logging.Logger log

INFO: Using database at /usr/local/neo4j/data/graph.db

==> data/log/console.log <==

2/3/12 5:11:13 PM org.neo4j.server.database.Database INFO: Using database at /usr/local/neo4j/data/graph.db

==> data/log/neo4j.0.0.log <==

Feb 3, 2012 5:11:13 PM org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery

INFO: Non clean shutdown detected on log [/usr/local/neo4j/data/graph.db/nioneo_logical.log.1]. Recovery started ...

Feb 3, 2012 5:11:14 PM org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery

INFO: Non clean shutdown detected on log [/usr/local/neo4j/data/graph.db/index/lucene.log.1]. Recovery started ...

Java version

 [deploy@store-1 ~]$ java -version

java version "1.6.0_20"

OpenJDK Runtime Environment (IcedTea6 1.9.8) (amazon-52.1.9.8.36.amzn1-x86_64)

OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

This is an AWS instance running CentOs

--
Thanks
Jithin Emmanuel

[Kevin Mai]

If you haven't already done so, you need to modify the /etc/security/limits.conf file

The problem you're facing is that the users have set up a limit on the number of files open simultaneously.

Once you modify the file, you need to log off and log back in.

Here is an example:

# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#        - NOTE: group and wildcard limits are not applied to root.
#          To apply a limit to the root user, <domain> must be
#          the literal username root.
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#        - chroot - change root to directory (Debian-specific)
#
#<domain>      <type>  <item>         <value>

neo4j               hard    nofile          100000
neo4j               soft    nofile          40000
root                hard    nofile          100000
root                soft    nofile          40000


Hope this helps!

Kevin

2012/2/3 Jithin Emmanuel <jithi...@gmail.com>

[Jithin Emmanuel]

Hi Kevin,

Even with relaxed limites I am facing same issue.

[jithin1987@store-1 ~]$ sudo  /usr/local/neo4j/bin/neo4j start

Starting Neo4j Server... process [28183]... waiting for server to be ready......................................................................................................................... BAD.

 another server-process is running with []

Is there anything amiss form my earlier mail. Is java version and all proper for use with neo4j. ?

--
Thanks
Jithin Emmanuel

[Jithin Emmanuel]

Also this is am AWS instance. Should I open up port before starting?

--
Thanks
Jithin Emmanuel

[thoma...@atns.de]

hi jinthn, 

when you start the neo4j-script as a limited user it will not report which process is using the port.

please do

$ netstart -nltp 

and/or  

$ ps xau

to check which process is all ready running.

as i wrote the script will change to the configured user by it self. see the configuration in ./conf/neo4j-wrapper.conf

the ulimit warning is just a warning: Problems ... may occur.

regards, thomas

[Jithin Emmanuel]

Before starting neo4j this is the output of  netstart -nltp 

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   

tcp        0      0 0.0.0.0:11211               0.0.0.0:*                   LISTEN      5443/memcached      

tcp        0      0 0.0.0.0:4369                0.0.0.0:*                   LISTEN      952/epmd            

tcp        0      0 0.0.0.0:45715               0.0.0.0:*                   LISTEN      7998/beam           

tcp        0      0 0.0.0.0:5269                0.0.0.0:*                   LISTEN      7998/beam           

tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      24851/sshd          

tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1211/sendmail: acce 

tcp        0      0 0.0.0.0:5280                0.0.0.0:*                   LISTEN      7998/beam           

tcp        0      0 0.0.0.0:5222                0.0.0.0:*                   LISTEN      7998/beam           

tcp        0      0 :::11211                    :::*                        LISTEN      5443/memcached      

tcp        0      0 :::22                       :::*                        LISTEN      24851/sshd          

tcp        0      0 :::8983                     :::*                        LISTEN      8909/java           

tcp        0      0 :::8984                     :::*                        LISTEN      24729/java       

When I started neo4j this happened.

sudo  /usr/local/neo4j/bin/neo4j start

Starting Neo4j Server... process [29451]... waiting for server to be ready......................................................................................................................... BAD.

 another server-process is running with []

And  netstart -nltp  showed 2 new processes

tcp        0      0 :::1337                     :::*                        LISTEN      29451/java

tcp        0      0 :::37246                    :::*                        LISTEN      29451/java

--
Thanks
Jithin Emmanuel

[thoma...@atns.de]

thats wired,  the jetty server seems not to start.

can you tell me which ami / zone / instance type you used? 

cheers, thomas

[Jithin Emmanuel]

AMI is centos 5.5, zone is US east and its a micro instance. This is used as a staging env of my project.

--
Thanks
Jithin Emmanuel

[Jithin Emmanuel]

Finally I was able to start. The host is really slow. Timeout of 120 seconds provided in neo4j startup script is inadequate for it. I doubled the value and server was able to start.

--
Thanks
Jithin Emmanuel

[Kevin Mai]

Jithin,

Just an advise: I use Neo4J on EC2 and actually running it on micro instances might be good for dev/testing, but be aware that micro instances have capped cpus which also means that on spike resource consumption performance will get even more degraded on micro instances as they don't have reserved resources.

Regards,

Kevin

pd: It is always encouraged to have all EC2 ports closed and to open them according to your needs.

2012/2/4 Jithin Emmanuel <jithi...@gmail.com>

[Jithin Emmanuel]

Thanks Kevin for the heads up. This is a dev env. Our actual deployment is on linode.

--
Thanks
Jithin Emmanuel

[divyanshu das]

@Jithin, how did u increase the timeout? which file did u edit?

[Jithin Emmanuel]

I don't recall it exactly. Just go through the conf files, u won't miss it.

--
 
 

[divyanshu das]

I cant find any reference to time-out thing. I have 1.8 version. 

Although I was able to start the neo4j serer by providing, sudo service neo4j-service start-no-wait on ec2.

Its not starting on my local machine.