Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Neo4j
Home
+ new post
About this group
Join this group
Neo4j REST service stops working
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   Messages 1 - 25 of 29 - Collapse all  -  Translate all to Translated (View all originals)   Newer >
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Khushbu Bhatewara  
View profile  
 More options Oct 3 2012, 5:53 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Wed, 3 Oct 2012 02:53:49 -0700 (PDT)
Local: Wed, Oct 3 2012 5:53 am
Subject: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Hi,

We are facing a very weird issue in Neo4j REST service hosted on windows
azure. The service becomes inaccessible through code and webadmin
console. On analyzing messages.log we found nothing and the most strange
thing was GC monitor log was being logged in messages.logs even when
service was not accessible. The service was in started state.

When nothing was found, we just restarted the service and it is accessible
now. But we need to find the root cause analysis for why this happened. Is
there any other logs/traces  where we can actually find why the service
behaved abnormally?

- Khushbu


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Peter Neubauer  
View profile  
 More options Oct 3 2012, 11:21 am
From: Peter Neubauer <peter.neuba...@neotechnology.com>
Date: Wed, 3 Oct 2012 17:20:57 +0200
Local: Wed, Oct 3 2012 11:20 am
Subject: Re: [Neo4j] Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author
I think you should connect JConsole to it in order to see what is
happening on the JVM via JMX.

Have you been running heavy load or big returns on the server when
this happened? Try to use the streaming REST format, see
http://docs.neo4j.org/chunked/snapshot/rest-api-streaming.html

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Wanna learn something new? Come to http://graphconnect.com

On Wed, Oct 3, 2012 at 11:53 AM, Khushbu Bhatewara


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 11, 7:10 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Thu, 11 Apr 2013 04:10:35 -0700 (PDT)
Local: Thurs, Apr 11 2013 7:10 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Hi Peter,

After a long time we faced the issue again. We ran threadprofiler of
newrelic to find the cause for the same. And found some thread locking
issue on an object.

Following is the screenshot of the same. Please suggest what can be done to
overcome the locking behavior or is there any way through which we can
drill down the goovy script which caused the issue.

<https://lh5.googleusercontent.com/-hjahQYm5kVA/UWaZxi8UaXI/AAAAAAAAA4...>
Regards,
Khushbu


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Peter Neubauer  
View profile  
 More options Apr 11, 7:15 am
From: Peter Neubauer <peter.neuba...@neotechnology.com>
Date: Thu, 11 Apr 2013 13:15:33 +0200
Local: Thurs, Apr 11 2013 7:15 am
Subject: Re: [Neo4j] Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Khushbu,
can't see the screenshot, can you maybe put it online somewhere, since it
might get filtered out on this list?

/peter

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

The authoritative book on graph databases - http://graphdatabases.com
Neo4j questions? Please use SO - http://stackoverflow.com/search?q=neo4j

On Thu, Apr 11, 2013 at 1:10 PM, Khushbu Bhatewara <


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 11, 10:47 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Thu, 11 Apr 2013 07:47:25 -0700 (PDT)
Local: Thurs, Apr 11 2013 10:47 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Hi Peter,

PFA the sceenshot.

Regards,
Khushbu

  Neo4j hang.png
23K Download

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 19, 1:16 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Thu, 18 Apr 2013 22:16:15 -0700 (PDT)
Local: Fri, Apr 19 2013 1:16 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Any findings?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Peter Neubauer  
View profile  
 More options Apr 19, 6:17 am
From: Peter Neubauer <peter.neuba...@neotechnology.com>
Date: Fri, 19 Apr 2013 12:17:01 +0200
Local: Fri, Apr 19 2013 6:17 am
Subject: Re: [Neo4j] Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Khushbu,
what kind of queries are you running?

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

The authoritative book on graph databases - http://graphdatabases.com
Neo4j questions? Please use SO - http://stackoverflow.com/search?q=neo4j

On Thu, Apr 11, 2013 at 4:47 PM, Khushbu Bhatewara <


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 19, 9:17 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Fri, 19 Apr 2013 06:17:57 -0700 (PDT)
Local: Fri, Apr 19 2013 9:17 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

We are running gremlin queries for get query. As well as, we are using
groovy scripts for transactions - creation and updation.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Peter Neubauer  
View profile  
 More options Apr 19, 9:49 am
From: Peter Neubauer <peter.neuba...@neotechnology.com>
Date: Fri, 19 Apr 2013 15:49:34 +0200
Local: Fri, Apr 19 2013 9:49 am
Subject: Re: [Neo4j] Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Mmh,
I wonder if you maybe are starting a transaction that spans the whole
database or so, and not committing it so that there is a transaction lock
hanging?

/peter

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

The authoritative book on graph databases - http://graphdatabases.com
Neo4j questions? Please use SO - http://stackoverflow.com/search?q=neo4j

On Fri, Apr 19, 2013 at 3:17 PM, Khushbu Bhatewara <


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 19, 11:36 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Fri, 19 Apr 2013 08:36:28 -0700 (PDT)
Local: Fri, Apr 19 2013 11:36 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

We are commiting transaction in case of success, in case of failure
transaction rollbacks. Please suggest if following query has any problems.

Sample query looks like:

import org.neo4j.graphdb.*;
neo4j = g.getRawGraph();
idxManager = neo4j.index();
msgIdxObj = idxManager.forNodes('Index_MessageId');
msgNode = msgIdxObj.get('MessageId',p0).getSingle();
if(!msgNode.equals(null))
{
try
{
tx = neo4j.beginTx();
userLockObj=null;
        Relationship messageOutRel =
msgNode.getSingleRelationship(DynamicRelationshipType.withName('MESSAGE'),D irection.OUTGOING);
        if(!messageOutRel.equals(null))
{
Node OutEndNode =messageOutRel.getEndNode();
if(!OutEndNode.hasProperty('Recepients'))
{
Lock userLockObj = tx.acquireWriteLock(OutEndNode);

};
};

        Relationship messageOutRelationship =
msgNode.getSingleRelationship(DynamicRelationshipType.withName('MESSAGE'),D irection.OUTGOING);
        Lock lockObj = tx.acquireWriteLock(msgNode);
if(!messageOutRelationship.equals(null))
{
            Relationship messageInRelationship =
msgNode.getSingleRelationship(DynamicRelationshipType.withName('MESSAGE'),D irection.INCOMING);
            if(!messageInRelationship.equals(null))
{
Node InStartNode =messageInRelationship.getStartNode();
Node OutEndNode =messageOutRelationship.getEndNode();
InStartNode.createRelationshipTo(OutEndNode,DynamicRelationshipType.withNam e('MESSAGE'));
messageInRelationship.delete();
};

messageOutRelationship.delete();
};    

        msgNode.delete();
lockObj.release();
if(!userLockObj.equals(null))
{
userLockObj.release();
};
tx.success();
}

catch(Exception e)
{
tx.failure();
throw e;
}

finally
{
tx.finish();


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
vyadav  
View profile  
 More options Apr 24, 2:21 am
From: vyadav <viya...@gmail.com>
Date: Tue, 23 Apr 2013 23:21:13 -0700 (PDT)
Local: Wed, Apr 24 2013 2:21 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

We are also facing similar issue with Neo4j. Is there any solution to the
problem?

We are using transactions to create relation with node. Committing and
rolling back transactions properly using try catch. But if we put load on
Neo4j and multiple nodes try to attachd with same node then transactions
are waiting to aquire lock so that they can take lock and create
relationship with same node... Neo4j is going in hang state if there is
long queue pending... Ideally it should process queue. It can take time as
other requests are getting processed... but unfortunately it going in hang
state...

We have to restart Neo4j to bring it up.....

-Vikas Y


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Hunger  
View profile  
 More options Apr 24, 2:45 am
From: Michael Hunger <michael.hun...@neopersistence.com>
Date: Wed, 24 Apr 2013 08:45:56 +0200
Local: Wed, Apr 24 2013 2:45 am
Subject: Re: [Neo4j] Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Which neo4j versions are you using?

Did you change any config?

Could you share the code which causes it to hang?

Thanks a lot!

Michael

Sent from mobile device

Am 24.04.2013 um 08:21 schrieb vyadav <viya...@gmail.com>:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
vyadav  
View profile  
 More options Apr 24, 2:51 am
From: vyadav <viya...@gmail.com>
Date: Tue, 23 Apr 2013 23:51:08 -0700 (PDT)
Local: Wed, Apr 24 2013 2:51 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Neo4j : 1.7
Yes, we increased the thread count per core to 25.
Code is something similar to snapshot which Khushbu shared.

-Vikas Y


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 24, 2:55 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Tue, 23 Apr 2013 23:55:56 -0700 (PDT)
Local: Wed, Apr 24 2013 2:55 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Hi,

I also made changes in thread configuration. Also, when neo4j hangs, if
REST URL of Neo4j is requested , it responds with 502 HTTP status code.

The messages.log does not show any error. Also java.exe is running with
optimal memory and CPU is at 100%.

But as soon as Neo4j is restarted it starts working. It is becoming a
mystery as no error is found at Neo4j machine. We also tried profiling but
did not found anything significant for help.

Regards,
Khushbu


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Hunger  
View profile  
 More options Apr 24, 3:30 am
From: Michael Hunger <michael.hun...@neotechnology.com>
Date: Wed, 24 Apr 2013 09:30:35 +0200
Local: Wed, Apr 24 2013 3:30 am
Subject: Re: [Neo4j] Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

How many threads / parallel requests are running against this node update?

And can you produce a thread-dump of the server when that happens?

kill -3 pid

or
jstack pid

Thanks

Michael

Am 24.04.2013 um 08:51 schrieb vyadav:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 24, 4:36 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Wed, 24 Apr 2013 01:36:26 -0700 (PDT)
Local: Wed, Apr 24 2013 4:36 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Around 10 parallel request coming for any update request similar to
mentioned below.

We just faced the same issue again and took the thread profiler result from
newrelic. PFA the screenshot.
Also, after restarting neo4j, messages.log has non-clean shutdown entry
with "Internal recovery completed, scanned 118549 log entries. Recovered
6571 transactions".In neo4j admin console, found some blank nodes created
at the end.

Does it indicate any problem?

Regards,
Khushbu

  neo4j hang.png
17K Download

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 25, 1:36 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Wed, 24 Apr 2013 22:36:46 -0700 (PDT)
Local: Thurs, Apr 25 2013 1:36 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Any suggestions?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Hunger  
View profile  
 More options Apr 25, 1:57 am
From: Michael Hunger <michael.hun...@neotechnology.com>
Date: Thu, 25 Apr 2013 07:57:00 +0200
Subject: Re: [Neo4j] Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

I would still love to see the JVM thread-dump when the service is in that state.

kill -3 pid
or
jstack pid

Cheers

Michael

Am 25.04.2013 um 07:36 schrieb Khushbu Bhatewara:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 25, 5:33 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Thu, 25 Apr 2013 02:33:21 -0700 (PDT)
Local: Thurs, Apr 25 2013 5:33 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Hi Michael,

Neo4j got in hang state again. PFA threaddump of the process.

-Khushbu

  threaddump.log
331K Download

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 25, 11:46 pm
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Thu, 25 Apr 2013 20:46:11 -0700 (PDT)
Local: Thurs, Apr 25 2013 11:46 pm
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Any suggestions?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 26, 2:07 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Thu, 25 Apr 2013 23:07:57 -0700 (PDT)
Local: Fri, Apr 26 2013 2:07 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Today again neo4j hanged. PFA another threaddump.

-Khushbu

  threaddump1.log
300K Download

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 26, 4:55 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Fri, 26 Apr 2013 01:55:06 -0700 (PDT)
Local: Fri, Apr 26 2013 4:55 am
Subject: Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

This dump was taken while load testing for different functionality wherein
NEO4J Rest batch request is executed for creating node and adding index to
it. This operation does not use any gremlin query.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Hunger  
View profile  
 More options Apr 26, 9:35 am
From: Michael Hunger <michael.hun...@neotechnology.com>
Date: Fri, 26 Apr 2013 15:35:57 +0200
Local: Fri, Apr 26 2013 9:35 am
Subject: Re: [Neo4j] Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Thanks so much for those logs. Sorry for the late reply I'm at a conference all day up until Saturday but I immediately report it to the team.

Michael

Am 26.04.2013 um 10:55 schrieb Khushbu Bhatewara:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Hunger  
View profile  
 More options Apr 26, 9:46 am
From: Michael Hunger <michael.hun...@neotechnology.com>
Date: Fri, 26 Apr 2013 15:46:45 +0200
Local: Fri, Apr 26 2013 9:46 am
Subject: Re: [Neo4j] Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Can you please read thouroughly through your code?

From a quick look you are acquiring locks which you never release (b/c those objects only live in that scope) and locks that are not really released will not get released or gc'ed.

This object only lives in this block and still holds a write lock.

You should also release your lock in finally blocks.

And certainly much more.

Michael

>                    if(!OutEndNode.hasProperty('Recepients'))
>                    {
>                            Lock userLockObj = tx.acquireWriteLock(OutEndNode);
>                    };

Am 19.04.2013 um 17:36 schrieb Khushbu Bhatewara:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Khushbu Bhatewara  
View profile  
 More options Apr 26, 9:49 am
From: Khushbu Bhatewara <khushbu.bhatew...@gmail.com>
Date: Fri, 26 Apr 2013 19:19:38 +0530
Local: Fri, Apr 26 2013 9:49 am
Subject: Re: [Neo4j] Re: Neo4j REST service stops working
Print | Individual message | Show original | Report this message | Find messages by this author

Thanks Michael. Looking forward for your response.

-Khushbu
On 26-Apr-2013 7:06 PM, "Michael Hunger" <michael.hun...@neotechnology.com>
wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Messages 1 - 25 of 29   Newer >
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google