Rexster REST API stops responding

[Quoc Lam]

Hi,

I have just downloaded the titan-server 0.4.2 and have it installed on a server. I was able to start the titan/cassandra/rexster server by running the script "bin/titan.sh start" as per the instruction. I was able to create graph, add vertex/edge through gremlin console as well as retrieve the graph data through rexster REST api normally. However, after around 5 minutes, the rexster REST api (all api) stops responding. I can still access the titan graph data normally through gremlin console and rexster console. The only issue is the Rexster REST API stops responding. I have checked the log at log/rexstitan.log and did not find any interesting information on to why the Rexster REST API stops responding.

Is there anyone encountering the same issue? Is there any other log I should look at to troubleshoot this?

Thanks,

Quoc

[Stephen Mallette]

Can you describe you problem some more please?  What kinds of requests are you making (e.g. standard REST API, requests to the Gremlin Extension, etc.) during those 5 minutes?  Do your REST requests timeout on the client side...what kind of error do you get back after it stops?  Are you making lots of concurrent requests from multiple clients or something less complex?

Thanks,

Stephen

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Quoc Lam]

Hi Stephen,

During those 5 minutes, I just issued a few requests to the Rexster REST API like /graphs, /graphs/graph, /graphs/graph/vertices. There were no concurrent requests. The requests were only issued from one client.

After around 5 minutes, the Rexster REST API stops responding. It doesn't timeout nor return any error. The request seemed to be running forever (at least I had waited for more than 5 minutes and the API was still not responded). Note that the issue should not be caused by large data as there were only a few vertices and edges.

There is one interesting note though, that issue seems only happens on the server running smartos. I have setup another server running ubuntu os, have titan installed there and it seems the titan/rexster run well on the ubuntu. Maybe the issue is due to the incompatibility between titan/rexster and smartos.

Regards,

Quoc

[Stephen Mallette]

I've heard of problems before with smartos.  Refer to the longest commented issue in TinkerPop history:

https://github.com/tinkerpop/rexster/issues/271

Rexster and smartos just don't seem to work well together and I've never been able to figure out why.  If you have new information not presented in that issue, that you think might be helpful in uncovering the problem please feel free to post it there.  We can continue this discussion in that issue.

Thanks,

Stephen 

[Shagun Sodhani]

I am having similar issues. I am inserting 80 vertices in my graph then checking each pair for a condition, and if condition is met, then I am inserting an edge between the nodes. I am using ubuntu 14.04 - 32-bit, though I tested this one a 64-bit machine as well. I was initially using bulbflow on top of Rexster. Then I tried with requests as well using the Rexster Rest Api as well. In my case the python program, which is making the requests to api, stops responding though I can still curl to the endpoints

[Stephen Mallette]

> python program, which is making the requests to api, stops responding though I can still curl to the endpoints

I'm not sure how to explain that one.  Rexster must be ok, if you can still curl the endpoints and get responses.  Doesn't that indicate something wrong with your program and not Rexster?

On Tue, Jan 6, 2015 at 5:52 AM, Shagun Sodhani <sshagun...@gmail.com> wrote:

I am having similar issues. I am inserting 80 vertices in my graph then checking each pair for a condition, and if condition is met, then I am inserting an edge between the nodes. I am using ubuntu 14.04 - 32-bit, though I tested this one a 64-bit machine as well. I was initially using bulbflow on top of Rexster. Then I tried with requests as well using the Rexster Rest Api as well. In my case the python program, which is making the requests to api, stops responding though I can still curl to the endpoints

--

You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/40283b55-97f1-4dc3-a13b-3e5eb99d2ba9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dmill]

How are your RAM/SWAP levels doing during these issues? I've experienced lockdowns as well when the machine running rexster ran out of memory. Thought it may be unrelated. The lockdown generally concerns a single keyspace and other keyspaces are still available (using titan).

[Waterloogarden]

Hi, Stephen,

I have the similar problem with Titan 0.4.4 server, the Rexster runs fine with the first 20min or about one hour (it seems random), and then stop responding. Even, curl server:8182, just no response. I checked the rexster.log, no any response, the server seems not receive the call. We have to reboot the rexster server, and then everything is back normal. The Titan server 0.4.4 runs perfectly on Google Compute before, and recently we migrate our data to IBM Softlayer, and the problem happened. Do you have some idea, the problem might be related to hardware issues?

Thank you very much!

Wei

[Stephen Mallette]

So you just moved it from one environment to another and it started behaving this way?  Sounds semi-familiar.  There was an old issue on Rexster about 200 comments long with different people saying they would have problems with Rexster on just certain operating systems.  They'd go from having a great time with Rexster on ubuntu to a big fat stack of problems on redhat.  There was a fair bit of collaboration on that issue but I don't think any of us ever really solved the problem (some got happy and others didn't).  And of course, i never could recreate it even on the OS's that people said they had trouble on.  I seem to remember that redhat liked to have a massive amount of worker threads configured on Rexster and then it would behave (like 256 or something like that), but even that didn't work for everyone.

anyway, I pretty much attributed that problem to grizzly and gave up trying to fix it as TinkerPop 3 was in play and we'd already moved forward with leaving grizzly behind for netty.  Sorry that isn't much help, but it's the best information i have.

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/d388995b-0b08-4841-9dab-ce037c37b90d%40googlegroups.com.

[Kevin Schmidt]

FWIW, I've seen similar behavior of Rexster not responding after a period of time running on AIX, but have not seen issues on Ubuntu, CentOS, OS X, or Windows.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/CAA-H438aZ0pfprzdf3wp3g6jHgdoV4aJRypOVU0sPexfVO%2BPzA%40mail.gmail.com.

[Stephen Mallette]

I've heard tales of CentOS acting this way for some folks on some versions - i've tested it myself though and never had a problem. 

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/CAJFiCb-xrvHeYF%2B9knrvKw3%2ByBP%2BtT9X0Wj1bn5soMqe%2BZgv-Q%40mail.gmail.com.

[Waterloogarden]

Hi, Stephen, Thank you!

I tried several different servers with Ubuntu, CentOs and Debian over IBM Softlayer,  all have the exact same problems. With the exact same configuration, I also tried with Ubuntu, CentOS and Debian over Google Compute and Racksapce respectively, and the Titan server runs perfectly in our product. So, for me, either I will have to switch back to Google or Racksapce or I will use Titan-server 1.0. For the latter, I will have to modify my restful API request according to the Tinkerpop 3 gremlin Restful API server, which seems not well be documented.  Since I use a lot of restful API queries to Titan server, do you have some suggestion about the Tinkerpop 3 gremlin Restful API? Right now the only source I found useful is http://tinkerpop.apache.org/docs/3.1.1-incubating/reference/#_connecting_via_rest.

Wei

[Stephen Mallette]

Gremlin Server only has one route - a Gremlin script endpoint.  There is no other REST API.  We found that even applications of minimum complexity almost always graduated away from the Rexster REST API and therefore, that fairly massive chunk of code was just a maintenance hole. It was dropped as something we would do for Gremlin Server pretty early on.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/9f864654-cdad-430b-853c-e460cb30c1e9%40googlegroups.com.

[Waterloogarden]

Hi, Stephen,

Here the restful API I mean is the Gremlin endpoint. I wonder whether scripts running through this endpoint are the same as that running through the Gremlin console, because sometimes I found that some scripts don't work on the endpoint while no problem over the console. 

[Stephen Mallette]

You would have to provide an example of what works in one place but not the other.  Certainly you can't send console command in a script to the server like :install or :plugin or anything like that, but most other scripts that work in the console should also work on the server.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/8d607070-cdd7-4594-9e24-cad9c3c5d2b1%40googlegroups.com.