Cluster Creation in Janus - Using Node JS

[Shanmugapriya M]

Hi , 

We are trying to create Janus cluster. We are following the below link for creating Janus cluster- https://tinkerpop.apache.org/docs/current/reference/#sparql-gremlin

This documentation has only details on Java & Groovy. However we are using Node JS as our application language . Could you please guide us how we can create and use clusters using Node JS.

Adding More information - We are reading data from Big query and using Node JS for the business logic and using Janus graph and Big table to store the data and gremlin visualizer to visualize the data. For improvising the performance , we wanted to create Janus cluster , but we dont find any documentation related to that. Kindly guide us 

[HadoopMarc]

Hi,

BigTable support is confirmed here:

https://docs.janusgraph.org/changelog/

Deployment options are found here:

https://docs.janusgraph.org/operations/deployment/

Best wishes,    Marc

Op maandag 8 november 2021 om 14:43:26 UTC+1 schreef sppri...@gmail.com:

[Shanmugapriya M]

Thank you so much .We found the second link useful and we were referring to it . Could you please guide us the node JS libraries that has the features to support the clusters?

[HadoopMarc]

Information about using javascript to connect to Gremlin Server (also the Gremlin Server shipped with JanusGraph):

http://tinkerpop.apache.org/docs/current/reference/#gremlin-javascript

Marc

Op dinsdag 9 november 2021 om 06:06:53 UTC+1 schreef sppri...@gmail.com:

[Shanmugapriya M]

Hi Marc, 

Thank you ! This is exactly what we are using for single Janus graph. 

The point we are stuck with is -clustering of Janus graph. The idea is to scale up the current architecture with multiple JanusGraph servers to speed up the execution. We have three google VM instances where we have have configured the same JanusGraph server setup. Now, how can I distribute the data to these three JanusGraph Servers.We are using "gremlin" npm module, but it doesn't have the option to connect with all three servers at the same time.

[Shanmugapriya M]

Just to add , in the method you shared , how do we add  more than 1 server?

[HadoopMarc]

It is surprising, indeed, that load balancing over Gremlin Server instances never ended up in the JanusGraph documentation. The user list thread below, more or less, covers the subject:

https://lists.lfaidata.foundation/g/janusgraph-users/topic/79936014#2876

Note that the gremlin npm module uses websockets for connecting, so load balancing can only be so granular as a websocket connection.

Best wishes,    Marc

Op dinsdag 9 november 2021 om 11:45:17 UTC+1 schreef sppri...@gmail.com:

[Shanmugapriya M]

Hi, 

We followed the suggestion provided ,as mentioned we tried to distribute the load between 2 Janus servers in 2 different VM's both pointing to same big table. We are using Janus  server 1  to create dynamically the graph and the other 2 servers are updating data into it . Despite pointing to same big table the other 2 servers are not able to access the graph 

[HadoopMarc]

Hi, how does the stacktrace for your connection attempt to BigTable look like?

Marc

Op woensdag 10 november 2021 om 11:22:33 UTC+1 schreef sppri...@gmail.com:

[Shanmugapriya M]

Thank you for the patience and help , i would share the stack trace but have another query now :-) ..

When we create the graph using one Janus server and create vertices and edges using the other .We observe that write operations are happening. But when we try to visualize the graph using the Gremlin visualizer from the second server it says the graph does not exist , while the first server it is able to read the graph.When we restart the server for the second one then try to visualize the graph then we are able to view it. 

So, the point is, we assume that graph is being rendered from the in-house memory of Janus server and not from the big table. Could you please advise how do we make the Janus server refer to the big table while reading the graph instead of the in-house memory 

[HadoopMarc]

On first sight I do not see a reason how you could experience this behavior:

• using gremlin-javascript with Gremlin Server, writing transactions are committed automatically and immediately, so new data should be visible to other JanusGraph instances
• Indeed, JanusGraph uses caching but this is at the level of vertex ids. If you do a full graph scan (g.V()), the backend is queried without using the cache. Even if you ask for specific vertices using an indexed property, the indexing and storage backend are accessed again if a vertex is not in the cache.

So it does not make sense to me how a restart of the reading server could make a difference. Even in a clean sheet situation with a cluster being spun up, the reading server would create a graph it there was none and the writing server would have to use it, if it came later. Strange things could maybe happen, if the writing server comes second and creates a schema on the graph not seen by the reading server?

Things to check anyway are:

• do the writing and the reading server really have the same configs? In particular, check the graph names  and indexing backend configs.
• ...

Marc

Op donderdag 11 november 2021 om 11:27:20 UTC+1 schreef sppri...@gmail.com:

[Shanmugapriya M]

Hi , 

We were able to cross the above issue removing the in-house memory from the configurations and also using configured graph factory command , but we are facing different issue now .Require your guidance here 

We tested the cluster portion with 3 Janus graph servers with same configurations. All three are hosted on Client VMs bound to one load balancer (nginx). Also, we deployed our NodeJS server with nginx and when we start the process after 3-4 batches it throws this error:-

/home/user_ak/nodejs_code/node_modules/ws/lib/websocket.js:270
throw err;
Error: WebSocket is not open: readyState 2 (CLOSING)

at WebSocket.ping (/home/user_ak/nodejs_code/node_modules/ws/lib/websocket.js:264:19)

at Timeout._onTimeout (/home/user_ak/nodejs_code/node_modules/gremlin/lib/driver/connection.js:229:16)
at listOnTimeout (internal/timers.js:557:17)
at processTimers (internal/timers.js:500:7)
[nodemon] app crashed - waiting for file changes before starting...