Run time error on larger volumes of data - running Gremlin queries in Jupyter notebook

[Neelima Tula]

Hi Team,

I am working with complex graph data and trying to count number of edges associated with 500k nodes.

I am using below query in Jupyter notebook and the data is loaded in Neptune cluster:

%%gremlin  
g.V().hasLabel('xyz').as('xydId').bothE('mno').label().as('mno').select('xyzId', 'mno').
groupCount().
    by('xyzId')

I get this below error:

{'error': RuntimeError("Received error on read: 'Message size 38529422 exceeds limit 10485760'")}

Any help/advice is really appreciated, I will have to do this on multiple different nodes with larger volumes.

Thanks.

This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions maybe unlawful. This e-mail may contain viruses. Provenir has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Provenir reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Provenir e-mail system.

[Kelvin Lawrence]

It looks like you are trying to return a massive amount of results (more than 38MB) . Right now the Jupyter notebooks are setup to use the Gremlin Python Client default maximum  of 10MB for the result set size.

That can be increased but would require patching the notebook runtime code for now.

Feel free to open a ticket in GitHub against the graph-notebook project to make that value configurable and we can easily make that change.

Cheers

Kelvin