why using unlimited LinkedBlockingQueue to receive org.apache.tinkerpop.gremlin.driver.Result

[jeremy...@gmail.com]

when I try to query the sql like "g.V()" which will return super big results and print every result,  OOM happen every time. Then I find that the LinkedBlockingQueue<Result> has lots of items. Does it make OOM?

The related code is located at org.apache.tinkerpop.gremlin.driver.Connection 214.

Thank you

[Stephen Mallette]

I suppose you could get an OOM if you did g.V() then realized all the results in memory on your client or perhaps didn't consume the results fast enough somehow (which would help fill the LinkedBlockingQueue). If you're just printing results it seems surprising that you would not be able to print as fast as the results arrived, so I'm not sure why that queue would fill up.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/43627af5-2984-40e5-81ea-7fae5dd5e9a2n%40googlegroups.com.

[jeremy...@gmail.com]

Thanks for your reply. I'm sure it's the print causes an OOM happened. I just wonder know  if there are other ways can replace the unlimited LinkedBlockingQueue. Because I don't think it's a good way using the unlimited LinkedBlockingQueue. Thank you again.

[Stephen Mallette]

Could you please show the code that is producing this problem? 

> Because I don't think it's a good way using the unlimited LinkedBlockingQueue.

What are you suggesting? Should it be changed to a bounded queue of some sort? Something else?

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/f3f7f4c6-a71a-4cdd-afad-9eba9297108fn%40googlegroups.com.

[jeremy...@gmail.com]

Why I don't think it's a good way just because it can cause OOM. I've tried to change it to be a bounded queue. And it's slower than the raw way. Besides, I can't find more ways to replace that for now. Here is the code and the pictures of memory analysis when OOM happened.

[Stephen Mallette]

Sorry - when I asked for code, I was asking to see the code that you were executing that produces the OOM. Could you please include that?

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/54b356a8-c531-4685-835e-07878a90e664n%40googlegroups.com.

[jeremy...@gmail.com]

yes, I just executed g.V().valueMap(true), and I had about 20 million vertices. I gave JVM arguments -Xms512m -Xmx2g.

[Stephen Mallette]

There are lots of ways to execute g.V().valueMap(true) - i think you said you were printing them out. Could you please provide a complete code example?

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/dfd62171-1208-4fca-994a-12ba3d624670n%40googlegroups.com.

[jeremy...@gmail.com]

sure, here you are.

ResultSet resultSet = client.submit("g.V().valueMap(true)");

Iterator<Result> iterator;

if (iterator == null || !iterator.hasNext())

{

List<Result> results = null;

try

{

results = resultSet.some(10).join();

}

iterator = results.iterator();

if (iterator.hasNext())

{

System.out.println(iterator.next());

[jeremy...@gmail.com]

Is there something found?

[Stephen Mallette]

So, I guess it is as I said earlier - the results may not be consumed from the queue at the rate they come in which can lead to an OOME. I didn't realize how fast it filled with a simple println though. Interesting. For now, I can only suggest that you increase your -Xmx to support both the queue size and results you expect to hold in memory. 

I did a rough hack at bounding the queue to 100,000 and it nicely processed about a billion results with -Xmx512m and no OOME. I'm not sure that my "rough hack" is the "right" way to go though - created an issue for further discussion:

https://issues.apache.org/jira/browse/TINKERPOP-2424

On Mon, Sep 14, 2020 at 11:39 PM jeremy...@gmail.com <jeremy...@gmail.com> wrote:

Is there something found?

在2020年9月10日星期四 UTC+8 上午11:20:33<jeremy...@gmail.com> 写道：

sure, here you are.

ResultSet resultSet = client.submit("g.V().valueMap(true)");

Iterator<Result> iterator;

if (iterator == null || !iterator.hasNext())

{

List<Result> results = null;

try

{

results = resultSet.some(10).join();

}

iterator = results.iterator();

if (iterator.hasNext())

{

System.out.println(iterator.next());

}

}

在2020年9月9日星期三 UTC+8 下午6:25:55<spmal...@gmail.com> 写道：

There are lots of ways to execute g.V().valueMap(true) - i think you said you were printing them out. Could you please provide a complete code example?

On Tue, Sep 8, 2020 at 10:19 PM jeremy...@gmail.com <jeremy...@gmail.com> wrote:

yes, I just executed g.V().valueMap(true), and I had about 20 million vertices. I gave JVM arguments -Xms512m -Xmx2g.

在2020年9月8日星期二 UTC+8 下午6:34:43<spmal...@gmail.com> 写道：

Sorry - when I asked for code, I was asking to see the code that you were executing that produces the OOM. Could you please include that?

On Tue, Sep 8, 2020 at 3:33 AM jeremy...@gmail.com <jeremy...@gmail.com> wrote:

Why I don't think it's a good way just because it can cause OOM. I've tried to change it to be a bounded queue. And it's slower than the raw way. Besides, I can't find more ways to replace that for now. Here is the code and the pictures of memory analysis when OOM happened.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/d1702a41-04da-4c33-8ba9-05008237c2f8n%40googlegroups.com.

[jeremy...@gmail.com]

If Using a bounded queue, the followed code in org.apache.tinkerpop.gremlin.driver.ResultQueue should be noticed. Because offer() will abandon items when the queue is full.

public void add(final Result result) {

        this.resultLinkedBlockingQueue.offer(result);

        tryDrainNextWaiting(false);

    }

[Stephen Mallette]

agreed - that's why my hack (shown in the diff on the JIRA issue) puts offer() in a while loop :/

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/15384bda-6799-4894-aece-4034d3ee894an%40googlegroups.com.