OOM and high CPU core usage

[Денис Черемисов]

Hi!

We have a set entities with 1 million elements, where each one is bound to ≈2000 elements of set groups, which consists of ≈110000 elements. Thus we have about 2 billion connections of entities and groups via members edge collection.

Group can be a subgroup of another group (and each does, except group `all`) via children edge collection with a length ≈300000 elements. It is also possible "subgroup" relation to form loop, i.e. A < B < C < D yet D < A, where `<` means `is subgroup of`. Amount of loops is fixed and equals 8000 or so.

The task is to get all groups certain entity belongs to: We made a query

FOR group IN 1..10000 INBOUND "entities/417641" children, members OPTIONS {uniqueVertices: "path", bfs: true}
  RETURN group

The upper limit is to cover loop lengths which are all elements 8000. 

It works really long, takes 100% of CPU core and eventually fails with OOM on Arango side.

`members` is large with its 2.000.000.000 elements, I understand, but `children` is just 300.000 items long. It is just O(410.000) algorithmic complexity.

What is wrong with our query?