ArangoDB read requests getting limited to a certain number

[Yash Bansal]

I'm using Arango in standalone mode(single node), with machine specs of 16gb RAM and 8 cores. No other process is running on the machine. I have some 4000 collections each having 15000 documents stored in it. My typical document is either a vertex (containing lat-long data), or an edge containing document hashes for two vertexes it connects.

I am doing some load testing where I have to see what is the read performance arango can give. For the load testing, I have taken 4 EC2 instances, and I'm running read-only geospatial-queries on the arango node (queries like nearest node to given lat-long). The queries are being made using go-clients with proper connection pooling.

Problem : Even using the 4 load testing machines, the max number of requests which shows up on arango _system-monitoring UI caps at 4000requests/sec. Even if I reduce the number of loadTesting machines, the number stays at 3000-4000, indicating that I'm indeed generating higher amounts of traffic from the machines.

My Arango instance CPU usage sticks at 100% all the time. Memory usage caps at 13-14GB(out of 16GB). The average request served time shown at arango is 20ms(which is acceptable given the un-optimised query). There are no network/disk restrictions (confirmed using the AWS network/disk monitoring).

Questions :

1. Can I get the stats on internal queue in arango, where it queues the message when all the CPUs/threads are busy. That will atleast tell me that I'm doing the loadTesting correctly.
2. If you see the screenshot, half of the requests are turning as writes, whereas my queries are strictly read-only. Does arango writes something in the background for any read query?
3. Hypothetically, if my arango didn't take 20ms to complete and just completed within a micro-second, will I be able to increase the number of incoming requests?
4. Will increasing number of CPU increase the number of incoming requests?

[Wilfried Gösgens]

Hi,

Please note that 3.7.2 which you appear to be using in your tests is EOL and thus no longer supported.

Later versions of ArangoDB come with prometheus integration; you should use that for system monitoring.

In this blog series I explain in which figures one could be interested in (though its using collectd for the gathering instead of prometheus):

https://www.arangodb.com/2017/10/performance-analysis-pyarango-usage-scenarios/

Since in the end everything boils down to disk I/O, This part of the documentation explains the way the Rocksdb storage engine block-cache behaves in that regard:

https://www.arangodb.com/docs/stable/administration-reduce-memory-footprint.html#linux-system-configuration

Please note that later versions of arangodb come with different capabilities to run geo queries:

https://www.arangodb.com/docs/stable/arangosearch-geospatial-search.html

If you run a cluster to increase disk I/O,  the capability to as well read from replication followers may be interesting.

Above resources will help you to identify the bottle necks of your installation.