Query performance issues

[A. Hébert]

Hi senx team,

    We have a single application with 2 or 3 single class names and each of them have between 4 and 5 millions unique time series. They have simply 3 labels. This makes query a single Time series slow on the user side: ~5s for each FIND statement (which of course is still fast when searching on 5 millions series). How could we speed up those kind of use cases on Warp 10?

Best,

[Mathias Herberts]

If you know the values associated with those labels for the GTS you wish to fetch you can use the 'gts' map key in the FETCH parameter map to specify an explicit list of GTS to fetch instead of performing a prior search on Directory.

[A. Hébert]

I just tried that and got the following exeption `ERROR line #8: Exception at '=>FETCH<=' in section [TOP]`

[Mathias Herberts]

There was an issue with key initialization which had been moved too early. Fixed on master.

[A. Hébert]

I have tried again a simple FETCH based on a FIND result with the current Warp10 master version and get an empty list with the 'gts' key. When I do the same fetch with the selectors I do get my results.

[Mathias Herberts]

Does your token have a single app/owner/producer?

The FETCH will inject .owner, .producer and .app labels from the value in the READ token provided.

[A. Hébert]

On my knowledge, our token are forged only on a single application base. As example with TOKENINFO, I get:

[{"application":"unicorn","issuance":1569....,"expiry":1591...,"type":"READ","apps":["unicorn"]}]

[Mathias Herberts]

Are you setting all the labels present in the GTS?

[A. Hébert]

All the gts in the 'gts' comes directly from a FIND statement

[Mathias Herberts]

What Warp 10 revision is the problem occurring on?

[A. Hébert]

I am on the actual master with last commit beeing: https://github.com/senx/warp10-platform/commit/7d3f57867fe269df1fd00dd3125c0d092d63bb61

[A. Hébert]

I do have the same behavior on both standalone as well as on a distributed setup.

--
You received this message because you are subscribed to a topic in the Google Groups "Warp 10 users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/warp10-users/Du8YZ8zHJNM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to warp10-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp10-users/66fa13a6-8655-47b2-8c76-2bbf0a8a76d9%40googlegroups.com.

--

Aurélien HÉBERT

[Mathias Herberts]

Are you performing the FETCH with the same token used for the FIND?

To unsubscribe from this group and all its topics, send an email to warp10-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/warp10-users/66fa13a6-8655-47b2-8c76-2bbf0a8a76d9%40googlegroups.com.

--

Aurélien HÉBERT

[A. Hébert]

Yes, of course. As soon as I find some time, I will provide more tests and descriptions here :). I use Worf to generate both standalone and Distributed tokens.

To unsubscribe from this group and all its topics, send an email to warp10...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/warp10-users/66fa13a6-8655-47b2-8c76-2bbf0a8a76d9%40googlegroups.com.

--

Aurélien HÉBERT

[A. Hébert]

I find the answer about this empty list. In our READ TOKEN, we do not have any "producer" labels set. However it's well set in our series labels stored. Applying this small change (https://github.com/senx/warp10-platform/compare/master...aurrelhebert:fix/gts/fetch?expand=1) fixes it.

[Mathias Herberts]

Did you see an improvement in terms of performance?

[A. Hébert]

For sure on a batch of hundred queries an average of 150ms, the best resolved in only 28ms, and the worst in 1s

[A. Hébert]

Hello Mathias,

I am back again on this story of performance for queries due to the Index size.

First of all, I wanted to say that in many use-cases the FETCH with the "gts" key improve a lot of query in our side.

However we still have massive slow queries, for example when we want to compute a FIND on a specific label on one of our most large application. The selector query is '~.*{id=42}' and this queries takes more than 70 seconds (server side). The FINDSTATS provides the following result "{"classes.estimate":14,"gts.estimate":14,"labelnames.estimate":13,"labelvalues.estimate":12,"error.rate":0.008125}".

Even when setting one series name on the selector 'test{id=42}', we still get series than takes more than 10 seconds to be executed (server side).

I would be to help to improve those queries. What can I look like to start with?

[Mathias Herberts]

Hi,

assuming you know the classes of those 14 series you want to access, you could very well use the same approach using the 'gts' parameter with a list of GTS you build manually, i.e.:

'gts' [

NEWGTS 'class-1' RENAME { 'id' '42' } RELABEL

NEWGTS 'class-2' RENAME { 'id' '42' } RELABEL

..

]

Of course you need to specify the full set of labels so this approach will only work if you know what the labels are.

Another approach, assuming you do not know what the labels are but you know the class names, you could select the GTS using a single class name then use the 'extra' parameter to specify the other 13 classes.

[A. Hébert]

Thanks for your proposal. Do you have an example for the 'extra' parameter? I couldn't make it work (only one series as result) on a Warp10 2.5.

As a matter of fact this time I am really in the second case (I can't know the full set of labels only the id) and I really would like to be able to reduce the 10 seconds to FIND a single series.

[A. Hébert]

I do apologise, I run my tests to fast, the 'extra' parameter isn't working with a FIND statement only with FETCH. I couldn't really reduce by a lot using this method (from 70 - 80 seconds) to (40 - 50 seconds) to retrieve the 14 last values of those series. I had to split my statement in 3 or 4 FETCH, as the labels keys are different for several classes.

[Mathias Herberts]

Can you provide the directory log lines related to the searches you are performing? They should look like:

Search returned AAA results in BBB ms, inspected CCC metadatas in DDD classes (EEE matched) and performed FFF comparisons

[A. Hébert]

I can't really provides those lines with certainty... We are using HA sharded directories. I could find an example of what may like a slow query:

Search returned 14 results in 25218.834073 ms, inspected 435067 metadatas in 1 classes (1 matched) and performed 1306154 comparisons.

For information in all the sharded directory logs, I didn't found a log with 14 results [ ... ] in 14 classes or with 14 matched neither.

[Mathias Herberts]

1.3M comparisons on 435k metadata usually does not take 25s, this is a typical log in one of our own systems:

Search returned 27 results in 183.488054 ms, inspected 182304 metadatas in 1 classes (1 matched) and performed 546912 comparisons

what is the load average on your Directory? What is the time a /find request actually takes (the path is slightly different so result may give some additional infos).

[A. Hébert]

[A. Hébert]

We do have fast queries on our indexes too (on average our find queries took 150 to 200ms). Our directories are not too overloaded (between 40 and 60 % of cpu load). In our use-cases we do have specific slow queries on high class names cardinalities. For instance if 20/30 accounts pushed the same class more than 1 millions times (ex: my.cpu, my.mem, ...). Then even a new account pushing this specific series is impacted by slow index queries too.

I was more searching for the root cause of this behaviour / an hint on how to improve those data retrieval.

For information with the /find request it took the same time (real: 1m13.770s)

[Mathias Herberts]

You can try fetching directly from one of your Directory using the class at https://gist.github.com/hbs/30bf6921b5145e0474adae3273027ef3

It is called like this:

java -cp warp10/build/libs/warp10-x.y.z.jar:. DirectoryStreamClient "{ 'urls' [ 'http://HOST:PORT/directory-streaming' ] 'psk' 'HEX ENCODED PRESHARED KEY' 'class' [ 'CLASS_SELECTOR' ] 'labels' [ { /* LABEL SELECTOR */ } ] }"

It will output the list of fetched GTS (their Metadata) and the time taken for the request in ns.

This should help you determine if the Directory instance itself is the bottleneck of something else down the line.

[A. Hébert]

I run this test and I have the following result:

- Retrieving 1 series takes 6269551620 ns (6 s)
- Retrieving all 14 series takes 197156864266 ns (3m 17s), I had to use a regular expression as I couldn't specified several classes (may be that's why it's a bit slower that previous find queries).

However I didn't get 1 or 14 series as result but 3 and 42.

This test leads me to think that the bottleneck is related to the directory component or at least our directories architecture. In production we have:
- 6 directories
- 3 replicas
- 2 shards per replica

- Between 210 and 230 millions of series per directory

[Mathias Herberts]

You can specify several selectors, 'class' and 'labels' are lists, see definition of DirectoryRequest thrift structure.

Can you share the number of comparisons which were performed for each request?

[A. Hébert]

How can I get such an informations? I would say that at least 3 applications have more than 1 millions occurrences of those specifics classes.

[Mathias Herberts]

The log output should appear in the directory logs.

Having an idea of the number of comparisons and the actual time spent should guide you towards a possible explanation of the slowness you experience, either lack of resources or actual huge number of comparisons.

[A. Hébert]

As we do not experience directories latencies (avg answer time is 200ms), I would say it's related to the number of comparisons. Our directories are too chatty (around 500 requests per seconds per directory) to find the specific related query output in it's logs.

[Mathias Herberts]

The log extract you gave a few weeks ago does not support the assumption that the culprit is the number of comparisons, if it is still 1.3M then they should not take 300s.

Unfortunately without more infos from the logs diagnosing the issue will be hard.