Parallelize multiple fetch requests in a single warpscript

[Alexandre Berthaud]

Hello everyone,

We have a warpscript where we do multiple requests which are completely independent:

{
  'cpu_fast' [
    "<TOKEN>"
    'fast_cpu.usage_idle'
    { 'cpu' 'cpu-total' 'app_id' '=app_fe5c3673-7574-4120-a116-74f1027f7264' 'host' '~(310531c5-1198-4dd1-9539-9165339664c3|6d7a7ebd-ce4d-40c3-8e62-1bd17e9c1cb9)' }
    NOW -199
    FETCH bucketizer.mean 0 1000000 0 ]                                                                  
          BUCKETIZE INTERPOLATE
  'mem' [
    "<TOKEN>"
    'mem.used'
    { 'app_id' '=app_fe5c3673-7574-4120-a116-74f1027f7264' 'host' '~(310531c5-1198-4dd1-9539-9165339664c3|6d7a7ebd-ce4d-40c3-8e62-1bd17e9c1cb9)' }
    NOW -199
    FETCH bucketizer.mean 0 1000000 0 ]
          BUCKETIZE INTERPOLATE
  'net' [        
    "<TOKEN>"
    '~net.(bytes_sent|bytes_recv)'
    { 'app_id' '=app_fe5c3673-7574-4120-a116-74f1027f7264' 'host' '~(310531c5-1198-4dd1-9539-9165339664c3|6d7a7ebd-ce4d-40c3-8e62-1bd17e9c1cb9)' }
    NOW -34
    FETCH
      mapper.rate 1 0 0
    ]  
    MAP INTERPOLATE
}

Given the performance we are getting (if we sum the runtime averages of each request we get the average of this warpscript), it's clear that those requests are run sequentially. Is there a way to make those run in parallel?

Thank you,

[Alexandre Berthaud]

Oh I guess we would have to use CEVAL for this.

[Pierre]

Hi, 

Did you try to fetch everything in a single instruction, then filter the result by classname ?  On a distributed backend, it will be faster (you asked one time directory, then the fetch is parallelized)

[ $fetchresult [] '~net.(bytes_sent|bytes_recv)' filter.byclass ] FILTER

beware : FETCH take a list : [ $token 'classname' { } x y ] FETCH . The signature you use is not supported anymore.

remark : you can write   1 s   instead of 1000000. 

remark : if you store millions of GTS, you can speed up the regexp search with REOPTALT function. try [ 'fast_cpu.usage_idle'  'mem.used' 'net.bytes_sent' 'net.bytes_recv' ] REOPTALT to generate the optimized regexp.

remark : = is not mandatory in the selector.

CEVAL is great for cpu intensive tasks, for example bucketize huge gts. Fetch is more i/o bound.

[Alexandre Berthaud]

Thanks for all this Pierre, we will try this out!

[Mathias Herberts]

Actually parallel scanners can also be configured for the standalone version if that is what you use.

[Alexandre Berthaud]

That's interesting to know but this is for a distributed cluster.