Performance issues Warp10 distributed version

[H.Azelmat]

I have setup a cluster contaning 4 Hbase region servers and 4 data nodes, each node has 8 vCores 2.3 GHZ, I have sensors with TOP10 minutes frequency and over 3 years of data , i have tried to fetch :

- 3 years worth of data of  10 sensors which is : ~1.1M datapoints it tooks  3.2 s,  which is way slower than 500k points /s which i have seen in another post in this forum .

- 70k datapoints  in 200 ms.

So I was wondering  how can I improve my warp10 distributed version performance ?

I have setup egress configuration like this with the hbase filter enabled :

egress.hbase.data.blockcache.gts.threshold = 1024

egress.hbase.parallelscanners.poolsize = 8

egress.hbase.parallelscanners.maxinflightperrequest = 8

egress.hbase.parallelscanners.min.gts.perscanner = 1

egress.hbase.parallelscanners.max.parallel.scanners = 8

my warpscript request looks like this :

[


[
'READ'
'elec'
{ 'SID' '~(c11_5217d16f-9f8b-3f5c-8e36-395a58e2c29e|c12_d358ac49-115e-3b10-8945-1ff99d99cef0|c13_d358ac49-115e-3b10-8945-1ff99d99cef0|c11_d358ac49-115e-3b10-8945-1ff99d99cef0|c11_d358ac49-115e-3b10-8945-1ff99d99cef0|c14_d358ac49-115e-3b10-8945-1ff99d99cef0|c11_d358ac49-115e-3b10-8945-1ff99d99cef0|c12_5217d16f-9f8b-3f5c-8e36-395a58e2c29e|c13_5217d16f-9f8b-3f5c-8e36-395a58e2c29e|c14_5217d16f-9f8b-3f5c-8e36-395a58e2c29e)'


}
'2013-01-01T00:00:00.000Z'
'2016-12-31T23:59:59.999Z'
] FETCH


bucketizer.mean // bucketizer
0 // lastbucket
1000000 3600 * 24 *
0 // bucketcount
] BUCKETIZE // BUCKETIZE
'' 1 ->LIST
reducer.sum
3 ->LIST
REDUCE
[ SWAP 24 mapper.mul 0 0 0 ] MAP

In a request like this, do i get use of the parallelisation of Hbase scanners ? I have also read in this forum that I can use wrapped GTS to improve the fetch performance but I can't seem to find more information on how to use the pack and unpack macros in the warp10 documentation.

Any help would be great.

[David Morin]

Hi,

Here after the post of Mathias concerning the use of pack/unpack: https://groups.google.com/d/msg/warp10-users/iBhpcSC7n6g/FS1k8LhqAQAJ

These 2 macros (with comments) are provided in this post.

With these macros you will be able to pack GTS into chunks of specific width.

[H.Azelmat]

Yes  i have added those warpscripts file pack.mc2 and unpack.mc2 on the serverside i have tried to use the pack macros like this :

[

'READ'

'elec'

{ 'SID' '~(c23_92ecf29f-b3a3-3000-8cb9-41397f673810|c22_92ecf29f-b3a3-3000-8cb9-41397f673810)'

}

'2016-05-01T00:00:00.000Z'

'2016-06-30T23:59:59.999Z'

] FETCH

'gts' STORE

$gts 5 false @gts/pack

but i got a : LASTTICK expects a Geo Time Serie on top of the stack. any idea how can I fix this ?

still  can I improve my 370 k points / s in my warp10 cluster ?

[Aurélien Hébert]

Hello Hamza,

Concerning your WarpScript, the pack function used in Mathias post expects a single time series on top of the stack instead of a list. 

The function FETCH leave on the stack a GTS List.

To correct it just apply a simple FOREACH and it should work : 

```

$gts

<%

  5 false @gts/pack

%>

FOREACH

```

By the way in which time unit is your plat-form ? Because if you pack your GTS in chunk of 5 microseconds (default time unit), you won't earn a lot.

Aurélien

---

Aurélien Hébert
Developer at Cityzen Data

--
You received this message because you are subscribed to the Google Groups "Warp 10 users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp10-users+unsubscribe@googlegroups.com.
To post to this group, send email to warp10...@googlegroups.com.
Visit this group at https://groups.google.com/group/warp10-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/warp10-users/64bb6f58-df46-4d5c-90a4-4f2f2cefa48c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Aurélien Hébert]

Hi again,

Let assume you have some series stored in Warp10.

When you execute a Fetch directly on them you get some points that math a duration, or a precise number of ticks.

The pack methods is used to make a compromise between storing each points and accelerating a FETCH for a large amount of points.

By adding "a chunk" (the duration to look for points around the current tick - here in microsecond), you will "pack" (put together) all the points founded and wrap them. For example instead of having one tick for each seconds, with a chunk of 10 seconds I will have only one tick every 10 seconds that has a value containing a wrap of all the 10 ticks values.

And finally by applying the unpack function on those packed GTS, the result will be the exact same series.

To conclude by executing an UPDATE on the packed series on WarpScript, it will push them on Warp10. New series will be created as the label "packed" "1" as been added to those series. Then the fetch on those will probably takes less times (depends on the size of the chunk you choose). However you will loose precision on the data that you can fetched as you have packed their ticks.

On Monday, November 7, 2016 at 10:57:05 AM UTC+1, H.Azelmat wrote:

[H.Azelmat]

My working request ( for anyone intetersted  ) :

[

'READ'

'elec'

{ 'SID' '~(c11_5217d16f-9f8b-3f5c-8e36-395a58e2c29e|c12_d358ac49-115e-3b10-8945-1ff99d99cef0|c13_d358ac49-115e-3b10-8945-1ff99d99cef0|c11_d358ac49-115e-3b10-8945-1ff99d99cef0|c11_d358ac49-115e-3b10-8945-1ff99d99cef0|c14_d358ac49-115e-3b10-8945-1ff99d99cef0|c11_d358ac49-115e-3b10-8945-1ff99d99cef0|c12_5217d16f-9f8b-3f5c-8e36-395a58e2c29e|c13_5217d16f-9f8b-3f5c-8e36-395a58e2c29e|c14_5217d16f-9f8b-3f5c-8e36-395a58e2c29e)'

}

'2014-01-01T00:00:00.000Z'

'2016-12-31T23:59:59.999Z'

] FETCH

'gts' STORE

$gts

<%

   1000000 false @gts/pack

%>

FOREACH

 it's really fast, and it's even better when you understood the magic behind it thank you for your detailled explanation.

[H.Azelmat]

Another question i encoutered while updating my GTS , how can I get only the series that dosent contains packed '1' ?

Le lundi 7 novembre 2016 14:35:24 UTC+1, Aurélien Hébert a écrit :

[Mathias Herberts]

Hi,

with the quick and dirty pack.mc2 I've posted earlier you can't. I've made another version which modifies the class name instead of simply adding an extra label (which is a bad practice).

I'll post it later on. In the mean time simply modify pack.mc2 so the class name ends in ':packed' and you'll be fine.

[Mathias Herberts]

Have you checked the Sensision metrics to confirm that you were indeed using the parallel scanning capability of Warp 10 ?

Have you checked that your network links still have some slack? We've seen on several occasions that using parallel scanners could easily saturate network links on Region Servers if the retrieved data are part of the same region.

On Monday, November 7, 2016 at 10:57:05 AM UTC+1, H.Azelmat wrote:

[H.Azelmat]

Can you please elaborate on how can I modify the class name on the pack.mc2 file as Im not familiar yet with warpscript ?

in my sensision metrics  Ihave this :

1478780585653000// warp.script.mobius.sessions.scheduled{} 0
1478780585653000// warp.hbase.client.scanners.parallel{} 3
1478780585653000// warp.store.hbase.time.nanos{} 406635
1478780585653000// warp.script.ops{} 78
1478780585653000// warp.store.hbase.puts.committed{} 0
1478780585653000// warp.directory.owners{} 1
1478780585653000// warp.fetch.bytes.values.perowner{app=42424242-4242-4242-4242-424242424242,owner=42424242-4242-4242-4242-424242424242,consumer=00000000-0000-0000-0000-000000000000} 1080711
1478780585653000// warp.fetch.datapoints.perowner{app=42424242-4242-4242-4242-424242424242,owner=42424242-4242-4242-4242-424242424242,consumer=00000000-0000-0000-0000-000000000000} 360237
1478780585653000// warp.directory.gts{} 18087
1478780585653000// warp.store.aborts{} 932
1478780585653000// warp.hbase.knownregions{table=continuum} 3
1478780585653000// warp.store.hbase.commits{} 1
1478780585653000// warp.directory.streaming.results{} 3
1478780585653000// warp.script.jvm.freememory{} 113143016
1478780585653000// warp.directory.client.cache.changed{} 4
1478780585654000// warp.fetch.bytes.values{app=42424242-4242-4242-4242-424242424242,consumer=00000000-0000-0000-0000-000000000000} 1080711
1478780585654000// warp.store.kafka.count{} 828
1478780585654000// warp.hbase.client.scanners.parallel.mutex{} 3
1478780585654000// warp.hbase.client.cells{} 360240
1478780585654000// warp.directory.jvm.freememory{} 452812216
1478780585654000// warp.hbase.client.iterators{} 3
1478780585654000// warp.directory.streaming.requests{} 2
1478780585654000// warp.script.repository.macros{} 3
1478780585654000// warp.script.requests{} 2
1478780585654000// warp.directory.streaming.time.us{} 32336
1478780585654000// warp.fetch.bytes.keys.perowner{app=42424242-4242-4242-4242-424242424242,owner=42424242-4242-4242-4242-424242424242,consumer=00000000-0000-0000-0000-000000000000} 9366162
1478780585654000// warp.directory.hbase.commits{} 178
1478780585654000// warp.hbase.client.scanners{} 3

I think I'm using parallel scanning of warp10 . 

worker1,16020,1478468267786	87	30	145888m	141083mb	189627k	407797k
worker2,16020,1478468264253	86	16	42106m	34188mb	55519k	165336k
worker3,16020,1478468263547	87	19	52374m	48217mb	72253k	157113k
worker4,16020,1478468274711	85	15	40477m	40071mb	55025k	102456k

The data is not evenly distributed in the hbase region servers  

The VM's are linked with 100 MBps , so one the nodes is using almost 40% but it's nothing severe 

[Aurélien Hébert]

To rename your series, just modify your update script and it should work.

$gts

<%

 // Pack your GTS according to a specific Time

 1 m false @gts/pack

 // New formed GTS is on stack

 // Duplicate it

 DUP

 // GET the name of the GTS on the stack

 NAME

 // Add suffix to the GTS name

 '.packed' +

 // Rename the GTS (using the String on top and the GTS on the stacked)

 RENAME

 // Optionally to delete the unnecessary added label

 {

   'packed' ''

 }

 RELABEL

%>

FOREACH

// Then apply the UPDATE
UPDATE

Let me know if you have questions.

On Monday, November 7, 2016 at 10:57:05 AM UTC+1, H.Azelmat wrote:

[Mathias Herberts]

You cannot expect good performances with 100Mbps network links.

[H.Azelmat]

@Aurelien Hébert Thank you it's working

Intra Vrack link is 2000 Mbps ,I have now checked that properly .

[H.Azelmat]

I have ingested the same dataset in a standalone version in the same node hosting the warp10 distributed instance and i have around 240 ms for 200k points , which the distributed version (egress + directory only components ) serve in 580 ms, beside packing GTS which is really useful I was wondering why theres such a difference x2 in my case ? how can I improve egress performance .

[Mathias Herberts]

Probably your HDFS/HBase set up which is not optimal.

We can provide consultancy services to help you fine tune your setup.