How can I improve bulkLoad performance

[Mark]

I'm experiencing a high amount of spillage in the map jobs when doing a bulkLoad. What can I do to minimize this? Does the number of input files make a difference - should I try splitting my data across more files?

Thanks

[Mark]

Splitting data across more files doesnt seem to have any major impact. I now have one mapper per file, rather than a file being assigned across two mappers. I double io.sort.factor to 200 and io.sort.mb to 2000. That seems to have helped a little bit.

[Mark]

I've now noticed immediately after the load I have around 100 regions, following day regions have increased to over 500. I'm guessing this mismatch is suggestive of the presplits were less than optimal and I need to reduce the split-limit-size to increase the number of presplits???

[Adam Sotona]

Hi Mark,

to better understand your case it would be better to know some context (approximate size of the bulk loaded data, number of files, mappers, how many regions has the target table, did you use Halyard Presplit, etc...

Generally speaking if each mapper is distributing a lot of statements to many reducers/regions - it has to spill a lot and it is nothing wrong. You can fine tune some options based on your cluster configuration, however the performance impact might not be so significant.

Thanks,

Adam

-------- Původní zpráva --------

Od: Mark <mark...@gmail.com>

Datum: 15.11.18 1:38 (GMT+01:00)

Komu: Halyard Users <halyar...@googlegroups.com>

Předmět: How can I improve bulkLoad performance

I'm experiencing a high amount of spillage in the map jobs when doing a bulkLoad. What can I do to minimize this? Does the number of input files make a difference - should I try splitting my data across more files?

Thanks

--
You received this message because you are subscribed to the Google Groups "Halyard Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to halyard-user...@googlegroups.com.
To post to this group, send email to halyar...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/halyard-users/503c9cf1-6df2-47c0-9c82-ce5c606c3514%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adam Sotona]

Hi Mark,

optimal granularity of the source has significant impact on performance. Source data split into more fragments gives more flexibility in calculation of optimal number of mappers, better load balancing across mappers and prevention of running out of local disk space on particular mappers.

The same job does Halyard Presplit for reducers. Optimally presplit table regions can easily receive huge amount of data, without overloading of some particular regions, without running out of local disk space on particular reducers, with significant performance effect on shuffle phase, and without following HBace region compactions and splits.

Adjusting particular options is up to your discretion, default values are based on some experience, that may significantly differ between clusters, and between datasets bulk loaded.

Thanks,

Adam

-------- Původní zpráva --------

Od: Mark <mark...@gmail.com>

Datum: 16.11.18 14:49 (GMT+01:00)

Komu: Halyard Users <halyar...@googlegroups.com>

Předmět: Re: How can I improve bulkLoad performance

Splitting data across more files doesnt seem to have any major impact. I now have one mapper per file, rather than a file being assigned across two mappers. I double io.sort.factor to 200 and io.sort.mb to 2000. That seems to have helped a little bit.

--
You received this message because you are subscribed to the Google Groups "Halyard Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to halyard-user...@googlegroups.com.
To post to this group, send email to halyar...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/halyard-users/57c26b6c-3aa7-408d-af5f-a4bf3271e2d5%40googlegroups.com.

[Adam Sotona]

Hi Mark,

in a standard case, where the presplit has been executed with the same data and same options as bulk load, and Hbase is not altering the presplits and their region size limits, you should not need to play with the split limit size and you should not observe any compactions and post-splits after the bulk load.

I can imagine a case where your Hbase is configured with much lower region size limit, so the presplit-requested table region size limit is ignored - then you would need to adjust the split limit according to you Hbase limit as you mention.

Thanks,

Adam

-------- Původní zpráva --------

Od: Mark <mark...@gmail.com>

Datum: 17.11.18 11:30 (GMT+01:00)

Komu: Halyard Users <halyar...@googlegroups.com>

Předmět: Re: How can I improve bulkLoad performance

I've now noticed immediately after the load I have around 100 regions, following day regions have increased to over 500. I'm guessing this mismatch is suggestive of the presplits were less than optimal and I need to reduce the split-limit-size to increase the number of presplits???

--

You received this message because you are subscribed to the Google Groups "Halyard Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to halyard-user...@googlegroups.com.
To post to this group, send email to halyar...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/halyard-users/894a6b99-b97f-408b-97f0-ca0d397855f7%40googlegroups.com.

[Mark]

Hi Adam,

Here are the details. According to the stats, I have 2865682721 triples currently split over 20 nt.gz files of about 1.3gb (compressed). I have a cluster of 5 nodes. I run presplit, then bulkload and this creates 20 mappers which takes up about 2/3 of my yarn resources. My min container size is 2gb, so I've modified halyard-defaults.xml to make the most and use 4gb for mappers and 6gb for reducers. hfile size is the default of 10gb. hadoop block size is 128mb. How can I "encourage" presplit/bulkload to create more mappers so at least I make full use of my yarn resources? Would I benefit from having a "tuned" halyard-defaults.xml specific to bulkload?

Currently I have just over 200 regions per node - I have done a couple of bulkupdates since the load.

Thanks for developing Halyard!

[Mark]

I found the problem with the excessive region splitting - a previous user had changed the max hfile size to 2.5gb. How does presplit determine the number of mappers to create?

[adam....@gmail.com]

Presplit as well as BulkLoad uses HalyardBulkLoad.RioFileInputFormat (slightly modified CombineFileInputFormat) here from line 214: https://github.com/Merck/Halyard/blob/master/tools/src/main/java/com/msd/gin/halyard/tools/HalyardBulkLoad.java

Originally the splits calculation is driven by mapreduce.input.fileinputformat.split.maxsize property, which is set to 200.000.000 in Halyard defaults.

FYI: RioFileInputFormat creates additional "cloned" splits for for files > 10xmapreduce.input.fileinputformat.split.maxsize

Such "cloned" splits for very big files are processed in parallel, however parser decimates (de-duplicates) the parsed statements. This approach spread load across cluster for very large files and speeds their bulk load for the price of the cluster resources.

So if you plan to decrease the value of mapreduce.input.fileinputformat.split.maxsize, be aware of the files 10x bigger to cost you extra mappers.

BTW: If I can add one more hint - disable CPU isolation (cgroups) in YARN in your cluster (if enabled). By disabling that single "ani-feature" you will see huge increase of performance in all MapReduce shuffles, in Halyard Bulk Update and Export tools (which are heavily multi-threaded), in Halyard RIOFileInputFormat parsing (which uses extra thread-pump), and in many other tasks based on YARN.

[Mark]

So, I've been playing with input file sizes. To help maximize data locality, I split my data into 128mb files to match the block size. This seemed to improve the load speed. However, when I tried a bulk update it appeared to run very slowly - it seemed two region servers were hotspotting. Anyway, it occured to me that with smaller files, the presplit decimation maybe too big leading to inaccurate presplits. So, for the avoidance of doubt, I presplitted wtih a decimation of 10. This had an unexpected consequence - it resulted in 10000 regions! I would expect decimation to only affect the accuracy of the presplits not the number of regions too - maybe it was something else I changed. Perhaps explicit support for .nt.lzo files could be added. As I understand it, such files would be inherently splittable and thus InputSplits would be less dependent on file boundaries.