Optimizing Hadoop fetch and swapping process

[Mohanjot Singh]

Hi Community,

We are working on a production read-only cluster using standard Voldemort 1.10 deployed with hadoop-2.3.0-cdh5-1.5 version.

server.properties has the following properties set specific to fetcher job:

fetcher.max.bytes.per.sec=10485760 // 10MB

hdfs.fetcher.buffer.size=262144    // 256KB 

The total size of the file for each node is around 80G. The whole swapping process takes more than 10hrs for all the nodes . 

While debugging the issue, I found out a couple of parameters in VoldemortConfig.java:

DEFAULT_FETCHER_SOCKET_TIMEOUT = 1000 * 60 *30; //30 minutes

DEFAULT_FETCHER_THROTTLE_INTERVAL_WINDOW_MS = 1000 — which by definition is :

* When measuring the download rate of HDFS fetches, this parameter defines
* the length in milliseconds of the two rolling windows. 

It definitely is not a socket timeout issue as the process gets completed. I am wondering if DEFAULT_FETCHER_THROTTLE_INTERVAL_WINDOW_MS parameter would help in increasing the throttling speed.

Is there's any other way to tweak any parameters to expedite the process? Any help/advice would be appreciated.

Thanks,

Mohanjot

[Arunachalam]

Did you try running the fetcher unthrottled ? The default for fetcher.max.bytes.per.sec is 0, which means no throttling at all. Other thing you can consider is try increasing the fetch download speed to 20 or 30, based on your online read throughput.

Thanks,

Arun.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Félix GV]

On Tue, Jul 12, 2016 at 3:59 PM, Arunachalam <arunac...@gmail.com> wrote:

fetcher.max.bytes.per.sec

Yeah, you should first tweak fetcher.max.bytes.per.sec rather than the other settings.

-F

--
--
Félix