Skip to first unread message
raghavendra kumar
Jul 7, 2016, 1:39:06 PM7/7/16
to Hadoop Learners from Hadoop-skills.com
Hi
I am new to Hadoop, just learnt Map Reduce. Have below doubts regarding it.
1) Does Map Reduce always does processing in (Key , Value ) pairs? I general any real time problem can be solved with (Key, Value) pair?
2)HDFS splits input file to multiple blocks. Suppose the file is split into 10 blocks, and then the output of mapper ( Key, Value) is converted to "Intermediate Data" ( which is sorted and shuffled) and then passed to reducer for aggregation.
What component is responsible for "Intermediate Data"
3) In case if 9 blocks have converted the data to (Key, Value) and 10th block is taking long time, will the sorting and shuffling be done on 9 block outputs or they are going to wait? This might occur generally because the block size is not same for all blocks.
4) Read HDFS is write once and read many times. Normally in DWH or ETL concepts we insert new records per each day into the table. In similar fashion, how can we achieve this in HDFS, because we cannot modify the file once loaded into HDFS. Do we need to add new records to our file and then upload to HDFS deleting the old file?
Sorry if I have asked too much, Thanks in advance.
uma ram
Dec 15, 2018, 1:35:15 AM12/15/18
to Hadoop Learners from Hadoop-skills.com
MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. A MapReduce program is composed of a map procedure, which performs filtering and sorting, and a reduce method, which performs a summary operation.MapReduce allows for distributed processing of the map and reduction operations. Maps can be performed in parallel, provided that each mapping operation is independent of the others; in practice, this is limited by the number of independent data sources and/or the number of CPUs near each source.
Reply all
Reply to author
Forward
0 new messages