Top Hadoop MapReduce features to manage failure

Hadoop MapReduce

The MapReduce concept is totally different from other distributed concepts. How hadoop managing server failures I am going to explain in this post.

MapReduce does not achieve high scalability with distributed processing and high fault tolerance at the same time.

Distributed computation is often a messy thing, so it is difficult to write a reliable distributed application by yourself.

MapReduce is basically two way model. One is map and the other one is reduce.

There are various kinds of failures that are introduced when distributed applications are running: Some servers might fail abruptly, whereas some disks may get out of order. Keep in mind that writing code to handle failures by yourself is very time consuming and can also cause new bugs in your application. You can read more on the Hadoop failed tasks.

Hadoop MapReduce, however, can take care of fault tolerance. When your application fails, the framework can handle the cause of failure and retry it or abort. Thanks to this feature, the application can complete its tasks while overcoming failures.

Architecture of Hadoop MapReduce

Hadoop MapReduce design

Hadoop has three Unique features:

  • High scalability

  • High fault tolerance

  • High level interface to achieve these two points

The process of MapReduce comprises of Five components:

  • map: Read the data from a storage system such as HDFS.

  • sort: Sort the input data from the map task according to their keys.

  • shuffle: Divide the sorted data and repartitioning among cluster nodes.

  • merge: Merge the input data sent from mapper on each node.

  • reduce: Read the merged data and integrate them into one result.


Also Read


Author: Srini

Experienced software developer. Skills in Development, Coding, Testing and Debugging. Good Data analytic skills (Data Warehousing and BI). Also skills in Mainframe.