---
title: "What is Adaptive MapReduce in Hadoop and How it Works"
description: "The performance and the approach of Adaptive MapReduce in Hadoop explained."
url: https://srinimf.com/2022/06/21/hadoop-adaptive-mapreduce-points-to-read-now/
date: 2022-06-21
modified: 2026-10-10
author: "Srini"
image: https://i0.wp.com/srinimf.com/wp-content/uploads/2022/06/cyclists-trail-bike-clouds-163407.jpeg?fit=1880%2C1251&ssl=1
categories: ["Hadoop"]
tags: ["Adaptive mapreduce"]
type: post
lang: en
---

# What is Adaptive MapReduce in Hadoop and How it Works

IBM Research has developed a concept called Adaptive MapReduce, which extends Hadoop by making individual mappers self-aware and aware of other mappers. This approach enables individual map tasks to adapt to their environment and make efficient decisions.

Related[Understanding Hadoop MapReduce Fault Tolerance](https://srinimf.com/2018/02/23/hadoop-mapreduce-fault-tolerance/)

## **Adaptive MapReduce**

When a MapReduce job is about to begin, Hadoop divides the data into many pieces, called splits. Each split is assigned a single mapper.

To ensure a balanced workload, these [mappers](https://hadoop.apache.org/docs/r2.7.4/api/org/apache/hadoop/mapreduce/Mapper.html) are deployed in waves, and new mappers start once old mappers finish processing their splits.

Related[How data is growing by 2020 in 10^11 times, a new wave to data Jobs](https://srinimf.com/2016/11/15/how-data-is-growing-by-2020-in-1011-times-a-new-wave-to-data-jobs/)

In this model, a small split size means more mappers, which helps ensure balanced workloads and minimizes failure costs.

However, smaller splits also result in increased cluster overhead due to the higher volumes of startup costs for each map task.

For workloads with high startup costs for map tasks, larger split sizes tend to be more efficient. An adaptive [approach to](https://srinimf.com/2018/06/19/ann-artificial-neural-network-real-useful-concept-to-design-robots/) running map tasks gives BigInsights the best of both worlds.

## **Adaptive Mapper**

One implementation of Adaptive MapReduce is the concept of an adaptive mapper. Adaptive Mappers extend the capabilities of conventional Hadoop mappers by tracking the state of file splits in a central repository.

*Each time an Adaptive Mapper finishes processing a split, it consults this central repository and locks another split for processing until the job is completed. This means that for Adaptive Mappers, only a single wave of mappers is deployed, since the individual mappers remain open to consume additional splits. *

The performance cost of locking a new split is far less than the startup cost for a new mapper, which accounts for a significant increase in performance. 

The [Adaptive Mappers](https://www.ibm.com/docs/en/idr/11.4.0?topic=tables-mapping-using-adaptive-apply) result (see the AM bar) was based on a low split size of 32 MB. Only a single wave of mappers was used, so there were significant performance savings based on avoiding the startup costs for additional mappers.

Further reading

[White paper on Adaptive map reducers in Hadoop](https://openproceedings.org/2012/conf/edbt/VernicaBBE12.pdf)

Related

- [Understanding Hadoop MapReduce Fault Tolerance](https://srinimf.com/2018/02/23/hadoop-mapreduce-fault-tolerance/)DateFebruary 23, 2018
- [Hadoop Quiz That Improves Your Knowledge](https://srinimf.com/2017/12/02/hadoop-knowledge-quiz-to-test-legacy-developers/)DateDecember 2, 2017
- [Hadoop Ecosystem – MapReduce Internal Process](https://srinimf.com/2013/04/16/hadoop-mapreduce-concept/)DateApril 16, 2013

### Share this:

-
- [Tweet](https://twitter.com/share)
- [![](https://i0.wp.com/assets.pinterest.com/images/pidgets/pinit_fg_en_rect_gray_20.png)](https://www.pinterest.com/pin/create/link/?url=https%3A%2F%2Fsrinimf.com%2F2022%2F06%2F21%2Fhadoop-adaptive-mapreduce-points-to-read-now%2F&media=https%3A%2F%2Fi0.wp.com%2Fsrinimf.com%2Fwp-content%2Fuploads%2F2022%2F06%2Fcyclists-trail-bike-clouds-163407.jpeg%3Ffit%3D1200%252C799%26ssl%3D1&description=What%20is%20Adaptive%20MapReduce%20in%20Hadoop%20and%20How%20it%20Works)
-
- [Share on Tumblr](https://www.tumblr.com/share)
- [Share on WhatsApp (Opens in new window) WhatsApp](https://srinimf.com/2022/06/21/hadoop-adaptive-mapreduce-points-to-read-now/?share=jetpack-whatsapp)
- [More](#)
-

- [Print (Opens in new window) Print](https://srinimf.com/2022/06/21/hadoop-adaptive-mapreduce-points-to-read-now/?share=print)
- [Share on Reddit (Opens in new window) Reddit](https://srinimf.com/2022/06/21/hadoop-adaptive-mapreduce-points-to-read-now/?share=reddit)
- [Share on Telegram (Opens in new window) Telegram](https://srinimf.com/2022/06/21/hadoop-adaptive-mapreduce-points-to-read-now/?share=telegram)
- [Email a link to a friend (Opens in new window) Email](mailto:?subject=%5BShared%20Post%5D%20What%20is%20Adaptive%20MapReduce%20in%20Hadoop%20and%20How%20it%20Works&body=https%3A%2F%2Fsrinimf.com%2F2022%2F06%2F21%2Fhadoop-adaptive-mapreduce-points-to-read-now%2F&share=email)
-

### Like this:

Like Loading…