Hadoop’s MapReduce model was introduced by Google. The processing of data in MapReduce is a 2-way process.
MapReduce internal process
- Map: It is an ingestion and transformation step. Initially, all input records are processed in parallel
- Reduce: It is an aggregation and stigmatization step. All associated records are processed together by a single entity.
Hadoop ecosystem
- Developed by apache software. It is open-source.
- Hadoop Core provides a distributed filesystem (HDFS) and support for the MapReduce distributed computing metaphor.
- HBase builds on Hadoop Core to provide a scalable, distributed database.
- Pig is a high-level data-flow language and execution framework for parallel computation. It is built on top of Hadoop Core.
- ZooKeeper is a highly available and reliable coordination system. Distributed applications use ZooKeeper to store and mediate updates for critical shared-state.
- Hive is a data warehouse infrastructure built on Hadoop Core that provides data summarization, ad-hoc querying, and analysis of datasets.
- HDFS: Hadoop distributed file system.
