Hadoop Technologies Explained: From MapReduce to HDFS

Hadoop ecosystem comes with a set of technologies. An ecosystem of Hadoop would solve big data problems.

Here is explained nature of the work of each technology in the context of the ecosystem.

Hadoop ecosystem technologies

1. HDFS
   1. HDFS is the one, which makes it possible to store different types of large data sets (i.e. structured, unstructured and semi structured data).
2. YARN
   1. The basic functions of YARN is ResourceManager and NodeManager.
3. MAP REDUCE
   1. MapReduce is a software framework which helps in writing applications that processes large data sets using distributed and parallel algorithms inside Hadoop environment.
4. SPARK
   1. In memory data processing
5. HIVE
   1. Facebook created HIVE for people who are fluent with SQL. Thus, HIVE makes them feel at home while working in a Hadoop Ecosystem.
6. PIG
   1. PIG has two parts: Pig Latin, the language and the pig runtime, for the execution environment. You can better understand it as Java and JVM.
   2. It supports pig latin language, which has SQL like command structure.
7. HBASE
   1. NOSQL database
8. MAHOUT
   1. Mahout provides an environment for creating machine learning applications which are scalable
9. SPARK
   1. Apache Spark is a framework for real time data analytics in a distributed computing environment.
10. DRILL
    1. It is a replica of Google Dremel.
    2. It supports different kinds NoSQL databases and file systems, which is a powerful feature of Drill.  For example: Azure Blob Storage, Google Cloud Storage, HBase, MongoDB, MapR-DB HDFS, MapR-FS, Amazon S3, Swift, NAS and local files.
11. ZOOKEEPER
    1. Apache Zookeeper is the coordinator of any Hadoop job which includes a combination of various services in a Hadoop Ecosystem.
12. OOZIE
    1. Consider Apache Oozie as a clock and alarm service inside Hadoop Ecosystem. For Apache jobs, Oozie has been just like a scheduler.
13. FLUME
    1. The Flume is a service which helps in ingesting unstructured and semi-structured data into HDFS.
14. SQOOP
    1. Flume only ingests unstructured data or semi-structured data into HDFS.
    2. While Sqoop can import as well as export structured data from RDBMS or Enterprise data warehouses to HDFS or vice versa.
15. SOLAR & LUCENE
    1. Apache Solr and Apache Lucene are the two services which are used for searching and indexing in Hadoop Ecosystem.
16. AMBARI
    1. Ambari is an Apache Software Foundation Project which aims at making Hadoop ecosystem more manageable.
17. HADOOP
    1. It is the overall architecture of Hadoop system

3 Top Hadoop Platforms

1. IBM
2. Cloudera
3. AWS

References

• 10 Top commercial Hadoop platforms read here.