--- title: "Hadoop Job Scheduling Tools: Capacity Vs. Fair" description: "Hadoop follows FIFO process to schedule the tasks. The tools Capacity and Fair play a key role in Scheduling the jobs. Read more." url: https://srinimf.com/2017/12/04/hadoop-workflow-process/ date: 2017-12-04 modified: 2026-10-10 author: "Srini" image: https://i0.wp.com/srinimf.com/wp-content/uploads/2017/12/pexels-photo-8447438.jpeg?fit=1880%2C1256&ssl=1 categories: ["Hadoop"] tags: ["Differences of Capacity Vs Fair", "Hadoop tips and Tricks", "Top Hadoop schedulers", "Workflow in Oozie"] type: post lang: en --- # Hadoop Job Scheduling Tools: Capacity Vs. Fair The two top tools to schedule a job in Hadoop are **Capacity** and **Fair**. Before diving into the scheduling tools, here is a handy guide to refer to on [job scheduling basics](https://www.slideshare.net/srinipdn/hadoop-job-scheduling)  > Hadoop is a batch processing ecosystem. So it cannot analyze data on-the-fly.Hadoop Scheduler ## **Job scheduling process in Hadoop** Moving on to tasks, Hadoop schedules tasks in FIFO order (Firs-in-First-out). Here is the best book [Hadoop workflow scheduler using Oozie](http://amzn.to/2yOhMyL) to get a fair idea on scheduling the jobs. ## **1. Capacity Scheduler** - Capacity handles large clusters that are shared among multiple organizations or groups. - *A*ccording to [Hadoop documentation](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html) the capacity scheduler is designed to run Hadoop applications as a shared, multi-tenant cluster in an operator-friendly manner while maximizing the throughput and the utilization of the cluster. ### Components in Capacity #### *In this multi-tenancy environment, a cluster can have many job types and multiple job priorities.* #### **Organization** As it is designed for situations in which clusters [need to](https://srinimf.com/2018/04/08/popular-influencers/) support multi-tenancy, its resource sharing is more stringent so as to meet capacity, security, and resource guarantees. #### **Capacity** Resources are allocated to queues and are shared among the jobs on that queue. It is possible to set soft and hard limits on queue-based resources. #### **Security** In a multi-tenancy cluster, security is a major concern. Capacity uses access control lists (ACLs) to manage queue-based job access. It also permits per-queue administration, so that you can have different settings on the queues. #### **Elasticity** Free resources from under-utilized queues [can be](https://srinimf.com/2017/09/08/the-story-develop-scala-applications-to-access-hadoop-data/) assigned to queues that have reached their capacities. When needed elsewhere, these resources can then be reassigned, thereby maximizing utilization. #### **Multi-tenancy** In a multi-tenancy environment, a single user’s rogue job could possibly soak up multiple tenants’ resources, which would have a serious impact on job-based service-level agreements (SLAs). The Capacity scheduler provides a range of limits for these multiple jobs, users, and queues so as to avoid this problem. #### **Resource-based Scheduling** Capacity uses an algorithm that supports memory-based resource scheduling for jobs that are resource intensive. #### **Hierarchical Queues** When used with Hadoop V2, Capacity supports a hierarchy of queues, so that underutilized resources are first shared among subqueues before they are then allocated to other cluster tenant queues. #### **Job Priorities** In Hadoop V1, the scheduler supports scheduling by job priority. #### **Operability** Capacity enables you to change the configuration of a queue at run-time via a console that permits viewing of the queues. In Hadoop V2, you can also stop a queue to let it drain. ![Workflow in Hadoop](https://i0.wp.com/srinimf.com/wp-content/uploads/2017/12/workflow-in-hadoop.jpg?resize=600%2C480&ssl=1) ## **2. Fair Scheduler** - Fair aims to do what its name implies: share resources fairly among all jobs within a cluster that is owned and used by a single organization. Over time, it aims to share resources evenly to job pools. - *According to [Hadoop documentation](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/FairScheduler.html)the Fair is – Fair scheduling is a method of assigning resources to applications such that all apps get, on average, an equal share of resources over time* ### Components in Fair #### **Organization** This scheduler organizes jobs into pools, with resources shared among the pools. Attributes, like priorities, act as weights when the resources are shared. #### **Resource Sharing** You can specify a minimum level of resources to a pool. If a pool is empty, then Fair shares the resources of other pools. #### **Resource Limits** With Fair, you can specify concurrent job limits by user and pool so as to limit the load on the cluster. ### **The role of work-flow manager** Capacity, Fair, and similar plug-in schedulers deal with resources allocated to individual jobs over a period of time. However, what about the relationships between jobs and the dependencies between them? **That’s where *workflow* managers, like Apache’s Oozie, come in.** The critical functionality of Oozie is – Oozie offer the ability to schedule jobs that are organized into workflows by time and event. The deisgn of work-flow in Oozie is done by using XML. [Refer more of design of work-flows](http://blog.cloudera.com/blog/2013/01/how-to-schedule-recurring-hadoop-jobs-with-apache-oozie/). **Related Posts** - [How Hadoop Process Data Uniquely](https://srinimf.com/2013/07/28/hadoop-a-tool-for-your-big-data-problems/)DateJuly 28, 2013 - [Hadoop Quiz That Improves Your Knowledge](https://srinimf.com/2017/12/02/hadoop-knowledge-quiz-to-test-legacy-developers/)DateDecember 2, 2017 - [6 Primary Skills That Make Easy Your Hadoop Career](https://srinimf.com/2014/01/06/skills-hadoop-developer/)DateJanuary 6, 2014 ### Share this: - - [Tweet](https://twitter.com/share) - [![](https://i0.wp.com/assets.pinterest.com/images/pidgets/pinit_fg_en_rect_gray_20.png)](https://www.pinterest.com/pin/create/link/?url=https%3A%2F%2Fsrinimf.com%2F2017%2F12%2F04%2Fhadoop-workflow-process%2F&media=https%3A%2F%2Fi0.wp.com%2Fsrinimf.com%2Fwp-content%2Fuploads%2F2017%2F12%2Fpexels-photo-8447438.jpeg%3Ffit%3D1200%252C802%26ssl%3D1&description=Hadoop%20Job%20Scheduling%20Tools%3A%20Capacity%20Vs.%20Fair) - - [Share on Tumblr](https://www.tumblr.com/share) - [Share on WhatsApp (Opens in new window) WhatsApp](https://srinimf.com/2017/12/04/hadoop-workflow-process/?share=jetpack-whatsapp) - [More](#) - - [Print (Opens in new window) Print](https://srinimf.com/2017/12/04/hadoop-workflow-process/?share=print) - [Share on Reddit (Opens in new window) Reddit](https://srinimf.com/2017/12/04/hadoop-workflow-process/?share=reddit) - [Share on Telegram (Opens in new window) Telegram](https://srinimf.com/2017/12/04/hadoop-workflow-process/?share=telegram) - [Email a link to a friend (Opens in new window) Email](mailto:?subject=%5BShared%20Post%5D%20Hadoop%20Job%20Scheduling%20Tools%3A%20Capacity%20Vs.%20Fair&body=https%3A%2F%2Fsrinimf.com%2F2017%2F12%2F04%2Fhadoop-workflow-process%2F&share=email) - ### Like this: Like Loading…