Hadoop Zone

YARN Overview YARN stands for Yet Another Resource Negotiator.  The Yarn was introduced in Hadoop 2.x.  It is purely for processing data and processing layer. It is called as Data Processing Framework (DPF). Yarn allows different data processing engines like graph processing, interactive processing, stream processing as well as batch processing to run and process data stored in HDFS.  Apart from resource management, Yarn is also used for job Scheduling. YARN Architecture Apache Yarn Framework consists of a master daemon known as “Resource Manager”, slave daemon called "Node Manager (one per slave node)".  Resource Manager, Node Manager are the two daemons of YARN.  Resource Manager contains in name node.  Node Manager contains in data node. Each data node will have one separate Node Manager. Resource Manager In General view, Resource Manager (RM) is responsible for tracking the resources in a cluster, and scheduling applications (e.g., MapReduce jobs).  Prior to

HDFS Services/Daemons           1. Name Node (NN)           2. Data node (DN)           3. Secondary Name Node (SNN)             4. Standby Name Node (Standby) These 3 nodes deal with HDFS which used to store the data into HDFS. V1 Map Reduce Daemons     Yarn Daemons (Map Reduce V2)           4. Job Tracker                  4. Resource Manager           5. Task Tracker                  5. Node Manager These daemons are responsible for running the MapReduce jobs. Start-dsf.sh service for starting hdfs daemons and start-yarn.sh for starting yarn daemons separately. start-all.sh service for starting all above daemons. Name Node (NN) Only one name node available for HDFS. It is a master node. Stores only Meta data of all the data. If name node fails then HDFS will be down, i.e., called Single Point Of Failure (SPOF). High Availability (HA) Here if Name node Active is down then Name Node Stand by acting like 'Active Name Node'. How memory management in NN , When we are doing any tra

HDFS Hadoop Distributed File System which is specially designed file system. Data storage framework. It is storage layer which is designed for only storing data. It is used to store huge amount of datasets with cluster of commodity hardware and with streaming access pattern. Why we are saying HDFS is specially designed ? File system means it is way of storing files and directories. Hard disk is having memory space like 500 GB. By default one block will be 4kb. Here I am storing 2kb of file then the remaining memory space will be wasted. It is a normal process for storing files which following in RDBMS. In HDFS, by default each block will have 64 MB size. When I am storing 35 MB of data in a block then the remaining memory space will be used to store another files or directories. This process is called Sharding. So that only we are calling HDFS as specially designed file system. If we are wasting these remaining spaces then we need more systems to store the huge datasets. You can make