Hadoop Zone

 1. Ctrl+Alt+T ,to open terminal and run the commands one by one,     a) sudo apt-get update                 If any problem for running above command then run below commands,                                      sudo rm -rf /var/lib/apt/lists/*                     sudo apt-get update                     sudo apt-get update -o Acquire::http::No-Cache=True                          b) sudo apt-get install openjdk-8-jdk    ...

Apache Spark is an open source framework for real-time processing. Also it is a powerful big data tool and used for various big data challenges.  Hadoop MapReduce is the best framework for processing data in batches. Spark could process data in real time and it is 100 times faster than Hadoop MapReduce in batch processing of large data sets.  Spark is able to achieve this speed through controlled partitioning. We could process that  partitioned  data with  minimal network traffic. Features Spark code can be written in Java, Scala, Python and R. Spark also supports and process structured and semi-structured data though Spark SQL. If it is absolutely necessary, then only spark will execute the code/function. This is called "Lazy Execution". Spark will be faster for processing the data because of its in-memory processing. Apache Spark is a separate tool and it has its own cluster. We can use Hadoop to implement Spark. Spark Components     ...

Transformation Spark Transformation is a function that produces new RDD from the existing RDDs.  It takes RDD as input and produces one or more RDD as output. Each time it creates new RDD when we apply any transformation. Thus, the so input RDDs, cannot be changed since RDD are immutable in nature. Transformations are lazy in nature i.e., they get execute when we call an action. They are not executed immediately. After the transformation, the resultant RDD is always different from its parent RDD. It can be smaller (e.g. filter, count, distinct, sample), bigger (e.g. flatMap(), union(), Cartesian()) or the same size (e.g. map). Action Transformations create RDDs from each other, but when we want to work with the actual dataset, at that point action is performed. When the action is triggered after the result, new RDD is not formed like transformation. Thus, Actions are Spark RDD operations that give non-RDD values. The values of action are stored to drivers or to the external storage...

Lazy Evaluation Evaluation means executing the logic. Lazy means don't execute immediately until some action is performed. So code will execute when the action is performed. Lazy evaluation means spark will not load or transform data unless an action is performed. Load file into RDD Filter the RDD Count no of elements (Only now loading filtering happens) Helps internally optimize operations and resource usage. We can write chaining operations and We can watch out during the troubleshooting. If you only used transformations like textFile then it will not execute immediately it will create DAG. Those transformations are compiled into DAG. DAG is a complex piece of module.  You can perform bunch of transformations before invoking action.  Until Action is performed the spark compiler checks whether required transformations code is compiled in DAG.  DAG is graph so it will be having multiple paths, it will take any one path and execute the transformations whenever the action i...

Job, Stage, Task Job A job in Spark represents a complete computation triggered by an action (e.g., collect(), count(), saveAsTextFile()). When you call an action on a Spark RDD (Resilient Distributed Dataset) or DataFrame, Spark starts to execute the transformations defined in your code. A job consists of one or more stages, which are formed based on the DAG (Directed Acyclic Graph) of transformations that need to be executed to fulfill the action's requirements. Spark may optimize the execution plan by breaking the job into multiple stages to minimize data shuffling and improve performance. Stage In Apache Spark, a stage is a logical division of a Spark job's execution plan. Stages are formed during the process of executing a Spark job, which involves transforming data through a series of RDD (Resilient Distributed Dataset) or DataFrame operations and actions. When an action is triggered, Spark analyzes the DAG (Directed Acyclic Graph) of transformations that need to be execu...

Mode of Execution Mode of Execution determine where your app’s resources are physically located when you run your application/job.  Cluster Mode Client Mode Local Mode Cluster Mode In cluster mode, the Spark driver program runs within the Spark cluster itself.  The driver program is launched on one of the nodes in the cluster, typically on a master node managed by the cluster manager (e.g., YARN, Mesos, or Spark's standalone cluster manager).  When a Spark application is submitted in cluster mode, the driver program is launched on a cluster node, and the application code is executed within the cluster's resources. Both the driver and executor processes run within the Cluster. Suitable for production deployments where resources are managed centrally by a cluster manager. Client Mode In client mode, the Spark driver program runs on the machine that submits the Spark application (often referred to as the client machine).  The client machine typically resides outsid...

What is RDD? RDD - Resilient Distributed Datasets Resilience : RDDs are resilient because they can recover automatically from failures. Spark achieves this by keeping track of the lineage of transformations applied to the base dataset, so it can recompute lost partitions due to node failures. In spark, lineage is built using the RDD (Resilient Distributed Dataset) abstraction, which keeps track of all transformations applied to it. This information is used to recompute lost data in the event of a failure, ensuring fault tolerance without the need for data replication. Distributed : RDDs are distributed across the nodes of the cluster. Resilient Distributed Datasets (RDDs) are the primary data structure in Spark. This data structure process any Python, Java, Scala, or user-created object. RDDs are partitioned across nodes in a cluster and can be operated on in parallel. RDDs are reliable and memory-efficient when it comes to parallel processing. By storing and processing data in RDDs,...

Apache Spark is an open-source distributed computing system that provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.  It's designed to handle large-scale data processing and analytics tasks efficiently. Spark's architecture is structured to maximize performance and scalability. Architecture Driver Program The driver program is the entry point of any Spark application.  The driver program is responsible for translating the user's code into tasks and distributing them to the executors.  The Driver Program is a process that runs the main() function of the application and creates the SparkContext object which represents the connection to the Spark cluster. When a user submits a Spark application, the Driver Program receives the application code and any associated configurations. Rest of the technical stuffs could be taken by SparkContext. The Driver Program interacts with the cluster manager (such as Apache Mesos or Y...

What is Apache Spark? Apache Spark is an open source big data processing framework built around speed, ease of use, and sophisticated analytics.  It was originally developed in 2009 in UC Berkeley's AMP Lab, and open sourced in 2010 as an Apache project. Apache Spark is a single node or cluster computing platform on top of any storage layer for large scale data processing. It is an End to End Analytics platform (Data Ingestion, ETL, Analytics and streaming). Developed to overcome limitations of Hadoop/MapReduce. Spark has several advantages compared to other big data and MapReduce technologies like Hadoop and Storm. First of all, Spark gives us a comprehensive, unified framework to manage big data processing requirements with a variety of data sets that are diverse in nature (text data, graph data etc) as well as the source of data (batch v. real-time streaming data). Major companies like Amazon, eBay, Yahoo uses Spark. Why do we prefer Spark over Hadoop? Hadoop as a big data proce...

01. What is the primary use of Textfile, Sequencefile, RCfile, ORCfile formats? Thus you can use the above four file formats depending on your data.                 For example,  If your data is delimited by some parameters then you can use TEXTFILE format.  If your data is in small files whose size is less than the block size then you can use SEQUENCEFILE format.  If you want to perform analytics on your data and you want to store your data efficiently for that then you can use RCFILE format.  If you want to store your data in an optimized way which lessens your storage and increases your performance then you can use ORCFILE format. 02. What is the difference between a block and a stripe? HDFS blocks is the lowest level, ORC stripe is upper level, these levels are completely independent, and stripes in ORC do not care about lower storage layer. HDFS blocks: HDFS blocks is the lowest level, independent from file forma...

 Hive supports several file formats: Text File SequenceFile RCFile Avro Files ORC Files Parquet Custom INPUTFORMAT and OUTPUTFORMAT      The hive.default.fileformat configuration parameter which is avaialble in hive-site.xml determines the format to use if it is not specified in a CREATE TABLE or ALTER TABLE statement.        Text file is the parameter's default value. What is File Format ? File Format is a way in which information is stored or encoded in a computer file.  In Hive it refers to how records are stored inside the file. These file formats mainly vary between data encoding, compression rate, usage of space and disk I/O. Hive does not verify whether the data that you are loading matches the schema for the table or not. However, it verifies if the file format matches the table definition or not. Text File TEXT FILE format is a famous input/output format used in Hadoop. In Hive if we define a table as TEXTFILE it can load data o...

 All the data types in Hive are classified into four types, given as follows: Column Types Literals Null Values Misc Types Complex Types Column Types Integral Types Integer type data can be specified using integral data types, INT. When the data range exceeds the range of INT, you need to use BIGINT and if the data range is smaller than the INT, you use SMALLINT. TINYINT is smaller than SMALLINT. The data ranges for TINYINT, SMALLINT, INT, BIGINT is below, TINYINT (1-byte signed integer, from -128 to 127) SMALLINT (2-byte signed integer, from -32,768 to 32,767) INT (4-byte signed integer, from -2,147,483,648 to 2,147,483,647) BIGINT (8-byte signed integer, from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807) Example create table test1(id tinyint) row format delimited fields terminated by ','; insert into test1 values(1); insert into test1 values(127); insert into test1 values(128); hive> select * from test1; OK 1 127 NULL Time taken: 0.055 seconds, Fetched: 3 row(s)...