Apache Spark Components

Apache Spark is an open source framework for real-time processing. Also it is a powerful big data tool and used for various big data challenges. 

Hadoop MapReduce is the best framework for processing data in batches. Spark could process data in real time and it is 100 times faster than Hadoop MapReduce in batch processing of large data sets. Spark is able to achieve this speed through controlled partitioning. We could process that partitioned data with minimal network traffic.

Features
  • Spark code can be written in Java, Scala, Python and R.
  • Spark also supports and process structured and semi-structured data though Spark SQL.
  • If it is absolutely necessary, then only spark will execute the code/function. This is called "Lazy Execution".
  • Spark will be faster for processing the data because of its in-memory processing.
  • Apache Spark is a separate tool and it has its own cluster. We can use Hadoop to implement Spark.
Spark Components

           • In addition to Map and Reduce operations, it supports SQL queries, streaming data, machine learning and graph data processing.


 Let's consider we want to process one hundreds of millions of data. Whenever we try to process it using old systems then it will pick the files sequentially. Whenever we try to process it through RDBMS then it will pick the files randomly. When we try with HDFS then the files will be processed by parallel mechanism. We can process it better than HDFS using Apache Spark.

Spark Core

Spark Core is an engine used to process large set of data's. It is responsible for Memory management, fault recovery, Scheduling, distributing and monitoring jobs on a cluster.

Spark Streaming

Spark Streaming is the component of Spark which is used to process real-time streaming data.

 Spark Streaming is used to stream real time data from various sources like Twitter, Stock Market, National Security, Telecom, Banking, Healthcare, etc and also perform powerful analytics to help business.

Spark SQL

Spark SQL is one of module in Spark which integrates relational processing with Spark's functional programming, It supports querying data either via SQL or via the Hive Query Language.

 Spark SQL queries are integrated with Spark programs. Spark SQL allows us to query structured data inside Spark programs, using SQL.

GraphX

GraphX is the Spark API for graphs and graph-parallel computation.

MLlib (Machine Learning)

MLlib stands for Machine Learning Library. Spark MLlib is used to perform machine learning in Apache Spark.

Typical flow of Spark


Comments

Post a Comment

Popular posts from this blog

Hive File Formats

Image
 Hive supports several file formats: Text File SequenceFile RCFile Avro Files ORC Files Parquet Custom INPUTFORMAT and OUTPUTFORMAT      The hive.default.fileformat configuration parameter which is avaialble in hive-site.xml determines the format to use if it is not specified in a CREATE TABLE or ALTER TABLE statement.        Text file is the parameter's default value. What is File Format ? File Format is a way in which information is stored or encoded in a computer file.  In Hive it refers to how records are stored inside the file. These file formats mainly vary between data encoding, compression rate, usage of space and disk I/O. Hive does not verify whether the data that you are loading matches the schema for the table or not. However, it verifies if the file format matches the table definition or not. Text File TEXT FILE format is a famous input/output format used in Hadoop. In Hive if we define a table as TEXTFILE it can load data o...
Read more

Why We Need Hadoop?

What is Hadoop? It is an open source framework or system which was fully developed by Java.  It is allow us to implement Big Data.  Hadoop can be used for storing and processing huge datasets with cluster of commodity hardware. Because its stores the data and processing in a distributed manner.  It made up of several components which entirely open source. Doug Cutting is the inventor of hadoop. He used elephant symbol. His child used to play with elephant toy. So he used that toy name as symbol and named for new invention. Why we need to go for Hadoop? 1. High Throughput and Low latency queries Less amount of time it will take to run/process real time queries or near real time queires. We can achieve high throughput in hadoop using below techniques.  Parallel Processing WORM Data Locality 2. Distributed computing A distributed computer system consists of multiple software components that are on multiple computers, but run as a single system.  The computers that ...
Read more

Hive Data Types

Image
 All the data types in Hive are classified into four types, given as follows: Column Types Literals Null Values Misc Types Complex Types Column Types Integral Types Integer type data can be specified using integral data types, INT. When the data range exceeds the range of INT, you need to use BIGINT and if the data range is smaller than the INT, you use SMALLINT. TINYINT is smaller than SMALLINT. The data ranges for TINYINT, SMALLINT, INT, BIGINT is below, TINYINT (1-byte signed integer, from -128 to 127) SMALLINT (2-byte signed integer, from -32,768 to 32,767) INT (4-byte signed integer, from -2,147,483,648 to 2,147,483,647) BIGINT (8-byte signed integer, from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807) Example create table test1(id tinyint) row format delimited fields terminated by ','; insert into test1 values(1); insert into test1 values(127); insert into test1 values(128); hive> select * from test1; OK 1 127 NULL Time taken: 0.055 seconds, Fetched: 3 row(s)...
Read more