Apache Pig Running & Execution Mode

Apache Pig Running Mode

You can run/execute Pig latin stetements and commands using three modes.

  1. Interactive mode
  2. Batch mode
  3. Embedded mode(UDF). 

Interactive mode

  • Interactive processing, we can use Apache Tez or Apache Spark as processing engine.
  • Interactive processing means that the person needs to provide the computer with instructions whilst it is doing the processing. i.e the user 'interacts' with the computer to complete the processing.
    • For example, if you use an online system to book a hotel room, you will fill in a web form, submit it and it will come back to inform you of the room you have booked.
  • You can run Pig in interactive mode using the Grunt shell. Invoke the Grunt shell using the "pig" command (as shown below) and then enter your Pig Latin statements and Pig commands interactively at the command line.

Batch mode

  • We can process the data set by set or batch by batch with limited number of records/data/rows Like mapreduce, hive.
  • We will do this in pig using mapreduce as processing engine.

Embedded mode(UDF)

  • Apache Pig provides the provision of defining our own functions (User Defined Functions) in programming languages such as Java, Python, Ruby and using them in our script. 
  • Running pig logics or loading data or manipulating data in pig using UDF of java or pythin or ruby.

Execution Modes of Pig

  1. Local Mode
  2. Tex Local Mode
  3. Spark Local Mode
  4. Mapreduce Mode
  5. Tez Mode
  6. Spark Mode

Local Mode

  • In this mode, all the files are installed and run from your local host and local file system. There is no need of Hadoop or HDFS. This mode is generally used for testing purpose.

            $ pig -x local

Tez Local Mode

  • To run pig in tez local mode.
  • It is similar to local mode except internally pig will invoke tez runtime engine.

    $ pig -x tez_local

Note: Tez local mode is experimental. There are some queries which just error out on bigger data in local mode.

Spark Local Mode

  • To run pig in spark local mode.
  • It is similar to local mode except internally pig will invoke spark runtime engine.

    $ pig -x spark_local

Note: Spark local mode is experimental. There are some queries which just error out on bigger data in local mode.

MapReduce Mode

  • MapReduce mode is where we load or process the data that exists in the Hadoop File System (HDFS) using Apache Pig. In this mode, whenever we execute the Pig Latin statements to process the data, a MapReduce job is invoked in the back-end to perform a particular operation on the data that exists in the HDFS.
  • You need access to a Hadoop Cluster and HDFS installation.
  • Mapreduce mode is default mode But dont need to specify it while logging into pig.

    $ pig -x mapreduce (or) $ pig

Tez Mode

  • To run pig in tez mode. you need access to a Hadoop Cluster and HDFS installation.
  • It will internally pig will invoke tez runtime engine for processing which is in cluster.

    $ pig -x tez

Spark Mode

  • To run pig in spark mode. you need access to a spark and Hadoop Cluster and HDFS installation.
  • It will internally pig will invoke spark runtime engine for processing which is in cluster.

    $ pig -x spark

  • pig scripts run on spark can take advantage of the "dynamic allocation" feature. This feature can be enabled using below.

spark.conf.set("spark.dynamicAllocation.enabled", "true") -- pyspark

spark.dynamicAllocation.enabled=true -- scala

  • In general all properties in the pig script prefixed with spark are copied to the Spark Application Configuration.
  • Please note that Yarn auxillary service need to be enabled on Spark for this to work.

Comments

Post a Comment

Popular posts from this blog

Hive File Formats

Image
 Hive supports several file formats: Text File SequenceFile RCFile Avro Files ORC Files Parquet Custom INPUTFORMAT and OUTPUTFORMAT      The hive.default.fileformat configuration parameter which is avaialble in hive-site.xml determines the format to use if it is not specified in a CREATE TABLE or ALTER TABLE statement.        Text file is the parameter's default value. What is File Format ? File Format is a way in which information is stored or encoded in a computer file.  In Hive it refers to how records are stored inside the file. These file formats mainly vary between data encoding, compression rate, usage of space and disk I/O. Hive does not verify whether the data that you are loading matches the schema for the table or not. However, it verifies if the file format matches the table definition or not. Text File TEXT FILE format is a famous input/output format used in Hadoop. In Hive if we define a table as TEXTFILE it can load data o...
Read more

Why We Need Hadoop?

What is Hadoop? It is an open source framework or system which was fully developed by Java.  It is allow us to implement Big Data.  Hadoop can be used for storing and processing huge datasets with cluster of commodity hardware. Because its stores the data and processing in a distributed manner.  It made up of several components which entirely open source. Doug Cutting is the inventor of hadoop. He used elephant symbol. His child used to play with elephant toy. So he used that toy name as symbol and named for new invention. Why we need to go for Hadoop? 1. High Throughput and Low latency queries Less amount of time it will take to run/process real time queries or near real time queires. We can achieve high throughput in hadoop using below techniques.  Parallel Processing WORM Data Locality 2. Distributed computing A distributed computer system consists of multiple software components that are on multiple computers, but run as a single system.  The computers that ...
Read more

Hive Data Types

Image
 All the data types in Hive are classified into four types, given as follows: Column Types Literals Null Values Misc Types Complex Types Column Types Integral Types Integer type data can be specified using integral data types, INT. When the data range exceeds the range of INT, you need to use BIGINT and if the data range is smaller than the INT, you use SMALLINT. TINYINT is smaller than SMALLINT. The data ranges for TINYINT, SMALLINT, INT, BIGINT is below, TINYINT (1-byte signed integer, from -128 to 127) SMALLINT (2-byte signed integer, from -32,768 to 32,767) INT (4-byte signed integer, from -2,147,483,648 to 2,147,483,647) BIGINT (8-byte signed integer, from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807) Example create table test1(id tinyint) row format delimited fields terminated by ','; insert into test1 values(1); insert into test1 values(127); insert into test1 values(128); hive> select * from test1; OK 1 127 NULL Time taken: 0.055 seconds, Fetched: 3 row(s)...
Read more