Apache Pig Operators

Load & Store Opertators

  • Load
  • Store

Diagnostic Opertators

  • Dump
  • Describe
  • Explain
  • Illustrate

Grouping & Joining Opertators

  • Group
  • Cogroup
  • Join - Single Key | Multi Key join
  • Cross

Combining & Splitting Opertators

  • Union
  • Split

Filtering Opertators

  • Distinct
  • Filter
  • Foreach

Sorting Opertators

  • order by 
  • limit

Built-In Functions

  • Concat
  • tokenize
  • flatten
  • bag, tuple, map, top
  • PigStorage
  • TextLoader
  • BinStorage


Comments

Post a Comment

Popular posts from this blog

Why We Need Hadoop?

What is Hadoop? It is an open source framework or system which was fully developed by Java.  It is allow us to implement Big Data.  Hadoop can be used for storing and processing huge datasets with cluster of commodity hardware. Because its stores the data and processing in a distributed manner.  It made up of several components which entirely open source. Doug Cutting is the inventor of hadoop. He used elephant symbol. His child used to play with elephant toy. So he used that toy name as symbol and named for new invention. Why we need to go for Hadoop? 1. High Throughput and Low latency queries Less amount of time it will take to run/process real time queries or near real time queires. We can achieve high throughput in hadoop using below techniques.  Parallel Processing WORM Data Locality 2. Distributed computing A distributed computer system consists of multiple software components that are on multiple computers, but run as a single system.  The computers that ...
Read more

Job, Stage, Task in Spark

Image
Job, Stage, Task Job A job in Spark represents a complete computation triggered by an action (e.g., collect(), count(), saveAsTextFile()). When you call an action on a Spark RDD (Resilient Distributed Dataset) or DataFrame, Spark starts to execute the transformations defined in your code. A job consists of one or more stages, which are formed based on the DAG (Directed Acyclic Graph) of transformations that need to be executed to fulfill the action's requirements. Spark may optimize the execution plan by breaking the job into multiple stages to minimize data shuffling and improve performance. Stage In Apache Spark, a stage is a logical division of a Spark job's execution plan. Stages are formed during the process of executing a Spark job, which involves transforming data through a series of RDD (Resilient Distributed Dataset) or DataFrame operations and actions. When an action is triggered, Spark analyzes the DAG (Directed Acyclic Graph) of transformations that need to be execu...
Read more

Apache Spark Components

Image
Apache Spark is an open source framework for real-time processing. Also it is a powerful big data tool and used for various big data challenges.  Hadoop MapReduce is the best framework for processing data in batches. Spark could process data in real time and it is 100 times faster than Hadoop MapReduce in batch processing of large data sets.  Spark is able to achieve this speed through controlled partitioning. We could process that  partitioned  data with  minimal network traffic. Features Spark code can be written in Java, Scala, Python and R. Spark also supports and process structured and semi-structured data though Spark SQL. If it is absolutely necessary, then only spark will execute the code/function. This is called "Lazy Execution". Spark will be faster for processing the data because of its in-memory processing. Apache Spark is a separate tool and it has its own cluster. We can use Hadoop to implement Spark. Spark Components     ...
Read more