Hadoop Zone

Numeric datatypes int Singned 32-bit integer Example: 8 (or) 5 (or) any number long Singned 64-bit integer Example: 5L (or) 5l float 32-bit floating point Example: 5.5F (or) 5.5f (or) 5.0f (or) any number with decimal       (early version, recently float has been dropped) double 64-bit floating point Example: 10.5 (or) 5.0 (or) 1.5 (or) 1.5e2 (or) 1.5E2 (or) any number with decimal Non-numeric or Arrays datatype chararray  character array(string) Example: 'karpagaraj' or 'hello there' or any string bytearray blob Example: any binary data Non-numeric datatype boolean true or false (case sensitive) datetime Example 1970-01-01T00:00:00.000+00:00 null unknown or data missing or any data element can be null. Complex datatype tuple ordered set of fields Example: (1,3) like that bag collection of tuples Example: {(1,3),(4,6)} map collection of tuples: set of key/value pairs Example: [k1#v1, k2#v2] atom  any single value in Pig Latin, irrespective of their data, t...

Apache Pig Running Mode You can run/execute Pig latin stetements and commands using three modes. Interactive mode Batch mode Embedded mode(UDF).  Interactive mode Interactive processing, we can use Apache Tez or Apache Spark as processing engine. Interactive processing means that the person needs to provide the computer with instructions whilst it is doing the processing. i.e the user 'interacts' with the computer to complete the processing. For example, if you use an online system to book a hotel room, you will fill in a web form, submit it and it will come back to inform you of the room you have booked. You can run Pig in interactive mode using the Grunt shell. Invoke the Grunt shell using the "pig" command (as shown below) and then enter your Pig Latin statements and Pig commands interactively at the command line. Batch mode We can process the data set by set or batch by batch with limited number of records/data/rows Like mapreduce, hive. We will do this in pig usi...

Load & Store Opertators Load Store Diagnostic Opertators Dump Describe Explain Illustrate Grouping & Joining Opertators Group Cogroup Join - Single Key | Multi Key join Cross Combining & Splitting Opertators Union Split Filtering Opertators Distinct Filter Foreach Sorting Opertators order by  limit Built-In Functions Concat tokenize flatten bag, tuple, map, top PigStorage TextLoader BinStorage

Apache Pig It is another hadoop framework for Non Java Developers. Originally developed at Yahoo! (2007).  PIG can eat anything that means it can handle structured and semi-structured. It is using Pig Latin Language. It is a data-flow language. It is Intermediate language between java and hive.  you want to play around with data in a Hadoop cluster without having to write hundreds or thousands of lines of Java MapReduce code, you most likely will use either Hive (using the  Hive Query Language HQL) or Pig. Hive is a SQL-like language which compiles to Java map-reduce code, while Pig is a data flow language which allows you to specify your map-reduce data pipelines using high level abstractions. Why Pig? Map Reduce requires programmers. For pig only less programming.  No Java knowledge.  Development time is very less.  Can process any kind of data (structured, semi-structured, un-structured).  Good for Ad-hoc queries.  Extensible by UDF by Java...

What is Impala Impala server is a distributed, massively parallel processing (MPP) database engine. Distributed Massively Parallel processing engine Developed by Cloudera. Based on Google 2010 dremel paper. Direct access to HDFS/HBase data via impala engine. Hive and Impala are similar. Both are meant for “SQL on Hadoop”. Any table create on impala can access from hive and impala Impala SQL is a subset of Hive Query Language. It provides high performance and low latency compared to other SQL engines for Hadoop. Why Impala when we have Hive Hive queries incurs overheads of starting MapReduce jobs: Job setup and creation Start JVMs Slot assignment Input split creation Map tasks generation Hive inherits the latency of MapReduce. The latency with a MapReduce jobs is higher since it has the overhead of creating a job, starting JVMs, calculating input splits etc. Higher latency, poor performance. So, Hive cannot be used interactively by the user. Impala does not build on MapReduce. It is its...

Impediments of Hadoop Developed to overcome limitations of Hadoop/MapReduce.  We should look at Spark as an alternative to Hadoop MapReduce rather than a replacement to Hadoop because of below reasons.  Intermediate Results gathering Processing Techniques - Almost Realtime Polyglot Deployment and Storage(Scalable) Powerful Cache and Good Speed (In-memory) Parallelize Lazy Evaluation Advantage of Spark over Other Frameworks In-Memory Processing - Many times it is faster than other processing engine like MapReduce, tez, mesos, etc. Keeps data In-Memory (RAM), as it will do In-memory processing. Iterative algorithms are faster as data is not being written to disk between jobs. Intermediate results will be maintained in in-memory. But In hadoop, it will keep intermediate results in disks. Processing Technique Sequencial Processing means, before RDBMS system introduced, FIFO technique. It will process data/job by sequencially. Random Processing means, RDBMS will do this. It will...

What is Hadoop? It is an open source framework or system which was fully developed by Java.  It is allow us to implement Big Data.  Hadoop can be used for storing and processing huge datasets with cluster of commodity hardware. Because its stores the data and processing in a distributed manner.  It made up of several components which entirely open source. Doug Cutting is the inventor of hadoop. He used elephant symbol. His child used to play with elephant toy. So he used that toy name as symbol and named for new invention. Why we need to go for Hadoop? 1. High Throughput and Low latency queries Less amount of time it will take to run/process real time queries or near real time queires. We can achieve high throughput in hadoop using below techniques.  Parallel Processing WORM Data Locality 2. Distributed computing A distributed computer system consists of multiple software components that are on multiple computers, but run as a single system.  The computers that ...