Lazy Evaluation & DAG in Spark

Lazy Evaluation

  • Evaluation means executing the logic. Lazy means don't execute immediately until some action is performed. So code will execute when the action is performed.
  • Lazy evaluation means spark will not load or transform data unless an action is performed.
    • Load file into RDD
    • Filter the RDD
    • Count no of elements (Only now loading filtering happens)
  • Helps internally optimize operations and resource usage.
  • We can write chaining operations and We can watch out during the troubleshooting.
  • If you only used transformations like textFile then it will not execute immediately it will create DAG. Those transformations are compiled into DAG. DAG is a complex piece of module. 
  • You can perform bunch of transformations before invoking action. 
  • Until Action is performed the spark compiler checks whether required transformations code is compiled in DAG. 
  • DAG is graph so it will be having multiple paths, it will take any one path and execute the transformations whenever the action is called.

DAG

  • In Apache Spark, a DAG (Directed Acyclic Graph) is a fundamental concept that represents the logical execution plan of a Spark job.
  • The DAG is "Directed" because it shows the flow of data from one transformation to another, and it's "Acyclic" because there are no cycles or loops in the graph. This means that each transformation depends only on its immediate predecessors and not on itself or any transformations further downstream.
  • The DAG is built when you create a Spark DataFrame or RDD and apply transformations to it.
  • Spark's Catalyst optimizer analyzes this DAG to generate an optimized physical execution plan that minimizes data movement and maximizes parallelism.

Comments

Post a Comment

Popular posts from this blog

HDFS Architecture

Image
HDFS Services/Daemons           1. Name Node (NN)           2. Data node (DN)           3. Secondary Name Node (SNN)             4. Standby Name Node (Standby) These 3 nodes deal with HDFS which used to store the data into HDFS. V1 Map Reduce Daemons     Yarn Daemons (Map Reduce V2)           4. Job Tracker                  4. Resource Manager           5. Task Tracker                  5. Node Manager These daemons are responsible for running the MapReduce jobs. Start-dsf.sh service for starting hdfs daemons and start-yarn.sh for starting yarn daemons separately. start-all.sh service for starting all above daemons. Name Node (NN) Only one name node available for HDFS. It is a master node. Stores only Meta data of all the data....
Read more

Hive Joins

 Map Join In Apache Hive, there is a feature that we use to speed up Hive queries. Basically, that feature is what we call Map join in Hive. Apache Hive Map Join is also known as Auto Map Join, or Map Side Join, or Broadcast Join. Also, we use Hive Map Side Join since one of the tables in the join is a small table and can be loaded into RAM. Map Join could be performed within a mapper without using a Map/Reduce step. However, map joins in Hive are way faster than the regular joins since no reducers are necessary. If we are joining two tables using map join then one table should have minimal data compare with the other one table. If that small table is having data upto or less than 25MB then it considered like small table. If it is exceeding 25MB but still it is small table(1TB) compare with the other one(10 TB) then we have to use some different method in map join to handle this case. In a Map join no reduce task is required that is why it improves the query performance significant...
Read more

Scala Overview & History & Popularity & Usage

Image
 Scala Overview Scala is an object-oriented and functional programming language. Scala is a general-purpose programming language.  It supports object oriented, functional and imperative programming approaches.  It is a pure object-oriented language in the sense that every value is an object and functional language in the sense that every function is a value. In Scala, everything is an object whether it is a function or a number.  It is a strong static type language.  It does not have concept of primitive data. The name of Scala is derived from word scalable which means it can grow with the demand of users. Scala is not an extension of Java, but it is completely interoperable with it. While compilation, Scala file translates to Java bytecode and runs on JVM (Java Virtual machine). History Scala was created and developed by Martin Odersky. Martin started working on Scala in 2001 at the Ecole Polytechnique Federale de Lausanne (EPFL). It was officially released on ...
Read more