Hadoop 3.3.6 Installation on Ubuntu 24.04.1 LTS

 1. Ctrl+Alt+T ,to open terminal and run the commands one by one,

    a) sudo apt-get update


                If any problem for running above command then run below commands,
                
                    sudo rm -rf /var/lib/apt/lists/*
                    sudo apt-get update
                    sudo apt-get update -o Acquire::http::No-Cache=True
                    
    b) sudo apt-get install openjdk-8-jdk
    
        At the moment, Apache Hadoop 3.x fully supports Java 8. The OpenJDK 8 package in Ubuntu contains both the runtime environment and development kit.
    
    c) sudo apt-get install openssh-server

2. after installation of opennsh-server , to generate the rsa key run the following commands and without any input just press enter 4 times,

    ssh-keygen -t rsa

3.cat .ssh/id_rsa.pub>.ssh/authorized_keys

    or
    
    cd .ssh
    ls
    cat authorized_keys

4.ssh localhost
    > Yes

 

5. Download Hadoop 3.3.6 and do below config file changes.

    Download hadoop 3.3.6 setup from below link.

    https://archive.apache.org/dist/hadoop/common/hadoop-3.3.6/

    Extract above downloaded zip file under below directory,

    /home/karpagaraj/hadoop_setup

    ('karpagaraj' is username and 'hadoop_setup' is directory we need to create under 'home' directory)

    once extracted above hadoop downloaded zip file then go the directory : /hadoop-3.3.6/etc/hadoop and follow below steps,

5.1) open the core-site.xml , by right click and open with gedit

Within the core-site.xml , in between the <configuration> tag paste the following code

<configuration>
 <property>
 <name>fs.defaultFS</name>
 <value>hdfs://localhost:9000</value>  </property>
 <property>
<name>hadoop.proxyuser.dataflair.groups</name> <value>*</value>
 </property>
 <property>
<name>hadoop.proxyuser.dataflair.hosts</name> <value>*</value>
 </property>
 <property>
<name>hadoop.proxyuser.server.hosts</name> <value>*</value>
 </property>
 <property>
<name>hadoop.proxyuser.server.groups</name> <value>*</value>
 </property>
</configuration>


5.2) create below directories,

mkdir -p /home/karpagaraj/hadoop_setup/yarn/namenode

mkdir -p /home/karpagaraj/hadoop_setup/yarn/datanode


5.3) open hdfs-site.xml, in between the <configuration> tag paste the following code and change the username.

<property>
 <name>dfs.replication</name>
 <value>1</value>
</property>

<property>
 <name>dfs.namenode.name.dir</name>
 <value>file:/home/karpagaraj/yarn/namenode</value>
</property>

<property>
 <name>dfs.datanode.data.dir</name>
 <value>file:/home/karpagaraj/yarn/datanode</value>
</property>

note: karpagaraj is your username.

5.4) open mapred-site.xml., in between the <configuration> tag paste the following code

 <property>
 <name>mapreduce.framework.name</name>  <value>yarn</value>
 </property>
 <property>
 <name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
 </property>

5.5) open yarn-site.xml, ,in between the <configuration> tag paste the following code

 <property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce_shuffle</value>
 </property>
 <property>
 <name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREP END_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
 </property>

 

6. Set the .bashrc

Press ctrl+h in ubuntu home and open .bashrc file then paste the below lines,

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=$HOME/hadoop_setup/hadoop-3.3.6
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_STREAMING=$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export PDSH_RCMD_TYPE=ssh

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin


07. type below in terminal,

bash (OR) source ~/.bashrc

Note : it will refresh .bashrc file.

 

08. Format the namenode and filesystem,

Go to cd $HOME/hadoop_setup/hadoop-3.3.6/bin in terminal
Note : only first time after the hadoop setup
Run the below command,
 ./hdfs namenode -format

format the file system,

export PDSH_RCMD_TYPE=ssh

 

09. Start all services,
Goto cd ../sbin

Run the below command,
./start-all.sh

 From next time on-wards you can run above start-all.sh command from anywhere in terminal. 

 

10. type jps in terminal, you can see all daemons running,

NameNode
DataNode
SecondaryNameNode
ResourceManager
NodeManager

 

11. Login to "localhost:9870" webconsole in browser -> under "utility" menu find the option -> "browse the file system" to view the hdfs file.



Comments

Post a Comment

Popular posts from this blog

Hive File Formats

Image
 Hive supports several file formats: Text File SequenceFile RCFile Avro Files ORC Files Parquet Custom INPUTFORMAT and OUTPUTFORMAT      The hive.default.fileformat configuration parameter which is avaialble in hive-site.xml determines the format to use if it is not specified in a CREATE TABLE or ALTER TABLE statement.        Text file is the parameter's default value. What is File Format ? File Format is a way in which information is stored or encoded in a computer file.  In Hive it refers to how records are stored inside the file. These file formats mainly vary between data encoding, compression rate, usage of space and disk I/O. Hive does not verify whether the data that you are loading matches the schema for the table or not. However, it verifies if the file format matches the table definition or not. Text File TEXT FILE format is a famous input/output format used in Hadoop. In Hive if we define a table as TEXTFILE it can load data o...
Read more

Why We Need Hadoop?

What is Hadoop? It is an open source framework or system which was fully developed by Java.  It is allow us to implement Big Data.  Hadoop can be used for storing and processing huge datasets with cluster of commodity hardware. Because its stores the data and processing in a distributed manner.  It made up of several components which entirely open source. Doug Cutting is the inventor of hadoop. He used elephant symbol. His child used to play with elephant toy. So he used that toy name as symbol and named for new invention. Why we need to go for Hadoop? 1. High Throughput and Low latency queries Less amount of time it will take to run/process real time queries or near real time queires. We can achieve high throughput in hadoop using below techniques.  Parallel Processing WORM Data Locality 2. Distributed computing A distributed computer system consists of multiple software components that are on multiple computers, but run as a single system.  The computers that ...
Read more

Hive Data Types

Image
 All the data types in Hive are classified into four types, given as follows: Column Types Literals Null Values Misc Types Complex Types Column Types Integral Types Integer type data can be specified using integral data types, INT. When the data range exceeds the range of INT, you need to use BIGINT and if the data range is smaller than the INT, you use SMALLINT. TINYINT is smaller than SMALLINT. The data ranges for TINYINT, SMALLINT, INT, BIGINT is below, TINYINT (1-byte signed integer, from -128 to 127) SMALLINT (2-byte signed integer, from -32,768 to 32,767) INT (4-byte signed integer, from -2,147,483,648 to 2,147,483,647) BIGINT (8-byte signed integer, from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807) Example create table test1(id tinyint) row format delimited fields terminated by ','; insert into test1 values(1); insert into test1 values(127); insert into test1 values(128); hive> select * from test1; OK 1 127 NULL Time taken: 0.055 seconds, Fetched: 3 row(s)...
Read more