This article describes steps one can take to install and use Apache Hadoop on Ubuntu Linux.
Apache Hadoop is an open-source, Java-based software platform that can used to manage and process large datasets for applications that require fast and scalable data processing.
It can be used across multiple distributed computing, delivering high availability in cluster environments for your applications.
There are many ways to install Apache Hadoop and some custom installations are also available. However, for most system administrators, the method below should be the standard way to install it in their environment.
How to install and use Apache Hadoop on Ubuntu Linux
As mentioned above, Apache Hadoop is an open-source, Java-based software platform that can used to manage and process large datasets for applications that require fast and scalable data processing.
Below is how to install it when using Ubuntu Linux.
Install JAVA OpenJDK
Apache Hadoop is a Java-based application and therefore requires JDK installed. The open-source version of Java is great to use with Hadoop.
Run the commands below to install the default OpenJDK packages in Ubuntu’s repositories.
sudo apt update sudo apt install openjdk-11-jdk
Once Java is installed, you can verify its version by running the commands below.
java -version
Additional resources on installing OpenJDK on Ubuntu Linux can be found in the post below.
How to install OpenJDK on Ubuntu Linux
Create a user account for Hadoop
As I said above, there are multiple ways to install Apache Hadoop. In this article, we’re going to be created a dedicated user account to run Hadoop services as well as being able to logon to its web interface.
The account will have a password and will also be able to login to Ubuntu.
Run the commands below to create a new user called hadoop.
sudo adduser hadoop
Complete the user details when prompted.
Adding user `hadoop' ... Adding new group `hadoop' (1001) ... Adding new user `hadoop' (1001) with group `hadoop' ... Creating home directory `/home/hadoop' ... Copying files from `/etc/skel' ... New password: Retype new password: passwd: password updated successfully Changing the user information for hadoop Enter the new value, or press ENTER for the default Full Name []: Apache Hadoop Room Number []: Work Phone []: Home Phone []: Other []: Is the information correct? [Y/n] Y
Once the user is created, switch to the newly created account by running the commands below.
su - hadoop
When prompted for password, type the password created when creating the user account above.
Next, generate a SSH key for hadoop and copy the key to the authorized_keys file in the user profile.
When prompted to enter passphrase, press ENTER and leave empty for no passphrase.
ssh-keygen -t rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 640 ~/.ssh/authorized_keys
Login to accept the SSH server’s key.
ssh localhost
Accept the key when prompted:
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'localhost' (ED25519) to the list of known hosts. Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-50-generic x86_64)
Install Apache Hadoop
We are now ready to install Apache Hadoop. At this time of writing this post, the latest Hadoop version is 3.3.4.
Run the commands below to download Hadoop current version.
cd /tmp wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz
Extract the downloaded file and move it into a new folder called hadoop inside the user home directory.
tar xzf hadoop-3.3.4.tar.gz mv hadoop-3.3.4 ~/hadoop
Next, run the commands below to open the bashrc file to setup Hadoop environment variables.
nano ~/.bashrc
You will have to add the variable JAVA_HOME location inside the file. You can quickly find the default JAVA_HOME location by running the commands below.
dirname $(dirname $(readlink -f $(which java)))
That should output similar line as below.
/usr/lib/jvm/java-11-openjdk-amd64
Copy and paste all the lines below and append to the bashrc file. Make sure the highlighted line matches your JAVA_HOME path.
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 export HADOOP_HOME=/home/hadoop/hadoop export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export HADOOP_YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
Save the load the new config.
source ~/.bashrc
You will also want to confirm JAVA_HOME is correct in the hadoop-env.sh file.
nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh
Under Generic settings for HADOOP, find the export JAVA_HOME line and uncomment. Then add the JAVA_HOME path and save.
### # Generic settings for HADOOP Many sites configure these options outside of Hadoop, # such as in /etc/profile.d # The java implementation to use. By default, this environment # variable is REQUIRED on ALL platforms except OS X! export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
Save and exit.
Configure Apache Hadoop
At this point, we’re ready to configure Hadoop to begin accepting connections. First, create two folders (namenode and datanode) inside the hdfs directory.
mkdir -p ~/hadoopdata/hdfs/{namenode,datanode}
Next, edit the core-site.xml file below.
nano $HADOOP_HOME/etc/hadoop/core-site.xml
Then between the configuration lines, enter the hostname value to reference your system name.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
Save the file and exit.
Next, edit the hdfs-site.xml file.
nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml
Then create configuration properties to reference the namenode and datanode folders created above.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<name>dfs.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
<name>dfs.data.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</configuration>
Save and exit.
Next, edit the mapred-site.xml file.
nano $HADOOP_HOME/etc/hadoop/mapred-site.xml
Then create configuration properties for the mapreduce framework.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Save and exit.
Next, edit the yarn-site.xml file.
nano $HADOOP_HOME/etc/hadoop/yarn-site.xml
Then create configuration properties for yarn.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
Save and exit.
Start and access Apache Hadoop portal
Finally, we’re ready to start and access Hadoop portal. First, run the commands below to format Hadoop namenode.
hdfs namenode -format
If successfully formatted, you should see a message in the results with similar line below:
Storage directory /home/hadoop/hadoopdata/hdfs/namenode has been successfully formatted.
Next, run the commands below to start Hadoop.
start-all.sh
A successful start message will look like the one below:
Starting namenodes on [localhost] Starting datanodes Starting secondary namenodes [Ubuntu2204] Ubuntu2204: Warning: Permanently added 'ubuntu2204' (ED25519) to the list of known hosts. Starting resourcemanager Starting nodemanagers
Next, open your web browser and browse to the server’s hostname or IP address followed by port number 9870 to view Hadoop summery portal.
Hadoop application portal can be accessed by going to the server’s hostname or IP address followed by port number 8088.
That should be it!
Conclusion:
This post showed you how to install and use Apache Hadoop on Ubuntu Linux. If you find any error above or have something to add, please use the comment form below.
