hadoop Getting started with hadoop

Remarks

Versions

Hadoop overview and HDFS

Hadoop is an open-source software framework for storage and large-scale processing of data-sets in a distributed computing environment. It is sponsored by Apache Software Foundation. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

History

• Hadoop was created by Doug Cutting and Mike Cafarella in 2005.
• Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant.
•  It was originally developed to support distribution for the search engine project.

Major modules of hadoop

Hadoop Distributed File System (HDFS): A distributed file system that provides high-throughput access to application data. Hadoop MapReduce: A software framework for distributed processing of large data sets on compute clusters.

Hadoop File System  Basic Features

Highly fault-tolerant. High throughput. Suitable for applications with large data sets. Can be built out of commodity hardware.

Namenode and Datanodes

Master/slave architecture. HDFS cluster consists of a single Namenode, a master server that manages the file system namespace and regulates access to files by clients. The DataNodes manage storage attached to the nodes that they run on. HDFS exposes a file system namespace and allows user data to be stored in files. A file is split into one or more blocks and set of blocks are stored in DataNodes. DataNodes: serves read, write requests, performs block creation, deletion, and replication upon instruction from Namenode.

HDFS is designed to store very large files across machines in a large cluster. Each file is a sequence of blocks. All blocks in the file except the last are of the same size. Blocks are replicated for fault tolerance. The Namenode receives a Heartbeat and a BlockReport from each DataNode in the cluster. BlockReport contains all the blocks on a Datanode.

Hadoop Shell Commands

Common commands used:-
ls Usage: hadoop fs –ls Path(dir/file path to list). Cat Usage: hadoop fs -cat PathOfFileToView

Link for hadoop shell commands:- https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/FileSystemShell.html

Installation of Hadoop on ubuntu

Creating Hadoop User:

sudo addgroup hadoop
 

Adding a user:

sudo adduser --ingroup hadoop hduser001
 

Configuring SSH:

su -hduser001
ssh-keygen -t rsa -P ""
cat .ssh/id rsa.pub >> .ssh/authorized_keys
 

Note: If you get errors [bash: .ssh/authorized_keys: No such file or directory] whilst writing the authorized key. Check here.

Add hadoop user to sudoer's list:

sudo adduser hduser001 sudo
 

Disabling IPv6:

Installing Hadoop:

sudo add-apt-repository ppa:hadoop-ubuntu/stable
sudo apt-get install hadoop
 

Installation or Setup on Linux

A Pseudo Distributed Cluster Setup Procedure

Prerequisites

• Install JDK1.7 and set JAVA_HOME environment variable.

• Create a new user as "hadoop".

  useradd hadoop

• Setup password-less SSH login to its own account

   su - hadoop
   ssh-keygen
   << Press ENTER for all prompts >>
   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
   chmod 0600 ~/.ssh/authorized_keys
   

• Verify by performing ssh localhost

• Disable IPV6 by editing /etc/sysctl.conf with the followings:

   net.ipv6.conf.all.disable_ipv6 = 1
   net.ipv6.conf.default.disable_ipv6 = 1
   net.ipv6.conf.lo.disable_ipv6 = 1
   

• Check that using cat /proc/sys/net/ipv6/conf/all/disable_ipv6

  (should return 1)

Installation and Configuration:

• Download required version of Hadoop from Apache archives using wget command.

   cd /opt/hadoop/
   wget http:/addresstoarchive/hadoop-2.x.x/xxxxx.gz
   tar -xvf hadoop-2.x.x.gz
   mv hadoop-2.x.x.gz hadoop 
  (or)
  
   ln -s hadoop-2.x.x.gz hadoop
   chown -R hadoop:hadoop hadoop
   

• Update .bashrc /.kshrc based on your shell with below environment variables

    export HADOOP_PREFIX=/opt/hadoop/hadoop
    export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
    export JAVA_HOME=/java/home/path
    export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$JAVA_HOME/bin
   

• In $HADOOP_HOME/etc/hadoop directory edit below files

  • core-site.xml

    <configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:8020</value>
        </property>
    </configuration> 
     

  • mapred-site.xml

    Create mapred-site.xml from its template

    cp mapred-site.xml.template mapred-site.xml

      <configuration>
          <property>
              <name>mapreduce.framework.name</name>
              <value>yarn</value>
          </property>
      </configuration> 
     

  • yarn-site.xml

     <configuration>
         <property>
             <name>yarn.resourcemanager.hostname</name>
             <value>localhost</value> 
         </property>
         <property>
             <name>yarn.nodemanager.aux-services</name>
             <value>mapreduce_shuffle</value>
         </property>
     </configuration>
     

  • hdfs-site.xml

     <configuration>
         <property>
             <name>dfs.replication</name>
             <value>1</value>
         </property>
         <property>
             <name>dfs.namenode.name.dir</name>
             <value>file:///home/hadoop/hdfs/namenode</value>
         </property>
         <property>
             <name>dfs.datanode.data.dir</name> 
             <value>file:///home/hadoop/hdfs/datanode</value> 
         </property>
     </configuration>
     

  Create the parent folder to store the hadoop data

  mkdir -p /home/hadoop/hdfs
   

• Format NameNode (cleans up the directory and creates necessary meta files)

  hdfs namenode -format
   

• Start all services:

  start-dfs.sh && start-yarn.sh
  mr-jobhistory-server.sh start historyserver
   

Instead use start-all.sh (deprecated).

• Check all running java processes

  jps 
   

• Namenode Web Interface: http://localhost:50070/

• Resource manager Web Interface: http://localhost:8088/

• To stop daemons(services):

  stop-dfs.sh && stop-yarn.sh
  mr-jobhistory-daemon.sh stop historyserver
   

Instead use stop-all.sh (deprecated).