II.2 HA Config (MS Azure)

Star InactiveStar InactiveStar InactiveStar InactiveStar Inactive


  • Lab : II.1 Hadoop Install (MS Azure)

Eliminate the Secondary Node


NB : In an HA cluster, the Standby NameNode also performs checkpoints of the namespace state, 
and thus it is not necessary to run a Secondary NameNode, CheckpointNode, or BackupNode in an HA cluster.
Now it would be incorrect to deploy a SecondaryNameNode alongside an HA NameNode pair as Standby Namenode does three things:
1. merging fsimage and edits-log files. (Secondary-namenode's work) 2. receive online updates of the file system meta-data, apply them to its memory state and persist them on disks just like the name-node does. 3. Cluster will switch over to the new name-node (this standby-node) if the active namenode die.


1.1 Remove Secondary Node informations from hdfs-site.xml configuration file


[root@vmnnmaster01 hadoop]# vi hdfs-site.xml 
#remove the following properties between "<configuration> ... </configuration>"
<property> <name>dfs.namenode.secondary.http-address</name> <value>vmnnsecondary01.northeurope.cloudapp.azure.com:50090</value> <description>SecondaryNameNodeHostname</description> </property>


1.2 Delete Secondary Node VM from Azure



[root@NNMaster01 ~]#cd  /opt/
[root@NNMaster01 opt]#mkdir hive
[root@NNMaster01 opt]#cd ~/Desktop/Hadoop
[root@NNMaster01 opt]#ls 
[root@NNMaster01 opt]#tar zxvf apache-hive-2.1.1-bin.tar.gz
[root@NNMaster01 opt]#ls 
[root@NNMaster01 opt]#cp -a apache-hive-2.1.1-bin/. /opt/hive




Setting up environment for Hive

Add to .bash_profile

[root@NNMaster01 opt]#cd ~ 
[root@NNMaster01 ~]#vi .bash_profile
#Append the following lines at the end of the file:
### HIVE env variables
export HIVE_HOME=/opt/hive export PATH=$PATH:$HIVE_HOME/bin export CLASSPATH=$CLASSPATH:/opt/Hadoop/lib/*:. export CLASSPATH=$CLASSPATH:/opt/hive/lib/*:.
[root@NNMaster01 ~]# source .bash_profile 
[root@NNMaster01 ~]# echo $HIVE_HOME 
[root@NNMaster01 ~]# cd $HIVE_HOME/conf 
[root@NNMaster01 conf]# cp  hive-env.sh.template hive-env.sh 
[root@NNMaster01 ~]# vi  hive-env.sh 



Configuring Hive


3.1. Create Dirctory tree :
[root@NNMaster01 ~]# hdfs dfs -mkdir -p /user/hive/warehouse 
[root@NNMaster01 ~]# hdfs dfs -mkdir -p /tmp/hive 
[root@NNMaster01 ~]# hdfs dfs -chmod 777 /tmp  
[root@NNMaster01 ~]# hdfs dfs -chmod 777 /user/hive/warehouse 
[root@NNMaster01 ~]# hdfs dfs -chmod 777 /tmp/hive 

Verification via WebBrowser (Port 50070)



3.2. Initialized the Schema:
[root@NNMaster01 ~]#rm /opt/hive/lib/log4j-slf4j-impl-2.4.1.jar
[root@NNMaster01 ~]#schematool -initSchema -dbType derby



3.3. Test Hive:
[root@NNMaster01 ~]#hive 
hive>show tables; 
hive>create database mytestdb; 
hive>show databases; 
### The database creates in a default location of the Hive warehouse. In Our Home Lab, Hive database store in a /user/hive/warehouse.
hive>USE mytestdb; #The command to use the database is USE database_name>



About Helix

Ball tip biltong pork belly frankfurter shankle jerky leberkas pig kielbasa kay boudin alcatra short loin.

Jowl salami leberkas turkey pork brisket meatball turducken flank bilto porke belly ball tip. pork belly frankf urtane bilto

Latest News

21 August 2017