标签:XML 成功 span type classpath xsl linux arc gre
10.11.84.4 web-crawler--1.novalocal master/slave 10.11.84.5 web-crawler--2.novalocal slave 10.11.84.6 web-crawler--3.novalocal slave
groupadd hadoop
4.创建用户hadoop
useradd -d /home/hadoop -g hadoop hadoop
chown -R hadoop:hadoop /usr/local/hadoop
5. 每台机器安装jdk(安装在相同的目录)
[hadoop@hadoop2 ~]$ java -version openjdk version "1.8.0_131" OpenJDK Runtime Environment (build 1.8.0_131-b12) OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode) [hadoop@hadoop2 ~]$ rpm -qa | grep java java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64 javapackages-tools-3.4.1-11.el7.noarch tzdata-java-2017b-1.el7.noarch java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64 java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64 java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 python-javapackages-3.4.1-11.el7.noarch
2)卸载openjdk
rpm -qa | grep java | xargs yum -y remove
3)安装sunjdk
vim /etc/profile 末尾添加如下内容 export JAVA_HOME=/usr/local/jdk export JRE_HOME=${JAVA_HOME}/jre export PATH=${JAVA_HOME}/bin:${PATH} export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:${CLASSPATH}
4)查看jdk是否安装成功
[root@hadoop2 local]# java -version java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
[hadoop@hadoop1 ~]$ ssh-keygen -t rsa -P ‘‘ #一路直接回车 发送master 机器~/.ssh/id_rsa.pub 到所有slave节点 Scp id_rsa.pub hadoop@10.11.84.5:~ Scp id_rsa.pub hadoop@10.11.84.6:~ 让后在每台slave节点hadoop 用户下执行如下命令 Cd ~/.ssh Cat ../id_rsa.pub >> authorized_keys Chmod 644 authorized_keys
7.关闭所有机器防火墙、selinux
1)centos 7 以下 防火墙 查看防火墙状态 service iptables status 临时关闭防火墙 service iptables stop 永久关闭防火墙 chkconfig iptables off selinux setenforce 0 #临时关闭 永久关闭需要修改如下文件 vim /etc/selinux/config 将SELINUX=enforcing改为SELINUX=disabled 2)centos 7 firewall-cmd --reload #重启firewall systemctl stop firewalld.service #停止firewall systemctl disable firewalld.service #禁止firewall开机启动 firewall-cmd --state #查看默认防火墙状态(关闭后显示notrunning,开启后显示running) selinux 同上
8.修改hadoop配置文件
Core-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/tmp</value> <description>Abase for other temporary directories.</description> </property> <property> <name>fs.default.name</name> <value>hdfs://web-crawler--1.novalocal:9000</value> </property> </configuration> hadoop-env.sh 添加java_home export JAVA_HOME=/usr/local/jdk hdfs-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.name.dir</name> <value>/home/hadoop/name</value> </property> <property> <name>dfs.data.dir</name> <value>/home/hadoop/data</value> </property> </configuration> mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>http://web-crawler--1.novalocal:9001</value>;;; </property> <property> <name>mapred.system.dir</name> <value>/home/hadoop/mapred/system</value> </property> <property> <name>mapred.local.dir</name> <value>/home/hadoop/mapred/local</value> </property> </configuration> 5)masters web-crawler--1.novalocal 6)slaves web-crawler--1.novalocal web-crawler--2.novalocal web-crawler--3.novalocal
Vim /etc/profile (root) export ANT_HOME=/usr/local/ant export PATH=${PATH}:${ANT_HOME}/bin export HADOOP_HOME=/usr/local/hadoop-1.2.1 export JAVA_HOME=/usr/local/jdk export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH export PATH=/usr/local/lib/phantomjs/bin:$PATH export ANT_HOME=/usr/local/ant export PATH=${PATH}:${ANT_HOME}/bin export PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin
10.启动hadoop
1)在master上format namenode hadoop namenode -format 2)启动hadoop sh /user/local/hadoop/bin/start-all.sh
11.进行简单测试
Hadoop fs -mkdir -p /user/hadoop/private/in Hadoop fs -put /user/local/hadoop/bin/start-all.sh /user/hadoop/private/in Cd /usr/local/hadoop Hadoop jar hadoop-examples-1.2.1.jar wordcount /user/hadoop/private/in /user/hadoop/private/out
标签:XML 成功 span type classpath xsl linux arc gre
原文地址:https://www.cnblogs.com/jason-dong/p/9447323.html