码迷,mamicode.com
首页 > 其他好文 > 详细

(02)Hadoop环境的搭建(伪分布模式)

时间:2020-02-10 13:55:50      阅读:75      评论:0      收藏:0      [点我收藏+]

标签:home   rman   exec   tfs   mis   mapreduce   false   shu   oca   

一、安装hadoop  

  1、将文件解压到/usr/local

[root@localhost src]# tar -zxvf hadoop-2.4.1.tar.gz -C /usr/local/

  2、配置环境变量

[root@localhost src]# vim /etc/profile

  增加以下配置

export HADOOP_HOME=/usr/local/hadoop-2.4.1
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

  使环境变量生效,执行以下语句

[root@localhost src]# source /etc/profile

  3、修改hadoop-env.sh,设置jdk

[root@localhost hadoop]# vim /usr/local/hadoop-2.4.1/etc/hadoop/hadoop-env.sh

  如图所示:

技术图片

  4、修改hdfs-site.xml,设置数据备份节点和权限检查

[root@localhost hadoop]# vim /usr/local/hadoop-2.4.1/etc/hadoop/hdfs-site.xml 

  添加以下内容:

<property>
        <name>dfs.replication</name>
        <value>1</value>
</property>
<property>
        <name>dfs.permissions</name>
        <value>false</value>
</property>

  如图所示:

技术图片

  5、修改core-site.xml,设置namenode的ip和端口,以及数据存放目录

[root@localhost hadoop]# vim /usr/local/hadoop-2.4.1/etc/hadoop/core-site.xml 

  添加以下内容:

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://192.168.7.151:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop-2.4.1/tmp</value>
</property>

  如图所示:

技术图片

   创建数据存放目录:

[root@localhost hadoop]# mkdir /usr/local/hadoop-2.4.1/tmp

  6、修改mapred-site.xml,设置

[root@localhost hadoop]# vim /usr/local/hadoop-2.4.1/etc/hadoop/mapred-site.xml

  添加以下内容:

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

  如图所示:

技术图片

  7、修改yarn-site.xml,设置

[root@localhost hadoop]# vim /usr/local/hadoop-2.4.1/etc/hadoop/yarn-site.xml

  添加以下内容:

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>192.168.7.151</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

  如图所示:

技术图片

  8、格式化namenode,使其产生一些必要的源信息

[root@localhost hadoop-2.4.1]# hdfs namenode -format

  执行成功如图所示:

技术图片

   执行成功以后,/usr/local/hadoop-2.4.1/tmp下面已经生成了目录dfs

  9、启动hadoop

  环境变量已经配置了hadoop/sbin,直接执行start-all.sh即可

[root@localhost hadoop-2.4.1]# start-all.sh

  启动成功如下所示:

[root@localhost hadoop-2.4.1]# start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It‘s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack‘.
20/02/10 11:46:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [192.168.7.151]
The authenticity of host ‘192.168.7.151 (192.168.7.151)‘ can‘t be established.
ECDSA key fingerprint is SHA256:vfvHgc0RFlOQtUhI3pdFozBK3iehM6/GmwsFaXaYMyo.
ECDSA key fingerprint is MD5:aa:63:31:82:31:04:51:35:d5:4d:46:36:59:21:b2:0d.
Are you sure you want to continue connecting (yes/no)? yes
192.168.7.151: Warning: Permanently added ‘192.168.7.151‘ (ECDSA) to the list of known hosts.
root@192.168.7.151‘s password: 
192.168.7.151: starting namenode, logging to /usr/local/hadoop-2.4.1/logs/hadoop-root-namenode-localhost.localdomain.out
192.168.7.151: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
192.168.7.151: It‘s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack‘.
The authenticity of host ‘localhost (::1)‘ can‘t be established.
ECDSA key fingerprint is SHA256:vfvHgc0RFlOQtUhI3pdFozBK3iehM6/GmwsFaXaYMyo.
ECDSA key fingerprint is MD5:aa:63:31:82:31:04:51:35:d5:4d:46:36:59:21:b2:0d.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added ‘localhost‘ (ECDSA) to the list of known hosts.
root@localhost‘s password: 
localhost: starting datanode, logging to /usr/local/hadoop-2.4.1/logs/hadoop-root-datanode-localhost.localdomain.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host ‘0.0.0.0 (0.0.0.0)‘ can‘t be established.
ECDSA key fingerprint is SHA256:vfvHgc0RFlOQtUhI3pdFozBK3iehM6/GmwsFaXaYMyo.
ECDSA key fingerprint is MD5:aa:63:31:82:31:04:51:35:d5:4d:46:36:59:21:b2:0d.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added ‘0.0.0.0‘ (ECDSA) to the list of known hosts.
root@0.0.0.0‘s password: 
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.4.1/logs/hadoop-root-secondarynamenode-localhost.localdomain.out
0.0.0.0: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
0.0.0.0: It‘s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack‘.
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It‘s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack‘.
20/02/10 11:47:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.4.1/logs/yarn-root-resourcemanager-localhost.localdomain.out
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It‘s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack‘.
root@localhost‘s password: 
localhost: starting nodemanager, logging to /usr/local/hadoop-2.4.1/logs/yarn-root-nodemanager-localhost.localdomain.out
localhost: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
localhost: It‘s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack‘.

  已经启动了DataNode、NodeManager、ResourceManager、SecondaryNameNode、NameNode,如图所示:

技术图片

 

(02)Hadoop环境的搭建(伪分布模式)

标签:home   rman   exec   tfs   mis   mapreduce   false   shu   oca   

原文地址:https://www.cnblogs.com/javasl/p/12290378.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!