hadoop单机简单搭建

系统:centos7

hadoop:3.0.1

java:10

下载并解压java包，编辑环境变量

vi /etc/profile

export JAVA_HOME=/data/java/jdk-10
export JRE_HOME=/data/java/jkd-10/jre
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

下载并解压hadoop包,进入hadoop etc/hadoop目录,vi hadoop-env.sh

新增或修改：

export JAVA_HOME=/data/java/jdk-10

export HADOOP_HOME=/data/hadoop/hadoop-3.0.1

export PATH=$PATH:/data/hadoop/hadoop-3.0.1/bin

vi core-site.xml,在<configuration></configuration>中间加入

<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/data/hadoop/tmp</value>
</property>

vi mapred-site.xml,在<configuration></configuration>中间加入

<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>

vi hdfs-site.xml,在<configuration></configuration>中间加入

<property>
<name>dfs.namenode.name.dir</name>
<value>file:/data/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/data/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>localhost:9001</value>
</property>

vi yarn-site.xml,在<configuration></configuration>中间加入

<property><name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>localhost:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>localhost:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>localhost:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>localhost:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>localhost:8088</value> </property>

进入hadoop bin目录,在 start-dfs.sh,stop-dfs.sh,新增

HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

在strat-yarn.sh,stop-yarn.sh,新增

YARN_RESOURCEMANAGER_USER="root"
YARN_NODEMANAGER_USER="root"

配置ssh免密登陆:

ssh-keygen -t rsa 生成密钥

cd ~/.ssh

cat ./id_rsa.pub >> ./authorized_keys

chmod 600 ./authorized_keys

ssh localhost验证免密登陆是否成功

切换到sbin目录，./start-all.sh启动hadoop

jps查看DataNode,NameNode是否启动

测试：

进入bin目录，

./hadoop fs -mkdir /input

./hadoop fs -ls /

echo hello hadoop,this is my first demo >> /data/text1.txt

./hadoop fs -put /data/text1.txt /input

./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.1.jar /input /output

./hadoop fs -ls /output 查看结果