标签:style http color java os 文件 io for
最近项目需要用到hadoop,边学习边应用,第一步无疑是安装hadoop,我安装的是hadoop-2.4.1,下面是详细步骤,做备忘以后查看
一、下载依赖软件
1、java
hadoop官网说明只支持java 6和7 ,下载最新的java7
http://download.oracle.com/otn-pub/java/jdk/7u65-b17/jdk-7u65-linux-x64.tar.gz
2、maven
从maven官网下载http://maven.apache.org/download.cgi
3、hadoop安装文件
http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.4.1/hadoop-2.4.1.tar.gz
4、hadoop源代码(用于编译生成hadoop的native包)
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.4.1/hadoop-2.4.1-src.tar.gz
5、hive
http://mirrors.cnnic.cn/apache/hive/hive-0.13.1/apache-hive-0.13.1-bin.tar.gz
6、protobuf(用于编译生成hadoop的native包)
http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz?1
二、把上述软件分别解压到soft目录下,然后设置环境变量
1、在管理员权限下centos 执行命令vi /etc/profile
2、在最后添加下面的路径
export HADOOP_HOME=/root/soft/hadoop/hadoop-2.4.1
export HIVE_HOME=/root/soft/hadoop/apache-hive-0.13.1-bin
export JAVA_HOME=/root/soft/java/jdk1.7.0_65
export M2_HOME=/root/soft/apache-maven-3.2.1
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH:$JRE_HOME/lib:$HADOOP_HOME/bin:$HIVE_HOME/bin:$M2_HOME/bin
根据解压后的具体路径修改
3、验证
java -version 验证java
mvn -version 验证maven
三、编译hadoop
1、安装必要工具
centos下:
yum -y install lzo-devel zlib-devel gcc autoconf automake libtool
安装g++:yum install gcc-c++
安装cmake:yum install cmake
安装openssl-devel:yum install openssl-devel
ubuntu下为
apt-get -y install maven build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev
2、进入下载的protobuf-2.5.0目录下
cd protobuf-2.5.0
./configure
make
make install
3、进入hadoop-2.4.1-src编译hadoop
cd hadoop-2.4.1-src
mvn clean package -Pdist,native -DskipTests -Dtar
4、一般情况下/root/soft/hadoop/hadoop-2.4.1-src/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads路径下的apache-tomcat-6.0.36.tar.gz下载不完全,
正常大小应该为6.47M左右,如果不对请官网手动下载,路径为http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.36/bin/apache-tomcat-6.0.36.tar.gz
5、替换hadoop-2.4.1/lib/native为编译后hadoop-2.4.1-src/hadoop-dist/target/hadoop-2.4.1/lib/native包
四、配置ssh
1、验证ssh
ssh localhost2、执行命令
ssh-keygen -t dsa -P ‘‘ -f ~/.ssh/id_dsa cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
五、配置hadoop
1、etc/hadoop/core-site.xml:
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration>2、etc/hadoop/hdfs-site.xml:
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>3、etc/hadoop/mapred-site.xml:
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>4、etc/hadoop/yarn-site.xml:
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>5、etc/hadoop/mapred-site.xml:
更改27行export JAVA_HOME=${JAVA_HOME}为
export JAVA_HOME=实际路径
六、启动hadoop
1、进入hadoop目录 cd $HADOOP_HOME 2、格式化namenode:bin/hdfs namenode -format 3、启动dfs:sbin/start-dfs.sh 4、验证dfs:http://localhost:50070/ 5、启动yarn:sbin/start-yarn.sh 6、官方小例子 bin/hdfs dfs -mkdir /user bin/hdfs dfs -mkdir /user/<username>(username必须为用户名) bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar grep input output ‘dfs[a-z.]+‘ bin/hdfs dfs -get output output cat output/*
自己动手一步一步安装hadoop(含编译hadoop的native本地包),布布扣,bubuko.com
自己动手一步一步安装hadoop(含编译hadoop的native本地包)
标签:style http color java os 文件 io for
原文地址:http://blog.csdn.net/zhulin40/article/details/38299249