标签：streaming 应用程序 access 吞吐量 基金会
Hadoop是一个由Apache基金会所开发的分布式系统基础架构。
用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力进行高速运算和存储。
Hadoop实现了一个分布式文件系统（Hadoop Distributed File System），简称HDFS。HDFS有高容错性的特点，并且设计用来部署在低廉的（low-cost）硬件上；而且它提供高吞吐量（high throughput）来访问应用程序的数据，适合那些有着超大数据集（large data set）的应用程序。HDFS放宽了（relax）POSIX的要求，可以以流的形式访问（streaming access）文件系统中的数据。
Hadoop的框架最核心的设计就是：HDFS和MapReduce。HDFS为海量的数据提供了存储，则MapReduce为海量的数据提供了计算。
namenode 192.168.31.243
datenode 192.168.31.165
实验环境
centos6_x64
实验软件
jdk-6u31-linux-i586.bin
hadoop-1.0.0.tar.gz
软件安装
yum install -y rsync* openssh*
yum install -y ld-linux.so.2
groupadd hadoop
useradd hadoop -g hadoop
mkdir /usr/local/hadoop
mkdir -p /usr/local/java
service iptables stop
ssh-keygen -t rsa 192.168.31.243配置 （192.168.31.165配置相同）
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
scp -r /root/.ssh/id_rsa.pub 192.168.31.165:/root/.ssh/authorized_keys
scp -r /root/.ssh/id_rsa.pub 192.168.31.243:/root/.ssh/authorized_keys
scp -r jdk-6u31-linux-i586.bin hadoop-1.0.0.tar.gz 192.168.31.165:/root/
mv jdk-6u31-linux-i586.bin /usr/local/java/
cd /usr/local/java/
chmod +x jdk-6u31-linux-i586.bin
./jdk-6u31-linux-i586.bin
vim /etc/profile 最后一行追加配置
# set java environment
export JAVA_HOME=/usr/local/java/jdk1.6.0_31
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
# set hadoop path
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
source /etc/profile
java -version
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) Client VM (build 20.6-b01, mixed mode, sharing)
tar zxvf hadoop-1.0.0.tar.gz
mv hadoop-1.0.0 /usr/local/hadoop
chown -R hadoop:hadoop /usr/local/hadoop
ll /usr/local/hadoop/
drwxr-xr-x 14 hadoop hadoop 4096 Dec 16 2011 hadoop-1.0.0
cp /usr/local/hadoop/conf/hadoop-env.sh /usr/local/hadoop/conf/hadoop-env.sh.bak
vim /usr/local/hadoop/conf/hadoop-env.sh
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME=/usr/local/java/jdk1.6.0_31 修改为
cd /usr/local/hadoop/conf
cp core-site.xml hdfs-site.xml mapred-site.xml core-site.xml 这几个文件都备份一下
vim core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration> 红色为需要修改的地方
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<!-- file system properties -->
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.31.243:9000</value>
</property>
</configuration>
vim hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
vim mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name> mapred.job.tracker</name>
<value>http://192.168.21.243:9001</value>
</property>
</configuration>
cp masters masters.bak
vim masters
localhost
192.168.31.243
cp slave slave.bak 192.168.31.165配置
vim /usr/local/hadoop/conf/slaves
localhost
192.168.31.165
scp -r core-site.xml hdfs-site.xml mapred-site.xml 192.168.31.165:/usr/local/hadoop/conf/
/usr/local/hadoop/bin/hadoop namenode -format
Warning: $HADOOP_HOME is deprecated.
16/09/21 22:51:13 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = java.net.UnknownHostException: centos6: centos6
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.0
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1214675; compiled by ‘hortonfo‘ on Thu Dec 15 16:36:35 UTC 2011
************************************************************/
16/09/21 22:51:14 INFO util.GSet: VM type = 32-bit
16/09/21 22:51:14 INFO util.GSet: 2% max memory = 19.33375 MB
16/09/21 22:51:14 INFO util.GSet: capacity = 2^22 = 4194304 entries
16/09/21 22:51:14 INFO util.GSet: recommended=4194304, actual=4194304
16/09/21 22:51:14 INFO namenode.FSNamesystem: fsOwner=root
16/09/21 22:51:14 INFO namenode.FSNamesystem: supergroup=supergroup
16/09/21 22:51:14 INFO namenode.FSNamesystem: isPermissionEnabled=true
16/09/21 22:51:14 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
16/09/21 22:51:14 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
16/09/21 22:51:14 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/09/21 22:51:14 INFO common.Storage: Image file of size 110 saved in 0 seconds.
16/09/21 22:51:14 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
16/09/21 22:51:14 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: centos6: centos6
************************************************************/
/usr/local/hadoop/bin/start-all.sh
Warning: $HADOOP_HOME is deprecated.
starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-namenode-centos6.out
The authenticity of host ‘localhost (::1)‘ can‘t be established.
RSA key fingerprint is 81:d9:c6:54:a9:99:27:c0:f7:5f:c3:15:d5:84:a0:99.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added ‘localhost‘ (RSA) to the list of known hosts.
root@localhost‘s password:
localhost: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-datanode-centos6.out
The authenticity of host ‘192.168.31.243 (192.168.31.243)‘ can‘t be established.
RSA key fingerprint is 81:d9:c6:54:a9:99:27:c0:f7:5f:c3:15:d5:84:a0:99.
Are you sure you want to continue connecting (yes/no)? yes
192.168.31.243: Warning: Permanently added ‘192.168.31.243‘ (RSA) to the list of known hosts.
root@192.168.31.243‘s password:
192.168.31.243: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-secondarynamenode-centos6.out
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-jobtracker-centos6.out
root@localhost‘s password:
localhost: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-tasktracker-centos6.out
ll /usr/local/hadoop/tmp/
drwxr-xr-x 5 root root 4096 Sep 21 22:53 dfs
drwxr-xr-x 3 root root 4096 Sep 21 22:53 mapred 看到这两项证明没有错误
jps
3237 SecondaryNameNode
3011 NameNode
3467 Jps
netstat -tuplna | grep 500
tcp 0 0 :::50070 :::* LISTEN 3011/java
tcp 0 0 :::50090 :::* LISTEN 3237/java
http://192.168.31.243:50070/dfshealth.jsp
本文出自 “mailfile” 博客，谢绝转载！
标签：streaming 应用程序 access 吞吐量 基金会
原文地址：http://mailfile.blog.51cto.com/6133205/1855204
