码迷,mamicode.com
首页 > Windows程序 > 详细

Step by step install and run Hadoop 2.9.1 on Windows 10 64 bit (最全步骤整理)

时间:2020-04-19 16:21:45      阅读:98      评论:0      收藏:0      [点我收藏+]

标签:设置   packages   下载   置配   error   set   windows   end   开发   

1. Java

Download the Java 1.8 64 bit version from https://java.com/en/download/  --------- If not 64 bit java there will be error when start resource node manager

Java 安装自选目录:比如 C:\java64 -----这里重要

Once installed confirm that you’re running the correct version from command line using ‘java -version’ command, output of which you can confirm in command line like this:

C:\Users>java -version
java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)

2. WinRAR/7-ZIP

Downloaded and installed WinRAR 64 bit release from http://www.rarlab.com/download.htm that will later allow me to decompress Linux type tar.gz packages on Windows. Or use 7-zip https://www.7-zip.org/download.html

3. Hadoop

The next step was to install a Hadoop distribution. Download a binary form:

https://archive.apache.org/dist/hadoop/common/hadoop-2.9.1/hadoop-2.9.1.tar.gz

Unzip it and rename folder name to hadoop and put it under C:\Learning  -------  如果用你自己的目录,名字中间不要有空格(因为Hadoop初始开发基于linux, linux 文件名字不能有空格)。 后续设置配置文件内容时目录都要修改

 

It‘s time to start the Hadoop cluster with a single node.

3. 1 Setup Environmental Variables

list as below:

技术图片

 

 

 技术图片

技术图片

 

Add following dir to path:

 

 技术图片

 

3.2 Edit Hadoop Configuration

注意:如果你不想自己配置下面 3.2 过程, 除了3.2.3 其它的步骤可以忽略, 那么下载下面链接(bin data etc)文件夹放到你的hadoop目录下覆盖(https://github.com/yjy24/bigdata_learning ), 然后从3.2.3 开始继续!

 

3.2.1 新建data 目录和子目录 

c:/Learning/hadoop/data

c:/Learning/hadoop/data/namenode

c:/Learning/hadoop/data/datanode

 

3.2.2 Configure Hadoop 

a) C:\Learning\hadoop\etc\hadoop\core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>
 

 

 

b)  C:\Learning\hadoop\etc\hadoop\mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>
 

c) C:\Learning\hadoop\etc\hadoop\hdfs-site.xml

<configuration>
    <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
      <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///Learning/hadoop/data/namenode</value>
  </property>
      <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///Learning/hadoop/data/datanode</value>
  </property>
</configuration>
 
d) C:\Learning\hadoop\etc\hadoop\yarn-site.xml
</configuration>
<configuration>
    <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
      <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///Learning/hadoop/data/namenode</value>
  </property>
      <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///Learning/hadoop/data/datanode</value>
  </property>
</configuration>
 
e) C:\Learning\hadoop\etc\hadoop\hadoop-env.cmd
显性设置JAVA_HOME
set JAVA_HOME=C:\java64
 

3.2.3 windows 需要的的 libs

1. install Visual C++ 2010 Redistributable Package (x64): https://www.microsoft.com/en-us/download/details.aspx?id=14632

2. 把 winutils.exe 等 windows 需要的文件放到 C:\Learning\hadoop\bin 下面
下载 https://github.com/yjy24/bigdata_learning 下面的bin 直接覆盖C:\Learning\hadoop\bin
里面有window需要的所有文件 -- 针对 2.9.1
 
3. 把里面的 hadoop.dll  放置到 C:\Windows\System32 一份

3.3 Start Hadoop all service

3.3.1 格式化 Hadoop namenode:

命令: hdfs namenode -format
 Log on scren:

20/04/19 10:40:13 INFO common.Storage: Storage directory \Learning\hadoop\data\namenode has been successfully formatted.
20/04/19 10:40:13 INFO namenode.FSImageFormatProtobuf: Saving image file \Learning\hadoop\data\namenode\current\fsimage.ckpt_0000000000000000000 using no compression
20/04/19 10:40:13 INFO namenode.FSImageFormatProtobuf: Image file \Learning\hadoop\data\namenode\current\fsimage.ckpt_0000000000000000000 of size 320 bytes saved in 0 seconds .
20/04/19 10:40:13 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/04/19 10:40:13 INFO namenode.NameNode: SHUTDOWN_MSG:

3.3.2 启动 Hadoop all service:

C:\Learning\hadoop\sbin\start-all.cmd      ---- 要用管理员身份运行

四个service窗口会打开运行 ------ 任何一个都不能有错误

恭喜你可以开始运行你的Hadoop程序了!

3.4 Open Hadoop GUI

Once all above steps were completed,opened browser and navigated to: http://localhost:8088/cluster

 这里可以查看job的运行情况 log 信息。

技术图片

--------------------------------------------------End---------------------------------------------------

Step by step install and run Hadoop 2.9.1 on Windows 10 64 bit (最全步骤整理)

标签:设置   packages   下载   置配   error   set   windows   end   开发   

原文地址:https://www.cnblogs.com/yjyyjy/p/12731968.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!