码迷，mamicode.com
首页 > 系统相关 > 详细

从零教你如何获取hadoop2.4源码并使用eclipse关联hadoop2.4源码

时间：2016-08-17 13:53:08      阅读：271      评论：0      收藏：0      [点我收藏+]

标签：

问题导读：
1.如何通过官网src包，获取hadoop的全部代码
2.通过什么样的操作，可以查看hadoop某个函数或则类的实现？
3.maven的作用是什么？

技术分享




我们如果想搞开发，研究源码对我们的帮助很大。不明白原理就如同黑盒子，遇到问题，我们也摸不着思路。所以这里交给大家
一.如何获取源码
二.如何关联源码

一.如何获取源码

1.下载hadoop的maven程序包

（1）官网下载
这里我们先从官网上下载maven包hadoop-2.4.0-src.tar.gz。
官网下载地址

对于不知道怎么去官网下载，可以查看：新手指导：hadoop官网介绍及如何下载hadoop（2.4）各个版本与查看hadoop API介绍

（2）网盘下载
也可以从网盘下载：
http://pan.baidu.com/s/1kToPuGB

2.通过maven获取源码
获取源码的方式有两种，一种是通过命令行的方式，一种是通过eclipse。这里主要讲通过命令的方式

通过命令的方式获取源码：
1.解压包

<ignore_js_op>技术分享 


解压包的时候遇到了下面问题。不过不用管，我们继续往下走

1        : 无法创建文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\applicationhistoryservice\ApplicationHistoryClientService$ApplicationHSClientProtocolHandler.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
2        : 无法创建文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\applicationhistoryservice\timeline\LeveldbTimelineStore$LockMap$CountingReentrantLock.class：系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
3        : 无法创建文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-applicationhistoryservice\target\test-classes\org\apache\hadoop\yarn\server\applicationhistoryservice\webapp\TestAHSWebApp$MockApplicationHistoryManagerImpl.class：系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
4        : 无法创建文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\monitor\capacity\TestProportionalCapacityPreemptionPolicy$IsPreemptionRequestFor.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
5        : 无法创建文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestFSRMStateStore$TestFSRMStateStoreTester$TestFileSystemRMStore.class：系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
6        : 无法创建文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestZKRMStateStore$TestZKRMStateStoreTester$TestZKRMStateStoreInternal.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
7        : 无法创建文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestZKRMStateStoreZKClientConnections$TestZKClient$TestForwardingWatcher.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
8        : 无法创建文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestZKRMStateStoreZKClientConnections$TestZKClient$TestZKRMStateStore.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
9        : 无法创建文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\rmapp\attempt\TestRMAppAttemptTransitions$TestApplicationAttemptEventDispatcher.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip



2.通过maven获取源码

这里需要说明的是，在使用maven的时候，需要先安装jdk，protoc ，如果没有安装可以参考win7如何安装maven、安装protoc

（1）进入hadoop-2.4.0-src\hadoop-maven-plugins，运行mvn install

  1. D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins>mvn install
复制代码



显示如下信息

  2. [INFO] Scanning for projects...
  3. [WARNING]
  4. [WARNING] Some problems were encountered while building the effective model for
  5. org.apache.hadoop:hadoop-maven-plugins:maven-plugin:2.4.0
  6. [WARNING] ‘build.plugins.plugin.(groupId:artifactId)‘ must be unique but found d
  7. uplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @
  8. org.apache.hadoop:hadoop-project:2.4.0, D:\hadoop2\hadoop-2.4.0-src\hadoop-proje
  9. ct\pom.xml, line 1015, column 15
  10. [WARNING]
  11. [WARNING] It is highly recommended to fix these problems because they threaten t
  12. he stability of your build.
  13. [WARNING]
  14. [WARNING] For this reason, future Maven versions might no longer support buildin
  15. g such malformed projects.
  16. [WARNING]
  17. [INFO]
  18. [INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethrea
  19. ded.SingleThreadedBuilder with a thread count of 1
  20. [INFO]
  21. [INFO] ------------------------------------------------------------------------
  22. [INFO] Building Apache Hadoop Maven Plugins 2.4.0
  23. [INFO] ------------------------------------------------------------------------
  24. [INFO]
  25. [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-maven-plugins
  26. ---
  27. [INFO] Executing tasks
  29. main:
  30. [INFO] Executed tasks
  31. [INFO]
  32. [INFO] --- maven-plugin-plugin:3.0:descriptor (default-descriptor) @ hadoop-mave
  33. n-plugins ---
  34. [INFO] Using ‘UTF-8‘ encoding to read mojo metadata.
  35. [INFO] Applying mojo extractor for language: java-annotations
  36. [INFO] Mojo extractor for language: java-annotations found 2 mojo descriptors.
  37. [INFO] Applying mojo extractor for language: java
  38. [INFO] Mojo extractor for language: java found 0 mojo descriptors.
  39. [INFO] Applying mojo extractor for language: bsh
  40. [INFO] Mojo extractor for language: bsh found 0 mojo descriptors.
  41. [INFO]
  42. [INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ hadoop-mav
  43. en-plugins ---
  44. [INFO] Using default encoding to copy filtered resources.
  45. [INFO]
  46. [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-maven-
  47. plugins ---
  48. [INFO] Nothing to compile - all classes are up to date
  49. [INFO]
  50. [INFO] --- maven-plugin-plugin:3.0:descriptor (mojo-descriptor) @ hadoop-maven-p
  51. lugins ---
  52. [INFO] Using ‘UTF-8‘ encoding to read mojo metadata.
  53. [INFO] Applying mojo extractor for language: java-annotations
  54. [INFO] Mojo extractor for language: java-annotations found 2 mojo descriptors.
  55. [INFO] Applying mojo extractor for language: java
  56. [INFO] Mojo extractor for language: java found 0 mojo descriptors.
  57. [INFO] Applying mojo extractor for language: bsh
  58. [INFO] Mojo extractor for language: bsh found 0 mojo descriptors.
  59. [INFO]
  60. [INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ ha
  61. doop-maven-plugins ---
  62. [INFO] Using default encoding to copy filtered resources.
  63. [INFO]
  64. [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoo
  65. p-maven-plugins ---
  66. [INFO] No sources to compile
  67. [INFO]
  68. [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hadoop-maven-plugins
  69. ---
  70. [INFO] No tests to run.
  71. [INFO]
  72. [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-maven-plugins ---
  73. [INFO] Building jar: D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\had
  74. oop-maven-plugins-2.4.0.jar
  75. [INFO]
  76. [INFO] --- maven-plugin-plugin:3.0:addPluginArtifactMetadata (default-addPluginA
  77. rtifactMetadata) @ hadoop-maven-plugins ---
  78. [INFO]
  79. [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hadoop-
  80. maven-plugins ---
  81. [INFO]
  82. [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ hadoop-maven-p
  83. lugins ---
  84. [INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\hadoop
  85. -maven-plugins-2.4.0.jar to C:\Users\hyj\.m2\repository\org\apache\hadoop\hadoop
  86. -maven-plugins\2.4.0\hadoop-maven-plugins-2.4.0.jar
  87. [INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\pom.xml to C:
  88. \Users\hyj\.m2\repository\org\apache\hadoop\hadoop-maven-plugins\2.4.0\hadoop-ma
  89. ven-plugins-2.4.0.pom
  90. [INFO] ------------------------------------------------------------------------
  91. [INFO] BUILD SUCCESS
  92. [INFO] ------------------------------------------------------------------------
  93. [INFO] Total time: 4.891 s
  94. [INFO] Finished at: 2014-06-23T14:47:33+08:00
  95. [INFO] Final Memory: 21M/347M
  96. [INFO] ------------------------------------------------------------------------
复制代码


部分截图如下：

<ignore_js_op>技术分享 


<ignore_js_op>技术分享 



（2）运行

  1. mvn eclipse:eclipse -DskipTests
复制代码

这时候注意，我们进入的是hadoop_home,我这里是D:\hadoop2\hadoop-2.4.0-src

部分信息如下

  1. [INFO]
  2. [INFO] ------------------------------------------------------------------------
  3. [INFO] Reactor Summary:
  4. [INFO]
  5. [INFO] Apache Hadoop Main ................................ SUCCESS [  0.684 s]
  6. [INFO] Apache Hadoop Project POM ......................... SUCCESS [  0.720 s]
  7. [INFO] Apache Hadoop Annotations ......................... SUCCESS [  0.276 s]
  8. [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [  0.179 s]
  9. [INFO] Apache Hadoop Assemblies .......................... SUCCESS [  0.121 s]
  10. [INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [  1.680 s]
  11. [INFO] Apache Hadoop MiniKDC ............................. SUCCESS [  1.802 s]
  12. [INFO] Apache Hadoop Auth ................................ SUCCESS [  1.024 s]
  13. [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [  0.160 s]
  14. [INFO] Apache Hadoop Common .............................. SUCCESS [  1.061 s]
  15. [INFO] Apache Hadoop NFS ................................. SUCCESS [  0.489 s]
  16. [INFO] Apache Hadoop Common Project ...................... SUCCESS [  0.056 s]
  17. [INFO] Apache Hadoop HDFS ................................ SUCCESS [  2.770 s]
  18. [INFO] Apache Hadoop HttpFS .............................. SUCCESS [  0.965 s]
  19. [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [  0.629 s]
  20. [INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [  0.284 s]
  21. [INFO] Apache Hadoop HDFS Project ........................ SUCCESS [  0.061 s]
  22. [INFO] hadoop-yarn ....................................... SUCCESS [  0.052 s]
  23. [INFO] hadoop-yarn-api ................................... SUCCESS [  0.842 s]
  24. [INFO] hadoop-yarn-common ................................ SUCCESS [  0.322 s]
  25. [INFO] hadoop-yarn-server ................................ SUCCESS [  0.065 s]
  26. [INFO] hadoop-yarn-server-common ......................... SUCCESS [  0.972 s]
  27. [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [  0.580 s]
  28. [INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [  0.379 s]
  29. [INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [  0.281 s]
  30. [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [  0.378 s]
  31. [INFO] hadoop-yarn-server-tests .......................... SUCCESS [  0.534 s]
  32. [INFO] hadoop-yarn-client ................................ SUCCESS [  0.307 s]
  33. [INFO] hadoop-yarn-applications .......................... SUCCESS [  0.050 s]
  34. [INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [  0.202 s]
  35. [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [  0.194 s]
  36. [INFO] hadoop-yarn-site .................................. SUCCESS [  0.057 s]
  37. [INFO] hadoop-yarn-project ............................... SUCCESS [  0.066 s]
  38. [INFO] hadoop-mapreduce-client ........................... SUCCESS [  0.091 s]
  39. [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [  1.321 s]
  40. [INFO] hadoop-mapreduce-client-common .................... SUCCESS [  0.786 s]
  41. [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [  0.456 s]
  42. [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [  0.508 s]
  43. [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [  0.834 s]
  44. [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [  0.541 s]
  45. [INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [  0.284 s]
  46. [INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [  0.851 s]
  47. [INFO] hadoop-mapreduce .................................. SUCCESS [  0.099 s]
  48. [INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [  0.742 s]
  49. [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [  0.335 s]
  50. [INFO] Apache Hadoop Archives ............................ SUCCESS [  0.397 s]
  51. [INFO] Apache Hadoop Rumen ............................... SUCCESS [  0.371 s]
  52. [INFO] Apache Hadoop Gridmix ............................. SUCCESS [  0.230 s]
  53. [INFO] Apache Hadoop Data Join ........................... SUCCESS [  0.184 s]
  54. [INFO] Apache Hadoop Extras .............................. SUCCESS [  0.217 s]
  55. [INFO] Apache Hadoop Pipes ............................... SUCCESS [  0.048 s]
  56. [INFO] Apache Hadoop OpenStack support ................... SUCCESS [  0.244 s]
  57. [INFO] Apache Hadoop Client .............................. SUCCESS [  0.590 s]
  58. [INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [  0.230 s]
  59. [INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [  0.650 s]
  60. [INFO] Apache Hadoop Tools Dist .......................... SUCCESS [  0.334 s]
  61. [INFO] Apache Hadoop Tools ............................... SUCCESS [  0.042 s]
  62. [INFO] Apache Hadoop Distribution ........................ SUCCESS [  0.144 s]
  63. [INFO] ------------------------------------------------------------------------
  64. [INFO] BUILD SUCCESS
  65. [INFO] ------------------------------------------------------------------------
  66. [INFO] Total time: 31.234 s
  67. [INFO] Finished at: 2014-06-23T14:55:08+08:00
  68. [INFO] Final Memory: 84M/759M
  69. [INFO] ------------------------------------------------------------------------
复制代码

这时候，我们已经把源码给下载下来了。这时候，我们会看到文件会明显增大。


<ignore_js_op>技术分享 



3.关联eclipse源码

加入我们以下程序
<ignore_js_op>技术分享 hadoop2.2mapreduce例子.rar (1.14 MB, 下载次数: 400, 售价: 1 云币) 
如下图示，对他们进行了打包
<ignore_js_op>技术分享 

这两个文件， MaxTemperature.zip为mapreduce例子，mockito-core-1.8.5.jar为mapreduce例子所引用的包
（这里需要说明的是，mapreduce为2.2，但是不影响关联源码，只是交给大家该如何关联源码）
我们解压之后，导入eclipse
（对于导入项目不熟悉，参考零基础教你如何导入eclipse项目）

<ignore_js_op>技术分享 


我们导入之后，看到很多的红线，这些其实都是没有引用包，下面我们开始解决这些语法问题。
一、解决导入jar包
（1）引入mockito-core-1.8.5.jar

（2）hadoop2.4编译包中的jar文件，这些文件的位置如下：

hadoop_home中share\hadoop文件夹下，具体我这的位置D:\hadoop2\hadoop-2.4.0\share\hadoop
找到里面的jar包，举例如下：lib文件中的jar包，以及下面的jar包都添加到buildpath中。
如果对于引用包，不知道该如何添加这些jar包，参考hadoop开发方式总结及操作指导。
（注意的是，我们这里是引入的是编译包，编译的下载hadoop--642.4.0.tar.gz
链接: http://pan.baidu.com/s/1c0vPjG0 密码:xj6l）

更多包下载可以参考hadoop家族、strom、spark、Linux、flume等jar包、安装包汇总下载

<ignore_js_op>技术分享 



<ignore_js_op>技术分享 


二、关联源码
1.我们导入jar包之后，就没有错误了，如下图所示

<ignore_js_op>技术分享 

2.找不到源码

当我们想看一个类或则函数怎么实现的时候，通过Open Call Hierarchy，却找不到源文件。

<ignore_js_op>技术分享 



<ignore_js_op>技术分享 


3.Attach Source

<ignore_js_op>技术分享 

上面三处，我们按照顺序添加即可，我们选定压缩包之后，单击确定，ok了，我们的工作已经完毕。

注意：对于hadoop-2.2.0-src.zip则是我们上面通过maven下载的源码，然后压缩的文件，记得一定是压缩文件zip的形式


4.验证关联后查看源码

我们再次执行上面操作，通过Open Call Hierarchy




看到下面内容



<ignore_js_op>技术分享 


然后我们双击上图主类，即红字部分，我们看到下面内容：

<ignore_js_op>技术分享 


问题：
细心的同学，这里面我们产生一个问题，因为我们看到的是.class文件，而不是.java文件。那么他会不会和我们所看到的.java文件不一样那。
其实是一样的，感兴趣的同学，可以验证一下。

从零教你如何获取hadoop2.4源码并使用eclipse关联hadoop2.4源码

标签：

原文地址：http://www.cnblogs.com/snowbook/p/5779485.html
踩
(0)
赞
(0)
   
举报
评论 一句话评论（0）
登录后才能评论！
分享档案
更多>
2021年07月29日 (22)
2021年07月28日 (40)
2021年07月27日 (32)
2021年07月26日 (79)
2021年07月23日 (29)
2021年07月22日 (30)
2021年07月21日 (42)
2021年07月20日 (16)
2021年07月19日 (90)
2021年07月16日 (35)
周排行
mamicode.com排行更多图片
更多
友情链接
兰亭集智   国之画   百度统计   站长统计   阿里云   chrome插件   新版天听网
关于我们 - 联系我们 - 留言反馈
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码！