Hadoop 2.3.0-cdh5.1.0重编译

标签：hadoop   java   重编译   cdh   

Hadoop版本有很多，这里我选择的是cdh版本。cdh是Cloudera公司在Apache原版基础经过加工的东西。chd具体的下载地址为：

http://archive-primary.cloudera.com/cdh5/cdh/5/

版本信息如下：
Hadoop：Hadoop 2.3.0-cdh5.1.0
JDK：1.7.0_79
Maven：apache-maven-3.2.5（3.3.1以及更高版本必须在JDK1.7以上）
Protobuf：protobuf-2.5.0
Ant：1.7.1

1.安装Maven
Maven可以自己到Maven官网上下载（http://maven.apache.org/download.cgi）。需要注意的是：如果JDK是1.6及其以下的，那Maven最多只能下载到3.2.5，如果下载3.3.1及其更高版本就必须要1.7的JDK。

2.安装Protobuf
Protobuf的安装参见：http://www.tuicool.com/articles/jM7Nn2，安装过程也很简单。

3.安装其他需要的组件
还需要安装ant、gcc等组件，运行如下命令，会根据需要自己到网上下载安装。

yum install autoconf automake libtool ant gcc gcc-c++ make cmake openssl-devel ncurses-devel

4.下载Hadoop源码并解压
源码下载地址为：http://archive-primary.cloudera.com/cdh5/cdh/5/

5.编译源码
运行：

mvn package -DskipTests -Pdist,native -Dtar

如果出现错误可以加上-X显示错误详细信息：

mvn -X package -DskipTests -Pdist,native –Dtar

重编译完成后，native包一般在如下的目录下：

../hadoop-2.3.0-cdh5.1.0/src/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/

如下图：

安装过程中常见的错误：

1、maven中的jar包下载失败。

Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/com/cloudera/cdh/cdh-root/5.1.0/cdh-root-5.1.0.pom
Downloading: http://repository.jboss.org/nexus/content/groups/public/com/cloudera/cdh/cdh-root/5.1.0/cdh-root-5.1.0.pom
Downloading: https://repo.maven.apache.org/maven2/com/cloudera/cdh/cdh-root/5.1.0/cdh-root-5.1.0.pom
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project org.apache.hadoop:hadoop-main:2.3.0-cdh5.1.0 (/home/hbase/install/hadoop-2.3.0-cdh5.1.0/src/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.1.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): repository.cloudera.com and 'parent.relativePath' points at wrong local POM @ line 19, column 11: Unknown host repository.cloudera.com -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

[ERROR] Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:2.4.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.felix:maven-bundle-plugin:jar:2.4.0 @ 

	at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:416)
	at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:672)
	at org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven.java:663)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:253)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[ERROR]   
[ERROR]   The project org.apache.hadoop:hadoop-main:2.3.0 (/usr/local/release-2.3.0/pom.xml) has 1 error

2、编译失败

通过mvn package总是失败，可以考虑通过mvn compile来，如下：

mvn compile -Pdist,native -Dskiptests -Dtar

开启：export HADOOP_ROOT_LOGGER=DEBUG,console
关闭：export HADOOP_ROOT_LOGGER=INFO,console

全文完，转载请注明出处：http://blog.csdn.net/ghuil/article/details/45371977

Hadoop 2.3.0-cdh5.1.0重编译

标签：hadoop   java   重编译   cdh   

原文地址：http://blog.csdn.net/ghuil/article/details/45371977