码迷,mamicode.com
首页 > 其他好文 > 详细

编译Flink 1.9.0

时间:2019-09-12 18:37:35      阅读:396      评论:0      收藏:0      [点我收藏+]

标签:proc   http   phi   upgrade   system   lib   tor   ade   col   

闲来无事,编个Flink 1.9 玩玩

1、下载flink、flink-shaded 源码、解压

  flink   flink-shaded 7.0

[venn@venn release]$ ll
total 19880
drwxrwxr-x. 44 venn venn     4096 Sep 12 14:00 flink-release-1.9.0
drwxrwxr-x. 13 venn venn     4096 Sep 11 11:05 flink-shaded-release-7.0
-rw-rw-r--.  1 venn venn 20318507 Sep 12 09:24 release-1.9.0.tar.gz
-rw-rw-r--.  1 venn venn    28196 Sep 11 11:05 release-7.0.tar.gz

 

最开始直接拿flink 的 releast-1.9分支,发现都带了SNAPSHOT,遂放弃

flink-shaded 包含flink 的很多依赖,比如 flink-shaded-hadoop-2,中央仓库只提供了几个hadoop 版本的包,可能没有与自己hadoop 对应的 flink-shaded-hadoop-2 的包。flink1.9 对应的flink-shaded 版本是 7.0

<dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-shaded-hadoop-2</artifactId>
            <version>${hadoop.version}-${flink.shaded.version}</version>
            <optional>true</optional>
        </dependency>

 

技术图片

我用的hadoop 版本是2.7.7,然而并没有(2.7.5应该可以用)。

2、编译flink-shaded

指定hadoop版本2.7.7, 把编译好的包打到本地maven仓库,后面编译flink 用

[venn@venn flink-shaded-release-7.0]$ mvn clean install -DskipTest -Dhadoop.version=2.7.7


[INFO] Reactor Summary:
[INFO] 
[INFO] flink-shaded 7.0 ................................... SUCCESS [  3.483 s]
[INFO] flink-shaded-force-shading 7.0 ..................... SUCCESS [  0.597 s]
[INFO] flink-shaded-asm-6 6.2.1-7.0 ....................... SUCCESS [  0.627 s]
[INFO] flink-shaded-guava-18 18.0-7.0 ..................... SUCCESS [  0.994 s]
[INFO] flink-shaded-netty-4 4.1.32.Final-7.0 .............. SUCCESS [  3.469 s]
[INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-7.0 SUCCESS [  0.572 s]
[INFO] flink-shaded-jackson-parent 2.9.8-7.0 .............. SUCCESS [  0.035 s]
[INFO] flink-shaded-jackson-2 2.9.8-7.0 ................... SUCCESS [  1.479 s]
[INFO] flink-shaded-jackson-module-jsonSchema-2 2.9.8-7.0 . SUCCESS [  0.857 s]
[INFO] flink-shaded-hadoop-2 2.7.7-7.0 .................... SUCCESS [  9.132 s]
[INFO] flink-shaded-hadoop-2-uber 2.7.7-7.0 ............... SUCCESS [ 17.173 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  38.612 s
[INFO] Finished at: 2019-09-12T15:14:42+08:00
[INFO] ------------------------------------------------------------------------

3、编译 flink

指定hadoop 版本 2.7.7

 

[venn@venn flink-release-1.9.0]$ mvn clean package -DskipTests -Dhadoop.version=2.7.7

[INFO] Reactor Summary for flink 1.9.0:
[INFO] 
[INFO] force-shading ...................................... SUCCESS [  1.281 s]
[INFO] flink .............................................. SUCCESS [  1.829 s]
[INFO] flink-annotations .................................. SUCCESS [  1.280 s]
[INFO] flink-shaded-curator ............................... SUCCESS [  1.501 s]
[INFO] flink-metrics ...................................... SUCCESS [  0.188 s]
[INFO] flink-metrics-core ................................. SUCCESS [  0.736 s]
[INFO] flink-test-utils-parent ............................ SUCCESS [  0.157 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [  0.928 s]
[INFO] flink-core ......................................... SUCCESS [ 33.771 s]
[INFO] flink-java ......................................... SUCCESS [  4.827 s]
[INFO] flink-queryable-state .............................. SUCCESS [  0.100 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [  0.643 s]
[INFO] flink-filesystems .................................. SUCCESS [  0.082 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [  1.168 s]
[INFO] flink-runtime ...................................... SUCCESS [01:58 min]
[INFO] flink-scala ........................................ SUCCESS [ 29.960 s]
[INFO] flink-mapr-fs ...................................... SUCCESS [  0.394 s]
[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [  2.916 s]
[INFO] flink-s3-fs-base ................................... SUCCESS [  5.740 s]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [  8.235 s]
[INFO] flink-s3-fs-presto ................................. SUCCESS [ 12.510 s]
[INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 13.618 s]
[INFO] flink-oss-fs-hadoop ................................ SUCCESS [  4.483 s]
[INFO] flink-azure-fs-hadoop .............................. SUCCESS [  5.729 s]
[INFO] flink-optimizer .................................... SUCCESS [ 11.589 s]
[INFO] flink-clients ...................................... SUCCESS [  1.179 s]
[INFO] flink-streaming-java ............................... SUCCESS [  8.882 s]
[INFO] flink-test-utils ................................... SUCCESS [  1.453 s]
[INFO] flink-runtime-web .................................. SUCCESS [02:06 min]
[INFO] flink-examples ..................................... SUCCESS [  0.112 s]
[INFO] flink-examples-batch ............................... SUCCESS [ 10.577 s]
[INFO] flink-connectors ................................... SUCCESS [  0.096 s]
[INFO] flink-hadoop-compatibility ......................... SUCCESS [  4.225 s]
[INFO] flink-state-backends ............................... SUCCESS [  0.070 s]
[INFO] flink-statebackend-rocksdb ......................... SUCCESS [  1.102 s]
[INFO] flink-tests ........................................ SUCCESS [ 28.820 s]
[INFO] flink-streaming-scala .............................. SUCCESS [ 26.143 s]
[INFO] flink-table ........................................ SUCCESS [  0.054 s]
[INFO] flink-table-common ................................. SUCCESS [  1.771 s]
[INFO] flink-table-api-java ............................... SUCCESS [  1.416 s]
[INFO] flink-table-api-java-bridge ........................ SUCCESS [  0.570 s]
[INFO] flink-table-api-scala .............................. SUCCESS [  5.054 s]
[INFO] flink-table-api-scala-bridge ....................... SUCCESS [  7.626 s]
[INFO] flink-sql-parser ................................... SUCCESS [  3.073 s]
[INFO] flink-libraries .................................... SUCCESS [  0.056 s]
[INFO] flink-cep .......................................... SUCCESS [  2.437 s]
[INFO] flink-table-planner ................................ SUCCESS [01:37 min]
[INFO] flink-orc .......................................... SUCCESS [  0.579 s]
[INFO] flink-jdbc ......................................... SUCCESS [  0.546 s]
[INFO] flink-table-runtime-blink .......................... SUCCESS [  3.868 s]
[INFO] flink-table-planner-blink .......................... SUCCESS [02:02 min]
[INFO] flink-hbase ........................................ SUCCESS [  1.725 s]
[INFO] flink-hcatalog ..................................... SUCCESS [  4.158 s]
[INFO] flink-metrics-jmx .................................. SUCCESS [  0.272 s]
[INFO] flink-connector-kafka-base ......................... SUCCESS [  1.671 s]
[INFO] flink-connector-kafka-0.9 .......................... SUCCESS [  0.768 s]
[INFO] flink-connector-kafka-0.10 ......................... SUCCESS [  0.547 s]
[INFO] flink-connector-kafka-0.11 ......................... SUCCESS [  0.692 s]
[INFO] flink-formats ...................................... SUCCESS [  0.054 s]
[INFO] flink-json ......................................... SUCCESS [  0.332 s]
[INFO] flink-connector-elasticsearch-base ................. SUCCESS [  0.810 s]
[INFO] flink-connector-elasticsearch2 ..................... SUCCESS [  8.239 s]
[INFO] flink-connector-elasticsearch5 ..................... SUCCESS [  9.715 s]
[INFO] flink-connector-elasticsearch6 ..................... SUCCESS [  0.747 s]
[INFO] flink-csv .......................................... SUCCESS [  0.251 s]
[INFO] flink-connector-hive ............................... SUCCESS [  1.942 s]
[INFO] flink-connector-rabbitmq ........................... SUCCESS [  0.332 s]
[INFO] flink-connector-twitter ............................ SUCCESS [  1.304 s]
[INFO] flink-connector-nifi ............................... SUCCESS [  0.349 s]
[INFO] flink-connector-cassandra .......................... SUCCESS [  2.205 s]
[INFO] flink-avro ......................................... SUCCESS [  1.565 s]
[INFO] flink-connector-filesystem ......................... SUCCESS [  0.861 s]
[INFO] flink-connector-kafka .............................. SUCCESS [  0.973 s]
[INFO] flink-connector-gcp-pubsub ......................... SUCCESS [  0.582 s]
[INFO] flink-sql-connector-elasticsearch6 ................. SUCCESS [  4.583 s]
[INFO] flink-sql-connector-kafka-0.9 ...................... SUCCESS [  0.256 s]
[INFO] flink-sql-connector-kafka-0.10 ..................... SUCCESS [  0.312 s]
[INFO] flink-sql-connector-kafka-0.11 ..................... SUCCESS [  0.413 s]
[INFO] flink-sql-connector-kafka .......................... SUCCESS [  0.641 s]
[INFO] flink-connector-kafka-0.8 .......................... SUCCESS [  0.700 s]
[INFO] flink-avro-confluent-registry ...................... SUCCESS [  0.843 s]
[INFO] flink-parquet ...................................... SUCCESS [  0.726 s]
[INFO] flink-sequence-file ................................ SUCCESS [  0.247 s]
[INFO] flink-examples-streaming ........................... SUCCESS [ 10.970 s]
[INFO] flink-examples-table ............................... SUCCESS [  5.554 s]
[INFO] flink-examples-build-helper ........................ SUCCESS [  0.069 s]
[INFO] flink-examples-streaming-twitter ................... SUCCESS [  0.464 s]
[INFO] flink-examples-streaming-state-machine ............. SUCCESS [  0.298 s]
[INFO] flink-examples-streaming-gcp-pubsub ................ SUCCESS [  3.640 s]
[INFO] flink-container .................................... SUCCESS [  0.304 s]
[INFO] flink-queryable-state-runtime ...................... SUCCESS [  0.555 s]
[INFO] flink-end-to-end-tests ............................. SUCCESS [  0.056 s]
[INFO] flink-cli-test ..................................... SUCCESS [  0.151 s]
[INFO] flink-parent-child-classloading-test-program ....... SUCCESS [  0.154 s]
[INFO] flink-parent-child-classloading-test-lib-package ... SUCCESS [  0.098 s]
[INFO] flink-dataset-allround-test ........................ SUCCESS [  0.153 s]
[INFO] flink-dataset-fine-grained-recovery-test ........... SUCCESS [  0.141 s]
[INFO] flink-datastream-allround-test ..................... SUCCESS [  1.119 s]
[INFO] flink-batch-sql-test ............................... SUCCESS [  0.136 s]
[INFO] flink-stream-sql-test .............................. SUCCESS [  0.183 s]
[INFO] flink-bucketing-sink-test .......................... SUCCESS [  0.233 s]
[INFO] flink-distributed-cache-via-blob ................... SUCCESS [  0.138 s]
[INFO] flink-high-parallelism-iterations-test ............. SUCCESS [  5.724 s]
[INFO] flink-stream-stateful-job-upgrade-test ............. SUCCESS [  0.636 s]
[INFO] flink-queryable-state-test ......................... SUCCESS [  1.688 s]
[INFO] flink-local-recovery-and-allocation-test ........... SUCCESS [  0.221 s]
[INFO] flink-elasticsearch2-test .......................... SUCCESS [  3.299 s]
[INFO] flink-elasticsearch5-test .......................... SUCCESS [  3.969 s]
[INFO] flink-elasticsearch6-test .......................... SUCCESS [  2.275 s]
[INFO] flink-quickstart ................................... SUCCESS [  0.781 s]
[INFO] flink-quickstart-java .............................. SUCCESS [  0.350 s]
[INFO] flink-quickstart-scala ............................. SUCCESS [  0.145 s]
[INFO] flink-quickstart-test .............................. SUCCESS [  0.323 s]
[INFO] flink-confluent-schema-registry .................... SUCCESS [  1.202 s]
[INFO] flink-stream-state-ttl-test ........................ SUCCESS [  3.172 s]
[INFO] flink-sql-client-test .............................. SUCCESS [  0.671 s]
[INFO] flink-streaming-file-sink-test ..................... SUCCESS [  0.221 s]
[INFO] flink-state-evolution-test ......................... SUCCESS [  0.822 s]
[INFO] flink-e2e-test-utils ............................... SUCCESS [  6.170 s]
[INFO] flink-mesos ........................................ SUCCESS [ 17.591 s]
[INFO] flink-yarn ......................................... SUCCESS [  0.915 s]
[INFO] flink-gelly ........................................ SUCCESS [  2.601 s]
[INFO] flink-gelly-scala .................................. SUCCESS [ 13.500 s]
[INFO] flink-gelly-examples ............................... SUCCESS [  7.711 s]
[INFO] flink-metrics-dropwizard ........................... SUCCESS [  0.193 s]
[INFO] flink-metrics-graphite ............................. SUCCESS [  0.123 s]
[INFO] flink-metrics-influxdb ............................. SUCCESS [  0.594 s]
[INFO] flink-metrics-prometheus ........................... SUCCESS [  0.371 s]
[INFO] flink-metrics-statsd ............................... SUCCESS [  0.179 s]
[INFO] flink-metrics-datadog .............................. SUCCESS [  0.237 s]
[INFO] flink-metrics-slf4j ................................ SUCCESS [  0.157 s]
[INFO] flink-cep-scala .................................... SUCCESS [  9.399 s]
[INFO] flink-table-uber ................................... SUCCESS [  1.794 s]
[INFO] flink-table-uber-blink ............................. SUCCESS [  2.050 s]
[INFO] flink-sql-client ................................... SUCCESS [  1.090 s]
[INFO] flink-state-processor-api .......................... SUCCESS [  0.598 s]
[INFO] flink-python ....................................... SUCCESS [  0.621 s]
[INFO] flink-scala-shell .................................. SUCCESS [  8.821 s]
[INFO] flink-dist ......................................... SUCCESS [ 10.765 s]
[INFO] flink-end-to-end-tests-common ...................... SUCCESS [  0.285 s]
[INFO] flink-metrics-availability-test .................... SUCCESS [  0.153 s]
[INFO] flink-metrics-reporter-prometheus-test ............. SUCCESS [  0.132 s]
[INFO] flink-heavy-deployment-stress-test ................. SUCCESS [  6.045 s]
[INFO] flink-connector-gcp-pubsub-emulator-tests .......... SUCCESS [  0.516 s]
[INFO] flink-streaming-kafka-test-base .................... SUCCESS [  0.192 s]
[INFO] flink-streaming-kafka-test ......................... SUCCESS [  5.923 s]
[INFO] flink-streaming-kafka011-test ...................... SUCCESS [  5.786 s]
[INFO] flink-streaming-kafka010-test ...................... SUCCESS [  5.000 s]
[INFO] flink-plugins-test ................................. SUCCESS [  0.095 s]
[INFO] flink-tpch-test .................................... SUCCESS [  1.210 s]
[INFO] flink-contrib ...................................... SUCCESS [  0.052 s]
[INFO] flink-connector-wikiedits .......................... SUCCESS [  0.258 s]
[INFO] flink-yarn-tests ................................... SUCCESS [  2.919 s]
[INFO] flink-fs-tests ..................................... SUCCESS [  0.330 s]
[INFO] flink-docs ......................................... SUCCESS [  0.531 s]
[INFO] flink-ml-parent .................................... SUCCESS [  0.052 s]
[INFO] flink-ml-api ....................................... SUCCESS [  0.284 s]
[INFO] flink-ml-lib ....................................... SUCCESS [  0.164 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15:33 min
[INFO] Finished at: 2019-09-12T17:41:19+08:00
[INFO] ------------------------------------------------------------------------

正常情况耗时15分钟,异常是可能会卡在 flink-runtime和 flink-runtime-web 上面

 

4、遇到的问题

1)、 flink 1.9 中移除了 map-r 的包,所有不用在这上面耗时间,但是我在编译的时候,还是遇到很多包无法下载(阿里云的镜像都有),只好在本地下载,上传上去(centos 的虚拟机里编译的,虚拟机下不下的包,本地都可以正常下载,就把本地的仓库覆盖上去了)

2)、 flink-runtime/flink-runtime-web 有时候会编译很久

3)、编译flink-table-api-java 的时候编译不过

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project flink-table-api-java: Compilation failure
[ERROR] /home/venn/git/release/flink-release-1.9.0/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/ut
ils/factories/AggregateOperationFactory.java:[550,53] unreported exception X; must be caught or declared to be thrown [ERROR] [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :flink-table-api-java

是低版本jdk的bug,下面这种用法有点问题

List<String> aliases = children.subList(1, children.size())
                .stream()
                .map(alias -> ExpressionUtils.extractValue(alias, String.class)
                    .orElseThrow(() -> new ValidationException("Unexpected alias: " + alias)))
                .collect(toList());

解决方法: 升级jdk 1.8 的小版本, 从jdk1.8.0_91 升到 jdk1.8.0_111 解决了

 4)、可以预装个 gcc、gcc-c++,中间会有报错,不过应该不影响

编译Flink 1.9.0

标签:proc   http   phi   upgrade   system   lib   tor   ade   col   

原文地址:https://www.cnblogs.com/Springmoon-venn/p/11512136.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!