溫馨提示×
您好，登錄后才能下訂單哦！
點擊 登錄注冊 即表示同意《億速云用戶服務條款》
您好，登錄后才能下訂單哦！
這篇文章將為大家詳細講解有關Flink 1.10.2如何編譯，小編覺得挺實用的，因此分享給大家做個參考，希望大家閱讀完這篇文章后可以有所收獲。
flink 1.10.2 編譯
macOS Catalina 10.15.3
Hadoop: 3.1.1.3.1.5.1-2（hortonworks）
flink-1.10.2
flink-shaded-9.0
可以從githup下載，也可以從官網直接下載
Flink on github: https://github.com/apache/flink
https://github.com/apache/flink/archive/release-1.10.2.tar.gz
flink-1.10.2-src.tgz：https://archive.apache.org/dist/flink/flink-1.10.2/flink-1.10.2-src.tgz
flink-shaded-9.0-src.tgz：https://archive.apache.org/dist/flink/flink-shaded-9.0/flink-shaded-9.0-src.tgz
其它相關鏈接：
building: https://ci.apache.org/projects/flink/flink-docs-release-1.11/flinkDev/building.html
hadoop integration: https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html
關于hadoop的集成，我們需要提前編譯flink-shaded-hadoop-2:jar
flink-1.10.2的pom.xml中可以發現默認的<flink.shaded.version>9.0</flink.shaded.version>
所以我們也選擇下載flink-shaded-9.0 編譯
我們編譯的hadoop基于hortonworks，所以這兩個project編譯前都需要在pom.xml中添加hdp repo，否則會有些依賴包找不到
<repositories> <repository> <id>jetty</id> <name>jetty Repository</name> <url>http://repo.hortonworks.com/content/groups/public/</url> <releases> <enabled>true</enabled> <updatePolicy>daily</updatePolicy> </releases> <snapshots> <enabled>false</enabled> <checksumPolicy>warn</checksumPolicy> </snapshots> <layout>default</layout> </repository> </repositories>
# 確定我們需要的hadoop版本 $ hadoop version Hadoop 3.1.1.3.1.5.1-2 # build 命令 # $ mvn clean install -DskipTests -Drat.skip=true -Dhadoop.version=3.1.1.3.1.0.0-78 -Dhive.version=3.1.0.3.1.0.0-78 #$ mvn clean install -DskipTests -Drat.skip=true -Dhadoop.version=3.1.1.3.1.5.1-2 $ mvn clean install -DskipTests -Dhadoop.version=3.1.1.3.1.5.1-2 -Dhive.version=3.1.0.3.1.5.1-2 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] flink-shaded 9.0 ................................... SUCCESS [ 1.809 s] [INFO] flink-shaded-force-shading 9.0 ..................... SUCCESS [ 0.532 s] [INFO] flink-shaded-asm-7 7.1-9.0 ......................... SUCCESS [ 0.685 s] [INFO] flink-shaded-guava-18 18.0-9.0 ..................... SUCCESS [ 1.086 s] [INFO] flink-shaded-netty-4 4.1.39.Final-9.0 .............. SUCCESS [ 4.428 s] [INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-9.0 SUCCESS [ 0.694 s] [INFO] flink-shaded-jackson-parent 2.10.1-9.0 ............. SUCCESS [ 0.024 s] [INFO] flink-shaded-jackson-2 2.10.1-9.0 .................. SUCCESS [ 1.266 s] [INFO] flink-shaded-jackson-module-jsonSchema-2 2.10.1-9.0 SUCCESS [ 0.923 s] [INFO] flink-shaded-hadoop-2 3.1.1.3.1.5.1-2-9.0 .......... SUCCESS [ 14.844 s] [INFO] flink-shaded-hadoop-2-uber 3.1.1.3.1.5.1-2-9.0 ..... SUCCESS [ 27.032 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 53.525 s [INFO] Finished at: 2020-12-05T16:32:20+08:00 [INFO] ------------------------------------------------------------------------
# mvn install -DskipTests -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Pvendor-repos -Dscala-2.11 -Dhadoop.version=3.1.1.3.1.0.0-78 -Dhive.version=3.1.0.3.1.0.0-78 $ mvn clean install -DskipTests -Dfast -Pvendor-repos -Dhadoop.version=3.1.1.3.1.5.1-2 -Dscala-2.11 -Dhive.version=3.1.0.3.1.5.1-2 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] force-shading [jar] [INFO] flink [pom] [INFO] flink-annotations [jar] [INFO] flink-shaded-curator [jar] [INFO] flink-test-utils-parent [pom] [INFO] flink-test-utils-junit [jar] [INFO] flink-metrics [pom] [INFO] flink-metrics-core [jar] [INFO] flink-core [jar] [INFO] flink-java [jar] [INFO] flink-queryable-state [pom] [INFO] flink-queryable-state-client-java [jar] [INFO] flink-filesystems [pom] [INFO] flink-hadoop-fs [jar] [INFO] flink-runtime [jar] [INFO] flink-scala [jar] [INFO] flink-mapr-fs [jar] [INFO] flink-filesystems :: flink-fs-hadoop-shaded [jar] [INFO] flink-s3-fs-base [jar] [INFO] flink-s3-fs-hadoop [jar] [INFO] flink-s3-fs-presto [jar] [INFO] flink-swift-fs-hadoop [jar] [INFO] flink-oss-fs-hadoop [jar] [INFO] flink-azure-fs-hadoop [jar] [INFO] flink-optimizer [jar] [INFO] flink-clients [jar] [INFO] flink-streaming-java [jar] [INFO] flink-test-utils [jar] [INFO] flink-runtime-web [jar] [INFO] flink-examples [pom] [INFO] flink-examples-batch [jar] [INFO] flink-connectors [pom] [INFO] flink-hadoop-compatibility [jar] [INFO] flink-state-backends [pom] [INFO] flink-statebackend-rocksdb [jar] [INFO] flink-tests [jar] [INFO] flink-streaming-scala [jar] [INFO] flink-table [pom] [INFO] flink-table-common [jar] [INFO] flink-table-api-java [jar] [INFO] flink-table-api-java-bridge [jar] [INFO] flink-table-api-scala [jar] [INFO] flink-table-api-scala-bridge [jar] [INFO] flink-sql-parser [jar] [INFO] flink-libraries [pom] [INFO] flink-cep [jar] [INFO] flink-table-planner [jar] [INFO] flink-table-runtime-blink [jar] [INFO] flink-table-planner-blink [jar] [INFO] flink-jdbc [jar] [INFO] flink-hbase [jar] [INFO] flink-hcatalog [jar] [INFO] flink-metrics-jmx [jar] [INFO] flink-formats [pom] [INFO] flink-json [jar] [INFO] flink-connector-kafka-base [jar] [INFO] flink-connector-kafka-0.9 [jar] [INFO] flink-connector-kafka-0.10 [jar] [INFO] flink-connector-kafka-0.11 [jar] [INFO] flink-connector-elasticsearch-base [jar] [INFO] flink-connector-elasticsearch3 [jar] [INFO] flink-connector-elasticsearch6 [jar] [INFO] flink-connector-elasticsearch7 [jar] [INFO] flink-connector-elasticsearch7 [jar] [INFO] flink-orc [jar] [INFO] flink-csv [jar] [INFO] flink-connector-hive [jar] [INFO] flink-connector-rabbitmq [jar] [INFO] flink-connector-twitter [jar] [INFO] flink-connector-nifi [jar] [INFO] flink-connector-cassandra [jar] [INFO] flink-avro [jar] [INFO] flink-connector-filesystem [jar] [INFO] flink-connector-kafka [jar] [INFO] flink-connector-gcp-pubsub [jar] [INFO] flink-connector-kinesis [jar] [INFO] flink-sql-connector-elasticsearch7 [jar] [INFO] flink-sql-connector-elasticsearch7 [jar] [INFO] flink-sql-connector-kafka-0.9 [jar] [INFO] flink-sql-connector-kafka-0.10 [jar] [INFO] flink-sql-connector-kafka-0.11 [jar] [INFO] flink-sql-connector-kafka [jar] [INFO] flink-connector-kafka-0.8 [jar] [INFO] flink-avro-confluent-registry [jar] [INFO] flink-parquet [jar] [INFO] flink-sequence-file [jar] [INFO] flink-compress [jar] [INFO] flink-examples-streaming [jar] [INFO] flink-examples-table [jar] [INFO] flink-examples-build-helper [pom] [INFO] flink-examples-streaming-twitter [jar] [INFO] flink-examples-streaming-state-machine [jar] [INFO] flink-examples-streaming-gcp-pubsub [jar] [INFO] flink-container [jar] [INFO] flink-queryable-state-runtime [jar] [INFO] flink-end-to-end-tests [pom] [INFO] flink-cli-test [jar] [INFO] flink-parent-child-classloading-test-program [jar] [INFO] flink-parent-child-classloading-test-lib-package [jar] [INFO] flink-dataset-allround-test [jar] [INFO] flink-dataset-fine-grained-recovery-test [jar] [INFO] flink-datastream-allround-test [jar] [INFO] flink-batch-sql-test [jar] [INFO] flink-stream-sql-test [jar] [INFO] flink-bucketing-sink-test [jar] [INFO] flink-distributed-cache-via-blob [jar] [INFO] flink-high-parallelism-iterations-test [jar] [INFO] flink-stream-stateful-job-upgrade-test [jar] [INFO] flink-queryable-state-test [jar] [INFO] flink-local-recovery-and-allocation-test [jar] [INFO] flink-elasticsearch3-test [jar] [INFO] flink-elasticsearch6-test [jar] [INFO] flink-elasticsearch7-test [jar] [INFO] flink-quickstart [pom] [INFO] flink-quickstart-java [maven-archetype] [INFO] flink-quickstart-scala [maven-archetype] [INFO] flink-quickstart-test [jar] [INFO] flink-confluent-schema-registry [jar] [INFO] flink-stream-state-ttl-test [jar] [INFO] flink-sql-client-test [jar] [INFO] flink-streaming-file-sink-test [jar] [INFO] flink-state-evolution-test [jar] [INFO] flink-rocksdb-state-memory-control-test [jar] [INFO] flink-mesos [jar] [INFO] flink-kubernetes [jar] [INFO] flink-yarn [jar] [INFO] flink-gelly [jar] [INFO] flink-gelly-scala [jar] [INFO] flink-gelly-examples [jar] [INFO] flink-metrics-dropwizard [jar] [INFO] flink-metrics-graphite [jar] [INFO] flink-metrics-influxdb [jar] [INFO] flink-metrics-prometheus [jar] [INFO] flink-metrics-statsd [jar] [INFO] flink-metrics-datadog [jar] [INFO] flink-metrics-slf4j [jar] [INFO] flink-cep-scala [jar] [INFO] flink-table-uber [jar] [INFO] flink-table-uber-blink [jar] [INFO] flink-sql-client [jar] [INFO] flink-state-processor-api [jar] [INFO] flink-python [jar] [INFO] flink-scala-shell [jar] [INFO] flink-dist [jar] [INFO] flink-end-to-end-tests-common [jar] [INFO] flink-metrics-availability-test [jar] [INFO] flink-metrics-reporter-prometheus-test [jar] [INFO] flink-heavy-deployment-stress-test [jar] [INFO] flink-connector-gcp-pubsub-emulator-tests [jar] [INFO] flink-streaming-kafka-test-base [jar] [INFO] flink-streaming-kafka-test [jar] [INFO] flink-streaming-kafka011-test [jar] [INFO] flink-streaming-kafka010-test [jar] [INFO] flink-plugins-test [pom] [INFO] dummy-fs [jar] [INFO] another-dummy-fs [jar] [INFO] flink-tpch-test [jar] [INFO] flink-streaming-kinesis-test [jar] [INFO] flink-elasticsearch7-test [jar] [INFO] flink-end-to-end-tests-common-kafka [jar] [INFO] flink-tpcds-test [jar] [INFO] flink-statebackend-heap-spillable [jar] [INFO] flink-contrib [pom] [INFO] flink-connector-wikiedits [jar] [INFO] flink-yarn-tests [jar] [INFO] flink-fs-tests [jar] [INFO] flink-docs [jar] [INFO] flink-ml-parent [pom] [INFO] flink-ml-api [jar] [INFO] flink-ml-lib [jar] [INFO] flink-walkthroughs [pom] [INFO] flink-walkthrough-common [jar] [INFO] flink-walkthrough-table-java [maven-archetype] [INFO] flink-walkthrough-table-scala [maven-archetype] [INFO] flink-walkthrough-datastream-java [maven-archetype] [INFO] flink-walkthrough-datastream-scala [maven-archetype] [INFO] [INFO] -------------------< org.apache.flink:force-shading >------------------- [INFO] Building force-shading 1.10.2 [1/176] [INFO] --------------------------------[ jar ]---------------------------------
<font color=red>為了和hadoop兼容使用，我們需要手動將上一步編譯的flink-shaded-hadoop-2-uber-3.1.1.3.1.5.1-2-9.0.jar拷貝至flink-1.10.2-bin/flink-1.10.2/lib</font>
然后手動將flink-1.10.2/flink-dist/target/flink-1.10.2-bin/flink-1.10.2打包成tar.gz去服務器部署安裝就可以了
<font >注意hadoop版本最好要一致，否則可能有不兼容地方</font>
# 最后生成編譯包所在路徑flink-1.10.2-bin $ ll flink-1.10.2/flink-dist/target total 264176 drwxr-xr-x 3 jiazz staff 96B 12 5 18:03 antrun drwxr-xr-x 2 jiazz staff 64B 12 5 18:04 archive-tmp -rw-r--r-- 1 jiazz staff 490K 12 5 18:04 bash-java-utils.jar drwxr-xr-x 5 jiazz staff 160B 12 5 18:03 classes drwxr-xr-x 3 jiazz staff 96B 12 5 18:04 flink-1.10.2-bin -rw-r--r-- 1 jiazz staff 116M 12 5 18:04 flink-dist_2.11-1.10.2.jar drwxr-xr-x 3 jiazz staff 96B 12 5 18:03 generated-test-sources drwxr-xr-x 3 jiazz staff 96B 12 5 18:03 maven-archiver drwxr-xr-x 3 jiazz staff 96B 12 5 18:03 maven-shared-archive-resources -rw-r--r-- 1 jiazz staff 206K 12 5 18:03 original-flink-dist_2.11-1.10.2.jar drwxr-xr-x 4 jiazz staff 128B 12 5 18:03 temporary drwxr-xr-x 5 jiazz staff 160B 12 5 18:03 test-classes # 添加flink-shaded-hadoop-2-uber到lib $ cp ~/.m2/repository/org/apache/flink/flink-shaded-hadoop-2-uber/3.1.1.3.1.5.1-2-9.0/flink-shaded-hadoop-2-uber-3.1.1.3.1.5.1-2-9.0.jar flink-1.10.2-bin/flink-1.10.2/lib #打包 $ tar czvf flink-1.10.2-3.1.5.1-2.tar.gz flink-1.10.2-bin/flink-1.10.2/
我明明已經編譯了flink-shaded，但是我編譯的是10，默認用的是9.0，所以去重新下載flink-shaded-9.0編譯好，再繼續編譯flink就好了，或者改成你編譯的flink-shaded版本
[ERROR] Failed to execute goal on project flink-hadoop-fs: Could not resolve dependencies for project org.apache.flink:flink-hadoop-fs:jar:1.10.2: Could not find artifact org.apache.flink:flink-shaded-hadoop-2:jar:3.1.1.3.1.5.1-2-9.0 in jetty (http://repo.hortonworks.com/content/groups/public/) -> [Help 1] [ERROR]
關于“Flink 1.10.2如何編譯”這篇文章就分享到這里了，希望以上內容可以對大家有一定的幫助，使各位可以學到更多知識，如果覺得文章不錯，請把它分享出去讓更多的人看到。
免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。
