hadoop-2.8.1如何編譯

這篇文章主要為大家展示了“hadoop-2.8.1如何編譯”，內容簡而易懂，條理清晰，希望能夠幫助大家解決疑惑，下面讓小編帶領大家一起研究并學習一下“hadoop-2.8.1如何編譯”這篇文章吧。

1. hadoop-2.8.1源代碼下載
    #先在/opt下面創建software和sourcecode文件夾
    [root@hadoop001 ~]# mkdir -p /opt/sourcecode /opt/software
    [root@hadoop001 sourcecode]# cd  /opt/sourcecode
    [root@hadoop001 sourcecode]# pwd
    /opt/sourcecode
    [root@hadoop001 sourcecode]# ll
    total 33756
    -rw-r--r--. 1 root root 34523353 Aug 20 12:14 hadoop-2.8.1-src.tar.gz

2. 解壓縮hadoop-2.8.1-src.tar.gz文件
    [root@hadoop001 sourcecode]# tar -xzvf hadoop-2.8.1-src.tar.gz
    [root@hadoop001 sourcecode]# ll
    total 33760
    drwxr-xr-x. 17 root root     4096 Jun  2 14:13 hadoop-2.8.1-src
    -rw-r--r--.  1 root root 34523353 Aug 20 12:14 hadoop-2.8.1-src.tar.gz

3.  打開hadoop-2.8.1-sr/BUILDING.txt文件，查看編譯hadoop-2.8.1需要的其他組件
    [root@hadoop001 sourcecode]# cd hadoop-2.8.1-src
    [root@hadoop001 hadoop-2.8.1-src]# ll
    total 224
    -rw-rw-r--.  1 root root 15623 May 24 07:14 BUILDING.txt
    drwxr-xr-x.  4 root root  4096 Aug 20 12:44 dev-support
    drwxr-xr-x.  3 root root  4096 Aug 20 12:44 hadoop-assemblies
    drwxr-xr-x.  3 root root  4096 Aug 20 12:44 hadoop-build-tools
    drwxrwxr-x.  2 root root  4096 Aug 20 12:44 hadoop-client
    drwxr-xr-x. 10 root root  4096 Aug 20 12:44 hadoop-common-project
    drwxr-xr-x.  2 root root  4096 Aug 20 12:44 hadoop-dist
    drwxr-xr-x.  8 root root  4096 Aug 20 12:44 hadoop-hdfs-project
    drwxr-xr-x.  9 root root  4096 Aug 20 12:44 hadoop-mapreduce-project
    drwxr-xr-x.  3 root root  4096 Aug 20 12:44 hadoop-maven-plugins
    drwxr-xr-x.  2 root root  4096 Aug 20 12:44 hadoop-minicluster
    drwxr-xr-x.  3 root root  4096 Aug 20 12:44 hadoop-project
    drwxr-xr-x.  2 root root  4096 Aug 20 12:44 hadoop-project-dist
    drwxr-xr-x. 18 root root  4096 Aug 20 12:44 hadoop-tools
    drwxr-xr-x.  3 root root  4096 Aug 20 12:44 hadoop-yarn-project
    -rw-rw-r--.  1 root root 99253 May 24 07:14 LICENSE.txt
    -rw-rw-r--.  1 root root 15915 May 24 07:14 NOTICE.txt
    drwxrwxr-x.  2 root root  4096 Jun  2 14:24 patchprocess
    -rw-rw-r--.  1 root root 20477 May 29 06:36 pom.xml
    -rw-r--r--.  1 root root  1366 May 20 13:30 README.txt
    -rwxrwxr-x.  1 root root  1841 May 24 07:14 start-build-env.sh
    [root@hadoop001 hadoop-2.8.1-src]# 
    [root@hadoop001 hadoop-2.8.1-src]# cat BUILDING.txt
    Build instructions for Hadoop


    ----------------------------------------------------------------------------------
    Requirements:


    * Unix System
    * JDK 1.7+
    * Maven 3.0 or later
    * Findbugs 1.3.9 (if running findbugs)
    * ProtocolBuffer 2.5.0
    * CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
    * Zlib devel (if compiling native code)
    * openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
    * Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
    * Internet connection for first build (to fetch all Maven and Hadoop dependencies)


4.JAVA安裝   必須是JDK 1.7+
    # 如果沒有安裝rz 需要先安裝rz
    [root@hadoop001 ~]# yum install -y lrzsz

    [root@hadoop001 ~]# rz #上傳jdk-8u45-linux-x64.gz
    [root@hadoop001 ~]# mkdir -p /usr/java
    [root@hadoop001 ~]# mv jdk-8u45-linux-x64.gz /usr/java
    [root@hadoop001 ~]# cd /usr/java
    [root@hadoop001 ~]# tar -xzvf jdk-8u45-linux-x64.gz


    #修改用戶和用戶組
    [root@hadoop001 java]# ll
    total 169388
    drwxr-xr-x. 8 uucp  143      4096 Apr 11  2015 jdk1.8.0_45
    -rw-r--r--. 1 root root 173271626 Mar 16 15:25 jdk-8u45-linux-x64.gz
    [root@hadoop001 java]# chown -R root:root jdk1.8.0_45
    [root@hadoop001 java]# ll
    total 169388
    drwxr-xr-x. 8 root root      4096 Apr 11  2015 jdk1.8.0_45
    -rw-r--r--. 1 root root 173271626 Mar 16 15:25 jdk-8u45-linux-x64.gz
    [root@hadoop001 java]# 
    #配置環境變量
    [root@hadoop001 java]# vi /etc/profile
    export JAVA_HOME=/usr/java/jdk1.8.0_45
    export PATH=$JAVA_HOME/bin:$PATH
    #使環境變量生效
    [root@hadoop001 java]# source /etc/profile
    [root@hadoop001 java]# which java
    /usr/java/jdk1.8.0_45/bin/java
    [root@hadoop001 java]# java -version
    java version "1.8.0_45"
    Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
    Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
    [root@hadoop001 java]# 

    # 如果系統還是顯示默認的jdk版本，需要修改默認的jdk版本為當前安裝的版本
    update-alternatives --install /usr/bin/java java /usr/java/jdk1.8.0_45/bin/java 300
    update-alternatives --install /usr/bin/javac javac /usr/java/jdk1.8.0_45/bin/javac 300
    update-alternatives --config java
    update-alternatives --config javac

5.Maven安裝   Maven 3.0 or later
    [root@hadoop001 ~]# cd /opt/software/
    #上傳apache-maven-3.3.9-bin.zip
    [root@hadoop001 software]# rz
    rz waiting to receive.
    Starting zmodem transfer.  Press Ctrl+C to cancel.
    Transferring apache-maven-3.3.9-bin.zip...
      100%    8415 KB    8415 KB/sec    00:00:01       0 Errors  

    [root@hadoop001 software]# 
    [root@hadoop001 software]# ll
    total 8432
    -rw-r--r--. 1 root root 8617253 Aug 20 12:35 apache-maven-3.3.9-bin.zip
    #解壓縮apache-maven-3.3.9-bin.zip
    [root@hadoop001 software]# unzip apache-maven-3.3.9-bin.zip
    #配置apache-maven-3.3.9的環境變量
    [root@hadoop001 software]# vi /etc/profile
    export MAVEN_HOME=/opt/software/apache-maven-3.3.9
    export MAVEN_OPTS="-Xms256m -Xmx512m"
    export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
    #使環境變量生效
    [root@hadoop001 software]# source /etc/profile
    #查看 maven 版本
    [root@hadoop001 software]# mvn -version
    Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
    Maven home: /opt/software/apache-maven-3.3.9
    Java version: 1.8.0_45, vendor: Oracle Corporation
    Java home: /usr/java/jdk1.8.0_45/jre
    Default locale: en_US, platform encoding: UTF-8
    OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
    [root@hadoop001 software]#


6.Findbugs安裝  Findbugs 1.3.9 (if running findbugs)
    #上傳findbugs-1.3.9.zip
    [root@hadoop001 software]# rz
    rz waiting to receive.
    Starting zmodem transfer.  Press Ctrl+C to cancel.
    Transferring findbugs-1.3.9.zip...
      100%    7369 KB    7369 KB/sec    00:00:01       0 Errors  
    #解壓縮findbugs-1.3.9.zip
    [root@hadoop001 software]# unzip findbugs-1.3.9.zip
    #配置環境變量
    [root@hadoop001 software]# vi /etc/profile
    export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
    export PATH=$FINDBUGS_HOME/bin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
    #使環境變量生效 
    [root@hadoop001 software]# source /etc/profile
    #查看 Findbugs版本
    [root@hadoop001 software]# findbugs -version
    1.3.9
    [root@hadoop001 software]# 


7.protobuf安裝   ProtocolBuffer 2.5.0 
    #上傳protobuf-2.5.0.tar.gz
    [root@hadoop001 software]# rz
    rz waiting to receive.
    Starting zmodem transfer.  Press Ctrl+C to cancel.
    Transferring protobuf-2.5.0.tar.gz...
      100%    2345 KB    2345 KB/sec    00:00:01       0 Errors  
    #解壓縮protobuf-2.5.0.tar.gz
    [root@hadoop001 software]# tar -xzvf protobuf-2.5.0.tar.gz
    [root@hadoop001 software]# ll
    total 10792
    drwxr-xr-x.  6 root   root    4096 Nov 10  2015 apache-maven-3.3.9
    -rw-r--r--.  1 root   root 8617253 Aug 20 12:35 apache-maven-3.3.9-bin.zip
    drwxr-xr-x. 10 109965 5000    4096 Feb 27  2013 protobuf-2.5.0
    -rw-r--r--.  1 root   root 2401901 Aug 20 13:03 protobuf-2.5.0.tar.gz
    [root@hadoop001 software]# cd protobuf-2.5.0
    [root@hadoop001 protobuf-2.5.0]# yum install -y gcc gcc-c++ make cmake   #已經安裝cmake
    [root@hadoop001 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf
    [root@hadoop001 protobuf-2.5.0]# make && make install
    #配置環境變量
    [root@hadoop001 protobuf-2.5.0]# vi /etc/profile
    export PROTOC_HOME=/usr/local/protobuf
    export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
    #使環境變量生效
    [root@hadoop001 protobuf-2.5.0]# source /etc/profile
    #查看 protobuf 版本
    [root@hadoop001 protobuf-2.5.0]# protoc --version
    libprotoc 2.5.0
    [root@hadoop001 protobuf-2.5.0]# 


8.其他依賴
    yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
    yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

9.編譯
    [root@hadoop001 sourcecode]# cd hadoop-2.8.1-src
    [root@hadoop001 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar
    #編譯成功的截圖
    [INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-dist ---
    [INFO] ------------------------------------------------------------------------
    [INFO] Reactor Summary:
    [INFO] 
    [INFO] Apache Hadoop Main ................................. SUCCESS [  7.770 s]
    [INFO] Apache Hadoop Build Tools .......................... SUCCESS [  4.990 s]
    [INFO] Apache Hadoop Project POM .......................... SUCCESS [  4.920 s]
    [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 12.936 s]
    [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.809 s]
    [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  5.013 s]
    [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 17.617 s]
    [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 23.551 s]
    [INFO] Apache Hadoop Auth ................................. SUCCESS [ 26.352 s]
    [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 10.305 s]
    [INFO] Apache Hadoop Common ............................... SUCCESS [05:58 min]
    [INFO] Apache Hadoop NFS .................................. SUCCESS [ 19.105 s]
    [INFO] Apache Hadoop KMS .................................. SUCCESS [ 27.479 s]
    [INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.174 s]
    [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:32 min]
    [INFO] Apache Hadoop HDFS ................................. SUCCESS [06:42 min]
    [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 20.370 s]
    [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 40.720 s]
    [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 21.298 s]
    [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 11.818 s]
    [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.148 s]
    [INFO] Apache Hadoop YARN ................................. SUCCESS [  0.192 s]
    [INFO] Apache Hadoop YARN API ............................. SUCCESS [ 41.517 s]
    [INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:19 min]
    [INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.192 s]
    [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 19.421 s]
    [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 42.398 s]
    [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  8.925 s]
    [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 16.120 s]
    [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 57.415 s]
    [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  3.869 s]
    [INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 14.325 s]
    [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 11.814 s]
    [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 10.027 s]
    [INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.276 s]
    [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  8.333 s]
    [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  5.473 s]
    [INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.160 s]
    [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 13.204 s]
    [INFO] Apache Hadoop YARN Project ......................... SUCCESS [  8.106 s]
    [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.514 s]
    [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [01:09 min]
    [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 40.479 s]
    [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 10.304 s]
    [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 27.335 s]
    [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 19.910 s]
    [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 16.657 s]
    [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  4.591 s]
    [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 12.346 s]
    [INFO] Apache Hadoop MapReduce ............................ SUCCESS [  5.966 s]
    [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  7.940 s]
    [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 15.245 s]
    [INFO] Apache Hadoop Archives ............................. SUCCESS [  5.380 s]
    [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  5.812 s]
    [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 11.785 s]
    [INFO] Apache Hadoop Gridmix .............................. SUCCESS [  9.890 s]
    [INFO] Apache Hadoop Data Join ............................ SUCCESS [  5.784 s]
    [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  3.254 s]
    [INFO] Apache Hadoop Extras ............................... SUCCESS [  5.495 s]
    [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 10.630 s]
    [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 11.234 s]
    [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 14.060 s]
    [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 10.535 s]
    [INFO] Apache Hadoop Client ............................... SUCCESS [ 13.519 s]
    [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  2.164 s]
    [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 10.405 s]
    [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.514 s]
    [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  9.201 s]
    [INFO] Apache Hadoop Tools ................................ SUCCESS [  0.129 s]
    [INFO] Apache Hadoop Distribution ......................... SUCCESS [01:07 min]
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 31:41 min
    [INFO] Finished at: 2017-12-10T11:55:28+08:00
    [INFO] Final Memory: 166M/494M
    [INFO] ------------------------------------------------------------------------
    [root@hadoop001 hadoop-2.8.1-src]# 

10.編譯好的tar包
     /opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz


提醒： 
    1、有時候編譯過程中會出現下載某個包的時間太久，這是由于連接網站的過程中會出現假死，
        此時按ctrl+c，重新運行編譯命令。 
    2、如果出現缺少了某個文件的情況，則要先清理maven(使用命令 mvn clean) 再重新編譯。

以上是“hadoop-2.8.1如何編譯”這篇文章的所有內容，感謝各位的閱讀！相信大家都有了一定的了解，希望分享的內容對大家有所幫助，如果還想學習更多知識，歡迎關注億速云行業資訊頻道！