cdh編譯安裝支持各種壓縮格式

在安裝CDH版本的hadoop的時候, 我們下載下來的CDH的hadoop安裝包中, 并沒有native包,不支持各個任務階段的壓縮功能，但在生產任務中是必須要支持壓縮功能的，所以要對CDH進行編譯以支持壓縮功能；

本次編譯使用的各軟件版本：

cdh：hadoop-2.6.0-cdh6.7.0-src.tar.gz
jdk：1.7（經多位同學踩坑，不要使用1.8版本的）
maven：apache-maven-3.3.9-bin.tar.gz?
protobuf：protobuf-2.5.0.tar.gz
系統：Centos?7.5

cdh官方下載地址：http://archive.cloudera.com/cdh6/cdh/5

依賴的軟件百度網盤地址

首先安裝各種依賴：

yum?install?-y?svn?ncurses-devel
yum?install?-y?gcc?gcc-c++?make?cmake
yum?install?-y?openssl?openssl-devel?svn?ncurses-devel?zlib-devel?libtool
yum?install?-y?snappy?snappy-devel?bzip2?bzip2-devel?lzo?lzo-devel?lzop?autoconf?automake?cmake

配置jdk的環境變量：

[root@hadoop004?soft]#?cat?/etc/profile.d/java.sh?
export?JAVA_HOME=/usr/java/jdk
export?CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export?PATH=$JAVA_HOME/bin:$PATH

[root@hadoop004?soft]#?java?-version
java?version?"1.7.0_79"
Java(TM)?SE?Runtime?Environment?(build?1.7.0_79-b15)
Java?HotSpot(TM)?64-Bit?Server?VM?(build?24.79-b02,?mixed?mode)

配置maven的環境變量：

#配置環境變量
[root@hadoop004?soft]#?cat?/etc/profile.d/maven.sh?
MAVEN_HOME=/usr/local/maven
export?PATH=$MAVEN_HOME/bin:$PATH

[root@hadoop004?local]#?mvn?--version
Apache?Maven?3.3.9?(bb52d8502b132ec0a5a3f4c09453c07478323dc5;?2015-11-11T00:41:47+08:00)
Maven?home:?/usr/local/maven
Java?version:?1.7.0_79,?vendor:?Oracle?Corporation
Java?home:?/usr/java/jdk1.7.0_79/jre
Default?locale:?en_US,?platform?encoding:?UTF-8
OS?name:?"linux",?version:?"3.10.0-862.3.2.el7.x86_64",?arch:?"amd64",?family:?"unix"

#配置mvn的本地存放地址
<localRepository>/usr/local/maven/repo</localRepository>
將上面下載repo.tar.gz解壓后的內容放在此本地倉庫目錄下，以便于cdh編譯時直接使用，否則下載將浪費大量的時間

#配置mvn下載源為阿里云的maven倉庫
<mirror>
	<id>alimaven</id>
	<name>aliyun?maven</name>
	<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
	<mirrorOf>central</mirrorOf>
</mirror>

編譯安裝protobuf并配置環境變量：

#編譯安裝命令
tar?-zxvf?protobuf-2.5.0.tar.gz
cd?protobuf-2.5.0
./configure??--prefix=/root/protobuf/
make
make?install

#配置環境變量
[root@hadoop004?soft]#?cat?/etc/profile.d/protobuf.sh?
PROTOC_HOME=/usr/local/protobuf
export?PATH=$PROTOC_HOME/bin:$PATH

#查看是否安裝成功
source?/etc/profile.d/protobuf.sh
[root@hadoop004?soft]#?protoc?--version
libprotoc?2.5.0

編譯cdh的hadoop源碼：

tar?-xzvf?hadoop-2.6.0-cdh6.7.0-src.tar.gz
cd?hadoop-2.6.0-cdh6.7.0
mvn?clean?package?-Pdist,native?-DskipTests?-Dtar

遇到的錯誤：

[ERROR]?Failed?to?execute?goal?org.apache.maven.plugins:maven-antrun-plugin:1.7:run?(make)?on?project?hadoop-pipes:?An?Ant?BuildException?has?occured:?exec?returned:?1
[ERROR]?around?Ant?part?...<exec?dir="/data/soft/hadoop-2.6.0-cdh6.7.0/hadoop-tools/hadoop-pipes/target/native"?executable="cmake"?failonerror="true">...?@?5:126?in?/data/soft/hadoop-2.6.0-cdh6.7.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml
[ERROR]?->?[Help?1]
[ERROR]?
[ERROR]?To?see?the?full?stack?trace?of?the?errors,?re-run?Maven?with?the?-e?switch.
[ERROR]?Re-run?Maven?using?the?-X?switch?to?enable?full?debug?logging.
[ERROR]?
[ERROR]?For?more?information?about?the?errors?and?possible?solutions,?please?read?the?following?articles:
[ERROR]?[Help?1]?http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]?
[ERROR]?After?correcting?the?problems,?you?can?resume?the?build?with?the?command
[ERROR]???mvn?<goals>?-rf?:hadoop-pipes

問題原因：在安裝依賴的過程，漏裝了openssl-devel；yum install -y?openssl-devel

再次執行上面的安裝命令：

tar?-xzvf?hadoop-2.6.0-cdh6.7.0-src.tar.gz
cd?hadoop-2.6.0-cdh6.7.0
mvn?clean?package?-Pdist,native?-DskipTests?-Dtar

注：由于已經提前將cdh需要下載的依賴，放入了maven的本地倉庫中了，所以節省了大量的下載時間

編譯后的文件：

????位置：./hadoop-dist/target目錄下

????文件：hadoop-2.6.0-cdh6.7.0

? ? 壓縮文件：hadoop-2.6.0-cdh6.7.0.tar.gz

????

測試查看是否支持各種壓縮：

[root@hadoop004?hadoop-2.6.0-cdh6.7.0]#?./bin/hadoop?checknative
19/04/18?15:09:34?INFO?bzip2.Bzip2Factory:?Successfully?loaded?&?initialized?native-bzip2?library?system-native
19/04/18?15:09:34?INFO?zlib.ZlibFactory:?Successfully?loaded?&?initialized?native-zlib?library
Native?library?checking:
hadoop:??true?/data/soft/hadoop-2.6.0-cdh6.7.0/hadoop-dist/target/hadoop-2.6.0-cdh6.7.0/lib/native/libhadoop.so.1.0.0
zlib:????true?/lib64/libz.so.1
snappy:??true?/lib64/libsnappy.so.1
lz4:?????true?revision:99
bzip2:???true?/lib64/libbz2.so.1
openssl:?true?/lib64/libcrypto.so

?????

移動到其它機器時的報錯及解決方案：

????注意如果拷貝tar包到新的機器安裝檢查checknative報錯

????openssl: false Cannot load libcrypto.so (libcrypto.so: 無法打開共享對象文件: 沒有那個文件或目錄)!

? ? ??

????centos中運行yum install openssl-devel -y

? ? ??