centos 7 靜態編譯docker-ce

目的

1）去dockerfile，處理網絡請求問題，縮短編譯耗時
2）方便內部統一版本維護
3）方便接入內部流水線作業
4）編譯static版本，避免系統庫動態依賴問題

一、準備工作編譯環境

1）系統環境

[root@localhost ~]# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
[root@localhost ~]# uname  -a
Linux localhost.localdomain 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
docker version ： docker-ce-18.09

2）docker相關

docker-ce變化
dokcer從17.06 項目由moby變更為docker-ce,docker客戶端與服務端項目分離開
docker客戶端代碼塊在cli目錄
docker服務端代碼塊在engine中

containerd runc proxy init未變化
相關項目文件見：
版本不同略有變化，最新版本（18.09），相關關聯的commit id也在installer文件中
docker-ce/components/engine/hack/dockerfile/install/
containerd.installer gometalinter.installer proxy.installer tini.installer vndr.installer
dockercli.installer install.sh runc.installer tomlv.installer
golang版本見：docker-ce/components/engine/Dockerfile.e2e

相關項目代碼庫：

https://github.com/opencontainers/runc.git docker-runc
https://github.com/krallin/tini.git docker-init
https://github.com/containerd/containerd.git docker-containerd || docker-containerd-shim || docker-containerd-ctr
https://github.com/docker/docker-ce.git docker || dockerd
https://github.com/docker/libnetwork.git docker-proxy

3）找到對應golang版本

 cat docker-ce/components/engine/Dockerfile.e2e  
    docker-ce/components/engine/Dockerfile.e2e:FROM golang:1.10.6-alpine3.7 as builder  
    從docker與golang對應dockerfile找到原始對應關系
    https://github.com/docker-library/golang/tree/366fe83ed839938cd04b2d546a06e2aee25a39a2

    這邊選擇直接下載不用編譯的go版本
    https://dl.google.com/go/go1.10.6.linux-amd64.tar.gz

4）配置基礎編譯環境

根據containerd的docker鏡像編譯方式可知redhat系列的gcc版本過低且不支持enable-default-pie選項，需要安裝編譯gcc 6.3.x版本;
編譯runc時報/usr/bin/ld: cannot find -lseccomp，默認的redhat系列無libseccomp-static支持

a)將原來的libseccomp軟件包刪掉

rpm -ivh https://cbs.centos.org/kojifiles/packages/libseccomp/2.3.2/3.el7/x86_64/libseccomp-2.3.2-3.el7.x86_64.rpm https://cbs.centos.org/kojifiles/packages/libseccomp/2.3.2/3.el7/x86_64/libseccomp-devel-2.3.2-3.el7.x86_64.rpm https://cbs.centos.org/kojifiles/packages/libseccomp/2.3.2/3.el7/x86_64/libseccomp-static-2.3.2-3.el7.x86_64.rpm
[root@localhost src]# rpm -qa|grep libseccomp
libseccomp-2.3.2-3.el7.x86_64
libseccomp-static-2.3.2-3.el7.x86_64
libseccomp-devel-2.3.2-3.el7.x86_64

b)編譯使用gcc 6.3.0環境

yum group install "Development Tools"
yum install redhat-lsb rpm-build rpm-sign check dejagnu expect zlib-devel
[root@localhost github.com]# git clone https://github.com/BobSteagall/gcc-builder.git
Cloning into 'gcc-builder'...
remote: Enumerating objects: 215, done.
remote: Total 215 (delta 0), reused 0 (delta 0), pack-reused 215
Receiving objects: 100% (215/215), 35.75 KiB | 0 bytes/s, done.
Resolving deltas: 100% (149/149), done.

[root@localhost github.com]# cd gcc-builder/

[root@localhost gcc-builder]# git checkout gcc6
Branch gcc6 set up to track remote branch gcc6 from origin.
Switched to a new branch 'gcc6'

[root@localhost gcc-builder]# vi gcc-build-vars.sh
export GCC_VERSION=6.3.0

root@localhost gcc-builder]# vi configure-gcc.sh
elif [ "$GCC_PLATFORM" == "Linux" ]
then
$GCC_SRC_DIR/configure -v \
--with-pkgversion="$GCC_PKG_NAME" \
--enable-default-pie \ ---增加enable-default-pie
--enable-languages=c,c++ \

[root@localhost gcc-builder]# ./build-gcc.sh | tee build.log

[root@localhost gcc-builder]# ./stage-gcc.sh

[root@localhost gcc-builder]# ./pack-gcc.sh

[root@localhost gcc-builder]# cd dist/usr/local
[root@localhost local]# cp -r bin/ gcc/ /usr/local/
[root@localhost local]# chown -R root:root /usr/local/gcc/6.3.0/
[root@localhost local]# chown root:root /usr/local/bin/gcc630
[root@localhost local]# source /usr/local/bin/setenv-for-gcc630.sh
[root@localhost local]# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/gcc/6.3.0/libexec/gcc/x86_64-kewb-linux-gnu/6.3.0/lto-wrapper
Target: x86_64-kewb-linux-gnu
Configured with: /usr/local/docker/src/github.com/gcc-builder/gcc-6.3.0/configure -v --with-pkgversion='KEWB Computing Build' --prefix=/usr/local/gcc/6.3.0 --program-suffix= --enable-tls --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-default-pie --enable-languages=c,c++ --enable-lto --enable-bootstrap --disable-nls --disable-multilib --disable-install-libiberty --disable-werror --with-system-zlib
Thread model: posix
gcc version 6.3.0 (KEWB Computing Build)

二、下載軟件

編譯環境配置

---

go: /usr/local/go1.10.6
dockerd: /usr/local/docker/src/github.com/docker/docker engine目錄
docker: /usr/local/docker/src/github.com/docker/cli
proxy: /usr/local/docker/src/github.com/docker/libnetwork
init: /usr/local/docker/src/github.com/tini
runc: /usr/local/docker/src/github.com/opencontainers/runc
containerd: /usr/local/docker/src/github.com/containerd/containerd/

下載go設置環境變量

cd /usr/local
wget https://dl.google.com/go/go1.10.6.linux-amd64.tar.gz
mkdir -p /usr/local/go1.10.6
tar -C /usr/local/go1.10.6 -zxvf go1.10.6.linux-amd64.tar.gz
go_version=/usr/local/go1.10.6/go
export PATH=${go_version}/bin/:$PATH
export GOROOT=${go_version}/
export GOPATH=/usr/local/docker
export DOCKER_GITCOMMIT=4c52b90/18.09

下載docker dockerd docker-proxy相關代碼

mkdir -p /usr/local/docker/src/github.com/docker
cd /usr/local/docker/src/github.com/docker
git clone https://github.com/docker/docker-ce.git
git clone https://github.com/docker/libnetwork.git
cp -r docker-ce/components/engine docker
cp -r docker-ce/components/cli cli

下載docker-init相關代碼

cd /usr/local/docker/src/github.com
git clone https://github.com/krallin/tini.git

下載docker-runc相關代碼

mkdir -p /usr/local/docker/src/github.com/opencontainers
cd /usr/local/docker/src/github.com/opencontainers
git clone https://github.com/opencontainers/runc.git

下載docker-containerd...相關代碼

mkdir -p /usr/local/docker/src/github.com/containerd
cd /usr/local/docker/src/github.com/containerd
git clone https://github.com/containerd/containerd.git

三、編譯二制文件

切換docker-ce至18.09版本
cd /usr/local/docker/src/github.com/docker/docker-ce
git checkout 18.09
Branch 18.09 set up to track remote branch 18.09 from origin.
Switched to a new branch '18.09'

1）據docker 編譯命令安裝基礎軟件包

docker-ce/components/packaging/image/Dockerfile.engine-dm
yum group install -y 'Development Tools'
yum install -y bash ca-certificates cmake gcc git glibc-static libtool make
yum install -y btrfs-progs-devel device-mapper-devel libseccomp-devel selinux-policy-devel systemd-devel

2）編譯runc

根據docker-ce/components/engine/hack/dockerfile/install/runc.installer切換至對應commit id(RUNC_COMMIT=96ec2177ae841256168fcf76954f7177af9446eb)
cd /usr/local/docker/src/github.com/opencontainers/runc
[root@localhost runc]# git checkout -q 96ec2177ae841256168fcf76954f7177af9446eb

#If using RHEL7 kernels (3.10.0 el7), disable kmem accounting/limiting
[root@localhost runc]# make BUILDTAGS="seccomp apparmor selinux nokmem" static
[root@localhost runc]# ldd runc
not a dynamic executable

3）編譯containerd

據編譯命令編譯

docker-ce/components/engine/hack/dockerfile/install/containerd.installer
CONTAINERD_COMMIT=9754871865f7fe2f4e74d43e2fc7ccd237edcbce # v1.2.2

cd /usr/local/docker/src/github.com/containerd/containerd/
[root@localhost containerd]# git checkout -q 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
[root@localhost containerd]# make EXTRA_FLAGS="-buildmode pie" EXTRA_LDFLAGS='-extldflags "-fno-PIC -static"' BUILDTAGS="netgo osusergo static_build"

[root@localhost containerd]# ldd bin/ctr
not a dynamic executable
[root@localhost containerd]# ldd bin/containerd*
bin/containerd:
not a dynamic executable
bin/containerd-shim:
not a dynamic executable
bin/containerd-shim-runc-v1:
not a dynamic executable
bin/containerd-stress:
not a dynamic executable

4)編譯docker-init

cd /usr/local/docker/src/github.com/tini
[root@localhost tini]# git checkout -q fec3683b971d9c3ef73f284f176672c44b44866
[root@localhost tini]# cmake .
[root@localhost tini]# make tini-static
[root@localhost tini]# ldd tini-static
not a dynamic executable
[root@localhost tini]# cp tini-static docker-init

5)編譯docker-proxy

cd /usr/local/docker/src/github.com/docker/libnetwork
[root@localhost libnetwork]# git checkout -q 2cfbf9b1f98162a55829a21cc603c76072a75382
[root@localhost libnetwork]# CGO_ENABLED=0 go build -o docker-proxy github.com/docker/libnetwork/cmd/proxy
[root@localhost libnetwork]# ldd docker-proxy
not a dynamic executable

6)編譯docker dockerd

cd /usr/local/docker/src/github.com/docker/cli
[root@localhost cli]#export VERSION=18.09
[root@localhost cli]#export GITCOMMIT=4c52b90
[root@localhost cli]# make binary

WARNING: you are not in a container.
Use "make -f docker.Makefile binary" or set
DISABLE_WARN_OUTSIDE_CONTAINER=1 to disable this warning.

Press Ctrl+C now to abort.

WARNING: binary creates a Linux executable. Use cross for macOS or Windows.
./scripts/build/binary
Building statically linked build/docker-linux-amd64
[root@localhost cli]# ldd build/docker
not a dynamic executable

[root@localhost cli]# build/docker -v
Docker version 18.09, build 4c52b90

cd /usr/local/docker/src/github.com/docker/docker
[root@localhost docker]# hack/make.sh binary
#WARNING! I don't seem to be running in a Docker container.
#The result of this command might be an incorrect build, and will not be
#officially supported.
#Try this instead: make all

Removing bundles/

---> Making bundle: binary (in bundles/binary)
Building: bundles/binary-daemon/dockerd-18.09
github.com/docker/docker/cmd/dockerd
/tmp/go-link-867197439/000008.o: In function mygetgrouplist':<br/>/usr/local/go1.10.6/go/src/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking<br/>/tmp/go-link-867197439/000007.o: In functionmygetgrgid_r':
/usr/local/go1.10.6/go/src/os/user/cgo_lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-867197439/000007.o: In function mygetgrnam_r':<br/>/usr/local/go1.10.6/go/src/os/user/cgo_lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking<br/>/tmp/go-link-867197439/000007.o: In functionmygetpwnam_r':
/usr/local/go1.10.6/go/src/os/user/cgo_lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-867197439/000007.o: In function `mygetpwuid_r':

/usr/local/go1.10.6/go/src/os/user/cgo_lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Created binary: bundles/binary-daemon/dockerd-18.09

[root@localhost docker]# bundles/binary-daemon/dockerd -v
Docker version 18.09, build 4c52b90/18.09
[root@localhost docker]# ldd bundles/binary-daemon/dockerd
not a dynamic executable

拷貝編譯完成的docker相關二制文件

[root@localhost ~]# mkdir /tmp/18.09-docker
[root@localhost ~]#cd /usr/local/docker/src/github.com
[root@localhost github.com]# cp docker/libnetwork/docker-proxy /tmp/18.09-docker/
[root@localhost github.com]# cp docker/cli/build/docker /tmp/18.09-docker/
[root@localhost github.com]# cp docker/docker/bundles/binary-daemon/dockerd /tmp/18.09-docker/
[root@localhost github.com]# cp containerd/containerd/bin/ctr containerd/containerd/bin/containerd containerd/containerd/bin/containerd-shim /tmp/18.09-docker/
[root@localhost github.com]# cp tini/docker-init /tmp/18.09-docker/
[root@localhost github.com]# cp opencontainers/runc/runc /tmp/18.09-docker/
[root@localhost github.com]# ls -lrt /tmp/18.09-docker/
total 160688
-rwxr-xr-x. 1 root root 2841376 Jan 25 01:38 docker-proxy
-rwxr-xr-x. 1 root root 50711753 Jan 25 01:38 docker
-rwxr-xr-x. 1 root root 53918880 Jan 25 01:39 dockerd
-rwxr-xr-x. 1 root root 28075792 Jan 25 01:40 containerd
-rwxr-xr-x. 1 root root 4968800 Jan 25 01:40 containerd-shim
-rwxr-xr-x. 1 root root 15816304 Jan 25 01:40 ctr
-rwxr-xr-x. 1 root root 845080 Jan 25 01:41 docker-init
-rwxr-xr-x. 1 root root 7352008 Jan 25 01:42 runc

[root@localhost ~]# docker version
Client:
Version: 18.09
API version: 1.39
Go version: go1.10.6
Git commit: 4c52b90
Built: Fri Jan 25 10:31:01 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.09
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 4c52b90/18.09
Built: Fri Jan 25 10:33:42 2019
OS/Arch: linux/amd64
Experimental: false

參考資料：

https://bobsteagall.com/2017/12/30/gcc-builder/
https://askubuntu.com/questions/1078516/disable-enable-default-pie-for-gcc
https://github.com/rust-lang/rust/issues/47037
http://gnu.mirror.constant.com/gcc/
https://github.com/containerd/containerd/blob/master/BUILDING.md
https://cbs.centos.org/koji/buildinfo?buildID=17550
https://download.docker.com/linux/static/stable/x86_64/