Centos7安裝部署Kubernetes（K8s）集群

Kubernetes集群的安裝有多種方式：下載源碼包編譯安裝、下載編譯好的二進制包安裝、使用kubeadm工具安裝等。本文是以二進制文件方式安裝Kubernetes集群。
系統環境

一、全局操作（所有機器執行）

1、安裝需要用到的工具

yum -y install vim bash-completion wget

注：安裝bash-completion工具后，使用tab鍵可以實現長格式參數補全，非常方便。kubectl命令的參數都是長格式，對于有些命令都記不住的我，更別說長格式參數了。
2、關閉firewalld防火墻
Kubernetes的master（管理主機）與node（工作節點）之間會有大量的網絡通信，安全的做法是在防火墻上配置各組件需要相互通信的端口號，關于防火墻的配置我會在后續博文中單獨講解。在一個安全的內部網絡環境中建議關閉防火墻服務，這里我們關閉防火墻來部署測試環境。

systemctl disable firewalld
systemctl stop firewalld

3、關閉SELinux
禁用SELinux的目的是讓容器可以讀取主機文件系統

sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
sed -i 's/SELINUXTYPE=targeted/#&/' /etc/selinux/config
setenforce 0

二、部署master管理節點

1、安裝CFSSL

[root@k8s-master ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl
[root@k8s-master ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson
[root@k8s-master ~]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/local/bin/cfssl-certinfo
[root@k8s-master ~]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

2、下載并解壓已編譯好的二進制包

[root@k8s-master tmp]# wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
[root@k8s-master tmp]# wget https://dl.k8s.io/v1.12.2/kubernetes-server-linux-amd64.tar.gz
[root@k8s-master tmp]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz
[root@k8s-master tmp]# tar zxvf kubernetes-server-linux-amd64.tar.gz

3、將可執行文件復制到/usr/bin目錄下

[root@k8s-master tmp]# cd etcd-v3.3.10-linux-amd64/
[root@k8s-master etcd-v3.3.10-linux-amd64]# cp -p etcd etcdctl /usr/bin/
[root@k8s-master etcd-v3.3.10-linux-amd64]# cd /tmp/kubernetes/server/bin/
[root@k8s-master bin]# cp -p kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/bin/

4、配置etcd服務
注：etcd作為kubernetes集群的數據庫，保存著所有資源對象的數據，安全起見，使用數字證書認證方式。生產環境建議將etcd獨立出來，單獨部署etcd集群。
（1）生成CA證書配置文件

[root@k8s-master bin]# mkdir -p /etc/{etcd/ssl,kubernetes/ssl}
[root@k8s-master bin]# cd /etc/etcd/ssl/
[root@k8s-master ssl]# cfssl print-defaults config > ca-config.json
[root@k8s-master ssl]# cfssl print-defaults csr > ca-csr.json

（2）修改配置文件
修改ca-config.json文件，設置有效期43800h（5年）

{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "kubernetes": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

"server auth","client auth"表示服務端和客戶端使用相同的證書驗證。
修改ca-csr.json文件，內容如下

{
    "CN": "k8s-master",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

（3）生成CA證書和私鑰相關文件

[root@k8s-master ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca

（4）簽發etcd證書文件

[root@k8s-master ssl]# cfssl print-defaults csr > etcd-csr.json

修改server-csr.json文件，內容如下

{
    "CN": "etcd",
    "hosts": [
        "127.0.0.1",
        "192.168.2.212"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

生成etcd證書和私鑰

[root@k8s-master ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname=127.0.0.1,192.168.2.212 etcd-csr.json | cfssljson -bare etcd

注："hosts"里填寫所有etcd主機的IP，-hostname填寫當前主機的IP，也可以填寫所有etcd主機的IP，這樣其他etcd節點就不需要再創建證書和私鑰了，拷貝過去直接使用。 -profile=kubernetes這個值根據對應ca-config.json文件中的profiles字段的值。
（5）創建生成etcd配置文件的腳本

[root@k8s-master ssl]# cd /root/
[root@k8s-master ~]# vim etcd.sh

#!/bin/bash
etcd_data_dir=/data/etcd
mkdir -p ${etcd_data_dir}

ETCD_NAME=${1:-"etcd"}
ETCD_LISTEN_IP=${2:-"192.168.2.212"}
ETCD_INITIAL_CLUSTER=${3:-}

cat <<EOF >//etc/etcd/etcd.conf
# [member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="${etcd_data_dir}/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://${ETCD_LISTEN_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_LISTEN_IP}:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_LISTEN_IP}:2380"
ETCD_INITIAL_CLUSTER="${ETCD_INITIAL_CLUSTER}"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_LISTEN_IP}:2379"
#
#[proxy]
#ETCD_PROXY="off"
#
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/etc/etcd/ssl/ca.pem"
ETCD_CERT_FILE="/etc/etcd/ssl/${ETCD_NAME}.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/${ETCD_NAME}-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/etc/etcd/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/${ETCD_NAME}.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/${ETCD_NAME}-key.pem"
EOF

cat <<EOF >//usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=${etcd_data_dir}
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/bin/bash -c "GOMAXPROCS=\$(nproc) /usr/bin/etcd"
Type=notify

[Install]
WantedBy=multi-user.target
EOF

（6）執行腳本，生成配置文件和啟動文件

[root@k8s-master ~]# sh etcd.sh

5、配置kube-apiserver服務
（1）創建生成apiserver配置文件的腳本
apiserver.sh腳本內容如下

#!/usr/bin/env bash
MASTER_ADDRESS=${1:-"192.168.2.212"}
ETCD_SERVERS=${2:-"https://127.0.0.1:2379"}
SERVICE_CLUSTER_IP_RANGE=${3:-"10.10.10.0/24"}
ADMISSION_CONTROL=${4:-""}
API_LOGDIR=${5:-"/data/apiserver/log"}

mkdir -p ${API_LOGDIR}
cat <<EOF >/etc/kubernetes/kube-apiserver
# --logtostderr=true: log to standard error instead of files
KUBE_LOGTOSTDERR="--logtostderr=false"
APISERVER_LOGDIR="--log-dir=${API_LOGDIR}"

# --v=0: log level for V logs
KUBE_LOG_LEVEL="--v=2"

# --etcd-servers=[]: List of etcd servers to watch (http://ip:port),
# comma separated. Mutually exclusive with -etcd-config
KUBE_ETCD_SERVERS="--etcd-servers=${ETCD_SERVERS}"

# --etcd-cafile="": SSL Certificate Authority file used to secure etcd communication.
KUBE_ETCD_CAFILE="--etcd-cafile=/etc/etcd/ssl/ca.pem"

# --etcd-certfile="": SSL certification file used to secure etcd communication.
KUBE_ETCD_CERTFILE="--etcd-certfile=/etc/etcd/ssl/etcd.pem"

# --etcd-keyfile="": key file used to secure etcd communication.
KUBE_ETCD_KEYFILE="--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem"

# --insecure-bind-address=127.0.0.1: The IP address on which to serve the --insecure-port.
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

# --insecure-port=8080: The port on which to serve unsecured, unauthenticated access.
KUBE_API_PORT="--insecure-port=8080"

# --kubelet-port=10250: Kubelet port
NODE_PORT="--kubelet-port=10250"

# --advertise-address=<nil>: The IP address on which to advertise
# the apiserver to members of the cluster.
KUBE_ADVERTISE_ADDR="--advertise-address=${MASTER_ADDRESS}"

# --allow-privileged=false: If true, allow privileged containers.
KUBE_ALLOW_PRIV="--allow-privileged=false"

# --service-cluster-ip-range=<nil>: A CIDR notation IP range from which to assign service cluster IPs.
# This must not overlap with any IP ranges assigned to nodes for pods.
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=${SERVICE_CLUSTER_IP_RANGE}"

# --admission-control="AlwaysAdmit": Ordered list of plug-ins
# to do admission control of resources into cluster.
KUBE_ADMISSION_CONTROL="--admission-control=${ADMISSION_CONTROL}"
EOF

KUBE_APISERVER_OPTS="   \${KUBE_LOGTOSTDERR}         \\
                        \${APISERVER_LOGDIR}         \\
                        \${KUBE_LOG_LEVEL}           \\
                        \${KUBE_ETCD_SERVERS}        \\
                        \${KUBE_ETCD_CAFILE}         \\
                        \${KUBE_ETCD_CERTFILE}       \\
                        \${KUBE_ETCD_KEYFILE}        \\
                        \${KUBE_API_ADDRESS}         \\
                        \${KUBE_API_PORT}            \\
                        \${NODE_PORT}                \\
                        \${KUBE_ADVERTISE_ADDR}      \\
                        \${KUBE_ALLOW_PRIV}          \\
                        \${KUBE_SERVICE_ADDRESSES}   \\
                        \${KUBE_ADMISSION_CONTROL}   \\
                        \${KUBE_API_CLIENT_CA_FILE}  \\
                        \${KUBE_API_TLS_CERT_FILE}   \\
                        \${KUBE_API_TLS_PRIVATE_KEY_FILE}"

cat <<EOF >/usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-apiserver
ExecStart=/usr/bin/kube-apiserver ${KUBE_APISERVER_OPTS}
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

啟動參數說明：
--logtostderr：設置為false，表示將日志寫入文件，不寫入stderr。
--log-dir：日志文件目錄。
--v：日志級別。
--etcd-servers：指定etcd服務的URL地址。
--etcd-cafile：連接etcd的ca根證書文件路徑。
--etcd-certfile：連接etcd的私鑰文件路徑。
--etcd-keyfile：連接etcd的key證書文件路徑。
--insecure-bind-address：apiserver的非安全IP地址，此參數已棄用，后面會替換成新的。
--insecure-port：apiserver的非安全端口號，此參數已棄用，后面會替換成新的。
--kubelet-port：kubelet的端口號，此參數已棄用，后面會去掉。
--advertise-address：apiserver主機的IP地址，用于通知其他集群成員。
--allow-privileged：是否允許容器運行在 privileged 模式，默認為false。
--service-cluster-ip-range：集群中service的虛擬IP地址范圍。
--admission-control：集群的準入控制設置，各控制模塊以插件的形式依次生效。此參數已棄用，后面會替換成新的。

（2）執行腳本，生成配置文件和啟動文件

[root@k8s-master ~]# sh apiserver.sh

6、配置kube-controller-manager服務
（1）創建生成controller-manager配置文件的腳本
controller-manager.sh腳本內容如下

#!/usr/bin/env bash
MASTER_ADDRESS=${1:-"192.168.2.212"}
CON_LOGDIR=${2:-"/data/controller-manager/log"}
mkdir -p ${CON_LOGDIR}

cat <<EOF >/etc/kubernetes/kube-controller-manager
KUBE_LOGTOSTDERR="--logtostderr=false"
CONTROLLER_LOGDIR="--log-dir=${CON_LOGDIR}"
KUBE_LOG_LEVEL="--v=2"
KUBE_MASTER="--master=${MASTER_ADDRESS}:8080"

# --root-ca-file="": If set, this root certificate authority will be included in
# service account's token secret. This must be a valid PEM-encoded CA bundle.
KUBE_CONTROLLER_MANAGER_ROOT_CA_FILE="--root-ca-file=/etc/kubernetes/ca.pem"

# --service-account-private-key-file="": Filename containing a PEM-encoded private
# RSA key used to sign service account tokens.
KUBE_CONTROLLER_MANAGER_SERVICE_ACCOUNT_PRIVATE_KEY_FILE="--service-account-private-key-file=/etc/kubernetes/k8s-server-key.pem"

# --leader-elect: Start a leader election client and gain leadership before 
# executing the main loop. Enable this when running replicated components for high availability.
KUBE_LEADER_ELECT="--leader-elect"
EOF

KUBE_CONTROLLER_MANAGER_OPTS="  \${KUBE_LOGTOSTDERR}  \\
                                \${CONTROLLER_LOGDIR} \\
                                \${KUBE_LOG_LEVEL}    \\
                                \${KUBE_MASTER}       \\
                                \${KUBE_CONTROLLER_MANAGER_ROOT_CA_FILE} \\
                                \${KUBE_CONTROLLER_MANAGER_SERVICE_ACCOUNT_PRIVATE_KEY_FILE}\\
                                \${KUBE_LEADER_ELECT}"

cat <<EOF >/usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-controller-manager
ExecStart=/usr/bin/kube-controller-manager ${KUBE_CONTROLLER_MANAGER_OPTS}
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

（2）執行腳本，生成配置文件和啟動文件

[root@k8s-master ~]# sh controller-manager.sh

7、配置kube-scheduler服務
（1）創建生成scheduler配置文件的腳本
scheduler.sh腳本內容如下

#!/usr/bin/env bash
MASTER_ADDRESS=${1:-"192.168.2.212"}
SCH_LOGDIR=${2:-"/data/scheduler/log"}
mkdir -p ${SCH_LOGDIR}

cat <<EOF >/etc/kubernetes/kube-scheduler
###
# kubernetes scheduler config

# --logtostderr=true: log to standard error instead of files
KUBE_LOGTOSTDERR="--logtostderr=false"
SCHEDULER_LOGDIR="--log-dir=${SCH_LOGDIR}"

# --v=0: log level for V logs
KUBE_LOG_LEVEL="--v=4"

# --master: The address of the Kubernetes API server (overrides any value in kubeconfig).
KUBE_MASTER="--master=${MASTER_ADDRESS}:8080"

# --leader-elect: Start a leader election client and gain leadership before 
# executing the main loop. Enable this when running replicated components for high availability.
KUBE_LEADER_ELECT="--leader-elect"

# Add your own!
KUBE_SCHEDULER_ARGS=""

EOF

KUBE_SCHEDULER_OPTS="   \${KUBE_LOGTOSTDERR}     \\
                        \${SCHEDULER_LOGDIR}     \\
                        \${KUBE_LOG_LEVEL}       \\
                        \${KUBE_MASTER}          \\
                        \${KUBE_LEADER_ELECT}    \\
                        \$KUBE_SCHEDULER_ARGS"

cat <<EOF >/usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-scheduler
ExecStart=/usr/bin/kube-scheduler ${KUBE_SCHEDULER_OPTS}
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

（3）執行腳本，生成配置文件和啟動文件

[root@k8s-master ~]# sh scheduler.sh

8、啟動master主機所有服務

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl enable etcd
[root@k8s-master ~]# systemctl start etcd
[root@k8s-master ~]# systemctl enable kube-apiserver
[root@k8s-master ~]# systemctl start kube-apiserver
[root@k8s-master ~]# systemctl enable kube-controller-manager
[root@k8s-master ~]# systemctl start kube-controller-manager
[root@k8s-master ~]# systemctl enable kube-scheduler 
[root@k8s-master ~]# systemctl start kube-scheduler

9、驗證etcd運行狀態

[root@k8s-master ~]# export ETCDCTL_API=3
[root@k8s-master ~]# etcdctl --key="/etc/etcd/ssl/etcd-key.pem" --cert="/etc/etcd/ssl/etcd.pem" --cacert="/etc/etcd/ssl/ca.pem" endpoint health
[root@k8s-master ~]# etcdctl --key="/etc/etcd/ssl/etcd-key.pem" --cert="/etc/etcd/ssl/etcd.pem" --cacert="/etc/etcd/ssl/ca.pem" member list

ETCDCTL_API=3表示使用etcd3.x版本的命令，由于我們配置了etcd的證書，所以etcdctl命令要帶上證書。

三、部署node工作節點

1、部署docker環境
（1）安裝docker
注：安裝的是docker社區版本，版本號18.06.1-ce

[root@k8s-node1 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@k8s-node1 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@k8s-node1 ~]# yum makecache fast
[root@k8s-node1 ~]# yum -y install docker-ce

（2）修改配置文件，添加私有倉庫地址和阿里云鏡像地址，并指定docker數據存儲目錄

[root@k8s-node1 ~]# mkdir -p /data/docker
[root@k8s-node1 ~]# mkdir -p /etc/docker
[root@k8s-node1 ~]# vim /etc/docker/daemon.json

{
  "registry-mirrors": ["https://registry.docker-cn.com"], "graph": "/data/docker",
  "insecure-registries": ["192.168.2.225:5000"]
}

（3）啟動docker，并加入開機啟動

[root@k8s-node1 ~]# systemctl start docker
[root@k8s-node1 ~]# systemctl enable docker

2、下載并解壓已編譯好的二進制包

[root@k8s-node1 tmp]# wget https://dl.k8s.io/v1.12.2/kubernetes-node-linux-amd64.tar.gz
[root@k8s-node1 tmp]# tar zxvf kubernetes-node-linux-amd64.tar.gz

3、將可執行文件復制到/usr/bin目錄下

[root@k8s-node1 tmp]# cd kubernetes/node/bin/
[root@k8s-node1 bin]# cp -p kubectl kubelet kube-proxy /usr/bin/

4、配置kubelet服務
（1）創建生成kubelet配置文件的腳本
kubelet.sh腳本內容如下

#!/usr/bin/env bash
MASTER_ADDRESS=${1:-"192.168.2.212"}
NODE_ADDRESS=${2:-"192.168.2.213"}
KUBECONFIG_DIR=${KUBECONFIG_DIR:-/etc/kubernetes}
NODE_LOGDIR=${3:-"/data/kubelet/log"}
mkdir -p ${KUBECONFIG_DIR}
mkdir -p ${NODE_LOGDIR}
# Generate a kubeconfig file
cat <<EOF > "${KUBECONFIG_DIR}/kubelet.kubeconfig"
apiVersion: v1
kind: Config
clusters:
  - cluster:
      server: http://${MASTER_ADDRESS}:8080/
    name: local
contexts:
  - context:
      cluster: local
    name: local
current-context: local
EOF

cat <<EOF >/etc/kubernetes/kubelet
# --logtostderr=true: log to standard error instead of files
KUBE_LOGTOSTDERR="--logtostderr=false"
KUBELET_LOGDIR="--log-dir=${NODE_LOGDIR}"

#  --v=0: log level for V logs
KUBE_LOG_LEVEL="--v=2"

# --hostname-override="": If non-empty, will use this string as identification instead of the actual hostname.
NODE_HOSTNAME="--hostname-override=${NODE_ADDRESS}"

# Path to a kubeconfig file, specifying how to connect to the API server.
KUBELET_KUBECONFIG="--kubeconfig=${KUBECONFIG_DIR}/kubelet.kubeconfig"

# Add your own!
KUBELET_ARGS="--kubeconfig=${KUBECONFIG_DIR}/kubelet.kubeconfig --hostname-override=${NODE_ADDRESS} --logtostderr=false --log-dir=${NODE_LOGDIR} --v=2"
EOF

KUBELET_OPTS="   \${KUBE_LOGTOSTDERR}  \\
                 \${KUBELET_LOGDIR}    \\
                 \${KUBE_LOG_LEVEL}    \\
                 \${NODE_HOSTNAME}     \\
                 \${KUBELET_KUBECONFIG}"

cat <<EOF >/usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet ${KUBELET_OPTS}
Restart=on-failure
KillMode=process
RestartSec=15s

[Install]
WantedBy=multi-user.target
EOF

（2）執行腳本，生成配置文件和啟動文件

[root@k8s-node1 ~]# sh kubelet.sh

5、配置kube-proxy服務
（1）創建生成kube-proxy配置文件的腳本
proxy.sh腳本內容如下

#!/usr/bin/env bash
MASTER_ADDRESS=${1:-"192.168.2.212"}
NODE_ADDRESS=${2:-"192.168.2.213"}

mkdir -p /data/proxy/log
cat <<EOF >/etc/kubernetes/kube-proxy
# --logtostderr=true: log to standard error instead of files
KUBE_LOGTOSTDERR="--logtostderr=false"
PROXY_LOGDIR="--log-dir=/data/proxy/log"

#  --v=0: log level for V logs
KUBE_LOG_LEVEL="--v=2"

# --hostname-override="": If non-empty, will use this string as identification instead of the actual hostname.
NODE_HOSTNAME="--hostname-override=${NODE_ADDRESS}"

# --master="": The address of the Kubernetes API server (overrides any value in kubeconfig)
KUBE_MASTER="--master=http://${MASTER_ADDRESS}:8080"
EOF

KUBE_PROXY_OPTS="   \${KUBE_LOGTOSTDERR} \\
                    \${PROXY_LOGDIR}   \\
                    \${KUBE_LOG_LEVEL}   \\
                    \${NODE_HOSTNAME}    \\
                    \${KUBE_MASTER}"

cat <<EOF >/usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/kube-proxy
ExecStart=/usr/bin/kube-proxy ${KUBE_PROXY_OPTS}
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

（2）執行腳本，生成配置文件和啟動文件

[root@k8s-node1 ~]# sh proxy.sh

6、啟動node節點服務

[root@k8s-node1 ~]# swapoff -a
[root@k8s-node1 ~]# systemctl daemon-reload
[root@k8s-node1 ~]# systemctl enable kubelet.service
[root@k8s-node1 ~]# systemctl start kubelet.service
[root@k8s-node1 ~]# systemctl enable kube-proxy.service
[root@k8s-node1 ~]# systemctl start kube-proxy.service

使用swapoff -a命令關閉swap交換分區，否則kubelet會啟動不了。
至此kubernetes（k8s）集群就搭建完成了，此時只有etcd是通過https連的，其他服務都是通過http連的，接下來我們就將其他服務也配置成https，也使用CA數字證書認證方式。

四、Kubernetes集群的安全設置

在一個安全的內網環境中，Kubernetes的各個組件與Master之間可以通過apiserver的非安全端口http://apiserver:8080 進行訪問。但如果apiserver需要對外提供服務，或者集群中的某些容器也需要訪問apiserver以獲取集群中的某些信息，則更安全的做法是啟用HTTPS安全機制。Kubernetes提供了基于CA簽名的雙向數字證書認證方式和簡單的基于HTTP BASE或TOKEN的認證方式，其中CA證書方式的安全性最高。現在我們就來配置基于CA簽名的數字證書認證方式。
1、關閉node節點所有服務

[root@k8s-node1 ~]# systemctl stop kube-proxy.service 
[root@k8s-node1 ~]# systemctl stop kubelet.service

2、關閉master主機除etcd外的所有服務

[root@k8s-master ~]# systemctl stop kube-scheduler.service 
[root@k8s-master ~]# systemctl stop kube-controller-manager.service 
[root@k8s-master ~]# systemctl stop kube-apiserver.service

3、生成各組件的證書和私鑰
（1）復制CA根證書和私鑰相關文件到存放kubernetes證書私鑰文件的目錄下

[root@k8s-master ~]# cd /etc/etcd/ssl/
[root@k8s-master ssl]# cp ca.pem ca-config.json ca-key.pem etcd-csr.json /etc/kubernetes/ssl/
[root@k8s-master ssl]# cd /etc/kubernetes/ssl/

（2）查看cluster role都有哪些用戶

[root@k8s-master ssl]# kubectl get clusterrole

注：apiserver使用admin用戶，controller-manager使用system:kube-controller-manager用戶，scheduler使用system:kube-scheduler用戶，kubelet和kube-proxy使用system:node用戶，用戶名對應json文件的CN（憑證）。記住用戶名和組件一定要一一對應，否則其他組件會連不上apiserver。
（3）編輯apiserver-csr.json文件

[root@k8s-master ssl]# mv etcd-csr.json apiserver-csr.json
[root@k8s-master ssl]# vim apiserver-csr.json

注：apiserver的證書和私鑰使用apiserver-csr.json文件來創建，使用admin用戶

{
    "CN": "admin",
    "hosts": [
        "127.0.0.1",
        "192.168.2.212"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai",
            "O": "admin",
            "OU": "system"
        }
    ]
}

（4）編輯kube-controller-manager的k8s-csr.json文件
注：kube-controller-manager的證書和私鑰使用k8s-csr.json文件來創建，使用system:kube-controller-manager用戶

{
    "CN": "system:kube-controller-manager",
    "hosts": [
        "127.0.0.1",
        "192.168.2.212"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai",
            "O": "controller",
            "OU": "system"
        }
    ]
}

（5）編輯scheduler-csr.json文件

{
    "CN": "system:kube-scheduler",
    "hosts": [
        "127.0.0.1",
        "192.168.2.212"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai",
            "O": "kube-scheduler",
            "OU": "system"
        }
    ]
}

（6）編輯node-csr.json文件

{
    "CN": "system:node",
    "hosts": [
        "127.0.0.1",
        "192.168.2.212",
        "192.168.2.213",
        "192.168.2.214",
        "192.168.2.215"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai",
            "O": "node",
            "OU": "system"
        }
    ]
}

（7）生成各組件的證書和私鑰文件

[root@k8s-master ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname=127.0.0.1,192.168.2.212 apiserver-csr.json | cfssljson -bare apiserver
[root@k8s-master ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname=127.0.0.1,192.168.2.212 k8s-csr.json | cfssljson -bare k8s
[root@k8s-master ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname=127.0.0.1,192.168.2.212 scheduler-csr.json | cfssljson -bare scheduler
[root@k8s-master ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname=127.0.0.1,192.168.2.212,192.168.2.213,192.168.2.214,192.168.2.215 node-csr.json | cfssljson -bare node

4、將證書文件和私鑰文件復制到三臺node節點上
（1）在node節點上創建存放證書文件的目錄

[root@k8s-node1 ~]# mkdir -p /etc/kubernetes/ssl

（2）將文件復制到node節點上

[root@k8s-master ssl]# scp ca.pem node.pem node-key.pem root@192.168.2.213:/etc/kubernetes/ssl/

注：另外兩臺請替換命令中的IP
5、配置kube-apiserver服務
（1）修改kube-apiserver配置文件，/etc/kubernetes/kube-apiserver

KUBE_LOGTOSTDERR="--logtostderr=false"
APISERVER_LOGDIR="--log-dir=/data/apiserver/log"
KUBE_LOG_LEVEL="--v=2"
KUBE_ETCD_SERVERS="--etcd-servers=https://127.0.0.1:2379"
KUBE_ETCD_CAFILE="--etcd-cafile=/etc/etcd/ssl/ca.pem"
KUBE_ETCD_CERTFILE="--etcd-certfile=/etc/etcd/ssl/etcd.pem"
KUBE_ETCD_KEYFILE="--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem"
KUBE_API_ADDRESS="--bind-address=0.0.0.0"
KUBE_API_PORT="--secure-port=6443"
KUBE_ADVERTISE_ADDR="--advertise-address=192.168.2.212"
KUBE_ALLOW_PRIV="--allow-privileged=true"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.10.10.0/24"
KUBE_MODE_CONTROL="--authorization-mode=Node,RBAC"
KUBE_ADMISSION_CONTROL="--enable-admission-plugins=NodeRestriction"
KUBE_API_CLIENT_CA_FILE="--client-ca-file=/etc/kubernetes/ssl/ca.pem"
KUBE_API_TLS_CERT_FILE="--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem"
KUBE_API_TLS_PRIVATE_KEY_FILE="--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem"

（2）修改systemd啟動配置文件，/usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-apiserver
ExecStart=/usr/bin/kube-apiserver    ${KUBE_LOGTOSTDERR}         \
                        ${APISERVER_LOGDIR}         \
                        ${KUBE_LOG_LEVEL}           \
                        ${KUBE_ETCD_SERVERS}        \
                        ${KUBE_ETCD_CAFILE}         \
                        ${KUBE_ETCD_CERTFILE}       \
                        ${KUBE_ETCD_KEYFILE}        \
                        ${KUBE_API_ADDRESS}         \
                        ${KUBE_API_PORT}            \
                        ${KUBE_ADVERTISE_ADDR}      \
                        ${KUBE_ALLOW_PRIV}          \
                        ${KUBE_SERVICE_ADDRESSES}   \
                        ${KUBE_MODE_CONTROL}        \
                        ${KUBE_ADMISSION_CONTROL}   \
                        ${KUBE_API_CLIENT_CA_FILE}  \
                        ${KUBE_API_TLS_CERT_FILE}   \
                        ${KUBE_API_TLS_PRIVATE_KEY_FILE}
Restart=on-failure

[Install]
WantedBy=multi-user.target

（3）啟動kube-apiserver服務

[root@k8s-master ssl]# systemctl daemon-reload
[root@k8s-master ssl]# systemctl start kube-apiserver.service

6、配置kube-controller-manager服務
（1）創建config配置文件，/etc/kubernetes/kubeconfig

[root@k8s-master kubernetes]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --kubeconfig=kubeconfig
[root@k8s-master kubernetes]# kubectl config set-credentials system:kube-controller-manager --client-certificate=/etc/kubernetes/ssl/k8s.pem --client-key=/etc/kubernetes/ssl/k8s-key.pem --embed-certs=true --kubeconfig=kubeconfig
[root@k8s-master kubernetes]# kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kubeconfig
[root@k8s-master kubernetes]# kubectl config use-context system:kube-controller-manager --kubeconfig=kubeconfig

（2）修改kube-controller-manager配置文件，/etc/kubernetes/kube-controller-manager

KUBE_LOGTOSTDERR="--logtostderr=false"
CON_LOGDIR="--log-dir=/data/controller-manager/log"
KUBE_LOG_LEVEL="--v=2"
KUBE_MASTER="--master=https://192.168.2.212:6443"
KUBE_CONTROLLER_MANAGER_ROOT_CA_FILE="--root-ca-file=/etc/kubernetes/ssl/ca.pem"
KUBE_CONTROLLER_MANAGER_SERVICE_ACCOUNT_PRIVATE_KEY_FILE="--service-account-private-key-file=/etc/kubernetes/ssl/k8s-key.pem"
KUBE_CONFIG_FILE="--kubeconfig=/etc/kubernetes/kubeconfig"
KUBE_LEADER_ELECT="--leader-elect"

（3）修改systemd啟動配置文件，/usr/lib/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-controller-manager
ExecStart=/usr/bin/kube-controller-manager   ${KUBE_LOGTOSTDERR} \
                                ${CON_LOGDIR}       \
                                ${KUBE_LOG_LEVEL}   \
                                ${KUBE_MASTER}      \
                                ${KUBE_CONTROLLER_MANAGER_ROOT_CA_FILE} \
                                ${KUBE_CONTROLLER_MANAGER_SERVICE_ACCOUNT_PRIVATE_KEY_FILE}\
                                ${KUBE_CONFIG_FILE}  \
                                ${KUBE_LEADER_ELECT}

Restart=on-failure

[Install]
WantedBy=multi-user.target

（4）啟動kube-controller-manager服務

[root@k8s-master kubernetes]# systemctl daemon-reload
[root@k8s-master kubernetes]# systemctl start kube-controller-manager.service

（5）創建角色綁定
kube-controller-manager服務啟動后，查看日志報如下錯誤

從報錯來看是rbac的授權錯誤，node信息的維護是屬于system:controller下面的用戶維護的，用戶system:kube-controller-manager沒有權限造成的。需要將system:kube-controller-manager綁定到system:controller:node-controller用戶下即可。

[root@k8s-master ssl]# kubectl create clusterrolebinding controller-node-clusterrolebing --clusterrole=system:controller:node-controller --user=system:kube-controller-manager

查看綁定信息

[root@k8s-master ssl]# kubectl describe clusterrolebinding controller-node-clusterrolebing

現在再查看日志就沒有上面的報錯了，但還是有cluster的錯誤。

將system:kube-controller-manager綁定到cluster-admin用戶下

[root@k8s-master ssl]# kubectl create clusterrolebinding controller-cluster-clusterrolebing --clusterrole=cluster-admin --user=system:kube-controller-manager

7、配置kube-scheduler服務
（1）創建config配置文件，/etc/kubernetes/scheconfig

[root@k8s-master kubernetes]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --kubeconfig=scheconfig
[root@k8s-master kubernetes]# kubectl config set-credentials system:kube-scheduler --client-certificate=/etc/kubernetes/ssl/scheduler.pem --client-key=/etc/kubernetes/ssl/scheduler-key.pem --embed-certs=true --kubeconfig=scheconfig
[root@k8s-master kubernetes]# kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=scheconfig
[root@k8s-master kubernetes]# kubectl config use-context system:kube-scheduler --kubeconfig=scheconfig

（2）修改kube-scheduler配置文件，/etc/kubernetes/kube-scheduler

KUBE_LOGTOSTDERR="--logtostderr=false"
KUBE_LOGDIR="--log-dir=/data/scheduler/log"
KUBE_LOG_LEVEL="--v=2"
KUBE_MASTER="--master=https://192.168.2.212:6443"
SCHEDULER_CONFIG_FILE="--kubeconfig=/etc/kubernetes/scheconfig"
KUBE_LEADER_ELECT="--leader-elect"
KUBE_SCHEDULER_ARGS=""

（3）修改systemd啟動配置文件，/usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-scheduler
ExecStart=/usr/bin/kube-scheduler    ${KUBE_LOGTOSTDERR}     \
                        ${KUBE_LOGDIR}          \
                        ${KUBE_LOG_LEVEL}       \
                        ${KUBE_MASTER}          \
                        ${SCHEDULER_CONFIG_FILE} \
                        ${KUBE_LEADER_ELECT}    \
                        $KUBE_SCHEDULER_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target

（4）啟動kube-scheduler服務

[root@k8s-master kubernetes]# systemctl daemon-reload 
[root@k8s-master kubernetes]# systemctl start kube-scheduler

8、配置kubelet服務
（1）創建kubelet和kube-proxy的config配置文件，/etc/kubernetes/nodeconfig

[root@k8s-node1 kubernetes]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.2.212:6443 --kubeconfig=nodeconfig
[root@k8s-node1 kubernetes]# kubectl config set-credentials system:node --client-certificate=/etc/kubernetes/ssl/node.pem --client-key=/etc/kubernetes/ssl/node-key.pem --embed-certs=true --kubeconfig=nodeconfig
[root@k8s-node1 kubernetes]# kubectl config set-context system:node --cluster=kubernetes --user=system:node --kubeconfig=nodeconfig
[root@k8s-node1 kubernetes]# kubectl config use-context system:node --kubeconfig=nodeconfig

（2）修改kubelet配置文件，/etc/kubernetes/kubelet

KUBE_LOGTOSTDERR="--logtostderr=false"
NODE_LOGDIR="--log-dir=/data/kubelet/log"
KUBE_LOG_LEVEL="--v=2"
NODE_HOSTNAME="--hostname-override=192.168.2.213"
KUBELET_KUBECONFIG="--kubeconfig=/etc/kubernetes/nodeconfig"

（3）啟動kubelet服務

[root@k8s-node1 kubernetes]# systemctl start kubelet.service

（4）創建角色綁定
kubelet啟動后，報如下錯誤

將system:node用戶也綁定到以上兩個角色中

[root@k8s-master ssl]# kubectl create clusterrolebinding node-node-clusterrolebing --clusterrole=system:controller:node-controller --user=system:node
[root@k8s-master ssl]# kubectl create clusterrolebinding node-cluster-clusterrolebing --clusterrole=cluster-admin --user=system:node

9、配置kube-proxy服務
（1）修改kube-proxy配置文件，/etc/kubernetes/kube-proxy

KUBE_LOGTOSTDERR="--logtostderr=false"
PROXY_LOGDIR="--log-dir=/data/proxy/log"
KUBE_LOG_LEVEL="--v=2"
NODE_HOSTNAME="--hostname-override=192.168.2.213"
KUBE_MASTER="--master=https://192.168.2.212:6443"
PROXY_KUBECONFIG="--kubeconfig=/etc/kubernetes/nodeconfig"

（2）修改systemd啟動配置文件，/usr/lib/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/kube-proxy
ExecStart=/usr/bin/kube-proxy    ${KUBE_LOGTOSTDERR} \
                    ${PROXY_LOGDIR}   \
                    ${KUBE_LOG_LEVEL}   \
                    ${NODE_HOSTNAME}    \
                    ${KUBE_MASTER}      \
                    ${PROXY_KUBECONFIG}
Restart=on-failure

[Install]
WantedBy=multi-user.target

（3）啟動kube-proxy服務

[root@k8s-node1 kubernetes]# systemctl daemon-reload
[root@k8s-node1 kubernetes]# systemctl start kube-proxy.service

如果報以下錯誤，需要先清空iptables的NAT規則

[root@k8s-node1 kubernetes]# iptables -F -t nat
[root@k8s-node1 kubernetes]# iptables -X -t nat
[root@k8s-node1 kubernetes]# iptables -Z -t nat
[root@k8s-node1 kubernetes]# systemctl restart kube-proxy.service

（4）查看node狀態

[root@k8s-master kubernetes]# kubectl get nodes

至此kubernetes集群的部署就全部完成了，各組件之間都是通過https協議連接的。通過上面各組件的配置，我們不難看出所有組件都需要連接apiserver服務，數據也是通過apiserver存儲到etcd中的，可以說apiserver是整個集群的中心組件。

五、測試

這里創建一個nginx的deployment用來測試
1、node節點下載pause鏡像
由于國內訪問不了k8s.gcr.io/pause:3.1，所以這里從kocker網站下載pause鏡像

[root@k8s-node1 log]# docker pull docker.io/kubernetes/pause
[root@k8s-node1 log]# docker tag kubernetes/pause:latest k8s.gcr.io/pause:3.1

2、創建nginx的Deployment定義文件
nginx.yaml文件內容如下

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myweb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myweb
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: myweb
    spec:
      containers:
      - name: myweb
        image: nginx
        ports:
        - containerPort: 80

3、創建deployment、RS、Pod和容器

[root@k8s-master ~]# kubectl create -f nginx.yaml

4、查看創建好的deployment運行情況

[root@k8s-master ~]# kubectl get deployment

5、查看ReplicaSet（RS）的運行情況

[root@k8s-master ~]# kubectl get rs

6、查看Pod的運行情況

[root@k8s-master ~]# kubectl get pod

7、查看容器的運行情況（node節點）

[root@k8s-node1 log]# docker ps -a

8、創建nginx的service定義文件
myweb-svc.yaml文件的內容如下

apiVersion: v1
kind: Service
metadata:
  name: myweb
spec:
  type: NodePort
  ports:
    - port: 80
      nodePort: 30001
  selector:
    app: myweb

9、創建Service

[root@k8s-master ~]# kubectl create -f myweb-svc.yaml

10、查看Service的運行情況

[root@k8s-master ~]# kubectl get svc

11、通過瀏覽器訪問
通過node節點的30001端口訪問，http://192.168.2.213:30001