kubeadm安裝Kubernetes1.15安裝部署詳解-Part 2

前提

Kubernetes發布了今年第二大版本 Kubernetes 1.15，此次版本共更新加強了25個相關功能，其中2個升級到GA版本，13個升級到beta版，10個alpha版。

Kubernetes 從1.14版本開始引入了新功能，用于動態地將主節點添加到群集。無需在節點之間復制證書和密鑰，從而減輕了自舉過程中的額外編排和復雜性。本文就使用這個新特性進行部署。整體部署過程多快好省！

初始化群集并系統環境 (所有節點上進行如下操作)

1.設置主機名hostname，管理節點設置主機名為 master 。
2.編輯 /etc/hosts 文件，添加域名解析。
3.關閉防火墻、selinux和swap。
4.配置內核參數，將橋接的IPv4流量傳遞到iptables的鏈
5.配置國內yum源

yum install -y wget
mkdir /etc/yum.repos.d/bak && mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos7_base.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.cloud.tencent.com/repo/epel-7.repo
yum clean all && yum makecache

6.配置國內Kubernetes源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

安裝必須軟件(所有節點上進行如下操作)

1:安裝docker

添加docker-ce repo文件

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce-18.06.1.ce-3.el7
systemctl enable docker && systemctl start docker

2:安裝kubeadm、kubelet、kubectl
Kubeadm是Kubernetes集群管理工具。
Kubelet負責與其他節點集群通信，并進行本節點Pod和容器生命周期的管理。
Kubeadm是Kubernetes的自動化部署工具，降低了部署難度，提高效率。

yum install -y kubelet kubeadm kubectl
systemctl enable kubelet

集群部署

節點角色介紹

[kub-master]        節點名稱                部署服務
172.20.101.157 name=kubm-01  docker、keepalived、nginx、etcd、kube-apiserver、kube-controller-manager、kube-scheduler
172.20.101.164 name=kubm-02  docker、keepalived、nginx、etcd、kube-apiserver、kube-controller-manager、kube-scheduler
172.20.101.165 name=kubm-03  docker、keepalived、nginx、etcd、kube-apiserver、kube-controller-manager、kube-scheduler

[kub-node]
172.20.101.160 name=kubnode-01 kubelet、docker、kube_proxy
172.20.101.166 name=kubnode-02 kubelet、docker、kube_proxy
172.20.101.167 name=kubnode-03 kubelet、docker、kube_proxy

新建安裝部署目錄

mkdir -p /etc/kuber/kubeadm

創建一個初始初始化文件 （kubm-01執行）

我使用的flannel 網絡插件需要配置網絡參數 --pod-network-cidr=10.244.0.0/16 。

vim /etc/kuber/kubeadm/kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "172.20.101.252:16443"
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.245.0.0/16

注意我使用nginx做的代理

master上面都配置Nginx反向代理 API Server

API Server使用 6443 端口；
Nginx 代理端口為 16443 端口；
172.20.101.252 是master節點的vip。

推薦清理環境

如果之前配置過k8s或者首次配置沒有成功等情況，推薦把系統環境清理一下,每一個節點。

systemctl stop kubelet
docker rm -f -v $(docker ps  -a -q)

rm -rf /etc/kubernetes
rm -rf  /var/lib/etcd
rm -rf   /var/lib/kubelet
rm -rf  $HOME/.kube/config
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

yum reinstall -y kubelet
systemctl daemon-reload
systemctl restart docker
systemctl enable kubelet

使用upload-certs和config指定初始化集群。

kubeadm init \
    --config=/etc/kuber/kubeadm/kubeadm-config.yaml \
    --upload-certs \
    --control-plane

第一臺master節點初始化返回結果

在執行節點上執行如下操作，初始化一下k8s環境。

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

添加master節點執行如下操作

  kubeadm join 172.20.101.252:16443 --token slsxxo.5aiu0lqpxy61n8ah \
    --discovery-token-ca-cert-hash sha256:2c3286ca0ac761ff7e29f590545d3370f801854158e7c6adde586ba96f1a6675 \
    --control-plane --certificate-key 1a139dc53b553091c59262b2f08b948848d7cda7e9cb0169c3f2e3db480ea255

添加nodes節點執行如下操作

kubeadm join 172.20.101.252:16443 --token slsxxo.5aiu0lqpxy61n8ah \
    --discovery-token-ca-cert-hash sha256:2c3286ca0ac761ff7e29f590545d3370f801854158e7c6adde586ba96f1a6675 

列出可用的令牌kubeadm。

[root@kubm-01 ~]# 

kubeadm  token list

TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION                                           EXTRA GROUPS
ezwzjn.9uslrdvu8y3o7hxc   23h       2019-06-27T13:29:21+08:00   authentication,signing   <none>                                                system:bootstrappers:kubeadm:default-node-token
nt7p9j.dgnf30gcr4bxg1le   1h        2019-06-26T15:29:20+08:00   <none>                   Proxy for managing TTL for the kubeadm-certs secret   <none>

檢查命名空間中的kubeadm-cert秘密kube-system。

kubectl get secrets -n kube-system kubeadm-certs -o yaml

配置kubectl訪問環境。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看系統環境鏡像啟動

[root@kubm-01 ~]# kubectl get pods -n kube-system
NAME                              READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-bdjh7          0/1     Pending   0          71s
coredns-5c98db65d4-xvvpl          0/1     Pending   0          71s
etcd-kubm-01                      1/1     Running   0          20s
kube-apiserver-kubm-01            1/1     Running   0          34s
kube-controller-manager-kubm-01   1/1     Running   0          14s
kube-proxy-l29hl                  1/1     Running   0          70s
kube-scheduler-kubm-01            1/1     Running   0          32s

只用dns服務pending，在添加網絡模塊后會自動修復。

部署flannel網絡

使用與podSubnet上面配置匹配的pod CIDR 安裝CNI插件,按照實際情況修改。

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

驗證1個master節點是Ready。

kubectl get nodes
NAME      STATUS   ROLES    AGE    VERSION   
kubm-01   Ready    master   7m2s   v1.15.0   

再次驗證kube-systempod是Running。

kubectl get pods -n kube-system

[root@kubm-01 ~]# kubectl get pods -n kube-system
NAME                              READY   STATUS    RESTARTS   AGE
etcd-kubm-01                      1/1     Running   0          80s
kube-apiserver-kubm-01            1/1     Running   0          94s
kube-controller-manager-kubm-01   1/1     Running   0          74s
kube-proxy-l29hl                  1/1     Running   0          2m10s
kube-scheduler-kubm-01            1/1     Running   0          92s
coredns-5c98db65d4-bdjh7          0/1     Running   0          2m11s
coredns-5c98db65d4-xvvpl          0/1     Running   0          2m11s
kube-flannel-ds-amd64-chb4p       1/1     Running   0          16s

發現dns和網絡插件容器都正常工作。

添加第二個 master

直接執行初始化集群時返回的命令添加即可，大致步驟如下。
1：清理系統環境
2：運行上一節中記錄的join命令。

  kubeadm join 172.20.101.252:16443 --token slsxxo.5aiu0lqpxy61n8ah \
    --discovery-token-ca-cert-hash sha256:2c3286ca0ac761ff7e29f590545d3370f801854158e7c6adde586ba96f1a6675 \
    --control-plane --certificate-key 1a139dc53b553091c59262b2f08b948848d7cda7e9cb0169c3f2e3db480ea255

3:初始化系統環境

[root@kubm-02 ~]# 
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

獲取集群狀態

[root@kubm-02 ~]# kubectl get nodes                                               
NAME      STATUS   ROLES    AGE     VERSION
kubm-01   Ready    master   7m58s   v1.15.0
kubm-02   Ready    master   103s    v1.15.0

[root@kubm-02 ~]# kubectl get pods -n kube-system

NAME                              READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-bdjh7          1/1     Running   0          8m29s
coredns-5c98db65d4-xvvpl          1/1     Running   0          8m29s
etcd-kubm-01                      1/1     Running   0          7m38s
etcd-kubm-02                      1/1     Running   0          80s
kube-apiserver-kubm-01            1/1     Running   0          7m52s
kube-apiserver-kubm-02            1/1     Running   0          72s
kube-controller-manager-kubm-01   1/1     Running   0          7m32s
kube-controller-manager-kubm-02   1/1     Running   0          72s
kube-flannel-ds-amd64-chb4p       1/1     Running   0          6m34s
kube-flannel-ds-amd64-h74k9       1/1     Running   1          2m30s
kube-proxy-kww9g                  1/1     Running   0          2m30s
kube-proxy-l29hl                  1/1     Running   0          8m28s
kube-scheduler-kubm-01            1/1     Running   0          7m50s
kube-scheduler-kubm-02            1/1     Running   0          94s

第三個 master 過程與店家第二個master節點保持一致即可。

node 節點添加

推薦清理環境

如果之前配置過k8s或者首次配置沒有成功等情況，推薦把系統環境清理一下,每一個節點。

systemctl stop kubelet
docker rm -f -v $(docker ps  -a -q)

rm -rf /etc/kubernetes
rm -rf  /var/lib/etcd
rm -rf   /var/lib/kubelet
rm -rf  $HOME/.kube/config
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

yum reinstall -y kubelet
systemctl daemon-reload
systemctl restart docker
systemctl enable kubelet

node節點接入

運行首次初始化時返回結果中，添加node的join命令。

kubeadm join 172.20.101.252:16443 --token slsxxo.5aiu0lqpxy61n8ah \
    --discovery-token-ca-cert-hash sha256:2c3286ca0ac761ff7e29f590545d3370f801854158e7c6adde586ba96f1a6675 \
    --node-name kubnode02

返回信息

[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

查看node節點加入集群 (master 節點執行)

kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
kubm-01      Ready    master   19m     v1.15.0
kubm-02      Ready    master   13m     v1.15.0
kubm-03      Ready    master   7m36s   v1.15.0
kubnode-01   Ready    <none>   71s     v1.15.0

再次查看系統服務

[root@kubm-01 ~]# kubectl get pods -n kube-system
NAME                              READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-bdjh7          1/1     Running   0          59m
coredns-5c98db65d4-xvvpl          1/1     Running   0          59m
etcd-kubm-01                      1/1     Running   0          58m
etcd-kubm-02                      1/1     Running   0          52m
etcd-kubm-03                      1/1     Running   0          45m
kube-apiserver-kubm-01            1/1     Running   0          58m
kube-apiserver-kubm-02            1/1     Running   0          52m
kube-apiserver-kubm-03            1/1     Running   0          46m
kube-controller-manager-kubm-01   1/1     Running   0          58m
kube-controller-manager-kubm-02   1/1     Running   0          52m
kube-controller-manager-kubm-03   1/1     Running   0          46m
kube-flannel-ds-amd64-6v5bw       1/1     Running   0          29m
kube-flannel-ds-amd64-chb4p       1/1     Running   0          57m
kube-flannel-ds-amd64-h74k9       1/1     Running   1          53m
kube-flannel-ds-amd64-nbx85       1/1     Running   2          47m
kube-flannel-ds-amd64-s4gv6       1/1     Running   0          40m
kube-flannel-ds-amd64-wv9gx       1/1     Running   0          30m
kube-proxy-hdkb9                  1/1     Running   0          47m
kube-proxy-kww9g                  1/1     Running   0          53m
kube-proxy-l29hl                  1/1     Running   0          59m
kube-proxy-ln5rj                  1/1     Running   0          29m
kube-proxy-rw22h                  1/1     Running   0          40m
kube-proxy-vbc6k                  1/1     Running   0          30m
kube-scheduler-kubm-01            1/1     Running   0          58m
kube-scheduler-kubm-02            1/1     Running   0          52m
kube-scheduler-kubm-03            1/1     Running   0          46m

按照添加添加node節點步驟繼續添加其它節點即可。

token處理命令

如果token超時或者自己想新建token，執行如下命令解決。

刪除token

[root@kubm-01 ~]# kubeadm token delete  yqds5b.gyiax5ntlzeiavrz
bootstrap token "yqds5b" deleted

新建 token 超時時間24h

[root@kubm-01 ~]# kubeadm token create --ttl 24h --print-join-command 

kubeadm join 172.20.101.252:16443 --token rgqz2k.q4529102hz5ctej5     --discovery-token-ca-cert-hash sha256:7b2c700b984b992fe96f50bb45ee782d68168a98197b79ecd87f7e68b089819f 

參考文檔：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
https://octetz.com/posts/ha-control-plane-k8s-kubeadm