kubernetes部署有狀態應用之zookeeper集群

部署集群過程中遇到的坑：

1 ?node(s) had taints that the pod didn't tolerate.

集群中有個節點被污染了，查看節點信息，發現其中一個節點沒有起來，導致只能啟動2個pod

2 PV和PVC上的坑

• 一個zk.yaml中需要創建3個zookeeper，根據zookeeper集群的特性，三個zookeeper需要三個不同的數據目錄（如果再添加日志目錄也是一樣），集群才會正常起來

• 創建PV的時候需要創建三個不同的PV

• 在一個zk.yaml中，PVC需要關聯到后端不同的三個PVC，如果按照傳統的創建PVC，并在zk.yaml中指定PVC的名字，只能指定一個名字，這樣是辦不到關聯到三個不同的PV，于是需要volumeClaimTemplates（PVC模板），模板accessModes需要ReadWriteOnce,ReadWriteMany會關聯失敗。

??volumeClaimTemplates:
??-?metadata:
??????name:?datadir
????spec:
??????accessModes:?[?"ReadWriteOnce"?]
??????resources:
????????requests:
??????????storage:?1Gi

• 如果是用動態pvc供給，也是一樣需要創建PVC模板

3 zookeeper提供給外部訪問

zookeeper由于是使用statefulSet部署，使用到了無頭服務，如果提供給外部訪問呢？辦法就是增加一個專門為外部訪問的service。

具體部署

1 在192.168.0.11上創建安裝nfs，并配置啟動nfs

2 在每個node節點上下載nfs

3 在matser節點創建zk使用的PV

apiVersion:?v1
kind:?PersistentVolume
metadata:
??name:?zk-data1
spec:
??capacity:
????storage:?1Gi
??accessModes:
????-?ReadWriteOnce
??nfs:
????server:?192.168.0.11
????path:?/data/zk/data1
---
apiVersion:?v1
kind:?PersistentVolume
metadata:
??name:?zk-data2
spec:
??capacity:
????storage:?1Gi
??accessModes:
????-?ReadWriteOnce
??nfs:
????server:?192.168.0.11
????path:?/data/zk/data2
---
apiVersion:?v1
kind:?PersistentVolume
metadata:
??name:?zk-data3
spec:
??capacity:
????storage:?1Gi
??accessModes:
????-?ReadWriteOnce
??nfs:
????server:?192.168.0.11
????path:?/data/zk/data3

4 在master節點上創建zookeeper的創建pod的yaml文件

參考官方文檔：https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/

官方文檔中的鏡像地址為國外地址，我這里網上找了一個可用的地址，跟官方鏡像一樣

[root@localhost?zk]#?cat?zk.yaml?
apiVersion:?v1
kind:?Service
metadata:
??name:?zk-hs
??labels:
????app:?zk
spec:
??ports:
??-?port:?2888
????name:?server
??-?port:?3888
????name:?leader-election
??clusterIP:?None
??selector:
????app:?zk
---
apiVersion:?v1
kind:?Service
metadata:
??name:?zk-cs
??labels:
????app:?zk
spec:
??type:?NodePort
??ports:
??-?port:?2181
????targetPort:?2181
????name:?client
????nodePort:?2181
??selector:
????app:?zk
---
apiVersion:?policy/v1beta1
kind:?PodDisruptionBudget
metadata:
??name:?zk-pdb
spec:
??selector:
????matchLabels:
??????app:?zk
??maxUnavailable:?1
---
apiVersion:?apps/v1
kind:?StatefulSet
metadata:
??name:?zok
spec:
??serviceName:?zk-hs
??replicas:?3
??selector:
????matchLabels:
??????app:?zk
??template:
????metadata:
??????labels:
????????app:?zk
????spec:
??????affinity:
????????podAntiAffinity:
??????????requiredDuringSchedulingIgnoredDuringExecution:
????????????-?labelSelector:
????????????????matchExpressions:
??????????????????-?key:?"app"
????????????????????operator:?In
????????????????????values:
????????????????????-?zk
??????????????topologyKey:?"kubernetes.io/hostname"
??????containers:
??????-?name:?kubernetes-zookeeper
????????imagePullPolicy:?Always
????????image:?leolee32/kubernetes-library:kubernetes-zookeeper1.0-3.4.10
????????resources:
??????????requests:
????????????memory:?"1Gi"
????????????cpu:?"0.5"
????????ports:
????????-?containerPort:?2181
??????????name:?client
????????-?containerPort:?2888
??????????name:?server
????????-?containerPort:?3888
??????????name:?leader-election
????????command:
????????-?sh
????????-?-c
????????-?"start-zookeeper?\
??????????--servers=3?\
??????????--data_dir=/var/lib/zookeeper/data?\
??????????--data_log_dir=/var/lib/zookeeper/data/log?\
??????????--conf_dir=/opt/zookeeper/conf?\
??????????--client_port=2181?\
??????????--election_port=3888?\
??????????--server_port=2888?\
??????????--tick_time=2000?\
??????????--init_limit=10?\
??????????--sync_limit=5?\
??????????--heap=512M?\
??????????--max_client_cnxns=60?\
??????????--snap_retain_count=3?\
??????????--purge_interval=12?\
??????????--max_session_timeout=40000?\
??????????--min_session_timeout=4000?\
??????????--log_level=INFO"
????????readinessProbe:
??????????exec:
????????????command:
????????????-?sh
????????????-?-c
????????????-?"zookeeper-ready?2181"
??????????initialDelaySeconds:?10
??????????timeoutSeconds:?5
????????livenessProbe:
??????????exec:
????????????command:
????????????-?sh
????????????-?-c
????????????-?"zookeeper-ready?2181"
??????????initialDelaySeconds:?10
??????????timeoutSeconds:?5
????????volumeMounts:
????????-?name:?datadir
??????????mountPath:?/var/lib/zookeeper
??volumeClaimTemplates:
??-?metadata:
??????name:?datadir
????spec:
??????accessModes:?[?"ReadWriteOnce"?]
??????resources:
????????requests:
??????????storage:?1Gi

5 查看集群狀態

for i in 0 1 2; do kubectl exec zok-$i zkServer.sh status; done

6 外部訪問zookeeper集群（使用本機電腦訪問）