在k8s中安裝jenkins并實現動態生成 jenkins slave

安裝jenkins
1、創建一個命名空間
$ kubectl create namespace kube-ops
2、為jenkins創建pvc（也可以使用存儲類創建）

apiVersion: v1
kind: PersistentVolume
metadata:
  name: opspv
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 192.168.1.244
    path: /data/k8s
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: opspvc
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

3、創建jenkins需要的rbac權限

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins2
  namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins2
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins2
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins2
subjects:
  - kind: ServiceAccount
    name: jenkins2
    namespace: kube-ops

也可以為ServiceAccoun綁定一個系統現有的 cluster-admin 集群角色權限
4、創建jenkins pod
$ docker pull docker.io/jenkins/jenkins:lts
$ docker pull cnych/jenkins:jnlp6

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins2
  namespace: kube-ops
spec:
  template:
    metadata:
      labels:
        app: jenkins2
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins2
      containers:
      - name: jenkins
        image: docker.io/jenkins/jenkins:lts
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkinshome
          subPath: jenkins2
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvision
er.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai      
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: opspvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins2
  namespace: kube-ops
  labels:
    app: jenkins2
spec:
  selector:
    app: jenkins2
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 30003
  - name: agent
    port: 50000
    targetPort: agent

$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 0/1 Running 0 #不能正常啟動
$ kubectl describe pod jenkins2-76644dbc9b-llcsp -n kube-ops
$ kubectl logs -f jenkins2-76644dbc9b-llcsp -n kube-ops
5、在nfs服務器上修改jenkins持久目錄的權限并重新創建jenkins pod
$ chown -R 1000 /data/k8s/jenkins2 #在192.168.1.244上
$ kubectl delete -f jenkins2.yaml
$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0
為什么是1000？
上述鏡像的Dockerfile文件中定義的是：user=jenkins group=jenkins uid=1000 gid=1000
Dockerfile文件的地址：
https://github.com/jenkinsci/docker/blob/master/Dockerfile
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000
ARG http_port=8080
ARG agent_port=50000
ARG JENKINS_HOME=/var/jenkins_home
$ kubectl get svc -n kube-ops
jenkins2 NodePort 10.105.121.176 <none> 8080:30003/TCP,50000:30936/TCP
http://192.168.1.243:30003
初始密碼在nfs服務器上
$ cat /data/k8s/jenkins2/secrets/initialAdminPassword

在jenkins上創建kubernetes云
enkins Master 和 Jenkins Slave 以 Pod 形式運行在 Kubernetes 集群的 Node 上，Master 運行在其中一個節點，并且將其配置數據存儲到一個 Volume 上去，Slave 運行在各個節點上，并且它不是一直處于運行狀態，它會按照需求動態的創建并自動刪除
這種方式的工作流程大致為：當 Jenkins Master 接受到 Build 請求時，會根據配置的 Label 動態創建一個運行在 Pod 中的 Jenkins Slave 并注冊到 Master 上，當運行完 Job 后，這個 Slave 會被注銷并且這個 Pod 也會自動刪除，恢復到最初狀態。
1、安裝插件
安裝kubernetes plugin， 點擊 Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin
2、增加kubernetes云
點擊 Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 選擇 Kubernetes，然后填寫 Kubernetes 和 Jenkins 配置信息----連接測試
name：kubernetes
Kubernetes 地址：https://kubernetes.default.svc.cluster.local
Kubernetes 命名空間：kube-ops
Jenkins 地址：http://jenkins2.kube-ops.svc.cluster.local:8080（jenkins2是svc）
3、添加pod模板
添加pod模板----Kubernetes Pod Template
名稱：jnlp
命名空間：kube-ops
標簽列表：dongyali-jnlp
4、添加容器模板
添加容器----Container Template
名稱：jnlp
Docker 鏡像：cnych/jenkins:jnlp6（Jenkins 版本在 2.176.x以下的鏡像名字去掉6）
工作目錄：/home/jenkins/agent
運行的命令：清空
命令參數：清空
5、添加兩個卷
添加卷----Host Path Volume
主機路徑：/var/run/docker.sock
掛載路徑：/var/run/docker.sock
主機路徑：/root/.kube
掛載路徑：/root/.kube
6、可能需要配置ServiceAccount
$ kubectl get sa -n kube-ops
jenkins2 1 14h
點擊添加卷下面的高級----Service Account----jenkins2
7、用shell測試 Kubernetes 動態生成 jenkins slave
新建任務----名字----自由風格
通用----勾選限制項目的運行節點----標簽表達式：dongyali-jnlp
構建----執行shell----輸入如下內容----保存立即構建

echo "測試 Kubernetes 動態生成 jenkins slave"
echo "==============docker in docker==========="
docker info
echo "=============kubectl============="
kubectl get pods

觀察 Kubernetes 集群中 Pod 的變化：
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0 3h59m
jnlp-tl1km 1/1 Running 0 44s
當任務運行完畢，jnlp這個slave pod就會自動消失。
8、用pipeline測試 Kubernetes 動態生成 jenkins slave
新建任務----名字----流水線
在流水線腳本中輸入如下內容----保存立即構建

node('dongyali-jnlp') {
    stage('Clone') {
      echo "1.Clone Stage"
    }
    stage('Test') {
      echo "2.Test Stage"
    }
    stage('Build') {
      echo "3.Build Stage"
    }
    stage('Deploy') {
      echo "4. Deploy Stage"
    }
}