在K8S中安装Jenkins并实现动态生成Jenkins Slave

发表于 更新于 阅读次数： 本文字数： 2.8k 阅读时长 ≈ 10 分钟

Kubernetes集群为Jenkins添加了一个新的自动化层。由Kubernetes来确保资源得到有效利用，并且服务器和基础架构不会过载。Kubernetes协调容器部署的能力可确保Jenkins始终拥有适量的可用资源。

先决条件

  • 一套可用的Kubernetes集群，测试环境可使用minikube
  • Kubernetes可提供持久化存储的能力，测试环境可使用NFS提供

在K8S中安装Jenkins

创建命名空间

为了方便对Jenkins进行管理，建议为Jenkins提供独立的命名空间。

创建namespace配置清单01-jenkins-namespace.yaml，内容如下：

1
2
3
4
5
---
apiVersion: v1
kind: Namespace
metadata:
  name: jenkins

应用配置

1
kubectl apply -f 01-jenkins-namespace.yaml

创建持久化存储卷

我们要为Jenkins提供一个持久化存储卷，用来保存Jenkins的配置和作业数据。

创建pvc配置清单02-jenkins-pvc.yaml，内容如下：

说明

  • storageClassName根据实际环境配置StorageClass名称
  • storage配置存储卷大小
1
2
3
4
5
6
7
8
9
10
11
12
13
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins-pvc
  namespace: jenkins
spec:
  storageClassName: managed-nfs-storage
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi

应用配置

1
kubectl apply -f 02-jenkins-pvc.yaml

创建服务账号

创建rbac配置清单03-jenkins-rbac.yaml，内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: jenkins

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-admin
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
roleRef:
  kind: ClusterRole
  name: jenkins-admin
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: jenkins

应用配置

1
kubectl apply -f 03-jenkins-rbac.yaml

创建Jenkins Deployment

创建deployment配置清单04-jenkins-deployment.yaml，内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins-admin
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts-jdk11
        imagePullPolicy: IfNotPresent
        env:
        - name: JAVA_OPTS
          value: -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 2
            memory: 4Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkins-home
          mountPath: /var/jenkins_home
        - name: timezone
          mountPath: /etc/localtime
          readOnly: true
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-pvc
      - name: timezone
        hostPath:
          path: /etc/localtime

应用配置

1
kubectl apply -f 04-jenkins-deployment.yaml

创建services

创建services配置清单05-jenkins-svc.yaml，内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: jenkins
spec:
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: 8080
    nodePort: 8080
  selector:
    app: jenkins

---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-jnlp
  namespace: jenkins
spec:
  selector:
    app: jenkins
  ports:
  - name: agent
    port: 50000
    targetPort: 50000

应用配置

1
kubectl apply -f 05-jenkins-svc.yaml

验证安装

执行如下命令查看Jenkins Pod的运行情况，确认STATUS为Running状态，READY为1/1

1
2
3
[root@k8s-master jenkins]# kubectl get pod -n jenkins
NAME                       READY   STATUS    RESTARTS   AGE
jenkins-57d95b78d5-7wzjk   1/1     Running   0          128s

执行如下命令查看Jenkins日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@k8s-master jenkins]# kubectl logs `kubectl get pod -n jenkins | grep -v NAME | awk '{print $1}'` -n jenkins
......
*************************************************************
*************************************************************
*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

2968a5310688484bb3ee2fd09fe6c5c1

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************

2021-05-07 16:15:23.177+0000 [id=29]	INFO	jenkins.InitReactorRunner$1#onAttained: Completed initialization
2021-05-07 16:15:23.273+0000 [id=22]	INFO	hudson.WebAppMain$3#run: Jenkins is fully up and running
2021-05-07 16:15:24.100+0000 [id=44]	INFO	h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
2021-05-07 16:15:24.101+0000 [id=44]	INFO	hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1
2021-05-07 16:15:24.105+0000 [id=44]	INFO	hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 16,700 ms

当看到“INFO: Jenkins is fully up and running”，就说明Jenkins已经运行好了，Jenkins的第一次启动需要一定时间，要耐心等待。

安装后设置向导

使用上述过程下载，安装和运行Jenkins之后，安装向导将启动。此设置向导将引导我们完成几个快速的“一次性”步骤，以解锁Jenkins，使用插件对其进行自定义，并创建第一个管理员用户，你可以通过该用户继续访问Jenkins。

解锁Jenkins

首次访问新的Jenkins实例时，系统会要求使用自动生成的密码将其解锁

  1. 浏览http://<node-ip>:8080（或安装时为Jenkins配置的nodePort端口）并等待，直到出现“解锁Jenkins”页面。

  1. 从Jenkins控制台日志输出中，复制自动生成的字母数字密码（在两组星号之间）。
1
kubectl logs `kubectl get pod -n jenkins | grep -v NAME | awk '{print $1}'` -n jenkins
  1. 在“解锁Jenkins”页面上，将此密码粘贴到“管理员密码”字段中，然后单击“继续”。

使用插件自定义Jenkins

解锁Jenkins后，出现“自定义Jenkins”页面，在这里，可以安装任何数量的插件，作为初始化配置的一部分。如果不确定所需的插件，直接选择“安装推荐的插件”，之后可以通过Jenkins中的“Manage Jenkins”>“插件管理”页面安装（或删除）其他Jenkins插件。

设置向导将显示正在配置的Jenkins的进度以及正在安装的所选Jenkins插件集。此过程可能需要几分钟。

创建第一个管理员用户

在使用插件自定义Jenkins之后，Jenkins要求你创建第一个管理员用户，在相应字段中指定管理员用户的详细信息，然后单击“保存并完成”

实例配置

创建管理员用户之后，最后会出现实例配置页面，确认“Jenkins URL”中的地址为Jenkins访问地址后单击“保存并完成”

出现“Jenkins已就绪”页面时，单击“开始使用Jenkins”进入Jenkins

HelloWorld Pipline

为了验证Jenkins是否可以正常使用pipline流水线，可以新建一个Item（流水线类型），输入如下Pipline script内容，保存后单击“立即构建”，若可以正常构建则表示Jenkins功能正常

1
2
3
4
5
6
7
8
9
10
11
pipeline {
    agent any

    stages {
        stage('Hello') {
            steps {
                echo 'Hello World'
            }
        }
    }
}

安装Kubernetes插件

用管理员用户登录Jenkins主页面后，找到“Manage Jenkins”>“插件管理”>“可选插件”，搜索“kubernetes”，勾选“Kubernetes”，单击“Download now and install after restart”，等待安装完成并重启Jenkins。

配置Kubernetes插件

用管理员用户登录Jenkins Master主页面后，找到“系统管理”>“节点管理”>“Configure Clouds”，单击“Add a new cloud”，选择“Kubernetes”

参考如下说明配置Kubernetes集群连接信息，然后单击“save”保存

说明

  • Name：集群名称，可自定义，也可使用默认的kubernetes
  • Kubernetes地址：因为Jenkins部署在Kubernetes中，所以此处配置为https://kubernetes.default即可，配置完可进行连接测试

说明

  • Jenkins地址：因为Jenkins部署在Kubernetes中，所以此处配置为http://jenkins.jenkins:8080即可，协议为HTTP协议
  • Jenkins通道：因为Jenkins部署在Kubernetes中，所以此处配置为jenkins-jnlp.jenkins:50000即可，协议为TCP协议，不需加http://

测试Jenkins

现在Jenkins已经全部安装好了，下面进行测试。在Jenkins主页面单击“新建任务”，输入项目名，然后选择“流水线”，单击“确定”

进入项目配置页面，在最下方输入如下Pipline script，最后单击“保存”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def POD_LABEL = "testpod-${UUID.randomUUID().toString()}"
podTemplate(label: POD_LABEL, cloud: 'kubernetes', containers: [
    containerTemplate(name: 'build', image: 'jfeng45/k8sdemo-backend:1.0', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'run', image: 'jfeng45/k8sdemo-backend:1.0', ttyEnabled: true, command: 'cat')
  ]) {

    node(POD_LABEL) {
        stage('build a go project') {
            container('build') {
                stage('Build a go project') {
                    sh 'echo hello'
                }
            }
        }

        stage('Run a Golang project') {
            container('run') {
                stage('Run a Go project') {
                    sh '/root/main.exe'
                }
            }
        }

    }
}

说明

  • “POD_LABEL”取任何名字都可以（在Kubernetes-plugin 1.17.0 版本之后，系统会自动命名，但以前需要自己取名）
  • “cloud: 'kubernetes'”要与前面定义的“Kubernetes Plugin”相匹配。它有两个stage，一个是“build”，另一个是“run”。
  • 在“podTemplate”里定义了每一个stage的镜像（这样后面的stage脚本里就可以引用），这里为了简便把两个镜像设成是一样的。因为是测试，第一个stage只是输出“echo > - hello”， 第二个运行镜像“jfeng45/k8sdemo-backend:1.0”里的main.exe程序。

在Jenkins主页面选择刚创建的任务，单击“立即构建”运行项目，再到“Console Output”中查看结果日志输出，若最后有“Finished: SUCCESS”字样则表示构建成功，测试阶段完成了。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
Started by user admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: kubernetes jenkins/testpod-651cfd26-9a81-4093-9f6e-f65526728fb8-728b1-5bnkn
Still waiting to schedule task
‘testpod-651cfd26-9a81-4093-9f6e-f65526728fb8-728b1-5bnkn’ is offline
Created Pod: kubernetes jenkins/testpod-651cfd26-9a81-4093-9f6e-f65526728fb8-728b1-twbzj
Agent testpod-651cfd26-9a81-4093-9f6e-f65526728fb8-728b1-twbzj is provisioned from template testpod-651cfd26-9a81-4093-9f6e-f65526728fb8-728b1
---
apiVersion: "v1"
kind: "Pod"
metadata:
  annotations:
    buildUrl: "http://jenkins.jenkins.svc.cluster.local:8080/job/test/1/"
    runUrl: "job/test/1/"
  labels:
    jenkins: "slave"
    jenkins/label-digest: "11cb2ea9ecb16723ac3ff1a6dbd27b41dee860c6"
    jenkins/label: "testpod-651cfd26-9a81-4093-9f6e-f65526728fb8"
  name: "testpod-651cfd26-9a81-4093-9f6e-f65526728fb8-728b1-twbzj"
spec:
  containers:
  - command:
    - "cat"
    image: "jfeng45/k8sdemo-backend:1.0"
    imagePullPolicy: "IfNotPresent"
    name: "build"
    resources:
      limits: {}
      requests: {}
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  - command:
    - "cat"
    image: "jfeng45/k8sdemo-backend:1.0"
    imagePullPolicy: "IfNotPresent"
    name: "run"
    resources:
      limits: {}
      requests: {}
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins-jnlp.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "testpod-651cfd26-9a81-4093-9f6e-f65526728fb8-728b1-twbzj"
    - name: "JENKINS_NAME"
      value: "testpod-651cfd26-9a81-4093-9f6e-f65526728fb8-728b1-twbzj"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins/agent"
    - name: "JENKINS_URL"
      value: "http://jenkins.jenkins.svc.cluster.local:8080/"
    image: "jenkins/inbound-agent:4.3-4"
    name: "jnlp"
    resources:
      limits: {}
      requests:
        memory: "256Mi"
        cpu: "100m"
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  nodeSelector:
    kubernetes.io/os: "linux"
  restartPolicy: "Never"
  volumes:
  - emptyDir:
      medium: ""
    name: "workspace-volume"

Running on testpod-651cfd26-9a81-4093-9f6e-f65526728fb8-728b1-twbzj in /home/jenkins/agent/workspace/test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build a go project)
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build a go project)
[Pipeline] sh
+ echo hello
hello
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Run a Golang project)
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run a Go project)
[Pipeline] sh
+ /root/main.exe
time="2021-05-07T20:24:45Z" level=debug msg="connect to database "
time="2021-05-07T20:24:45Z" level=debug msg="dataSourceName::@tcp(:)/?charset=utf8"
time="2021-05-07T20:24:45Z" level=debug msg="FindAll()"
time="2021-05-07T20:24:45Z" level=debug msg="user registere failed:dial tcp :0: connect: connection refused"
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

参考文档