kubeadm部署Kubernetes集群更新证书

默认情况下,kubeadm会生成一个集群所需的全部证书。而由于kubeadm生成的CA根证书有效期为10年,客户端证书有效期只有1年,因此在一年之内就需要对所有的客户端证书进行续签。所有证书都存放在/etc/kubernetes/pki目录下。

查看证书过期时间


方式一(非推荐方式)

在Master节点执行如下命令查看各个证书过期时间

1
2
3
4
5
6
for tls in `find /etc/kubernetes/pki -maxdepth 2 -name "*.crt" | sort`
do
echo ==== $tls ====
openssl x509 -in $tls -noout -dates
echo
done

命令输出如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
==== /etc/kubernetes/pki/apiserver.crt ====
notBefore=Aug 6 19:03:07 2021 GMT
notAfter=Aug 6 19:03:08 2022 GMT

==== /etc/kubernetes/pki/apiserver-etcd-client.crt ====
notBefore=Aug 6 19:03:08 2021 GMT
notAfter=Aug 6 19:03:09 2022 GMT

==== /etc/kubernetes/pki/apiserver-kubelet-client.crt ====
notBefore=Aug 6 19:03:07 2021 GMT
notAfter=Aug 6 19:03:08 2022 GMT

==== /etc/kubernetes/pki/ca.crt ====
notBefore=Aug 6 19:03:07 2021 GMT
notAfter=Aug 4 19:03:07 2031 GMT

==== /etc/kubernetes/pki/etcd/ca.crt ====
notBefore=Aug 6 19:03:08 2021 GMT
notAfter=Aug 4 19:03:08 2031 GMT

==== /etc/kubernetes/pki/etcd/healthcheck-client.crt ====
notBefore=Aug 6 19:03:08 2021 GMT
notAfter=Aug 6 19:03:09 2022 GMT

==== /etc/kubernetes/pki/etcd/peer.crt ====
notBefore=Aug 6 19:03:08 2021 GMT
notAfter=Aug 6 19:03:08 2022 GMT

==== /etc/kubernetes/pki/etcd/server.crt ====
notBefore=Aug 6 19:03:08 2021 GMT
notAfter=Aug 6 19:03:08 2022 GMT

==== /etc/kubernetes/pki/front-proxy-ca.crt ====
notBefore=Aug 6 19:03:08 2021 GMT
notAfter=Aug 4 19:03:08 2031 GMT

==== /etc/kubernetes/pki/front-proxy-client.crt ====
notBefore=Aug 6 19:03:08 2021 GMT
notAfter=Aug 6 19:03:08 2022 GMT

方式二(推荐方式)

使用check-expiration子命令来检查证书何时过期

1
kubeadm certs check-expiration

输出类似以下内容:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 06, 2022 19:03 UTC 344d no
apiserver Aug 06, 2022 19:03 UTC 344d ca no
apiserver-etcd-client Aug 06, 2022 19:03 UTC 344d etcd-ca no
apiserver-kubelet-client Aug 06, 2022 19:03 UTC 344d ca no
controller-manager.conf Aug 06, 2022 19:03 UTC 344d no
etcd-healthcheck-client Aug 06, 2022 19:03 UTC 344d etcd-ca no
etcd-peer Aug 06, 2022 19:03 UTC 344d etcd-ca no
etcd-server Aug 06, 2022 19:03 UTC 344d etcd-ca no
front-proxy-client Aug 06, 2022 19:03 UTC 344d front-proxy-ca no
scheduler.conf Aug 06, 2022 19:03 UTC 344d no

CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 04, 2031 19:03 UTC 9y no
etcd-ca Aug 04, 2031 19:03 UTC 9y no
front-proxy-ca Aug 04, 2031 19:03 UTC 9y no

该命令显示/etc/kubernetes/pki文件夹中的客户端证书以及kubeadm(admin.conf,controller-manager.confscheduler.conf)使用的KUBECONFIG文件中嵌入的客户端证书的到期时间/剩余时间。

另外,kubeadm会显示用户证书是否由外部管理(EXTERNALLY MANAGED)。在这种情况下,用户应该小心的手动/使用其他工具来管理证书更新。

说明
上面的列表中没有包含kubelet.conf,因为kubeadm将kubelet配置为自动更新证书。轮换的证书位于目录/var/lib/kubelet/pki。要修复过期的kubelet客户端证书,请参阅kubelet客户端证书轮换失败

警告
在通过kubeadm init创建的节点上,在kubeadm 1.17版本之前有一个缺陷,该缺陷使得必须手动修改/etc/kubernetes/kubelet.conf文件的内容。kubeadm init操作结束之后,必须更新/etc/kubernetes/kubelet.conf文件将client-certificate-dataclient-key-data改为如下所示的内容以便使用轮换后的kubelet客户端证书

1
2
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

证书类别


集群根证书

1
2
3
# ll /etc/kubernetes/pki/ca*
-rw-r--r-- 1 root root 1066 Aug 7 03:03 /etc/kubernetes/pki/ca.crt
-rw------- 1 root root 1675 Aug 7 03:03 /etc/kubernetes/pki/ca.key

由此集群根证书签发的证书有:

  1. kube-apiserver组件持有的服务端证书
1
2
3
# ll /etc/kubernetes/pki/apiserver.*
-rw-r--r-- 1 root root 1277 Aug 7 03:03 /etc/kubernetes/pki/apiserver.crt
-rw------- 1 root root 1675 Aug 7 03:03 /etc/kubernetes/pki/apiserver.key
  1. kubelet组件持有的客户端证书
1
2
3
# ll /etc/kubernetes/pki/apiserver-kubelet-client.*
-rw-r--r-- 1 root root 1143 Aug 7 03:03 /etc/kubernetes/pki/apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Aug 7 03:03 /etc/kubernetes/pki/apiserver-kubelet-client.key

说明
kubelet的/var/lib/kubelet/config.yaml配置文件中一般不会明确指定服务端证书,而是只指定ca根证书, 让kubelet根据本地主机信息自动生成服务端证书并保存到配置的cert-dir文件夹中。

汇聚层根证书

1
2
3
# ll /etc/kubernetes/pki/front-proxy-ca.*
-rw-r--r-- 1 root root 1078 Aug 7 03:03 /etc/kubernetes/pki/front-proxy-ca.crt
-rw------- 1 root root 1675 Aug 7 03:03 /etc/kubernetes/pki/front-proxy-ca.key

由此汇聚层根证书签发的证书有:

  1. 代理端使用的客户端证书, 用作代用户与kube-apiserver认证
1
2
3
# ll /etc/kubernetes/pki/front-proxy-client.*
-rw-r--r-- 1 root root 1103 Aug 7 03:03 /etc/kubernetes/pki/front-proxy-client.crt
-rw------- 1 root root 1675 Aug 7 03:03 /etc/kubernetes/pki/front-proxy-client.key

etcd集群根证书

1
2
3
# ll /etc/kubernetes/pki/etcd/ca.*
-rw-r--r-- 1 root root 1058 Aug 7 03:03 /etc/kubernetes/pki/etcd/ca.crt
-rw------- 1 root root 1679 Aug 7 03:03 /etc/kubernetes/pki/etcd/ca.key

由此etcd根证书签发的证书有:

  1. etcd server服务端证书
1
2
3
# ll /etc/kubernetes/pki/etcd/server.*
-rw-r--r-- 1 root root 1188 Aug 7 03:03 /etc/kubernetes/pki/etcd/server.crt
-rw------- 1 root root 1679 Aug 7 03:03 /etc/kubernetes/pki/etcd/server.key
  1. etcd集群中peer节点互相通信使用的客户端证书
1
2
3
# ll /etc/kubernetes/pki/etcd/peer.*
-rw-r--r-- 1 root root 1188 Aug 7 03:03 /etc/kubernetes/pki/etcd/peer.crt
-rw------- 1 root root 1675 Aug 7 03:03 /etc/kubernetes/pki/etcd/peer.key
  1. Pod中定义Liveness探针使用的客户端证书
1
2
3
# ll /etc/kubernetes/pki/etcd/healthcheck-client.*
-rw-r--r-- 1 root root 1139 Aug 7 03:03 /etc/kubernetes/pki/etcd/healthcheck-client.crt
-rw------- 1 root root 1675 Aug 7 03:03 /etc/kubernetes/pki/etcd/healthcheck-client.key
  1. 配置在kube-apiserver中用来与etcd server做双向认证的客户端证书
1
2
3
# ll /etc/kubernetes/pki/apiserver-etcd-client.*
-rw-r--r-- 1 root root 1135 Aug 7 03:03 /etc/kubernetes/pki/apiserver-etcd-client.crt
-rw------- 1 root root 1679 Aug 7 03:03 /etc/kubernetes/pki/apiserver-etcd-client.key

Serveice Account密钥

1
2
3
# ll /etc/kubernetes/pki/sa.*
-rw------- 1 root root 1675 Aug 7 03:03 /etc/kubernetes/pki/sa.key
-rw------- 1 root root 451 Aug 7 03:03 /etc/kubernetes/pki/sa.pub

说明
Serveice Account密钥对仅提供给kube-controller-manager使用。kube-controller-manager通过sa.key对token进行签名, Master节点通过公钥sa.pub进行签名的验证。

延伸——API Server身份验证过程

API Server的Authentication环节支持多种身份校验方式:Client Cert、Bearer Token、Static Password Auth等,这些方式中只要有一种方式通过Authentication(Kubernetes API Server会逐个方式尝试),那么身份校验就会通过。

一旦API Server发现Client发起的Request使用的是Service Account Token的方式,API Server就会自动采用Signed Bearer Token方式进行身份校验。而Request就会使用携带的Service Account Token参与验证。该Token是API Server在创建Service Account时用kube-controller-manager启动参数--service-account-private-key-file指定的私钥签署(sign)的,同时必须指定kube-apiserver参数--service-account-key-file(如果没指定的话,会使用--tls-private-key-file替代)为该私钥对应的公钥,用来在认证阶段验证Token,也就是说该证书对通过CN和O指定了ServiceAccount的授权权限。

通过Authentication后,API Server将根据Pod所属ServiceAccount的用户名(以system:serviceaccount:为前缀)和组(以system:serviceaccounts:前缀)的权限对其进行Authorization和Admission Control两个环节的处理。

不管是自动生成的Token还是手动创建的Token的值都是一样的,因为进行签署Token的-–service-account-key-file是同一个。

ServiceAccount中的Token是API server私钥签署的,Pod在对API Server发起请求的时候会带上该Token,以确保能够通过API Server的认证。对ServiceAccount的授权通过对ServiceAccount对应的用户或组进行RBAC控制即可

更新证书方式一


说明
此方式采用kubeadm默认延期1年时间的策略,若要自定义更长时间,如100年,参考方式二

备份集群文件

说明
如果你运行了一个HA集群,以下命令需要在所有Master节点上执行。

1
2
cp -a /etc/kubernetes/ /etc/kubernetes-`date +%Y%m%d`
kubectl get cm kubeadm-config -n kube-system -o yaml > /root/kubeadm-config.yaml

更新证书

查看证书更新命令帮助

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# kubeadm certs renew --help 
This command is not meant to be run on its own. See list of available subcommands.

Usage:
kubeadm certs renew [flags]
kubeadm certs renew [command]

Available Commands:
admin.conf Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
all Renew all available certificates
apiserver Renew the certificate for serving the Kubernetes API
apiserver-etcd-client Renew the certificate the apiserver uses to access etcd
apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet
controller-manager.conf Renew the certificate embedded in the kubeconfig file for the controller manager to use
etcd-healthcheck-client Renew the certificate for liveness probes to healthcheck etcd
etcd-peer Renew the certificate for etcd nodes to communicate with each other
etcd-server Renew the certificate for serving etcd
front-proxy-client Renew the certificate for the front proxy client
scheduler.conf Renew the certificate embedded in the kubeconfig file for the scheduler manager to use

Flags:
-h, --help help for renew

Global Flags:
--add-dir-header If true, adds the file directory to the header of the log messages
--log-file string If non-empty, use this log file
--log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--one-output If true, only write logs to their native severity level (vs also writing to each lower severity level
--rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem.
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files
-v, --v Level number for the log level verbosity

Use "kubeadm certs renew [command] --help" for more information about a command.

说明
由HELP可知,证书更新可针对单个证书更新。

执行如下命令更新所有证书

说明
如果你运行了一个HA集群,这个命令需要在所有Master节点上执行。

1
kubeadm certs renew all

出现类似以下输出说明证书更新完成,并且最后一行Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.提示要求要重启kube-apiserverkube-controller-managerkube-scheduleretcd使其使用新证书。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

确认证书是否续期成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 29, 2022 04:20 UTC 364d no
apiserver Aug 29, 2022 04:20 UTC 364d ca no
apiserver-etcd-client Aug 29, 2022 04:20 UTC 364d etcd-ca no
apiserver-kubelet-client Aug 29, 2022 04:20 UTC 364d ca no
controller-manager.conf Aug 29, 2022 04:20 UTC 364d no
etcd-healthcheck-client Aug 29, 2022 04:20 UTC 364d etcd-ca no
etcd-peer Aug 29, 2022 04:20 UTC 364d etcd-ca no
etcd-server Aug 29, 2022 04:20 UTC 364d etcd-ca no
front-proxy-client Aug 29, 2022 04:20 UTC 364d front-proxy-ca no
scheduler.conf Aug 29, 2022 04:20 UTC 364d no

CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 04, 2031 19:03 UTC 9y no
etcd-ca Aug 04, 2031 19:03 UTC 9y no
front-proxy-ca Aug 04, 2031 19:03 UTC 9y no

说明
从命令输出可以看到,所有客户端证书的到期时间均发生了变化,不过不是顺延一年, 而是从执行renew成功的时间开始续签一年。

启用证书

依次在所有Master节点上执行如下命令重启kube-apiserverkube-controller-managerkube-scheduleretcd使其启用新证书。

警告
执行完一台后请务必通过kubectl get pod -n kube-system确认对应Master节点上的kube-apiserverkube-controller-managerkube-scheduleretcd服务Pod均处于Running状态并就绪后再开始操作下一台

1
mv /etc/kubernetes/manifests/ /etc/kubernetes/manifests.bak && sleep 30 && mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests/

替换config文件

执行如下命令替换config文件

说明
如果你运行了一个HA集群,这个命令需要在所有Master节点上执行。

1
cp -a /etc/kubernetes/admin.conf /root/.kube/config

更新证书方式二


说明
此方式采用在Kubernetes源码中自定义证书时间(如100年)然后重新编译kubeadm的方法。

查看当前版本

在任意一台Master节点执行以下命令查看当前版本,GitVersion对应Kubernetes版本,GoVersion对应使用的Golang版本

1
2
# kubeadm version 
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:57:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

获取源码

获取源码,以v1.20.0为例

1
2
3
yum install git wget -y
cd /root/
git clone --branch v1.20.0 --single-branch --depth 1 https://github.com/kubernetes/kubernetes

查看kube-cross的TAG版本号,Kubernetes使用的Golang版本与之对应,省略最后的-1

Kubernetes v1.20.0使用的Golang版本为1.15.5。各个Kubernetes版本使用的Golang版本是不一样的,编译时安装的Golang版本要严格按照文件中的指定版本

1
2
# cat /root/kubernetes/build/build-image/cross/VERSION
v1.15.5-1

修改CA证书时间

编辑cert.go文件

1
vim /root/kubernetes/staging/src/k8s.io/client-go/util/cert/cert.go

修改CA证书时间为100年

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
...
// NewSelfSignedCACert creates a CA certificate
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
now := time.Now()
tmpl := x509.Certificate{
SerialNumber: new(big.Int).SetInt64(0),
Subject: pkix.Name{
CommonName: cfg.CommonName,
Organization: cfg.Organization,
},
NotBefore: now.UTC(),
NotAfter: now.Add(duration365d * 100).UTC(), //修改duration365d * 10为duration365d * 100
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
BasicConstraintsValid: true,
IsCA: true,
}

certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key)
if err != nil {
return nil, err
}
return x509.ParseCertificate(certDERBytes)
}
...

修改其他证书时间

编辑constants.go文件

1
vim /root/kubernetes/cmd/kubeadm/app/constants/constants.go

修改CertificateValidity的值为100年

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
...
const (
// KubernetesDir is the directory Kubernetes owns for storing various configuration files
KubernetesDir = "/etc/kubernetes"
// ManifestsSubDirName defines directory name to store manifests
ManifestsSubDirName = "manifests"
// TempDirForKubeadm defines temporary directory for kubeadm
// should be joined with KubernetesDir.
TempDirForKubeadm = "tmp"

// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
// 修改time.Hour * 24 * 365为time.Hour * 24 * 365 * 100
CertificateValidity = time.Hour * 24 * 365 * 100

// CACertAndKeyBaseName defines certificate authority base name
CACertAndKeyBaseName = "ca"
...

安装Golang

https://golang.org/dl/下载对应版本的Golang

解压Golang

1
tar zxf go1.15.5.linux-amd64.tar.gz -C /usr/local

配置Golang环境变量

1
2
3
4
5
cat >> /etc/profile << EOF
export PATH=$PATH:/usr/local/go/bin
export GOPATH=/root/go
EOF
source /etc/profile

验证Golang是否安装成功

1
2
# go version
go version go1.15.5 linux/amd64

安装Docker(可选)

警告
如果你直接在集群所在节点上编译,原本已安装有Docker,千万不要执行此节操作!!!千万不要执行此节操作!!!千万不要执行此节操作!!!

清理原有Docker环境(如果有的话)

1
2
3
4
5
6
7
8
yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine

配置docker-ce repository

1
2
3
4
5
6
7
8
9
# 安装所需要的包,yum-utils提供了yum-config-manager工具,device-mapper-persistent-data和lvm2是设备映射存储驱动所需要的
yum install -y yum-utils \
device-mapper-persistent-data \
lvm2

# 设置稳定版的repo仓库
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo

说明
若无法访问国外网站,可配置国内阿里云的docker源**

1
>wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

配置好Docker仓库后,执行如下命令安装Docker v18.09.9

1
yum install docker-ce-18.09.9 docker-ce-cli-18.09.9 containerd.io -y

启动Docker并设置开机自启

1
2
3
systemctl start docker
systemctl enable docker
systemctl status docker

设置阿里云镜像加速器(可选)

1
2
3
4
5
6
7
8
9
10
mkdir -p /etc/docker
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://lerc8rqe.mirror.aliyuncs.com"]
}
EOF

systemctl daemon-reload
systemctl restart docker
docker info

编译前准备

执行下面命令去掉dirty,否则因为我们获取后修改了源码,编译出的version信息会带有-dirty字样,例如v1.20.0-dirty

1
2
cd /root/kubernetes
sed -ri 's#KUBE_GIT_TREE_STATE="dirty"#KUBE_GIT_TREE_STATE="clean"#g' hack/lib/version.sh

拉取kube-cross镜像,镜像标签与/root/kubernetes/build/build-image/cross/VERSION中保持一致

1
2
docker pull us.gcr.io/k8s-artifacts-prod/build-image/kube-cross:v1.15.5-1
docker tag us.gcr.io/k8s-artifacts-prod/build-image/kube-cross:v1.15.5-1 k8s.gcr.io/build-image/kube-cross:v1.15.5-1

如果因网络问题无法拉取,可拉取我推送到阿里云镜像仓库的镜像然后重新打TAG

1
2
docker pull registry.cn-hangzhou.aliyuncs.com/koenli/kube-cross:v1.15.5-1
docker tag registry.cn-hangzhou.aliyuncs.com/koenli/kube-cross:v1.15.5-1 k8s.gcr.io/build-image/kube-cross:v1.15.5-1

编译kubeadm

一切准备就绪,执行下面的命令开始编译。KUBE_BUILD_PLATFORMS指定目标平台;加上WHAT=cmd/kubeadm指定只编译kubeadm;GOFLAGS=-v开启verbose日志,GOGCFLAGS=”-N -l”禁止编译优化和内联,减小可执行程序大小。

1
2
3
cd /root/kubernetes
make clean
KUBE_BUILD_PLATFORMS=linux/amd64 KUBE_GIT_VERSION=v1.20.0 ./build/run.sh make all WHAT=cmd/kubeadm GOFLAGS=-v GOGCFLAGS="-N -l"

出现类似如下输出表示编译完成,生成的kubeadm二进制可执行程序发布在_output/dockerized/bin/linux/amd64/目录下

1
2
3
4
5
6
7
8
9
10
...
k8s.io/kubernetes/cmd/kubeadm/app/discovery/token
k8s.io/kubernetes/cmd/kubeadm/app/discovery
k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade
k8s.io/kubernetes/cmd/kubeadm/app/util/output
k8s.io/kubernetes/cmd/kubeadm/app/cmd
k8s.io/kubernetes/cmd/kubeadm/app
k8s.io/kubernetes/cmd/kubeadm
+++ [0829 11:41:45] Placing binaries
+++ [0829 11:41:45] Syncing out of container

替换kubeadm

将编译好的kubeadm二进制文件拷贝到所有节点的/root目录下并进行替换

1
2
mv /usr/bin/kubeadm  /usr/bin/kubeadm.bak
mv /root/kubeadm /usr/bin/kubeadm

验证kubeadm版本与原来是否保持一致,确认BuildDate为重新编译的时间

1
2
# kubeadm version 
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2021-08-29T03:54:29Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

备份集群文件

说明
如果你运行了一个HA集群,以下命令需要在所有Master节点上执行。

1
2
cp -a /etc/kubernetes/ /etc/kubernetes-`date +%Y%m%d`
kubectl get cm kubeadm-config -n kube-system -o yaml > /root/kubeadm-config.yaml

更新证书

执行如下命令更新所有证书

说明
如果你运行了一个HA集群,这个命令需要在所有Master节点上执行。

1
kubeadm certs renew all

出现类似以下输出说明证书更新完成,并且最后一行Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.提示要求要重启kube-apiserverkube-controller-managerkube-scheduleretcd使其使用新证书。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

确认证书是否续期成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 05, 2121 04:15 UTC 99y no
apiserver Aug 05, 2121 04:15 UTC 99y ca no
apiserver-etcd-client Aug 05, 2121 04:15 UTC 99y etcd-ca no
apiserver-kubelet-client Aug 05, 2121 04:15 UTC 99y ca no
controller-manager.conf Aug 05, 2121 04:15 UTC 99y no
etcd-healthcheck-client Aug 05, 2121 04:15 UTC 99y etcd-ca no
etcd-peer Aug 05, 2121 04:15 UTC 99y etcd-ca no
etcd-server Aug 05, 2121 04:15 UTC 99y etcd-ca no
front-proxy-client Aug 05, 2121 04:15 UTC 99y front-proxy-ca no
scheduler.conf Aug 05, 2121 04:15 UTC 99y no

CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 04, 2031 19:03 UTC 9y no
etcd-ca Aug 04, 2031 19:03 UTC 9y no
front-proxy-ca Aug 04, 2031 19:03 UTC 9y no

说明
从命令输出可以看到,所有客户端证书的到期时间均从执行renew成功的时间开始续签了100年。

说明
所有根证书caetcd-cafront-proxy-ca只有在init初始化的时候才会更新时间,因此建议对于kubeadm部署Kubernetes,可以在初始化之前使用编译的方式将证书设置为更长时间,如100年。

启用证书

依次在所有Master节点上执行如下命令重启kube-apiserverkube-controller-managerkube-scheduleretcd使其启用新证书。

警告
执行完一台后请务必通过kubectl get pod -n kube-system确认对应Master节点上的kube-apiserverkube-controller-managerkube-scheduleretcd服务Pod均处于Running状态并就绪后再开始操作下一台

1
mv /etc/kubernetes/manifests/ /etc/kubernetes/manifests.bak && sleep 30 && mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests/

替换config文件

执行如下命令替换config文件

说明
如果你运行了一个HA集群,这个命令需要在所有Master节点上执行。

1
cp -a /etc/kubernetes/admin.conf /root/.kube/config

参考文档