0%

在CentOS7上使用kubeadm快速部署Kubernetes高可用集群

环境要求


Mater节点

  • 物理机虚拟机均可,至少1台,高可用集群至少2台(etcd集群必须奇数台)

  • 配置推荐:实验环境2核2G、测试环境2核4G、生产环境8核16G

  • 关闭所有swap分区或不划分swap分区

Node节点

  • 物理机虚拟机均可,大于等于1台

  • 配置推荐:实验环境2核2G、测试环境4核8G、生产环境16核64G

  • 关闭所有swap分区或不划分swap分区

操作系统版本


CentOS7.5及以上

实验环境信息


主机名 配置 操作系统 IP地址 角色
k8s-master1 2核2G CentOS7.5 10.211.55.4 master,node
k8s-master2 2核2G CentOS7.5 10.211.55.5 master,node
k8s-master3 2核2G CentOS7.5 10.211.55.6 master,node

VIP:10.211.55.10

系统初始化


关闭防火墙

1
2
systemctl stop firewalld
systemctl disable firewalld

关闭selinux

1
2
setenforce 0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

关闭swap

1
2
swapoff -a
echo 'swapoff -a ' >> /etc/rc.d/rc.local

配置主机名

1
hostnamectl set-hostname <hostname>

添加所有节点的本地host解析

1
2
3
4
5
cat >> /etc/hosts << EOF
x.x.x.x hostname1
y.y.y.y hostname2
...
EOF

安装基础软件包

1
yum install vim net-tools lrzsz unzip dos2unix telnet sysstat iotop pciutils lsof tcpdump psmisc bc wget socat -y

内核开启网络支持

1
2
3
4
5
6
7
8
cat >  /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
EOF
modprobe br_netfilter
sysctl -p

配置所有Master到所有节点(包括自身)的ssh免密登录

依此在所有的master节点上做如下操作:

1
2
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-master1

节点之间时间同步

Server端

注:如果环境可以访问互联网,可以不需要自己搭建server端,参考后面的client端部分设置所有节点与公网ntp时间服务器(例如time1.cloud.tencent.com)同步时间即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#安装chrony并备份配置文件
yum install chrony ntpdate -y
cp -a /etc/chrony.conf /etc/chrony.conf.bak

#修改server端配置文件如下,标注的地方需要修改
cat > /etc/chrony.conf << EOF
stratumweight 0
driftfile /var/lib/chrony/drift
rtcsync
makestep 10 3
allow 10.211.55.0/24 #设置为实际环境客户端所属IP网段
smoothtime 400 0.01

bindcmdaddress 127.0.0.1
bindcmdaddress ::1

local stratum 8
manual
keyfile /etc/chrony.keys
#initstepslew 10 client1 client3 client6
noclientlog
logchange 0.5
logdir /var/log/chrony
EOF

#启动服务,设置开机自启
systemctl restart chronyd.service
systemctl enable chronyd.service
systemctl status chronyd.service

Client端

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#安装chrony并备份配置文件
yum install chrony ntpdate -y
cp -a /etc/chrony.conf /etc/chrony.conf.bak

#修改client端配置文件
sed -i "s%^server%#server%g" /etc/chrony.conf
echo "server 10.211.55.4 iburst" >> /etc/chrony.conf #添加一行,其中的IP地址替换为实际环境server端的IP地址

ntpdate 10.211.55.4 #手动同步一次时间,其中的IP地址替换为实际环境server端的IP地址

#启动服务,设置开机自启
systemctl restart chronyd.service
systemctl enable chronyd.service
systemctl status chronyd.service

chronyc sources #查看ntp_servers状态
chronyc tracking #查看ntp详细信息

所有节点(Master和Node)安装docker


卸载旧版本的Docker

1
2
3
4
5
6
7
8
yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine

配置docker-ce repository

1
2
3
4
5
6
7
8
9
#安装所需要的包,yum-utils提供了yum-config-manager工具,device-mapper-persistent-data和lvm2是设备映射存储驱动所需要的
yum install -y yum-utils \
device-mapper-persistent-data \
lvm2

#设置稳定版的repo仓库
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo

安装docker-ce

1
2
#安装最新版本的docker-ce
yum install docker-ce docker-ce-cli containerd.io -y

注:若要安装指定版本的docker,按照如下步骤:

1
2
3
4
5
#列出repo仓库中可用的docker版本并降序排列
yum list docker-ce --showduplicates | sort -r

#确认好要安装的版本,例如为18.09.9,则替换yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io -y中的<VERSION_STRING>进行安装
例如:yum install docker-ce-18.09.9 docker-ce-cli-18.09.9 containerd.io -y

启动Docker并设置开机自启

1
2
3
systemctl start docker
systemctl enable docker
systemctl status docker

所有节点(Master和Node)安装kubeadm、kubelet和kubectl


  • Kubelet:负责与其他节点集群通信,并进行本节点Pod和容器生命周期的管理。

  • Kubeadm:Kubernetes的自动化部署工具,降低了部署难度,提高效率。

  • Kubectl:Kubernetes集群管理工具。

配置kubernetes repository

1
2
3
4
5
6
7
8
9
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

注:若无法访问国外网站,可配置国内的kubernetes源

1
2
3
4
5
6
7
8
9
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

关闭selinux

关闭selinux主要是为了允许容器访问主机文件系统,在kubelet对SELINUX的支持改进前需要先关闭selinux,由于在前期准备中已做,此步可忽略

打开net.bridge.bridge-nf-call-iptables内核参数

配置net.bridge.bridge-nf-call-iptables内核参数为1,不开启的话可能出现流量绕过iptables而导致流量路由错误的问题,由于在前期准备中已做,此步可忽略

安装kubeadm、kubelet和kubectl

1
2
3
4
#安装最新版本的kubelet、kubeadm、kubectl
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable --now kubelet
#kubelet现在每隔几秒钟就重新启动一次,因为它在一个crashloop中等待kubeadm告诉它该做什么

注:若要安装指定版本的kubelet、kubeadm、kubectl,可参考如下步骤:

1
2
3
4
#安装1.14.6版本的kubelet、kubeadm、kubectl
yum install -y kubelet-1.14.6 kubeadm-1.14.6 kubectl-1.14.6 --disableexcludes=kubernetes
systemctl enable --now kubelet
#kubelet现在每隔几秒钟就重新启动一次,因为它在一个crashloop中等待kubeadm告诉它该做什么

所有的Master上配置Keepalived和HAProxy(单Master节点部署忽略此步)


Keepalived

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#安装keeplived并备份配置文件
yum install -y keepalived
cp -a /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

#修改keepalived配置文件如下,标注的地方需要修改
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived

global_defs {
router_id k8s-master1 #标识,可用机器主机名作为标识
}

vrrp_instance VI_1 {
state MASTER #设置角色,第一个master为MASTER,剩余的节点均为BACKUP
interface eth0 #设置vip绑定端口
virtual_router_id 51 #让master和backup在同一个虚拟路由里,id号必须相同
priority 150 #优先级,谁的优先级高谁就是master,值越大优先级越高
advert_int 1 #心跳间隔时间
authentication {
auth_type PASS #认证
auth_pass k8s #密码
}
virtual_ipaddress {
10.211.55.10 #虚拟ip
}
}
EOF

#启动keepalived并设置开机自启
systemctl restart keepalived.service
systemctl enable keepalived.service
systemctl status keepalived.service

HAProxy

注:所有master节点上haproxy配置文件都是一样的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#安装haproxy并备份配置文件
yum install -y haproxy
cp -a /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak

#修改haproxy配置文件如下,标注的地方需要修改
cat > /etc/haproxy/haproxy.cfg << EOF
global
chroot /var/lib/haproxy
daemon
group haproxy
user haproxy
log 127.0.0.1:514 local0 warning
pidfile /var/lib/haproxy.pid
maxconn 20000
spread-checks 3
nbproc 8
defaults
log global
mode tcp
retries 3
option redispatch
listen k8s-apiserver
bind 0.0.0.0:8443 # 指定绑定的IP和端口,端口建议用非6443端口(此处用8443),因为如果haproxy是和k8s apiserver部署在同一台服务器上,用6443会产生端口冲突,若不是部署在同一台机器上则此处端口可以使用6443
mode tcp
balance roundrobin
timeout server 15s
timeout connect 15s
server k8sapiserver1 10.211.55.4:6443 check port 6443 inter 5000 fall 5 #转发到k8s-master1的apiserver上,apiserver端口默认是6443
server k8sapiserver2 10.211.55.5:6443 check port 6443 inter 5000 fall 5 #转发到k8s-master2的apiserver上,apiserver端口默认是6443
server k8sapiserver3 10.211.55.6:6443 check port 6443 inter 5000 fall 5 #转发到k8s-master3的apiserver上,apiserver端口默认是6443
EOF

#启动haproxy并设置开机自启
systemctl restart haproxy
systemctl enable haproxy
systemctl status haproxy

初始化第一台Master


新版本kubeadm

初始化第一个master

1
2
3
4
5
6
7
kubeadm init \
--apiserver-advertise-address=10.211.55.4 \
--kubernetes-version v1.16.4 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16 \
--control-plane-endpoint 10.211.55.10:8443 \
--upload-certs

参数描述:

-–apiserver-advertise-address:指定kube-apiserver监听的ip地址,就是master本机IP地址。

-–kubernetes-version:指定k8s版本;

-–service-cidr:指定SVC的网络范围,注意不要与服务器的网络范围重叠;

-–pod-network-cidr:指定Pod的网络范围,默认为10.244.0.0/16,安装网络时yaml文件中指定的网络范围要与此参数设置的一致;

-–control-plane-endpoint:单Master部署时配置为Master节点IP,端口为6443;多Master部署时指定keepalived的虚拟ip和端口

-–upload-certs:上传证书

初始化成功后,会看到大概如下提示,下面信息先保留,后续添加master节点和node节点需要用到

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

kubeadm join 10.211.55.10:6443 --token y55nex.s699ytnwr28o1vu1 \
--discovery-token-ca-cert-hash sha256:dd0932f4d17a864cf4ea3a7eff44c77695b47f39de03eaaf1dfd27762d7dd48b \
--control-plane --certificate-key b2ef4609c5af0d51334d49b20a3713e073ef818593d830b5b17ac58b294cc5c0

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use kubeadm init phase upload-certs to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.211.55.10:6443 --token y55nex.s699ytnwr28o1vu1 \
--discovery-token-ca-cert-hash sha256:dd0932f4d17a864cf4ea3a7eff44c77695b47f39de03eaaf1dfd27762d7dd48b

配置kubectl的config文件

kubectl默认会在执行的用户家目录下面的.kube目录下寻找config文件。按照上面输出的提示执行如下指令将在初始化时[kubeconfig]步骤生成的admin.conf拷贝到.kube/config,因为此配置文件中记录了apiserver的访问地址,所以后面直接执行kubectl命令就可以正常连接到APIServer中

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

此时使用kubectl get node能看到第一个master节点还处于NotReady状态,这是因为现在还没安装网络

安装网络

在任意一台master节点执行如下指令下载网络yaml文件,k8s支持多种网络类型,本文安装的是flannel网络,更多网络类型可参考https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network

1
2
cd /root/
wget https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

根据实际环境情况修改kube-flannel.yml文件,比如Network、Backend、hostNetwork配置等等,Network配置的网络范围要与kubeadm init时指定的-–pod-network-cidr一致,修改完成后进行安装

1
2
cd /root/
kubectl apply -f kube-flannel.yml

查看flannel的pod创建情况,等待到所有node节点上的flannel pod都处于running状态后表示cni部署完成

1
kubectl get pods -n kube-system -o wide

旧版本kubeadm

修改初始化配置文件

注:由于之前版本的kubeadm(例如1.14.6)不支持–control-plane-endpoint参数,不能像上面那样通过kubeadm init直接初始化第一个master,需要先通过如下指令导出默认配置,然后在根据自己的实际环境修改配置。

1
kubeadm config print init-defaults > kubeadm-init.yml

编辑kubeadm-init.yml,标注的地方需要修改或增加

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s ####token有效期,添加节点如果token过期需要重新生成
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 10.211.55.4 #填写本地真实IP
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: k8s-master1 #填写本地主机名
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "10.211.55.10:8443" #单master节点时这里写“本地真实IP:6443”;多master节点时写“VIP:端口”,端口要与haproxy配置中的bind字段的端口一致
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io #如果无法访问国外网址,可将此处的镜像仓库地址改为国内阿里云镜像仓库地址
kind: ClusterConfiguration
kubernetesVersion: v1.14.6 #指定k8s版本
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16" #指定Pod的网络范围为10.244.0.0/16,若没有则flannel网络启动失败
serviceSubnet: 10.1.0.0/16 #指定SVC的网络范围
scheduler: {}

预下载镜像

1
kubeadm config images pull --config kubeadm-init.yml

初始化第一个master

执行如下指令进行初始化第一个master节点

1
kubeadm init --config kubeadm-init.yml

初始化成功后,会看到大概如下提示,下面信息先保留,后续添加master节点和node节点需要用到

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

kubeadm join 10.211.55.10:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:a0816d4859bcb4e84e25c5c87c5495e9b9e38486d3bfb3d71bc28e1ff8451e13 \
--experimental-control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.211.55.10:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:a0816d4859bcb4e84e25c5c87c5495e9b9e38486d3bfb3d71bc28e1ff8451e13

配置kubectl的config文件

kubectl默认会在执行的用户家目录下面的.kube目录下寻找config文件。按照上面输出的提示执行如下指令将在初始化时[kubeconfig]步骤生成的admin.conf拷贝到.kube/config,因为此配置文件中记录了apiserver的访问地址,所以后面直接执行kubectl命令就可以正常连接到APIServer中

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

此时使用kubectl get node能看到第一个master节点还处于NotReady状态,这是因为现在还没安装网络

安装网络

在任意一台master节点执行如下指令下载网络yaml文件,k8s支持多种网络类型,本文安装的是flannel网络,更多网络类型可参考https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network

1
2
cd /root/
wget https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

根据实际环境情况修改kube-flannel.yml文件,比如Network、Backend、hostNetwork配置等等,Network配置的网络范围要与kubeadm-init.yml中的podSubnet一致,修改完成后进行安装

1
2
cd /root/
kubectl apply -f kube-flannel.yml

查看flannel的pod创建情况,等待到所有node节点上的flannel pod都处于running状态后表示cni部署完成

1
kubectl get pods -n kube-system -o wide

加入剩余的master


新版本kubeadm

在剩余的master节点上运行如下指令进行初始化并配置kube的config文件:

1
2
3
4
5
6
7
kubeadm join 10.211.55.10:6443 --token y55nex.s699ytnwr28o1vu1 \
--discovery-token-ca-cert-hash sha256:dd0932f4d17a864cf4ea3a7eff44c77695b47f39de03eaaf1dfd27762d7dd48b \
--control-plane --certificate-key b2ef4609c5af0d51334d49b20a3713e073ef818593d830b5b17ac58b294cc5c0

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

旧版本kubeadm

在第一台master节点上创建如下内容的脚本(假设为add_master.sh):

1
2
3
4
5
6
7
8
9
10
CONTROL_PLANE_IPS="k8s-master2 k8s-master3"
for host in ${CONTROL_PLANE_IPS}
do
ssh root@${host} "mkdir -p /etc/kubernetes/pki/etcd"
scp -r /etc/kubernetes/pki/ca.* root@${host}:/etc/kubernetes/pki/
scp -r /etc/kubernetes/pki/sa.* root@${host}:/etc/kubernetes/pki/
scp -r /etc/kubernetes/pki/front-proxy-ca.* root@${host}:/etc/kubernetes/pki/
scp -r /etc/kubernetes/pki/etcd/ca.* root@${host}:/etc/kubernetes/pki/etcd/
scp -r /etc/kubernetes/admin.conf root@${host}:/etc/kubernetes/
done

CONTROL_PLANE_IPS变量设置其余master的主机名或者IP,之间用空格隔开(若设置主机名需要确保配置有解析)

配置好ONTROL_PLANE_IPS后运行脚本

1
sh add_master.sh

在剩余的master节点上运行如下指令进行初始化并配置kube的config文件:

1
2
3
4
5
6
7
8
kubeadm join 10.211.55.10:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:a0816d4859bcb4e84e25c5c87c5495e9b9e38486d3bfb3d71bc28e1ff8451e13 \
--experimental-control-plane


mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

注:默认情况下出于安全的考虑业务pods不会调度到控制节点上,如果想要让业务pods能够调度到控制节点上的话,可以在其中一台master节点上执行如下指令:

1
kubectl taint nodes --all node-role.kubernetes.io/master-

加入Node


依此在node节点上运行如下指令:

1
2
kubeadm join 10.211.55.10:6443 --token y55nex.s699ytnwr28o1vu1 \
--discovery-token-ca-cert-hash sha256:dd0932f4d17a864cf4ea3a7eff44c77695b47f39de03eaaf1dfd27762d7dd48b

环境测试验证


在任意一个master节点上执行如下指令创建一个nginx pod并暴露端口测试是否可以从外部正常访问

1
2
3
4
5
6
7
8
9
10
#创建nginx deployment
kubectl create deployment web --image=nginx

#暴露端口
kubectl expose deployment web --port=80 --type=NodePort

#查看对应的访问端口
kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web NodePort 10.1.153.72 <none> 80:30612/TCP 4s

浏览器访问:http://<Node_IP>:30612若能正常返回nginx欢迎页面,则表示环境一切正常。

部署Web UI


Github地址:https://github.com/kubernetes/dashboard
官方地址:https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

在任意一台master节点上下载Web UI的yaml文件到/root目录下

1
2
cd /root/
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta6/aio/deploy/recommended.yaml

编辑recommended.yaml文件,找到kubernetes-dashboard这个Service的部分,设置其type为NodePort,nodePort为30001(可自定义)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
......

kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard

......

修改完成后,执行如下指令开始部署Web UI

1
2
3
4
5
6
7
kubectl apply -f /root/recommended.yaml

#查看kubernetes-dashboard的pod,确认pod的STATUS均为running再继续下面的步骤
kubectl get pods -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-76585494d8-v4gd9 1/1 Running 0 63s
kubernetes-dashboard-b65488c4-pwkc2 1/1 Running 0 63s

创建service account并绑定默认cluster-admin管理员集群角色:

1
2
3
kubectl create serviceaccount dashboard-admin -n kubernetes-dashboard
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin
kubectl describe secrets -n kubernetes-dashboard $(kubectl -n kubernetes-dashboard get secret | awk '/dashboard-admin/{print $1}')

使用上面输出的token登录Dashboard(https://<NODE_IP>:30001)。注意协议一定要用https

**至此已经完成一套k8s集群的部署。若还需要再安装Helm、ingress-nginx、配置k8s对接外部存储做持久化存储,比如Ceph,可继续参考如下内容。

安装helm


helm官网:https://helm.sh/

helm github:https://github.com/helm/helm

下载所需版本的helm安装包(本文以2.16.1版本为例),上传到所有的master节点的/root/helm目录下(若没有此目录先创建),执行如下指令安装helm客户端

1
2
3
4
cd /root/helm
tar zxf helm-v2.16.1-linux-amd64.tar.gz
cd linux-amd64/
cp -a helm /usr/local/bin/

在其中一台master运行如下指令安装helm服务端

1
helm init

执行如下指令设置tiller的rbac权限

1
2
3
4
5
6
7
8
9
kubectl create serviceaccount -n kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl --namespace kube-system patch deploy tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
helm init --upgrade --service-account tiller

#稍等片刻待tiller的pod处于均Ready后执行helm list看是否可以正常列出所有的release
kubectl get pods -n kube-system |grep tiller
helm list
helm version

若执行helm version出现类似如下报错

1
2
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
E1213 15:58:40.605638 10274 portforward.go:400] an error occurred forwarding 34583 -> 44134: error forwarding port 44134 to pod 1e92153b279110f9464193c4ea7d6314ac69e70ce60e7319df9443e379b52ed4, uid : unable to do port forwarding: socat not found

解决办法:

在所有node节点上安装socat

1
yum install socat -y

安装ingress-nginx


ingress-nginx官网:https://kubernetes.github.io/ingress-nginx/

ingress-nginx github:https://github.com/kubernetes/ingress-nginx

在任意一台master节点上下载ingress-nginx的yaml文件到/root目录下

1
2
cd /root/
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

编辑mandatory.yaml文件,设置ingress-nginx的部署模式,本文采用hostNetwork模式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
hostNetwork: true #添加此行设置ingress-nginx的部署模式为hostNetwork
# wait up to five minutes for the drain of connections
terminationGracePeriodSeconds: 300
serviceAccountName: nginx-ingress-serviceaccount
nodeSelector:
kubernetes.io/os: linux
...

运行如下指令安装ingress-nginx

1
2
3
4
5
cd /root/
kubectl apply -f mandatory.yaml

#确认ingress-nginx的pod是否正常启动,处于running状态
kubectl get pods -n ingress-nginx

配置rbd-provisioner


K8S要对接Ceph RBD存储做持久化存储,首先必须要搭建Ceph存储集群,并在K8S的所有节点上安装对应版本的ceph-common客户端命令。关于Ceph集群的搭建和ceph-common此处不进行赘述,可参考Ceph官网文档进行。

Ceph集群和ceph-common安装都完成后,在其中一台master上创建/root/rbd-provisioner目录下,并执行如下指令创建rbd-provisioner所需的yaml文件,标注部分根据实际情况进行修改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
mkdir /root/rbd-provisioner
cd /root/rbd-provisioner

cat > clusterrole.yaml << EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns","coredns"]
verbs: ["list", "get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
EOF


cat > clusterrolebinding.yaml << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: ceph
roleRef:
kind: ClusterRole
name: rbd-provisioner
apiGroup: rbac.authorization.k8s.io
EOF

cat > deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: rbd-provisioner
namespace: ceph
spec:
progressDeadlineSeconds: 600
revisionHistoryLimit: 10
replicas: 1
selector:
matchLabels:
app: rbd-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: rbd-provisioner
spec:
containers:
- name: rbd-provisioner
imagePullPolicy: IfNotPresent
image: "quay.io/external_storage/rbd-provisioner:latest"
env:
- name: PROVISIONER_NAME
value: ceph.com/rbd
serviceAccount: rbd-provisioner
restartPolicy: Always
EOF

cat > role.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-provisioner
namespace: ceph
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
EOF


cat > rolebinding.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-provisioner
namespace: ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: ceph
EOF

cat > serviceaccount.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-provisioner
namespace: ceph
EOF


cat > storageclass.yaml << EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
name: rbd
provisioner: ceph.com/rbd
parameters:
monitors: 10.211.55.4:6789,10.211.55.5:6789,10.211.55.6:6789 #配置Ceph集群的monitor节点信息
pool: k8s #配置要连接的pool,若没有需要先在ceph集群上创建
adminId: admin
adminSecretNamespace: ceph
adminSecretName: ceph-secret
fsType: ext4
userId: admin
userSecretNamespace: ceph
userSecretName: ceph-secret
imageFormat: "2"
imageFeatures: layering
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

cat > secrets.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
namespace: ceph
type: "ceph.com/rbd"
data:
# ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kube'
# ceph auth get-key client.admin | base64
key: QVFEcTN5VmRvK28xRHhBQUlKNW5zQ0xwcTd3N0Q5OTJENm9YeGc9PQ== #配置Ceph集群的kering,此处填的是经过base64编码后的值
EOF

执行如下执行创建rbd storageclass

1
2
3
4
cd /root/rbd-provisioner
kubectl create namespace ceph
kubectl apply -f storageclass.yaml -f clusterrolebinding.yaml -f clusterrole.yaml -f deployment.yaml -f rolebinding.yaml -f role.yaml -f secrets.yaml -f serviceaccount.yaml
kubectl get pods -n ceph | grep rbd-provisioner

配置cephfs-provisioner


K8S要对接CephFS存储做持久化存储,首先必须要搭建Ceph存储集群,并在K8S的所有节点上安装对应版本的ceph-common客户端命令。关于Ceph集群的搭建和ceph-common此处不进行赘述,可参考Ceph官网文档进行。

Ceph集群和ceph-common安装都完成后,在其中一台master上创建/root/cephfs-provisioner目录下,并执行如下指令创建cephfs-provisioner所需的yaml文件,标注部分根据实际情况进行修改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
mkdir /root/cephfs-provisioner
cd /root/cephfs-provisioner

cat > clusterrole.yaml << EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cephfs-provisioner
namespace: ceph
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns","coredns"]
verbs: ["list", "get"]
EOF

cat > clusterrolebinding.yaml << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cephfs-provisioner
subjects:
- kind: ServiceAccount
name: cephfs-provisioner
namespace: ceph
roleRef:
kind: ClusterRole
name: cephfs-provisioner
apiGroup: rbac.authorization.k8s.io
EOF


cat > deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: cephfs-provisioner
namespace: ceph
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: cephfs-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: cephfs-provisioner
spec:
containers:
- name: cephfs-provisioner
image: "quay.io/external_storage/cephfs-provisioner:latest"
imagePullPolicy: IfNotPresent
env:
- name: PROVISIONER_NAME
value: ceph.com/cephfs
- name: PROVISIONER_SECRET_NAMESPACE
value: ceph
command:
- "/usr/local/bin/cephfs-provisioner"
args:
- "-id=cephfs-provisioner-1"
- "-disable-ceph-namespace-isolation=true"
- "-enable-quota=true"
serviceAccount: cephfs-provisioner
restartPolicy: Always
terminationGracePeriodSeconds: 30
EOF

cat > role.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cephfs-provisioner
namespace: ceph
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
EOF


cat > rolebinding.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cephfs-provisioner
namespace: ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cephfs-provisioner
subjects:
- kind: ServiceAccount
name: cephfs-provisioner
EOF


cat > serviceaccount.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephfs-provisioner
namespace: ceph
EOF


cat > storageclass.yaml << EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: cephfs
provisioner: ceph.com/cephfs
parameters:
monitors: 10.211.55.4:6789,10.211.55.5:6789,10.211.55.6:6789 #配置Ceph集群的monitor节点信息
adminId: admin
adminSecretName: ceph-secret
adminSecretNamespace: "ceph"
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

执行如下指令创建cephfs storageclass

1
2
3
cd /root/cephfs-provisioner
kubectl apply -f storageclass.yaml -f clusterrolebinding.yaml -f clusterrole.yaml -f deployment.yaml -f rolebinding.yaml -f role.yaml -f serviceaccount.yaml
kubectl get pods -n ceph | grep cephfs-provisioner

参考链接:

https://kubernetes.io/docs/setup/

https://docs.docker.com/install/linux/docker-ce/centos/

https://blog.csdn.net/fuck487/article/details/102783300

https://github.com/kubernetes-incubator/external-storage


- - - - - - - - - 本文结束啦感谢您阅读 - - - - - - - - -