Fork me on GitHub

kubernetes系列之《使用Kubeadm快速部署集群》

kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具。
本文将实战Kubeadm完整部署一套Kubernetes集群,这种部署方式官网在现在以及将来也可用于生产环境使用了。但如果你想快速体验Kubernetes集群, 不妨采用Kubeadm试试!

这几个工具能通过两条指令完成一个kubernetes集群的部署:

1
2
3
4
5
# 创建一个Master节点
$ kubeadm init

# 将一个Node节点加入到当前集群中
$ kubeadm join <Master节点的IP和端口>

1、安装要求

在开始之前,部署kubernetes集群机器需要满足一下几个条件:

  • 一台或多台机器,操作系统CentOS7.x-86_x64;
  • 硬件配置:2GB或更多内存,2个CPU或更多,30G硬盘或更;
  • 集群中所有机器之间网络互通
  • 可以访问外网,需要拉取镜像
  • 禁止使用swap分区
  • 关闭iptables防火墙
  • 做好时间定期同步

2、部署进程

  1. 在所有节点上安装docker和kubeadm
  2. 部署kubernetes Master
  3. 部署容器网络插件
  4. 部署kubernetes Node,将节点加入Kubernetes集群中
  5. 部署Dashboard Web页面,可视化查看Kubernetes资源

3、环境准备

关闭防火墙:

1
2
$ systemctl stop firewalld
$ systemctl disable firewalld

关闭selinux:

1
2
3
4
$ vim /etc/selinux/config
SELINUX=disabled
# 临时禁用selinux
$ setenforce 0

关闭swap:

1
2
$ swapoff -a
$ swapon -s # 没有分区信息即可。

添加主机对应IP对应关系

1
2
3
4
vim /etc/hosts
172.16.194.128 k8s-master-128
172.16.194.129 k8s-node-129
172.16.194.130 k8s-node-130

根据上面主机规范请自行更改主机名哈。

将桥接的IPv4流量传递到iptables的链:

1
2
3
4
5
$ cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sysctl --system

配置时间同步:

1
2
3
$ crontab -e
# 时间同步
*/5 * * * * /usr/sbin/ntpdate ntp1.aliyun.com >/dev/null 2>&1

4、所有节点安装Docker/kubeadm/kubelet

Kubernetes默认CRI(容器运行时)为Docker,因此先安装Docker。

4.1、安装Docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ yum install -y yum-utils \
device-mapper-persistent-data \
lvm2

# 官网的repo源在中国用不了,咱们还是乖乖使用马爸爸提供的源好了
$ yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
$ yum makecache fast
$ yum install docker-ce -y
$ mkdir /etc/docker/
$ cat << EOF > /etc/docker/daemon.json
{
"registry-mirrors": ["https://registry.docker-cn.com"]
}
EOF
$ systemctl enable docker && systemctl start docker

4.2、添加阿里云YUM软件源

1
2
3
4
5
6
7
$ cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
EOF

4.3、安装kubeadm,kubelet和kubectl

由于版本更新频繁,指定版本号进行部署:

1
2
$ yum install -y kubelet-1.14.1 kubeadm-1.14.1 kubectl-1.14.1
$ systemctl enable kubelet

5、部署kubernetes Master

使用kubeadm init来安装Master节点

1
2
3
4
5
6
$ kubeadm init \
--apiserver-advertise-address=172.16.194.128 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.14.1 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16

安装成功后提示:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.194.128:6443 --token zbqk3l.fe1dywo3ufnrcwff \
--discovery-token-ca-cert-hash sha256:a7497e212a62f6bd87b4f3b1fb23333fb725c9958990a54bed131185d667e9ed

非常顺利的就部署成功了,如果我们想使用非root用户操作kubectl,可以使用以下命令,这也是kubeadm init输出的一部分:

1
2
3
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

此时Master节点安装并启动了如下组件:

  • kubelet
  • kube-scheduler
  • kube-controller-manager
  • kube-apiserver
  • etcd

通过ps -ef|grep kube能看到进程信息
通过docker image 和 docker ps 能看到镜像与启动的容器

6、安装Pod网络插件(CNL)

为了让Pods间可以相互通信,我们必须安装一个网络插件,并且必须在部署任何应用之前安装,CoreDNS也是在网络插件安装之后才会启动的。

网络的插件完整列表,请参考: Networking and Network Policy

在安装之前,我们先查看一下当前Pods的状态:

1
2
3
4
5
6
7
8
9
[root@k8s-master-128 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-8686dcc4fd-84dfh 0/1 Pending 0 18m
kube-system coredns-8686dcc4fd-ldw8r 0/1 Pending 0 18m
kube-system etcd-k8s-master-128 1/1 Running 0 18m
kube-system kube-apiserver-k8s-master-128 1/1 Running 0 18m
kube-system kube-controller-manager-k8s-master-128 1/1 Running 0 17m
kube-system kube-proxy-qqchb 1/1 Running 0 18m
kube-system kube-scheduler-k8s-master-128 1/1 Running 0 18m

如上,可以看到CoreDND的状态是Pending,这是因为我们还没有安装网络插件。

默认情况下,flannel网络插件使用的的网段是10.244.0.0/16,在init的时候,我们保持了默认配置–pod-network-cidr=10.244.0.0/16,当然你也可以修改flannel.yml文件来指定不同的网段。

可以使用如下命令命令来安装flannel插件:

1
2
3
4
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

# 通过wget 下载kube-flannel.yml文件,发现里面有如下镜像所国外源,可以手动docker pull 尝试拉取,如果拉取失败则需要更换为国内的镜像才能用。
$ docker pull quay.io/coreos/flannel:v0.11.0-amd64

稍等片刻,再使用kubectl get pods –all-namespaces命令来查看网络插件的安装情况:

1
2
3
4
5
6
7
8
9
10
[root@k8s-master-128 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-8686dcc4fd-84dfh 1/1 Running 0 33m
kube-system coredns-8686dcc4fd-ldw8r 1/1 Running 0 33m
kube-system etcd-k8s-master-128 1/1 Running 0 32m
kube-system kube-apiserver-k8s-master-128 1/1 Running 0 32m
kube-system kube-controller-manager-k8s-master-128 1/1 Running 0 32m
kube-system kube-flannel-ds-amd64-wsrbp 1/1 Running 0 7m48s
kube-system kube-proxy-qqchb 1/1 Running 0 33m
kube-system kube-scheduler-k8s-master-128 1/1 Running 0 32m

如上,STATUS全部变为了Running,表示安装成功,接下来就可以加入其他节点以及部署应用了。

7、加入Kubernetes Node

如果我们忘记了Master节点的加入token,可以使用如下命令来查看:

1
2
3
[root@k8s-master-128 ~]# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
zbqk3l.fe1dywo3ufnrcwff 23h 2019-05-17T11:28:25+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token

默认情况下,token的有效期是24小时,如果我们的token已经过期的话,可以使用以下命令重新生成:

1
2
3
[root@k8s-master-128 ~]# kubeadm token create
# 输出
4j6te5.bja8setduvotvjm7

如果我们也没有–discovery-token-ca-cert-hash的值,可以使用以下命令生成:

1
2
3
[root@k8s-master-128 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
# 输出
a7497e212a62f6bd87b4f3b1fb23333fb725c9958990a54bed131185d667e9ed

现在,我们登录到工作节点服务器,然后运行如下命令加入集群(这也是上面init输出的一部分):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ kubeadm join 172.16.194.128:6443 --token 4j6te5.bja8setduvotvjm7 --discovery-token-ca-cert-hash sha256:a7497e212a62f6bd87b4f3b1fb23333fb725c9958990a54bed131185d667e9ed

# 输出
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

等待一会,我们可以在Master节点上使用kubectl get nodes命令来查看节点的状态:

1
2
3
4
5
[root@k8s-master-128 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-128 Ready master 47m v1.14.1
k8s-node-129 Ready <none> 4m10s v1.14.1
k8s-node-130 Ready <none> 99s v1.14.1

如上全部Ready,大功告成,我们可以运行一些命令来测试一下集群是否正常。

8、测试Kubernetes集群

首先验证kube-apiserver, kube-controller-manager, kube-scheduler, pod network 是否正常:

1
2
3
4
5
6
7
8
9
10
11
12
# 部署一个 Nginx Deployment,包含两个Pod
# https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
$ kubectl create deployment nginx --image=nginx:alpine
$ kubectl scale deployment nginx --replicas=2

# 验证Nginx Pod是否正确运行,并且会分配192.168.开头的集群IP
$ kubectl get pods -l app=nginx -o wide

# 输出
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-77595c695-w6smk 1/1 Running 0 67s 10.244.2.2 k8s-node-130 <none> <none>
nginx-77595c695-xnhps 1/1 Running 0 72s 10.244.1.2 k8s-node-129 <none> <none>

再验证一下kube-proxy是否正常:

1
2
3
4
5
6
7
8
9
10
11
12
13
# 以 NodePort 方式对外提供服务 https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
$ kubectl expose deployment nginx --port=80 --type=NodePort

# 查看集群外可访问的Port
$ kubectl get svc nginx

# 输出
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx NodePort 10.1.42.3 <none> 80:31319/TCP 21s

# 可以通过任意 NodeIP:Port 在集群外部访问这个服务,本示例中部署的2台集群IP分别是172.16.194.129和172.16.194.130
curl http://172.16.194.129:31319
curl http://172.16.194.130:31319

最后验证一下dns, pod network是否正常:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 运行Busybox并进入交互模式
kubectl run -it curl --image=radial/busyboxplus:curl

# 输入`nslookup nginx`查看是否可以正确解析出集群内的IP,已验证DNS是否正常
[ root@curl-66bdcf564-rxkpp:/ ]$ nslookup nginx

# 输出
Server: 10.1.0.10
Address 1: 10.1.0.10 kube-dns.kube-system.svc.cluster.local

Name: nginx
Address 1: 10.1.42.3 nginx.default.svc.cluster.local

# 通过服务名进行访问,验证kube-proxy是否正常
[ root@curl-66bdcf564-rxkpp:/ ]$ curl http://nginx/

# 输出如下:
<!DOCTYPE html> ---省略

# 分别访问一下2个Pod的内网IP,验证跨Node的网络通信是否正常
[ root@curl-66bdcf564-rxkpp:/ ]$ curl http://10.244.2.2
[ root@curl-66bdcf564-rxkpp:/ ]$ curl http://10.244.1.2

验证通过,集群搭建成功,接下来我们就可以参考官方文档来部署其他服务,愉快的玩耍了。

9、部署Dashboard

官网有安装文件:https://github.com/kubernetes/dashboard

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

# 默认镜像国内无法访问,将文件下载下来,修改镜像:
$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
$ vim kubernetes-dashboard.yaml # 搜索关键字:image
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1# 将这里的官网镜像更改成国内镜像
# 并且在Dashboard Service 配置项里(配置文件末尾)增加NodePort,对外暴露访问:
# ------------------- Dashboard Service ------------------- #

kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort # 增加了这一项配置
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard


# 然后执行部署
$ kubectl apply -f kubernetes-dashboard.yaml

等待一会,使用kubeadm get查看pod启动情况:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@k8s-master-128 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-8686dcc4fd-84dfh 1/1 Running 0 3h32m
coredns-8686dcc4fd-ldw8r 1/1 Running 0 3h32m
etcd-k8s-master-128 1/1 Running 0 3h31m
kube-apiserver-k8s-master-128 1/1 Running 0 3h31m
kube-controller-manager-k8s-master-128 1/1 Running 0 3h31m
kube-flannel-ds-amd64-qs89g 1/1 Running 0 167m
kube-flannel-ds-amd64-tsj6r 1/1 Running 0 170m
kube-flannel-ds-amd64-wsrbp 1/1 Running 0 3h7m
kube-proxy-bmwms 1/1 Running 0 170m
kube-proxy-fwphm 1/1 Running 0 167m
kube-proxy-qqchb 1/1 Running 0 3h32m
kube-scheduler-k8s-master-128 1/1 Running 0 3h32m
kubernetes-dashboard-5d9599dc98-bdg8q 1/1 Running 0 94s

pod已经运行起来了,继续查看暴露的服务:

1
2
3
4
[root@k8s-master-128 ~]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.1.0.10 <none> 53/UDP,53/TCP,9153/TCP 3h38m
kubernetes-dashboard NodePort 10.1.239.63 <none> 443:31459/TCP 2m54s

dashboard已经运行起来并且对外提供服务。

访问UI界面:
https://Node节点IP:31459(节点IP之间,能负载均衡,所有随便访问哪个节点IP+端口 都能访问到UI)
https://172.16.194.130:31459

创建Service account并绑定默认cluster-admin管理员集群角色:

1
2
3
4
5
6
$ kubectl create serviceaccount dashboard-admin -n kube-system
$ kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
$ kubectl describe secrets -n kube-system $(kubectl get secrets -n kube-system |awk '/dashboard-admin/{print $1}')

输出:
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tNDZzbnoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOTg0ODVkNTctNzdhYy0xMWU5LThiNDQtMDAwYzI5ZjRkYWE5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.BaJSW1taC633s6ZEqlnkSfA-L7c59VHUIfUEzaIK8CqiU64R1b1zgsQYiIjiIX465pZoWEeR2ao7_NiTtpP3E1dK8dZzGRhIwLvTrvTUJLj3p1YNDkM3z-kd7OWyo6pgP3yf3RdEWqhXGidf5_2GORF-9dSp3PJghuHc5CZVMu_64-PeT7ZTtyPvcMJw7b8UfW39K7DPzr36liT0y-vZNvmO17Gn-Q4KgIxxTzIwd5zwisXIA7qGY8bf9P6VYymfiacov0jKp3_KlLyrUTeit5OYIOVLuFLak22mPutPYuFre3YFnEgycRSj_QW9QynXms1OEvA3TgEVXgWQytG1rQ

将这个token放入Dashboard认证,即可登录到UI页面。

10、卸载集群

想要撤销kubeadm执行的操作,首先要排除节点,并确保该节点为空, 然后再将其关闭。

在Master节点上运行:

1
2
kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>

然后在需要移除的节点上,重置kubeadm的安装状态:

1
sudo kubeadm reset

如果你想重新配置集群,使用新的参数重新运行kubeadm init或者kubeadm join即可。

参考文章:

-------------本文结束感谢您的阅读-------------