Fork me on GitHub

kubernetes系列之《集群部署(下)》.md

本文继承上一篇文章《集群部署(上)》, 继续Kubernetes集群实战
本文将实现:

  1. 完整部署一套Kubernetes
  2. 部署Kubernetes UI(Dashboard)
  3. 部署多Master节点的Kubernetes集群
  4. 部署Kubernetes集群内部DNS(CoreDNS )

七、部署Master组件

根据环境规划,Master节点应有以下组件:

  • kube-apiserver
  • kube-controller-manager
  • kube-sheduler
  • etcd

etcd在上面已经部署完成,本文里Master节点是k8s-master-128,那该节点就剩下其他三个组件需部署。

7.1、获取二进制包

官网:https://kubernetes.io/,
Github: https://github.com/kubernetes/kubernetes/releases
二进制包:https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz

注意:二进制包比较大,400M大小,且有墙,我这里手动下载然后上传到服务器;

7.2、部署二进制文件

1
2
3
4
5
6
7
8
9
10
11
# 部署Master组件
[root@k8s-master-128 ~]# cd soft/
[root@k8s-master-128 soft]# tar zxf kubernetes-server-linux-amd64.tar.gz
[root@k8s-master-128 soft]# cp kubernetes/server/bin/{kubectl,kube-scheduler,kube-controller-manager,kube-apiserver} /opt/kubernetes/bin/
[root@k8s-master-128 soft]# echo "export PATH=$PATH:/opt/kubernetes/bin" >>/etc/profile
[root@k8s-master-128 soft]# source /etc/profile
[root@k8s-master-128 soft]# which kubectl
/opt/kubernetes/bin/kubectl

# 部署Node组件(方便后面使用)
[root@k8s-master-128 soft]# scp kubernetes/server/bin/{kubelet,kube-proxy} k8s-node-129:/opt/kubernetes/bin/

7.3、生成Kubeconfig文件

7.3.1、api-server认证方式

这个是知识了解,api-server有如下几种认证方式:

  1. CA证书认证
  2. Token认证:token-auth
  3. 基本认证:basic-auth

kubernetes 认证主要分为上面三种,可以同时配置多种认证方式,只要其中任意一个方式认证通过即可。

参考:

本次部署使用第二种:Token认证。

7.3.2、创建配置文件

需要创建以下三个配置文件。

  • 1、创建TLS Bootstrapping Token
  • 2、创建kubelet kubeconfig
  • 3、创建kube-proxy kubeconfig

我们使用一个脚本将所有的配置文件一起创建出来

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
[root@k8s-master-128 ~]# mkdir ssl/k8s-cret
[root@k8s-master-128 ~]# cd ssl/k8s-cret/
[root@k8s-master-128 k8s-cret]# vim kubeconfig.sh
# 创建 TLS Bootstrapping Token
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ') # 生成随机token(随机字符串)

cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

#----------------------
# 创建kubelet bootstrapping kubeconfig
export KUBE_APISERVER="https://172.16.194.128:6443" # 这个脚本唯一需要更改的就是这个地方,填写成你的Master IP地址

# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#----------------------

# 创建kube-proxy kubeconfig文件
kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
--client-certificate=./kube-proxy.pem \
--client-key=./kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

执行脚本,创建配置文件:

1
2
3
4
5
6
7
8
9
10
[root@k8s-master-128 k8s-cret]# chmod +x kubeconfig.sh
[root@k8s-master-128 k8s-cret]# ./kubeconfig.sh
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".

配置文件的用途: 总共创建了三个文件,用途分别为:

  • bootstrap.kubeconfig # Node节点的kubelet组件使用
  • kube-proxy.kubeconfig # Node节点的kube-proxy组件使用
  • token.csv # Master节点的apiserver组件使用

根据上面的用途说明,将生成的文件推送到各个节点的目录中(以备后面部署组件使用):

1
2
3
4
5
6
# Master节点只需要token.csv
[root@k8s-master-128 k8s-cret]# cp token.csv /opt/kubernetes/cfg/

# Node节点推送文件
[root@k8s-master-128 k8s-cret]# scp bootstrap.kubeconfig kube-proxy.kubeconfig k8s-node-129:/opt/kubernetes/cfg/
[root@k8s-master-128 k8s-cret]# scp bootstrap.kubeconfig kube-proxy.kubeconfig k8s-node-130:/opt/kubernetes/cfg/

7.3.3、查看配置文件

token.csv

1
2
[root@k8s-master-128 k8s-cret]# cat token.csv
38435b41e3861251dce8c2cbf968ca67,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

注解:

  • 38435b41e3861251dce8c2cbf968ca67:随机生成的token,也可以固定一个Token来使用
  • kubelet-bootstrap :用户名
  • 10001:用户ID
  • system:kubelet-bootstrap:用户组

bootstrap.kubeconfig

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@k8s-master-128 k8s-cret]# cat bootstrap.kubeconfig
apiVersion: v1 # api的版本
clusters: # 集群的内容
- cluster:
certificate-authority-data: # 集群CA数字证书,有很大一堆,略······
server: https://172.16.194.128:6443 # server的地址
name: kubernetes # 集群的名字
contexts: # 上下文内容
- context:
cluster: kubernetes
user: kubelet-bootstrap # k8s用户(可改变,是从token.csv里定义的)
name: default # 上下文名称
current-context: default # 当前默认使用的上下文
kind: Config
preferences: {}
users: # 用户的信息
- name: kubelet-bootstrap # 用户名
user:
token: 38435b41e3861251dce8c2cbf968ca67 # token,关键!必须要与token.csv里的token对应!不对应则没有相应的权限,会认证失败。

kube-proxy.kubeconfig

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@k8s-master-128 k8s-cret]# cat kube-proxy.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: # 集群CA数字证书,有很大一堆,略······
server: https://192.16.194.128:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kube-proxy
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-proxy
user:
client-certificate-data: # 客户端CA数字证书,有很大一堆,略······
client-key-data: # 客户端key证书,有很大一堆,略······

7.4、启动Master组件

7.4.1、启动kube-apiserver

1、启动的脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
[root@k8s-master-128 ~]# cat apiserver.sh 
#!/bin/bash
MASTER_ADDRESS=${1:-"192.168.1.195"}
ETCD_SERVERS=${2:-"http://127.0.0.1:2379"}

cat <<EOF >/opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs \
--v=4 \
--etcd-servers=${ETCD_SERVERS} \
--insecure-bind-address=127.0.0.1 \
--bind-address=${MASTER_ADDRESS} \
--insecure-port=8080 \
--secure-port=6443 \
--advertise-address=${MASTER_ADDRESS} \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--service-node-port-range=30000-50000 \
--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--kubelet-https=true \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF

cat <<EOF >/usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

2、启动apiserver

1
2
3
4
5
6
7
# apiserver.sh 接收两个参数,第一个是apiserver的地址,第二个是ETCD集群的地址,我们集群有三个节点,都填写上。
[root@k8s-master-128 ~]# chmod +x apiserver.sh
[root@k8s-master-128 ~]# ./apiserver.sh 172.16.194.128 https://172.16.194.128:2379,https://172.16.194.129:2379,https://172.16.194.130:2379
[root@k8s-master-128 ~]# systemctl daemon-reload
[root@k8s-master-128 ~]# systemctl enable kube-apiserver
[root@k8s-master-128 ~]# systemctl start kube-apiserver
[root@k8s-master-128 ~]# systemctl status kube-apiserver

3、查看启动状态

1
2
3
4
5
6
[root@k8s-master-128 logs]# ls -lh /opt/kubernetes/logs/  # 详细日志
[root@k8s-master-128 logs]# netstat -lntup|grep kube
tcp 0 0 172.16.194.128:6443 0.0.0.0:* LISTEN 28201/kube-apiserve
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 28201/kube-apiserve
[root@k8s-master-128 logs]# ps -ef|grep kube-apiserver
# 信息太多,不贴了

注意:

  1. 如果你照着上面的脚本执行后,如没有进程存活,你应该查看下日志和检查上面的证书都是否存在。
  2. 一定要先启动apiserver!因为后面两个组件依赖apiserver,controller-manager和scheduler可以没有先后顺序。

7.4.2、启动kube-controller-manager

1、启动的脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[root@k8s-master-128 ~]# cat controller-manager.sh
#!/bin/bash

MASTER_ADDRESS=${1:-"127.0.0.1"}

cat <<EOF >/opt/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs \
--v=4 \
--master=${MASTER_ADDRESS}:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"
EOF

cat <<EOF >/usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

2、启动controller-manager

1
2
3
4
5
[root@k8s-master-128 ~]# chmod +x controller-manager.sh
[root@k8s-master-128 ~]# ./controller-manager.sh
[root@k8s-master-128 ~]# systemctl daemon-reload
[root@k8s-master-128 ~]# systemctl enable kube-controller-manager
[root@k8s-master-128 ~]# systemctl start kube-controller-manager

启动失败请检查这两个文件是否配置正确,以及启动日志:
/opt/kubernetes/cfg/kube-controller-manager
/usr/lib/systemd/system/kube-controller-manager.service
/opt/kubernetes/logs/

3、查看启动状态

1
2
3
4
5
# controller-manager启动的端口是10252
[root@k8s-master-128 ~]# netstat -lntup|grep kube-c
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 7699/kube-controlle
tcp6 0 0 :::10257 :::* LISTEN 7699/kube-controlle
[root@k8s-master-128 ~]# ps -ef|grep controller-manager

7.4.3、启动kube-sheduler

1、启动的脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@k8s-master-128 ~]# cat scheduler.sh
#!/bin/bash

MASTER_ADDRESS=${1:-"127.0.0.1"}

cat <<EOF >/opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs \
--v=4 \
--master=${MASTER_ADDRESS}:8080 \
--leader-elect"
EOF

cat <<EOF >/usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

2、启动scheduler

1
2
3
4
5
[root@k8s-master-128 ~]# chmod +x scheduler.sh
[root@k8s-master-128 ~]# ./scheduler.sh
[root@k8s-master-128 ~]# systemctl daemon-reload
[root@k8s-master-128 ~]# systemctl enable kube-scheduler
[root@k8s-master-128 ~]# systemctl start kube-scheduler

3、查看启动状态

1
2
3
4
5
# scheduler启动的端口是10251
[root@k8s-master-128 ~]# netstat -lntup|grep kube-sc
tcp6 0 0 :::10251 :::* LISTEN 7897/kube-scheduler
tcp6 0 0 :::10259 :::* LISTEN 7897/kube-scheduler
[root@k8s-master-128 ~]# ps -ef|grep scheduler

7.5、查看组件运行状态

kubectl get cs # 查看k8s集群资源的健康信息
相关阅读:Kubernetes kubectl get 命令详解

1
2
3
4
5
6
7
[root@k8s-master-128 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}

看到以上信息都为健康状态,到此,Kubernetes集群的Master节点即部署完成。

八、部署Node组件

根据前面环境规划,Node节点将启动kubelet和kube-proxy组件。

8.1、创建Token租户并绑定角色

这是我们创建的tuken信息,需要把租户信息创建出来,Node节点在部署kubelet组件时需要通过token租户进行权限验证

1
2
3
4
5
6
7
8
9
[root@k8s-master-128 ~]# kubectl create --help
[root@k8s-master-128 ~]# kubectl create clusterrolebinding --help
Usage:
kubectl create clusterrolebinding NAME --clusterrole=NAME [--user=username] [--group=groupname]
[--serviceaccount=namespace:serviceaccountname] [--dry-run] [options]

[root@k8s-master-128 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
[root@k8s-master-128 ~]# kubectl get clusterrole # 查看集群角色(system:node-bootstrapper由此得来)

8.2、部署Kubelet

8.2.1、启动Kubelet组件

1、启动的脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[root@k8s-node-129 ~]# cat kubelet.sh
#!/bin/bash

NODE_ADDRESS=${1:-"127.0.0.1"} # 节点的IP地址,通过参数传入
DNS_SERVER_IP=${2:-"10.0.0.2"} # DNS地址,在apiserver.sh里指定的地址端,可以写10.10.10.0/24段的任意IP

cat <<EOF >/opt/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs \
--v=4 \
--address=${NODE_ADDRESS} \
--hostname-override=${NODE_ADDRESS} \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--cert-dir=/opt/kubernetes/ssl \
--allow-privileged=true \
--cluster-dns=${DNS_SERVER_IP} \
--cluster-domain=cluster.local \
--fail-swap-on=false \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
EOF

cat <<EOF >/usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

2、启动kubelet

1
2
3
4
5
6
7
8
9
# 两个Node节点同时启动kubelet
[root@k8s-node-129 ~]# chmod +x kubelet.sh
[root@k8s-node-129 ~]# ./kubelet.sh 172.16.194.129
[root@k8s-node-130 ~]# ./kubelet.sh 172.16.194.130

# 启动kubelet
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet

3、查看启动状态

1
2
3
[root@k8s-node-129 ~]# ps -ef|grep kubelet
[root@k8s-node-129 ~]# ls -lh /opt/kubernetes/cfg/kubelet.kubeconfig
ls: 无法访问/opt/kubernetes/cfg/kubelet.kubeconfig: 没有那个文件或目录

注意:你会发现没有kubelet.kubeconfig文件,这是为什么呢?是因为k8s-Master需要给Node节点的kubelet组件颁发证书,Node节点才会生成这个证书文件。

承上,我们继续到Master节点看看证书请求状况:

1
2
3
4
[root@k8s-master-128 ~]# kubectl get csr  # 查看Node节点的kubelet证书申请请求
NAME AGE REQUESTOR CONDITION
node-csr-K50rilZHx2Gn_MHesRDr_wIJ5hOrg3buFjWuq3RNRhE 3m45s kubelet-bootstrap Pending
node-csr-xW7Q15gU-88wPvObRXmia6y-eMyx5dbcMHbGIwINzD0 15m kubelet-bootstrap Pending # 等待颁发状态

8.2.2、k8s集群为kubelet颁发证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@k8s-master-128 ~]# kubectl --help|grep certificate
certificate 修改 certificate 资源.
[root@k8s-master-128 ~]# kubectl certificate --help
approve 同意一个自签证书请求
deny 拒绝一个自签证书请求

[root@k8s-master-128 ~]# kubectl certificate approve --help
Usage:
kubectl certificate approve (-f FILENAME | NAME) [options]

# 上面几步help我们找到了用法
[root@k8s-master-128 ~]# kubectl certificate approve node-csr-K50rilZHx2Gn_MHesRDr_wIJ5hOrg3buFjWuq3RNRhE
certificatesigningrequest.certificates.k8s.io/node-csr-K50rilZHx2Gn_MHesRDr_wIJ5hOrg3buFjWuq3RNRhE approved
[root@k8s-master-128 ~]# kubectl certificate approve node-csr-xW7Q15gU-88wPvObRXmia6y-eMyx5dbcMHbGIwINzD0
certificatesigningrequest.certificates.k8s.io/node-csr-xW7Q15gU-88wPvObRXmia6y-eMyx5dbcMHbGIwINzD0 approved

[root@k8s-master-128 ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-K50rilZHx2Gn_MHesRDr_wIJ5hOrg3buFjWuq3RNRhE 7m8s kubelet-bootstrap Approved,Issued
node-csr-xW7Q15gU-88wPvObRXmia6y-eMyx5dbcMHbGIwINzD0 18m kubelet-bootstrap Approved,IssuedCRGvCskSEnG13F4BPrqQbDiR2epHRg4 7m kubelet-bootstrap Approved,Issued # 同意办法证书后,变为:批准状态。

我这里有两个Node节点,都为其颁发证书允许加入集群。

8.2.3、查看Node节点加入k8s集群

1
2
3
4
[root@k8s-master-128 ~]# kubectl get nodes  # 查看集群中的Node节点信息,Ready表示该节点健康,已准备就绪
NAME STATUS ROLES AGE VERSION
172.16.194.129 Ready <none> 54s v1.14.0
172.16.194.130 Ready <none> 63s v1.14.0

8.2.4、查看Node节点签发的证书

1
2
3
4
5
6
7
8
9
[root@k8s-node-129 ~]# ls -lh /opt/kubernetes/cfg/kubelet.kubeconfig  # 已经有这个文件了,你可以cat查看内容细节
-rw------- 1 root root 2.3K 5月 7 17:04 /opt/kubernetes/cfg/kubelet.kubeconfig

# ssl目录生成如下证书
[root@k8s-node-129 ~]# ls -lh /opt/kubernetes/ssl/kubelet*
-rw------- 1 root root 1.3K 5月 7 17:04 /opt/kubernetes/ssl/kubelet-client-2019-05-07-17-04-27.pem
lrwxrwxrwx 1 root root 58 5月 7 17:04 /opt/kubernetes/ssl/kubelet-client-current.pem -> /opt/kubernetes/ssl/kubelet-client-2019-05-07-17-04-27.pem
-rw-r--r-- 1 root root 2.2K 5月 7 16:36 /opt/kubernetes/ssl/kubelet.crt
-rw------- 1 root root 1.7K 5月 7 16:36 /opt/kubernetes/ssl/kubelet.key

整个自签发证书颁发流程完结。

8.3、部署kube-proxy

1、启动的脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@k8s-node-129 ~]# cat kube-proxy.sh
#!/bin/bash
NODE_ADDRESS=${1:-"127.0.0.1"} # kube-proxy的节点IP

cat <<EOF >/opt/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs \
--v=4 \
--hostname-override=${NODE_ADDRESS} \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
EOF

cat <<EOF >/usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

2、启动kube-proxy

1
2
3
4
5
6
7
8
9
10
# 两个Node节点都启动kube-proxy
[root@k8s-node-129 ~]# chmod +x kube-proxy.sh

[root@k8s-node-129 ~]# ./kube-proxy.sh 172.16.194.129
[root@k8s-node-130 ~]# ./kube-proxy.sh 172.16.194.130

# 启动kube-proxy
systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy

3、查看启动状态

1
2
[root@k8s-node-129 ~]# ps -ef|grep kube-proxy
[root@k8s-node-130 ~]# systemctl status kube-proxy

九、部署一个测试示例

9.1、创建一个Nginx pods

1
2
3
4
5
6
7
8
[root@k8s-master-128 ~]# kubectl run --help  # 能看到有很多用法
[root@k8s-master-128 ~]# kubectl run nginx --image=nginx
[root@k8s-master-128 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7db9fccd9b-kckml 0/1 ContainerCreating 0 45s # 第一次创建K8s会在Node节点上拉取镜像,启动时间稍长;
[root@k8s-master-128 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7db9fccd9b-kckml 1/1 Running 0 3m57s

查看pod被分配到了哪个节点上:

1
2
3
[root@k8s-master-128 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-7db9fccd9b-bhldc 1/1 Running 0 7m7s 172.17.64.2 172.16.194.129 <none> <none>

9.2、创建一个services

用于将pod封装成一个service,提供外界访问;

1
2
3
4
5
6
7
[root@k8s-master-128 ~]# kubectl expose --help
[root@k8s-master-128 ~]# kubectl expose deployment nginx --port=80 --target-port=80 --type=NodePort
service/nginx exposed
[root@k8s-master-128 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 22h
nginx NodePort 10.0.0.86 <none> 80:43364/TCP 13s

9.3、访问services

1
2
3
4
5
6
7
8
[root@k8s-node-129 ~]# curl 10.0.0.86   # 对内访问
[root@k8s-master-128 ~]# curl 172.16.194.129:43364 # 对外访问

# 查看访问日志
[root@k8s-master-128 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7db9fccd9b-bhldc 1/1 Running 0 13m
[root@k8s-master-128 ~]# kubectl logs nginx-7db9fccd9b-bhldc

十、部署Web UI(Dashboard)

部署UI这部分网上有很多方法,如果下文有你看不懂的地方,可以不用参考我这里的方案

在kubernetes源码里有关于Dashboard的部署文件:https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dashboard
https://github.com/kubernetes/dashboard

文件的细节这里不展开解释,需要具备一定阅读能力。

10.1、获取YAML文件

从源码包获取文件

1
2
3
4
5
6
7
8
9
[root@k8s-master-128 ~]# cd /root/soft/kubernetes
[root@k8s-master-128 kubernetes]# tar zxf kubernetes-src.tar.gz
[root@k8s-master-128 kubernetes]# ls -lh cluster/addons/dashboard/ # 在源码包里获取这些文件
总用量 32K
-rw-rw-r-- 1 root root 264 3月 21 13:51 dashboard-configmap.yaml
-rw-rw-r-- 1 root root 1.8K 3月 21 13:51 dashboard-controller.yaml
-rw-rw-r-- 1 root root 1.4K 3月 21 13:51 dashboard-rbac.yaml
-rw-rw-r-- 1 root root 551 3月 21 13:51 dashboard-secret.yaml
-rw-rw-r-- 1 root root 322 3月 21 13:51 dashboard-service.yaml

10.2、更改YAML文件

dashboard-controller配置文件里引用的国外镜像源,因此需要更改成国内的源才可启动

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
[root@k8s-master-128 kubernetes]# cd cluster/addons/dashboard
[root@k8s-master-128 dashboard]# vim dashboard-controller.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
addonmanager.kubernetes.io/mode: Reconcile
name: kubernetes-dashboard
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
spec:
priorityClassName: system-cluster-critical
containers:
- name: kubernetes-dashboard
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1 # 将这里的官网镜像更改成国内镜像
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 50m
memory: 100Mi
ports:
- containerPort: 8443
protocol: TCP
args:
# PLATFORM-SPECIFIC ARGS HERE
- --auto-generate-certificates
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
- name: tmp-volume
mountPath: /tmp
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"

dashboard-service.yaml 文件里需要新增:type: NodePort来对外提供服务;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@k8s-master-128 dashboard]# cat dashboard-service.yaml
apiVersion: v1
kind: Service
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
type: NodePort
selector:
k8s-app: kubernetes-dashboard
ports:
- port: 443
targetPort: 8443

10.3、执行部署

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-configmap.yaml
configmap/kubernetes-dashboard-settings created
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-rbac.yaml
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-secret.yaml
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-key-holder created
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-controller.yaml
serviceaccount/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-service.yaml
service/kubernetes-dashboard created

查看部署情况

1
2
3
4
5
6
7
8
9
[root@k8s-master-128 dashboard]# kubectl get pods -n kube-system # 指定命名空间查看Pod
NAME READY STATUS RESTARTS AGE
kubernetes-dashboard-7d5f7c58f5-mn44f 1/1 Running 0 2m44s

[root@k8s-master-128 dashboard]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.0.0.47 <none> 443:34756/TCP 11s # 对外暴露了38276端口,这是能访问UI界面的端口

[root@k8s-master-128 dashboard]# kubectl describe pods/kubernetes-dashboard-7d5f7c58f5-mn44f -n kube-system # 这个命令能查看pods创建过程信息

10.4、UI界面

https://Node节点IP:34756(节点IP之间,能负载均衡,所有随便访问哪个节点IP+端口 都能访问到UI)
https://172.16.194.130:34756

创建Service account并绑定默认cluster-admin管理员集群角色:

1
2
3
4
5
6
7
$ kubectl create serviceaccount dashboard-admin -n kube-system
$ kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
$ kubectl describe secrets -n kube-system $(kubectl get secrets -n kube-system |awk '/dashboard-admin/{print $1}')


输出:
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tOXA4ODciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMmRkODQyNzUtN2M2OS0xMWU5LTkzYzQtMDAwYzI5ZjRkYWE5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.e96EQXa1YIbhRXeckNthFpqNaVw7jrQFqlQTgvVIropqCGZyw_PiM9CF3F5fWwGKZmrojo1_OV4xPNEgxkA1pA-FizwBsUo6flPeVnvkxgaSw2ME6D_z0nj8rIPwyNoDe6x2mLGZNtZi4JNDu5ehoKhZqSF60-rDsoYmlwhY8WVq6uuNVSu086i25q3UU8Wz963TyRZgiywP5fTIbCfikBA5Aj7Mjar9IcCsrPGKeWOm0CxaF_IFPidMWR0scNOZfwdTHC2gU6MUxwMjAQ3KRcC1j7sNQnjXd_mPuJg96SDJsWT8T9IKaMXfXa0etb_b9F5FEZ3qAdFFsKjh-pbJ7g

将这个token放入Dashboard认证,即可登录到UI页面。

十一、部署多Master节点集群

环境规划中,选定172.16.194.127来作为另一个Master节点,主要是实现kube-apiserver的负载均衡,首先对Master节点进行部署,然后使用nginx来进行对kube-apiserver进行负载代理,实现apiserver的高可用;

11.1、部署Master组件

在172.16.194.128服务器上已经部署好了一台Master所需要的组件,这里可以直接将配置文件和启动服务文件直接拉过来即可使用

11.1.1、拷贝文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 配置免密要环境
[root@k8s-master-128 ~]# ssh-copy-id k8s-master-127
# kube-apiserver需要用到etcd的证书,因此我们将128上etcd所有文件拷贝到127
[root@k8s-master-128 ~]# ssh k8s-master-127 mkdir -p /opt/etcd/{bin,cfg,ssl}
[root@k8s-master-128 ~]# scp -r /opt/etcd/{bin,cfg,ssl} k8s-master-127:/opt/etcd/
# 拷贝k8s配置文件
[root@k8s-master-128 ~]# ssh k8s-master-127 mkdir -p /opt/kubernetes/{bin,cfg,logs,ssl}
[root@k8s-master-128 ~]# scp -r /opt/kubernetes/{bin,cfg,ssl} k8s-master-127:/opt/kubernetes/
[root@k8s-master-128 ~]# scp -r /usr/lib/systemd/system/kube-{apiserver,controller-manager,scheduler}.service k8s-master-127:/usr/lib/systemd/system/

# 更改配置信息
[root@k8s-master-127 ~]# cd /opt/kubernetes/cfg/
[root@k8s-master-127 cfg]# vim kube-apiserver
# 将以下参数的IP值更改为对应主机IP,其他的不变
--bind-address=172.16.194.127
--advertise-address=172.16.194.127

# kube-controller-manager与kube-schedule组件的配置不用更改,并且这两个组件都有负载均衡功能:
--leader-elect

11.1.2、启动组件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@k8s-master-127 ~]# systemctl start kube-apiserver
[root@k8s-master-127 ~]# systemctl start kube-controller-manager
[root@k8s-master-127 ~]# systemctl start kube-scheduler
[root@k8s-master-127 ~]# systemctl enable kube-apiserver
[root@k8s-master-127 ~]# systemctl enable kube-controller-manager
[root@k8s-master-127 ~]# systemctl enable kube-scheduler

[root@k8s-master-127 ~]# netstat -lntup|grep kube
tcp 0 0 172.16.194.127:6443 0.0.0.0:* LISTEN 7344/kube-apiserver
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 7362/kube-controlle
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 7344/kube-apiserver
tcp6 0 0 :::10251 :::* LISTEN 7375/kube-scheduler
tcp6 0 0 :::10257 :::* LISTEN 7362/kube-controlle
tcp6 0 0 :::10259 :::* LISTEN 7375/kube-scheduler

启动完成。
配置环境变量和查看集群状态:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@k8s-master-127 ~]# echo "export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/kubernetes/bin" >>/etc/profile
[root@k8s-master-127 ~]# source /etc/profile
[root@k8s-master-127 ~]# which kubectl
/opt/kubernetes/bin/kubectl

[root@k8s-master-127 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7db9fccd9b-bhldc 1/1 Running 0 4d7h
[root@k8s-master-127 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
kubernetes-dashboard-7d5f7c58f5-dvz5m 1/1 Running 5 3d13h
[root@k8s-master-127 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 5d6h
nginx NodePort 10.0.0.86 <none> 80:43364/TCP 4d7h
[root@k8s-master-127 ~]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.0.0.47 <none> 443:34756/TCP 4d6h
[root@k8s-master-127 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}

能看到127节点可以正常接入集群并使用。

11.2、Nginx负载Master节点

官网安装文档:http://nginx.org/en/linux_packages.html

选择自己的OS环境并进行nginx安装,因为实验环境机器有限,这里使用127和128两台k8s的Master节点来安装Nginx来进行实验。与前期环境规划一致。

安装:

1
2
3
4
5
6
7
8
9
[root@k8s-master-127 ~]# vim /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key

[root@k8s-master-127 ~]# yum install nginx -y

启动nginx:

1
2
[root@k8s-master-127 ~]#  systemctl start nginx
[root@k8s-master-127 ~]# systemctl enable nginx

配置:
nginx配置目录在/etc/nginx/下,主配置文件nginx.conf,生产环境需要对这个配置文件进行调优,而这里做实验的话默认配置就行。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@k8s-master-127 nginx]# cat nginx.conf
···略···
events {
worker_connections 1024;
}

# stream模块用于代理4层请求,且不能存在http模块内
stream {
log_format main '$remote_addr $upstream_addr $time_local $status';
access_log /var/log/nginx/k8s-apiserver.com.access_log main;
error_log /var/log/nginx/k8s-apiserver.com.error_log warn;
upstream k8s-apiserver {
server 172.16.194.127:6443;
server 172.16.194.128:6443;
}
server {
listen 172.16.194.127:6444;
proxy_pass k8s-apiserver; # 代理4层请求,不用加http://
}
}

http {
···略···
}
···略···

查看启动:

1
2
[root@k8s-master-127 nginx]# netstat -lntup|grep 6444
tcp 0 0 172.16.194.127:6444 0.0.0.0:* LISTEN 7061/nginx: master

因为本机有6443,想要与代理理共存的话,nginx更改下端口即可。

11.3、Node节点配置使用

更改Node节点的配置:

1
2
3
4
5
6
7
8
9
10
11
[root@k8s-node-129 ~]# cd /opt/kubernetes/cfg/
[root@k8s-node-129 cfg]# grep "6443" ./* # 将这三个文件的地址更改为api-server的代理地址
./bootstrap.kubeconfig: server: https://172.16.194.128:6443
./kubelet.kubeconfig: server: https://172.16.194.128:6443
./kube-proxy.kubeconfig: server: https://172.16.194.128:6443

# 更改后:(所有节点都需要更改)
[root@k8s-node-129 cfg]# grep "6444" ./*
./bootstrap.kubeconfig: server: https://172.16.194.127:6444
./kubelet.kubeconfig: server: https://172.16.194.127:6444
./kube-proxy.kubeconfig: server: https://172.16.194.127:6444

重启Node节点组件:

1
2
[root@k8s-node-129 cfg]# systemctl restart kubelet
[root@k8s-node-129 cfg]# systemctl restart kube-proxy

查看nginx代理日志:

1
2
3
4
5
6
[root@k8s-master-127 nginx]# tail -f k8s-apiserver.com.access_log
172.16.194.130 172.16.194.127:6443 13/May/2019:03:12:07 +0800 200
172.16.194.130 172.16.194.127:6443 13/May/2019:03:12:07 +0800 200
172.16.194.130 172.16.194.128:6443 13/May/2019:03:12:13 +0800 200
172.16.194.130 172.16.194.128:6443 13/May/2019:03:12:13 +0800 200
172.16.194.130 172.16.194.128:6443 13/May/2019:03:12:13 +0800 200

代理成功。

查看集群Node节点状态:

1
2
3
4
[root@k8s-master-127 nginx]# kubectl get node
NAME STATUS ROLES AGE VERSION
172.16.194.129 Ready <none> 5d10h v1.14.0
172.16.194.130 Ready <none> 5d10h v1.14.0

只有代理连接是正常的,集群节点才会是Ready状态,否则就是上面配置环境有问题。

11.4、Keepalived实现Nginx高可用

目前只部署了一个Nginx节点,只有一个节点就存在单点故障的风险,一旦这个节点挂掉,那么整个集群将会不可用,
市场有些可选的高可用开源解决方案,如:Keepalived、heartbeat等,这里我们选择使用前者。

大致的思路就是:
1、将上面的Nginx代理节点配置多个(两个以上);
2、通过Keepalived控制VIP来进行漂移,如果Nginx代理节点挂掉后,Keepalived会将VIP漂移到正常的Nginx代理节点上,从而实现集群高可用,提高k8s集群的健壮性。

127节点已经安装和配置好Nginx,128节点按照上面的配置即可。

11.4.1、Keepalived安装与配置

127Master节点部署:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
[root@k8s-master-127 ~]# yum install keepalived -y
[root@k8s-master-127 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}

vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight 2
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 50
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.194.111
}
track_script {
check_nginx
}
}

检查nginx的脚本:

1
2
3
4
5
6
7
[root@k8s-master-127 ~]# cat /etc/keepalived/check_nginx.sh
count=$(ps -C nginx --no-header |wc -l)

if [ $count -eq 0 ];then
systemctl stop keepalived
fi
[root@k8s-master-127 ~]# chmod +x /etc/keepalived/check_nginx.sh

启动主节点的Keepalived:

1
2
[root@k8s-master-127 ~]# systemctl enable keepalived
[root@k8s-master-127 ~]# systemctl start keepalived

注意:Keepalived不可设置成开机自启动,一旦发生VIP漂移,则需要运维工程师介入排查问题,如果设置成开启启动,有可能会给业务带来二次伤害。

查看Keepalived绑定的VIP:

1
2
3
4
5
6
7
8
9
[root@k8s-master-127 ~]# ip a
···略···
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:41:09:16 brd ff:ff:ff:ff:ff:ff
inet 172.16.194.127/24 brd 172.16.194.255 scope global noprefixroute eth0 # 这是本机IP
valid_lft forever preferred_lft forever
inet 172.16.194.111/32 scope global eth0 # 这是VIP
valid_lft forever preferred_lft forever
···略···

128Backup节点部署:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
[root@k8s-master-128 nginx]# yum install keepalived -y
[root@k8s-master-128 nginx]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}

vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight 2
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 50 # 这个参数要与Master一样,不然不能识别及通讯
priority 50
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.194.111/24
}
track_script {
check_nginx
}
}

检查nginx的脚本:

1
2
3
4
5
6
7
[root@k8s-master-128 nginx]# vim /etc/keepalived/check_nginx.sh
count=$(ps -C nginx --no-header |wc -l)

if [ $count -eq 0 ];then
systemctl stop keepalived
fi
[root@k8s-master-128 nginx]# chmod +x /etc/keepalived/check_nginx.sh

启动备节点的Keepalived:

1
2
[root@k8s-master-128 nginx]# systemctl enable keepalived
[root@k8s-master-128 nginx]# systemctl start keepalived

11.4.2、故障模拟测试

现在状态:127Master节点和128Backup节点都已经启动,VIP绑定在127的eth0网卡上;
理论上,将nginx停掉,VIP会从当前节点消失,漂移到备节点上,那来看实际情况:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 模拟故障停掉nginx,看看VIP是否会漂移到备用节点上
[root@k8s-master-127 ~]# systemctl stop nginx
[root@k8s-master-127 ~]# ip a
···略···
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:41:09:16 brd ff:ff:ff:ff:ff:ff
inet 172.16.194.127/24 brd 172.16.194.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
···略···

# 查看备节点是否有VIP
[root@k8s-master-128 ~]# ip a
···略···
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:f4:da:a9 brd ff:ff:ff:ff:ff:ff
inet 172.16.194.128/24 brd 172.16.194.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 172.16.194.111/24 scope global secondary eth0
valid_lft forever preferred_lft forever
···略···

跟理论情况一样,因故障后,VIP漂移正常,那也就意味着负载均衡加高可用集群搭建完成。

11.5、k8s集群接入负载均衡使用

kube-apiserver的负载均衡主备节点都已经搭建完成并测试成功,接下来就需要配置下k8s的Node组件使用即可

11.5.1、更改Nginx配置

127和128节点的nginx更改如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@k8s-master-127 ~]# vim /etc/nginx/nginx.conf
stream {
log_format main '$remote_addr $upstream_addr $time_local $status';
access_log /var/log/nginx/k8s-apiserver.com.access_log main;
error_log /var/log/nginx/k8s-apiserver.com.error_log warn;
upstream k8s-apiserver {
server 172.16.194.127:6443;
server 172.16.194.128:6443;
}
server {
listen 0.0.0.0:6444; # 将这里更改为监听所有网段
proxy_pass k8s-apiserver;
}
}

重新加载nginx,使配置生效:

1
2
3
4
5
6
7
[root@k8s-master-127 ~]# systemctl reload nginx
[root@k8s-master-127 ~]# netstat -lntup|grep nginx
tcp 0 0 0.0.0.0:6444 0.0.0.0:* LISTEN 20523/nginx: master

[root@k8s-master-128 ~]# systemctl reload nginx
[root@k8s-master-128 ~]# netstat -lntup|grep nginx
tcp 0 0 0.0.0.0:6444 0.0.0.0:* LISTEN 21604/nginx: master

11.5.2、Node节点配置使用

更改api-server地址的配置

1
2
3
4
5
6
7
8
9
[root@k8s-node-129 cfg]# grep '111' ./*
./bootstrap.kubeconfig: server: https://172.16.194.111:6444
./kubelet.kubeconfig: server: https://172.16.194.111:6444
./kube-proxy.kubeconfig: server: https://172.16.194.111:6444

[root@k8s-node-130 cfg]# grep '111' ./*
./bootstrap.kubeconfig: server: https://172.16.194.111:6444
./kubelet.kubeconfig: server: https://172.16.194.111:6444
./kube-proxy.kubeconfig: server: https://172.16.194.111:6444

重启kubelet和kube-proxy组件:

1
2
3
4
[root@k8s-node-129 cfg]# systemctl restart kubelet
[root@k8s-node-129 cfg]# systemctl restart kube-proxy
[root@k8s-node-130 cfg]# systemctl restart kubelet
[root@k8s-node-130 cfg]# systemctl restart kube-proxy

查看127上的代理日志:

1
2
3
4
5
[root@k8s-master-127 ~]# tail -f /var/log/nginx/k8s-apiserver.com.access_log
172.16.194.130 172.16.194.127:6443 13/May/2019:05:32:55 +0800 200
172.16.194.129 172.16.194.128:6443 13/May/2019:05:32:55 +0800 200
172.16.194.129 172.16.194.127:6443 13/May/2019:05:32:55 +0800 200
172.16.194.130 172.16.194.128:6443 13/May/2019:05:32:55 +0800 200

nginx默认是轮训代理。

11.5.3、K8集群状态

能看到Node节点状态是Ready就没事

1
2
3
4
[root@k8s-master-128 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
172.16.194.129 Ready <none> 7d v1.14.0
172.16.194.130 Ready <none> 7d v1.14.0

遇到的坑:
1、在生成server证书的时候,需要将VIP填写进去并生成证书,不然就会报错如下:

1
2
# 启动kubelet查看/var/log/messages
certificate is valid for 127.0.0.1, 10.0.0.1, 172.16.194.127, 172.16.194.128, 172.16.194.129, 172.16.194.130, not 172.16.194.111

解决办法:将server证书重新生成(填写VIP进去),并重启api-server进程即可解决。

十二、部署集群内部DNS解析服务(CoreDNS)

在kubernetes1.12之后的版本中,使用了CoreDNS作为默认的DNS;
官网:https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/coredns/

12.1、下载CoreDNS部署文件

https://github.com/coredns/deployment/tree/master/kubernetes

1
2
3
[root@k8s-master-128 ~]# mkdir coredns && cd coredns
[root@k8s-master-128 coredns]# wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
[root@k8s-master-128 coredns]# wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh

deploy.sh是一个便捷的脚本,用于生成用于在当前运行标准kube-dns的集群上运行CoreDNS的清单。使用coredns.yaml.sed文件作为模板,它创建一个ConfigMap和一个CoreDNS deployment,然后更新 Kube-DNS service selector以使用CoreDNS deployment。 通过重新使用现有服务,服务请求不会中断。

12.2、部署CoreDNS

1
2
3
4
5
6
7
8
9
10
11
12
[root@k8s-master-128 coredns]# chmod +x deploy.sh
[root@k8s-master-128 coredns]# ./deploy.sh -i 10.0.0.2 >coredns.yaml
tips:
少了个jq命令:yum install -y jq

[root@k8s-master-128 coredns]# kubectl create -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created

查看生成的文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182

apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/name: "CoreDNS"
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
spec:
priorityClassName: system-cluster-critical
serviceAccountName: coredns
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
nodeSelector:
beta.kubernetes.io/os: linux
containers:
- name: coredns
image: coredns/coredns:1.5.0
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
readOnly: true
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.0.0.2
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
- name: metrics
port: 9153
protocol: TCP

早在部署kubelet的时候就通过--cluster-dns=10.0.0.2 --cluster-domain=cluster.local参数指定了dns地址和解析的域,因此直接部署即可使用,如果kubelet启动参数里没有配置这两个dns参数的话,上面部署coredns之后还需要将所有kubelet重新添加配置并重启进程。

12.3、查看CoreDNS状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[root@k8s-master-128 coredns]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-55f46dd959-q47j5 1/1 Running 0 3m59s
coredns-55f46dd959-vcj4w 1/1 Running 0 3m59s
kubernetes-dashboard-7d5f7c58f5-xqvmh 1/1 Running 1 94m

[root@k8s-master-128 coredns]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.0.0.2 <none> 53/UDP,53/TCP,9153/TCP 4m12s
kubernetes-dashboard NodePort 10.0.0.47 <none> 443:34756/TCP 14d

[root@k8s-master-128 coredns]# kubectl get deploy -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
coredns 2/2 2 2 5m32s
kubernetes-dashboard 1/1 1 1 14d

[root@k8s-master-128 coredns]# kubectl get ep -n kube-system kube-dns
NAME ENDPOINTS AGE
kube-dns 172.17.17.4:53,172.17.50.2:53,172.17.17.4:53 + 3 more... 10m

[root@k8s-master-128 coredns]# kubectl -n kube-system get configmap coredns
NAME DATA AGE
coredns 1 12m

CoreDNS所创建的服务均已正常启动。

12.4、测试CoreDNS

在安装完Kubernetes cluster环境后,如何验证coreDNS是否在正常工作?这是一项很重要的工作,将会影响将来在容器中部署的服务能否被正常调用。

我们可以通过创建一个busybox 的pod,再在busybox里去解析服务名的方式来验证coreDNS是否正常工作。

具体可参考kubernetes官方文档《Debugging DNS Resolution》

busybox的yaml文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@k8s-master-128 coredns]# cat busybox.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: busybox:1.28
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always

创建Buxybox pod:

1
2
3
4
5
[root@k8s-master-128 coredns]# kubectl create -f busybox.yaml
pod/busybox created
[root@k8s-master-128 coredns]# kubectl get pods busybox
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 26s

busybox的resolv.conf内容:

1
2
3
4
[root@k8s-master-128 coredns]# kubectl exec busybox cat /etc/resolv.conf
nameserver 10.0.0.2
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

在busybox 的pod里解析不同名字空间的服务:
解析规则是:my-svc.my-namespace.svc.cluster.local
因此每个空间的服务都需要指明自己所在的名字空间才可进行访问

1
2
3
4
5
6
7
8
9
10
11
12
[root@k8s-master-128 coredns]# kubectl exec -it busybox nslookup kubernetes.default
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: kubernetes
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
[root@k8s-master-128 coredns]# kubectl exec -it busybox nslookup kube-dns.kube-system
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: kube-dns.kube-system
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

实验证明,用 my-svc.my-namespace.svc.cluster.local方法即可访问服务。

在busybox 的pod里解析公网:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@k8s-master-128 coredns]# kubectl exec -it busybox nslookup qq.com
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: qq.com
Address 1: 111.161.64.48 dns48.online.tj.cn
Address 2: 111.161.64.40 dns40.online.tj.cn
[root@k8s-master-128 coredns]# kubectl exec -it busybox nslookup www.baidu.com
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: www.baidu.com
Address 1: 220.181.38.150
Address 2: 220.181.38.149

在busybox 的pod里解析外部IP地址 ,按照前文CoreDNS的配置,是通过pod所在node上的/etc/resolv.conf 来代理解析的。
通过以上例子可见,coredns工作正常。coredns既可以管理新生成的service的域名,又可以解析出外部域名。

参考文章:

-------------本文结束感谢您的阅读-------------