Kubernetes 高可用集群搭建(一) | ETCD集群搭建

环境准备#

服务器准备及系统配置调整, 这个参考前一篇基础配置.
ssh配置

1
> ssh-keygen -b 2048 -t rsa

以上环境的准备可在一台vm上创建好后, 后面只需要克隆这台就可以了.

containerd

这里不在使用 Docker 作为cri的依赖. 而是使用containerd.

1
> sudo apt-get update && sudo apt-get install -y containerd
2
> sudo mkdir -p /etc/containerd
3
> sudo containerd config default | sudo tee /etc/containerd/config.toml

1
* cni配置

1
> sudo mkdir -p /etc/cni/net.d
2
> sudo cat <<EOF | sudo tee /etc/cni/net.d/99-loopback.conf
3
{
4
    "cniVersion": "0.3.1",
5
    "type": "loopback"
6
}
7
EOF

1
* 镜像加速配置及默认pause镜像配置

1
# /etc/containerd/config.toml 大概91行
2
    [plugins."io.containerd.grpc.v1.cri".registry]
3
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
4
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
5
          endpoint = ["https://wlij5bh2.mirror.aliyuncs.com"]
6
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
7
          endpoint = ["https://registry.cn-hangzhou.aliyuncs.com/morse_k8s"]
8

9
# sandbox_image 修改
10
sandbox_image = "k8s.gcr.io/pause:3.1"
11
# 改为
12
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/morse_k8s/pause:3.1"
13

14

15
# systemd
16
# 结合 runc 使用 systemd cgroup 驱动，在 /etc/containerd/config.toml 中设置
17
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
18
  ...
19
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
20
    SystemdCgroup = true

1
* containerd的几个重要命令

1
ctr image ls
2
# 简写: ctr i ls
3
ctr image remove docker.io/library/hello-world:latest
4
> ctr container list
5
# 简写: ctr c list
6
ctr c remove demo
7
# 命名空间
8
> sudo ctr ns ls
9
# 配置
10
> sudo containerd config default

怎么查其他信息, 这个时候我们可以用这个工具crictl.

-n: 命名空间, kubernetes的命名空间为 k8s.io. 注意 -n 的位置, 放在ctr后面 * 镜像准备

1
k8s.gcr.io/pause:3.1
2
k8s.gcr.io/etcd:3.4.13-0
3
# registry.cn-hangzhou.aliyuncs.com/morse_k8s/etcd:3.4.13-0

参考 Container runtimes | Kubernetes Containerd 使用教程 – 云原生实验室 - Kubernetes|Docker|Istio|Envoy|Hugo|Golang|云原生容器运行时 | Kubernetes containerd/ops.md at master · containerd/containerd · GitHub https://github.com/containerd/containerd/blob/master/docs/cri/config.md

Etcd集群搭建#

前置条件#

2 GB 或更多的 RAM, 2 CPU 核或更多
集群中的所有机器的网络彼此均能相互连接(公网和内网都可以)
节点之中不可以有重复的主机名、MAC 地址或 product_uuid。
禁用交换分区。为了保证 kubelet 正常工作，你必须禁用交换分区
开启机器上的某些端口。(可以关闭防火墙)

sudo cat /sys/class/dmi/id/product_uuid用这个命令校验机器ID

这里的ProductId在虚拟机copy的时候可能会重复, 记得清理

修改hosts

1
# 每台机器添加彼此的host解析
2
192.168.10.5 vm5.fuxiao.dev  vm5
3
192.168.10.6 vm6.fuxiao.dev  vm6
4
192.168.10.7 vm7.fuxiao.dev  vm7

安装kubeadm, kubelet (每台) 因为使用kubelet来管理 etcd, 即通过容器运行Etcd. 所以需要先安装kubelet. CRI(即上面的containerd). 因为kubeadm 不能帮你安装或者管理 kubelet 或 kubectl. 所以这里kubectl, kubelet, kubeadm就一起安装了.

1
sudo apt-get install -y kubelet kubeadm kubectl

将 kubelet 配置为 etcd 的服务管理器。(每台)

1
> sudo cd /etc/systemd/system/kubelet.service.d
2
> # 修改10-kubeadm.conf文件
3
> # 屏蔽文件中的Environment, Environment 参数
4
> sudo mkdir -p /var/lib/kubelet
5
> sudo cat<<EOF|sudo tee /var/lib/kubelet/kubeadm-flags.env
6
KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd --pod-manifest-path=/etc/kubernetes/manifests --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --runtime-request-timeout=15m"
7
EOF
8
> sudo systemctl daemon-reload
9
> sudo systemctl restart kubelet

当CRI非Docker时, 需要添加以下启动参数.

1
--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --runtime-request-timeout=15m

不同的CRI sock 文件位置不同

1
Docker: /var/run/docker.sock
2
containerd: /run/containerd/containerd.sock
3
CRI-O: /var/run/crio/crio.sock

只是创建, kubelet这一步并不能运行

配置脚本

1
# cat create_etcd.sh
2

3
#!/bin/bash
4
# 使用 IP 或可解析的主机名替换 HOST0、HOST1 和 HOST2
5
export HOST0=192.168.10.5
6
export HOST1=192.168.10.6
7
export HOST2=192.168.10.7
8

9
# 创建临时目录来存储将被分发到其它主机上的文件
10
rm -rf /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/
11
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/
12

13
ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
14
NAMES=("vm5" "vm6" "vm7")
15
# 注意这里的names中的值与下面initial-cluster中参数的key保持一致
16

17
for i in "${!ETCDHOSTS[@]}"; do
18
HOST=${ETCDHOSTS[$i]}
19
NAME=${NAMES[$i]}
20
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
21
apiVersion: "kubeadm.k8s.io/v1beta2"
22
kind: ClusterConfiguration
23
etcd:
24
    local:
25
        serverCertSANs:
26
        - "${HOST}"
27
        peerCertSANs:
28
        - "${HOST}"
29
        extraArgs:
30
            initial-cluster: vm5=https://${ETCDHOSTS[0]}:2380,vm6=https://${ETCDHOSTS[1]}:2380,vm7=https://${ETCDHOSTS[2]}:2380
31
            initial-cluster-state: new
32
            name: ${NAME}
33
            listen-peer-urls: https://${HOST}:2380
34
            listen-client-urls: https://${HOST}:2379
35
            advertise-client-urls: https://${HOST}:2379
36
            initial-advertise-peer-urls: https://${HOST}:2380
37
EOF
38
done

注意: 1. NAMES与initial-cluster的key要匹配, initial-cluster中多个值不能用空格 2. 执行脚本: . create_etcd.sh, (注意执行命令前的那个点, 没有那个点导不出shell中的环境变量)

etcd证书

1
sudo kubeadm init phase certs etcd-ca
2
# ls -la /etc/kubernetes/pki/etcd/
3
#  /etc/kubernetes/pki/etcd/ca.key

为每个成员创建证书

1
sudo kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
2
sudo kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
3
sudo kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
4
sudo kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
5
sudo cp -R /etc/kubernetes/pki /tmp/${HOST2}/
6
# 清理不可重复使用的证书
7
sudo find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
8

9
sudo kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
10
sudo kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
11
sudo kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
12
sudo kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
13
sudo cp -R /etc/kubernetes/pki /tmp/${HOST1}/
14
sudo find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
15

16
sudo kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
17
sudo kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
18
sudo kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
19
sudo kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
20
# 不需要移动 certs 因为它们是给 HOST0 使用的
21

22
# 清理不应从此主机复制的证书
23
sudo find /tmp/${HOST2} -name ca.key -type f -delete
24
sudo find /tmp/${HOST1} -name ca.key -type f -delete

复制证书和 kubeadm 配置

1
#!/bin/bash
2
export USER=ubuntu
3
export HOST1=192.168.10.22
4
export HOST2=192.168.10.23
5
ETCDHOSTS=(${HOST1} ${HOST2})
6
for i in "${!ETCDHOSTS[@]}"; do
7
  sudo scp -r /tmp/${ETCDHOSTS[$i]}/* ${USER}@${ETCDHOSTS[$i]}:
8
done

1
USER=ubuntu
2
HOST=${HOST1}
3
ssh ${USER}@${HOST}
4
USER@HOST $ sudo -Es && mkdir /etc/kubernetes/pki
5
root@HOST $ chown -R root:root pki
6
root@HOST $ mv pki /etc/kubernetes/

确保已经所有预期的文件都存在
创建静态 Pod 清单

1
root@HOST0 $ sudo kubeadm init phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
2
root@HOST1 $ sudo kubeadm init phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
3
root@HOST2 $ sudo kubeadm init phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml

修改每个节点中etcd静态清单配置

1
# 将image的值改为下面的值
2
image: registry.cn-hangzhou.aliyuncs.com/morse_k8s/etcd:3.4.13-0

可选：检查群集运行状况

1
HOST=192.168.10.5
2
sudo etcdctl \
3
--cert /etc/kubernetes/pki/etcd/peer.crt \
4
--key /etc/kubernetes/pki/etcd/peer.key \
5
--cacert /etc/kubernetes/pki/etcd/ca.crt \
6
--endpoints https://${HOST}:2379 endpoint health --cluster
7
...
8
# 注意: 正常可以看到全部主机列表
9
https://[HOST0 IP]:2379 is healthy: successfully committed proposal: took = 16.283339ms
10
https://[HOST1 IP]:2379 is healthy: successfully committed proposal: took = 19.44402ms
11
https://[HOST2 IP]:2379 is healthy: successfully committed proposal: took = 35.926451ms

etcdctl 命令可到etcd的github release页面下载

扩展#

crictl工具

初始配置如下:

1
sudo touch /etc/crictl.yaml
2
sudo cat << EOF | sudo tee /etc/crictl.yaml
3
runtime-endpoint: /run/containerd/containerd.sock
4
EOF

使用方式基本等同docker的命令.

1
> sudo crictl pods
2
> sudo crictl images
3
> sudo crictl ps

参考:

SAN: Subject Alternative Names : 可选附加主题备用名称, 用于 API Server 服务证书的可选附加主题备用名称 #01 Blog/云原生# #01 Blog/Kubernetes#