CentOS 7 ETCD集群配置大全

本次环境，是用于k8s集群中的etcd，由于二进制文档对etcd并没有进行详细的介绍安装配置，故抽出时间进行etcd集群深入总结。本次主要是针对etcd静态发现和DNS动态发现两种，并结合自签的TLS证书来创建集群。

前言

Etcd 是 CoreOS 基于 Raft 开发的分布式 key-value 存储，可用于服务发现、共享配置以及一致性保障（如数据库选主、分布式锁等）

本次环境，是用于k8s集群，由于在二进制部署 k8s 中，由于 Etcd 集群导致各种各样的问题，特意抽出时间来研究 Etcd 集群。

k8s

k8s

Kubernetes 1.14 二进制集群安装

g-1新闻联播老司机
  • 1
  • 16.9k

Etcd 集群配置分为三种:

  1. 静态发现
  2. Etcd 动态发现
  3. DNS 动态发现 通过DNS的SRV解析动态发现集群

本次主要基于 静态发现 和 DNS动态发现 两种，并结合自签的TLS证书来创建集群。

环境准备

此环境实际用于 k8s 中的ETCD集群使用，用于本次文档

在三台机器上均执行

  1. [root@node01 ~]# yum install etcd –y
  2. [root@node01 ~]# rpm –qa etcd
  3. etcd–3.3.11–2.el7.centos.x86_64

创建Etcd所需目录，在三台机器上均执行

  1. mkdir /data/k8s/etcd/{data,wal} –p
  2. mkdir –p /etc/kubernetes/cert
  3. chown –R etcd.etcd /data/k8s/etcd

静态集群

配置

node01 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“http://192.168.1.91:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“http://192.168.1.91:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd1”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“http://192.168.1.91:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“http://192.168.1.91:2379”
  14.  
  15. ETCD_INITIAL_CLUSTER=“etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380”
  16. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  17. ETCD_INITIAL_CLUSTER_STATE=“new”

node02 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“http://192.168.1.92:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“http://192.168.1.92:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd2”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“http://192.168.1.92:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“http://192.168.1.92:2379”
  14.  
  15. ETCD_INITIAL_CLUSTER=“etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380”
  16. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  17. ETCD_INITIAL_CLUSTER_STATE=“new”

node03 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“http://192.168.1.93:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“http://192.168.1.93:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd3”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“http://192.168.1.93:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“http://192.168.1.93:2379”
  14.  
  15. ETCD_INITIAL_CLUSTER=“etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380”
  16. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  17. ETCD_INITIAL_CLUSTER_STATE=“new”

启动测试

  1. [root@node01 etcd]# systemctl start etcd
  2. [root@node01 etcd]# systemctl status etcd
  3. ● etcd.service – Etcd Server
  4. Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
  5. Active: active (running) since Thu 2019–11–07 09:28:54 CST; 5s ago
  6. Main PID: 1546 (etcd)
  7. Tasks: 8
  8. Memory: 41.3M
  9. CGroup: /system.slice/etcd.service
  10. └─1546 /usr/bin/etcd —name=etcd1 —data–dir=/data/k8s/etcd/data —listen–client–urls=http://192.168.1.91:2379
  11.  
  12. Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 [term: 1] received a MsgVote message with higher term from 9c64fba479c5e94 [term: 2]
  13. Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 became follower at term 2
  14. Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 [logterm: 1, index: 3, vote: 0] cast MsgVote for 9c64fba479c5e94 [logterm: 1, index: 3] at term 2
  15. Nov 07 09:28:54 node01.k8s.com etcd[1546]: raft.node: 3b8b38de05e2c497 elected leader 9c64fba479c5e94 at term 2
  16. Nov 07 09:28:54 node01.k8s.com etcd[1546]: published {Name:etcd1 ClientURLs:[http://192.168.1.91:2379]} to cluster 19456f0bfd57284e
  17. Nov 07 09:28:54 node01.k8s.com etcd[1546]: ready to serve client requests
  18. Nov 07 09:28:54 node01.k8s.com etcd[1546]: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!
  19. Nov 07 09:28:54 node01.k8s.com systemd[1]: Started Etcd Server.
  20. Nov 07 09:28:54 node01.k8s.com etcd[1546]: set the initial cluster version to 3.3
  21. Nov 07 09:28:54 node01.k8s.com etcd[1546]: enabled capabilities for version 3.3

查看 /var/log/message 日志中，会有日下体现：

  1. Nov 7 09:28:53 node02 etcd: added member 9c64fba479c5e94 [http://192.168.1.92:2380] to cluster 19456f0bfd57284e
  2. Nov 7 09:28:53 node02 etcd: added member 3b8b38de05e2c497 [http://192.168.1.91:2380] to cluster 19456f0bfd57284e
  3. Nov 7 09:28:53 node02 etcd: added member 76ea8679db7365b3 [http://192.168.1.93:2380] to cluster 19456f0bfd57284e

查看集群状态

  1. [root@node01 etcd]# ETCDCTL_API=3 etcdctl —endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 endpoint health
  2. http://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.103545ms
  3. http://192.168.1.93:2379 is healthy: successfully committed proposal: took = 2.122478ms
  4. http://192.168.1.91:2379 is healthy: successfully committed proposal: took = 2.690215ms
  5. [root@node01 etcd]# etcdctl —endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 cluster-health
  6. member 9c64fba479c5e94 is healthy: got healthy result from http://192.168.1.92:2379
  7. member 3b8b38de05e2c497 is healthy: got healthy result from http://192.168.1.91:2379
  8. member 76ea8679db7365b3 is healthy: got healthy result from http://192.168.1.93:2379
  9. cluster is healthy

生成TLS证书

使用自签证书

CA(Certificate Authority)是自签名的根证书，用来签名后续创建的其他证书。本文章使用CloudFlare的PKI工具cfssl创建所有证书。

etcd证书创建

整个证书的创建过程均在 node01 上操作;

安装cfssl工具集

  1. mkdir –p /opt/k8s/cert && cd /opt/k8s
  2. wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
  3. mv cfssl_linux–amd64 /opt/k8s/bin/cfssl
  4. wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
  5. mv cfssljson_linux–amd64 /opt/k8s/bin/cfssljson
  6. wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
  7. mv cfssl–certinfo_linux–amd64 /opt/k8s/bin/cfssl–certinfo
  8. chmod +x /opt/k8s/bin/*
  9. echo ‘export PATH=/opt/k8s/bin:$PATH’ >> ~/.bash_profile
  10. source ~/.bash_profile

生成证书

创建根证书 (CA)

CA证书是集群所有节点共享的，只需要创建一个CA证书，后续创建的所有证书都是由它签名

创建配置文件

CA配置文件用于配置根证书的使用场景(profile)和具体参数

(usage、过期时间、服务端认证、客户端认证、加密等)

  1. cd /opt/k8s/work
  2. cat > ca–config.json <<EOF
  3. {
  4. “signing”: {
  5. “default”: {
  6. “expiry”: “87600h”
  7. },
  8. “profiles”: {
  9. “kubernetes”: {
  10. “usages”: [
  11. “signing”,
  12. “key encipherment”,
  13. “server auth”,
  14. “client auth”
  15. ],
  16. “expiry”: “87600h”
  17. }
  18. }
  19. }
  20. }
  21. EOF
  22.  
  23.  
  24. ######################
  25. signing 表示该证书可用于签名其它证书，生成的ca.pem证书找中CA=TRUE
  26. server auth 表示client可以用该证书对server提供的证书进行验证
  27. client auth 表示server可以用该证书对client提供的证书进行验证

创建证书签名请求文件

  1. cd /opt/k8s/work
  2. cat > ca–csr.json <<EOF
  3. {
  4. “CN”: “kubernetes”,
  5. “key”: {
  6. “algo”: “rsa”,
  7. “size”: 2048
  8. },
  9. “names”: [
  10. {
  11. “C”: “CN”,
  12. “ST”: “BeiJing”,
  13. “L”: “BeiJing”,
  14. “O”: “k8s”,
  15. “OU”: “4Paradigm”
  16. }
  17. ],
  18. “ca”: {
  19. “expiry”: “876000h”
  20. }
  21. }
  22. EOF
  23.  
  24.  
  25.  
  26. #######################
  27. CN CommonName,kube–apiserver从证书中提取该字段作为请求的用户名(User Name)，浏览器使用该字段验证网站是否合法
  28. O Organization,kube–apiserver 从证书中提取该字段作为请求用户和所属组(Group)
  29. kube–apiserver将提取的User、Group作为RBAC授权的用户和标识

生成CA证书和私钥

  1. cd /opt/k8s/work
  2. cfssl gencert –initca ca–csr.json | cfssljson –bare ca
  3. ls ca*

创建etcd证书和私钥

  1. cd /opt/k8s/work
  2. cat > etcd–csr.json <<EOF
  3. {
  4. “CN”: “etcd”,
  5. “hosts”: [
  6. “127.0.0.1”,
  7. “192.168.1.91”,
  8. “192.168.1.92”,
  9. “192.168.1.93”,
  10. “k8s.com”,
  11. “etcd1.k8s.com”,
  12. “etcd2.k8s.com”,
  13. “etcd3.k8s.com”
  14. ],
  15. “key”: {
  16. “algo”: “rsa”,
  17. “size”: 2048
  18. },
  19. “names”: [
  20. {
  21. “C”: “CN”,
  22. “ST”: “BeiJing”,
  23. “L”: “BeiJing”,
  24. “O”: “k8s”,
  25. “OU”: “4Paradigm”
  26. }
  27. ]
  28. }
  29. EOF
  30.  
  31. #host字段指定授权使用该证书的etcd节点IP或域名列表，需要将etcd集群的3个节点都添加其中

生成证书和私钥

  1. cd /opt/k8s/work
  2.  
  3. cfssl gencert –ca=/opt/k8s/work/ca.pem \
  4. –ca–key=/opt/k8s/work/ca–key.pem \
  5. –config=/opt/k8s/work/ca–config.json \
  6. –profile=kubernetes etcd–csr.json | cfssljson –bare etcd
  7.  
  8. ls etcd*pem –l
  9. –rw——- 1 root root 1675 Nov 7 09:52 etcd–key.pem
  10. –rw–r—r— 1 root root 1444 Nov 7 09:52 etcd.pem

etcd 使用的TLS证书创建完成

分发证书到各节点上

要做所有节点上创建对应的目录

  1. mkdir /data/k8s/etcd/{data,wal} –p
  2. mkdir –p /etc/kubernetes/cert
  3. chown –R etcd.etcd /data/k8s/etcd

分发证书

  1. cd /opt/k8s/work
  2. scp ca*.pem ca–config.json 192.168.1.91:/etc/kubernetes/cert
  3. scp ca*.pem ca–config.json 192.168.1.92:/etc/kubernetes/cert
  4. scp ca*.pem ca–config.json 192.168.1.93:/etc/kubernetes/cert
  5. scp etcd*pem 192.168.1.91:/etc/etcd/cert/
  6. scp etcd*pem 192.168.1.92:/etc/etcd/cert/
  7. scp etcd*pem 192.168.1.93:/etc/etcd/cert/

在所有节点上执行:

  1. chown –R etcd.etcd /etc/etcd/cert

静态TLS集群

etcd 配置

node01 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“https://192.168.1.91:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“https://192.168.1.91:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd1”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://192.168.1.91:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“https://192.168.1.91:2379”
  14. ETCD_INITIAL_CLUSTER=“etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380”
  15. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  16. ETCD_INITIAL_CLUSTER_STATE=“new”
  17.  
  18. ETCD_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  19. ETCD_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  20. ETCD_CLIENT_CERT_AUTH=“true”
  21. ETCD_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  22. ETCD_AUTO_TLS=“true”
  23. ETCD_PEER_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  24. ETCD_PEER_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  25. ETCD_PEER_CLIENT_CERT_AUTH=“true”
  26. ETCD_PEER_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  27. ETCD_PEER_AUTO_TLS=“true”

node02 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“https://192.168.1.92:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“https://192.168.1.92:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd2”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://192.168.1.92:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“https://192.168.1.92:2379”
  14. ETCD_INITIAL_CLUSTER=“etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380”
  15. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  16. ETCD_INITIAL_CLUSTER_STATE=“new”
  17.  
  18. ETCD_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  19. ETCD_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  20. ETCD_CLIENT_CERT_AUTH=“true”
  21. ETCD_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  22. ETCD_AUTO_TLS=“true”
  23. ETCD_PEER_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  24. ETCD_PEER_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  25. ETCD_PEER_CLIENT_CERT_AUTH=“true”
  26. ETCD_PEER_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  27. ETCD_PEER_AUTO_TLS=“true”

node03 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“https://192.168.1.93:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“https://192.168.1.93:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd3”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://192.168.1.93:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“https://192.168.1.93:2379”
  14. ETCD_INITIAL_CLUSTER=“etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380”
  15. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  16. ETCD_INITIAL_CLUSTER_STATE=“new”
  17.  
  18. ETCD_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  19. ETCD_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  20. ETCD_CLIENT_CERT_AUTH=“true”
  21. ETCD_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  22. ETCD_AUTO_TLS=“true”
  23. ETCD_PEER_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  24. ETCD_PEER_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  25. ETCD_PEER_CLIENT_CERT_AUTH=“true”
  26. ETCD_PEER_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  27. ETCD_PEER_AUTO_TLS=“true”

启动测试

  1. [root@node01 work]# systemctl start etcd
  2. [root@node01 work]# systemctl status etcd
  3. ● etcd.service – Etcd Server
  4. Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
  5. Active: active (running) since Thu 2019–11–07 10:15:58 CST; 5s ago
  6. Main PID: 2078 (etcd)
  7. Tasks: 8
  8. Memory: 28.9M
  9. CGroup: /system.slice/etcd.service
  10. └─2078 /usr/bin/etcd —name=etcd1 —data–dir=/data/k8s/etcd/data —listen–client–urls=https://192.168.1.91:2379
  11.  
  12. Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe [term: 1] received a MsgVote message with higher term from af05139f75a68867 [term: 2]
  13. Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe became follower at term 2
  14. Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe [logterm: 1, index: 3, vote: 0] cast MsgVote for af05139f75a68867 [logterm: 1, index: 3] at term 2
  15. Nov 07 10:15:58 node01.k8s.com etcd[2078]: raft.node: 2a40d8ba966d12fe elected leader af05139f75a68867 at term 2
  16. Nov 07 10:15:58 node01.k8s.com etcd[2078]: published {Name:etcd1 ClientURLs:[https://192.168.1.91:2379]} to cluster f3e9c54e1aafb3c1
  17. Nov 07 10:15:58 node01.k8s.com etcd[2078]: ready to serve client requests
  18. Nov 07 10:15:58 node01.k8s.com etcd[2078]: serving client requests on 192.168.1.91:2379
  19. Nov 07 10:15:58 node01.k8s.com systemd[1]: Started Etcd Server.
  20. Nov 07 10:15:58 node01.k8s.com etcd[2078]: set the initial cluster version to 3.3
  21. Nov 07 10:15:58 node01.k8s.com etcd[2078]: enabled capabilities for version 3.3

查看 /var/log/message 日志中，会有日下体现：

  1. Nov 7 10:15:57 node01 etcd: added member 2a40d8ba966d12fe [https://192.168.1.91:2380] to cluster f3e9c54e1aafb3c1
  2. Nov 7 10:15:57 node01 etcd: added member af05139f75a68867 [https://192.168.1.92:2380] to cluster f3e9c54e1aafb3c1
  3. Nov 7 10:15:57 node01 etcd: added member c3bab7c20fba3f60 [https://192.168.1.93:2380] to cluster f3e9c54e1aafb3c1

检查TLS集群状态

  1. ETCDCTL_API=3 etcdctl \
  2. —endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \
  3. —cacert=/etc/kubernetes/cert/ca.pem \
  4. —cert=/etc/etcd/cert/etcd.pem \
  5. —key=/etc/etcd/cert/etcd–key.pem endpoint health
  6.  
  7. # 输出
  8. https://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.317022ms
  9. https://192.168.1.91:2379 is healthy: successfully committed proposal: took = 1.59958ms
  10. https://192.168.1.93:2379 is healthy: successfully committed proposal: took = 1.453049ms
  11. etcdctl \
  12. —endpoint=https://etcd1.k8s.com:2379 \
  13. —ca–file=/etc/kubernetes/cert/ca.pem \
  14. —cert–file=/etc/etcd/cert/etcd.pem \
  15. —key–file=/etc/etcd/cert/etcd–key.pem cluster–health
  16.  
  17. # 输出
  18. member 40a8f19a5db99534 is healthy: got healthy result from https://etcd2.k8s.com:2379
  19. member 9888555207dbf0e0 is healthy: got healthy result from https://etcd3.k8s.com:2379
  20. member a0d541999e9eb3b3 is healthy: got healthy result from https://etcd1.k8s.com:2379
  21. cluster is healthy

ETCD 动态集群基于DNS的SRV解析自动发现

需要局域网内部有DNS服务器

添加SRV解析

目前常用的内部DNS服务有两种，bind、dnsmasq

在下面都会列出具体的配置，但只需要配置其中之一即可；

方法一: 使用bind配置SRV解析

如果内部没有bind服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11806962.html

使用域名为 : k8s.com，在bind的zone文件中添加如下解析:

  1. etcd1 IN A 192.168.1.91
  2. etcd2 IN A 192.168.1.92
  3. etcd3 IN A 192.168.1.93
  4. _etcd–server._tcp.k8s.com. IN SRV 10 10 2380 etcd1
  5. _etcd–server._tcp.k8s.com. IN SRV 10 10 2380 etcd2
  6. _etcd–server._tcp.k8s.com. IN SRV 10 10 2380 etcd3
  7. _etcd–client._tcp.k8s.com. IN SRV 10 10 2379 etcd1
  8. _etcd–client._tcp.k8s.com. IN SRV 10 10 2379 etcd2
  9. _etcd–client._tcp.k8s.com. IN SRV 10 10 2379 etcd3

修改之后重新加载配置文件:

  1. [root@jenkins named]# named–checkzone k8s.com k8s.com.zone
  2. zone k8s.com/IN: loaded serial 0
  3. OK
  4. [root@jenkins named]# rndc reload
  5. server reload successful

方法二: 使用dnsmasq配置SRV解析

如果内部没有dnsmasq服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11809066.html

使用域名为 : k8s.com，具体修改如下:

在/etc/dnsmasq_hosts新增下面内容

  1. 192.168.1.91 etcd1 etcd1.k8s.com
  2. 192.168.1.92 etcd2 etcd2.k8s.com
  3. 192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析内容

  1. srv–host=_etcd–server._tcp.k8s.com,etcd1.k8s.com,2380,0,100
  2. srv–host=_etcd–server._tcp.k8s.com,etcd2.k8s.com,2380,0,100
  3. srv–host=_etcd–server._tcp.k8s.com,etcd3.k8s.com,2380,0,100
  4. srv–host=_etcd–client._tcp.k8s.com,etcd1.k8s.com,2379,0,100
  5. srv–host=_etcd–client._tcp.k8s.com,etcd2.k8s.com,2379,0,100
  6. srv–host=_etcd–client._tcp.k8s.com,etcd3.k8s.com,2379,0,100

修改之后重启服务 systemctl restart dnsmasq

验证SRV解析是否正常

查询SRV记录

  1. [root@node01 ~]# dig @192.168.1.122 +noall +answer SRV _etcd–server._tcp.k8s.com
  2. _etcd–server._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd2.k8s.com.
  3. _etcd–server._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd1.k8s.com.
  4. _etcd–server._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd3.k8s.com.

查询域名解析结果

  1. [root@node01 ~]# dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com
  2. etcd1.k8s.com. 86400 IN A 192.168.1.91
  3. etcd2.k8s.com. 86400 IN A 192.168.1.92
  4. etcd3.k8s.com. 86400 IN A 192.168.1.93

如上述显示，则表示SRV解析正常

配置ETCD

node01 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“http://192.168.1.91:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“http://192.168.1.91:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd1”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“http://etcd1.k8s.com:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“http://etcd1.k8s.com:2379”
  14. ETCD_DISCOVERY_SRV=“k8s.com”
  15.  
  16. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  17. ETCD_INITIAL_CLUSTER_STATE=“new”

node02 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“http://192.168.1.92:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“http://192.168.1.92:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd2”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“http://etcd2.k8s.com:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“http://etcd2.k8s.com:2379”
  14. ETCD_DISCOVERY_SRV=“k8s.com”
  15.  
  16. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  17. ETCD_INITIAL_CLUSTER_STATE=“new”

node03 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“http://192.168.1.93:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“http://192.168.1.93:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd3”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“http://etcd3.k8s.com:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“http://etcd3.k8s.com:2379”
  14. ETCD_DISCOVERY_SRV=“k8s.com”
  15.  
  16. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  17. ETCD_INITIAL_CLUSTER_STATE=“new”

启动并测试

启动

  1. [root@node01 etcd]# systemctl start etcd
  2. [root@node01 etcd]# systemctl status etcd
  3. ● etcd.service – Etcd Server
  4. Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
  5. Active: active (running) since Thu 2019–11–07 11:25:29 CST; 4s ago
  6. Main PID: 14203 (etcd)
  7. Tasks: 8
  8. Memory: 16.9M
  9. CGroup: /system.slice/etcd.service
  10. └─14203 /usr/bin/etcd —name=etcd1 —data–dir=/data/k8s/etcd/data —listen–client–urls=http://192.168.1.91:2379
  11.  
  12. Nov 07 11:25:29 node01.k8s.com etcd[14203]: d79e9ae86b2a1de1 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
  13. Nov 07 11:25:29 node01.k8s.com etcd[14203]: d79e9ae86b2a1de1 became leader at term 2
  14. Nov 07 11:25:29 node01.k8s.com etcd[14203]: raft.node: d79e9ae86b2a1de1 elected leader d79e9ae86b2a1de1 at term 2
  15. Nov 07 11:25:29 node01.k8s.com etcd[14203]: published {Name:etcd1 ClientURLs:[http://etcd1.k8s.com:2379 http://etcd1.k8s.com:4001]} to cluster 42cecf80e3791d6c
  16. Nov 07 11:25:29 node01.k8s.com etcd[14203]: ready to serve client requests
  17. Nov 07 11:25:29 node01.k8s.com etcd[14203]: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!
  18. Nov 07 11:25:29 node01.k8s.com systemd[1]: Started Etcd Server.
  19. Nov 07 11:25:29 node01.k8s.com etcd[14203]: setting up the initial cluster version to 3.3
  20. Nov 07 11:25:29 node01.k8s.com etcd[14203]: set the initial cluster version to 3.3
  21. Nov 07 11:25:29 node01.k8s.com etcd[14203]: enabled capabilities for version 3.3

日志 vim /var/log/messages 表现如下:

  1. Nov 7 11:25:27 node01 etcd: got bootstrap from DNS for etcd–server at 0=http://etcd3.k8s.com:2380
  2. Nov 7 11:25:27 node01 etcd: got bootstrap from DNS for etcd–server at 1=http://etcd2.k8s.com:2380
  3. Nov 7 11:25:27 node01 etcd: got bootstrap from DNS for etcd–server at etcd1=http://etcd1.k8s.com:2380
  4. Nov 7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380
  5. Nov 7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380
  6. Nov 7 11:25:28 node01 etcd: name = etcd1
  7. Nov 7 11:25:28 node01 etcd: data dir = /data/k8s/etcd/data
  8. Nov 7 11:25:28 node01 etcd: member dir = /data/k8s/etcd/data/member
  9. Nov 7 11:25:28 node01 etcd: dedicated WAL dir = /data/k8s/etcd/wal
  10. Nov 7 11:25:28 node01 etcd: heartbeat = 100ms
  11. Nov 7 11:25:28 node01 etcd: election = 1000ms
  12. Nov 7 11:25:28 node01 etcd: snapshot count = 100000
  13. Nov 7 11:25:28 node01 etcd: advertise client URLs = http://etcd1.k8s.com:2379,http://etcd1.k8s.com:4001
  14. Nov 7 11:25:28 node01 etcd: initial advertise peer URLs = http://etcd1.k8s.com:2380
  15. Nov 7 11:25:28 node01 etcd: initial cluster = 0=http://etcd3.k8s.com:2380,1=http://etcd2.k8s.com:2380,etcd1=http://etcd1.k8s.com:2380

测试：

  1. [root@node01 etcd]# etcdctl —endpoints=http://192.168.1.91:2379 cluster-health
  2. member 184beca37ca32d75 is healthy: got healthy result from http://etcd2.k8s.com:2379
  3. member d79e9ae86b2a1de1 is healthy: got healthy result from http://etcd1.k8s.com:2379
  4. member f7662e609b7e4013 is healthy: got healthy result from http://etcd3.k8s.com:2379
  5. cluster is healthy

ETCD TLS动态集群基于DNS的SRV解析自动发现

需要局域网内部有DNS服务器

添加SRV解析

目前常用的内部DNS服务有两种，bind、dnsmasq

在下面都会列出具体的配置，但只需要配置其中之一即可；

方法一: 使用bind配置SRV解析

如果内部没有bind服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11806962.html

使用域名为 : k8s.com，在bind的zone文件中添加如下解析:

  1. etcd1 IN A 192.168.1.91
  2. etcd2 IN A 192.168.1.92
  3. etcd3 IN A 192.168.1.93
  4. _etcd–server–ssl._tcp.k8s.com. IN SRV 10 10 2380 etcd1
  5. _etcd–server–ssl._tcp.k8s.com. IN SRV 10 10 2380 etcd2
  6. _etcd–server–ssl._tcp.k8s.com. IN SRV 10 10 2380 etcd3
  7. _etcd–client–ssl._tcp.k8s.com. IN SRV 10 10 2379 etcd1
  8. _etcd–client–ssl._tcp.k8s.com. IN SRV 10 10 2379 etcd2
  9. _etcd–client–ssl._tcp.k8s.com. IN SRV 10 10 2379 etcd3

修改之后重新加载配置文件:

  1. [root@jenkins named]# named–checkzone k8s.com k8s.com.zone
  2. zone k8s.com/IN: loaded serial 0
  3. OK
  4. [root@jenkins named]# rndc reload
  5. server reload successful

方法二: 使用dnsmasq配置SRV解析

如果内部没有dnsmasq服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11809066.html

使用域名为 : k8s.com，具体修改如下:

在/etc/dnsmasq_hosts新增下面内容

  1. 192.168.1.91 etcd1 etcd1.k8s.com
  2. 192.168.1.92 etcd2 etcd2.k8s.com
  3. 192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析内容

  1. srv–host=_etcd–server–ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100
  2. srv–host=_etcd–server–ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100
  3. srv–host=_etcd–server–ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100
  4. srv–host=_etcd–client–ssl._tcp.k8s.com,etcd1.k8s.com,2379,0,100
  5. srv–host=_etcd–client–ssl._tcp.k8s.com,etcd2.k8s.com,2379,0,100
  6. srv–host=_etcd–client–ssl._tcp.k8s.com,etcd3.k8s.com,2379,0,100

修改之后重启服务 systemctl restart dnsmasq

验证SRV解析是否正常

查询SRV记录

  1. [root@node01 etcd]# dig @192.168.1.122 +noall +answer SRV _etcd–server–ssl._tcp.k8s.com
  2. _etcd–server–ssl._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd3.k8s.com.
  3. _etcd–server–ssl._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd2.k8s.com.
  4. _etcd–server–ssl._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd1.k8s.com.

查询域名解析结果

  1. [root@node01 ~]# dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com
  2. etcd1.k8s.com. 86400 IN A 192.168.1.91
  3. etcd2.k8s.com. 86400 IN A 192.168.1.92
  4. etcd3.k8s.com. 86400 IN A 192.168.1.93

ETCD 配置

node01 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“https://192.168.1.91:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“https://192.168.1.91:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd1”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://etcd1.k8s.com:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“https://etcd1.k8s.com:2379,https://etcd1.k8s.com:4001”
  14. ETCD_DISCOVERY_SRV=“k8s.com”
  15. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  16. ETCD_INITIAL_CLUSTER_STATE=“new”
  17.  
  18. ETCD_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  19. ETCD_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  20. ETCD_CLIENT_CERT_AUTH=“true”
  21. ETCD_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  22. ETCD_AUTO_TLS=“true”
  23. ETCD_PEER_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  24. ETCD_PEER_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  25. ETCD_PEER_CLIENT_CERT_AUTH=“true”
  26. ETCD_PEER_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  27. ETCD_PEER_AUTO_TLS=“true”

node02 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“https://192.168.1.92:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“https://192.168.1.92:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd2”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://etcd2.k8s.com:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“https://etcd2.k8s.com:2379”
  14. ETCD_DISCOVERY_SRV=“k8s.com”
  15. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  16. ETCD_INITIAL_CLUSTER_STATE=“new”
  17.  
  18. ETCD_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  19. ETCD_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  20. ETCD_CLIENT_CERT_AUTH=“true”
  21. ETCD_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  22. ETCD_AUTO_TLS=“true”
  23. ETCD_PEER_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  24. ETCD_PEER_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  25. ETCD_PEER_CLIENT_CERT_AUTH=“true”
  26. ETCD_PEER_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  27. ETCD_PEER_AUTO_TLS=“true”

node03 配置文件

  1. ETCD_DATA_DIR=“/data/k8s/etcd/data”
  2. ETCD_WAL_DIR=“/data/k8s/etcd/wal”
  3. ETCD_LISTEN_PEER_URLS=“https://192.168.1.93:2380”
  4. ETCD_LISTEN_CLIENT_URLS=“https://192.168.1.93:2379”
  5. ETCD_MAX_SNAPSHOTS=“5”
  6. ETCD_MAX_WALS=“5”
  7. ETCD_NAME=“etcd3”
  8. ETCD_SNAPSHOT_COUNT=“100000”
  9. ETCD_HEARTBEAT_INTERVAL=“100”
  10. ETCD_ELECTION_TIMEOUT=“1000”
  11.  
  12. ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://etcd3.k8s.com:2380”
  13. ETCD_ADVERTISE_CLIENT_URLS=“https://etcd3.k8s.com:2379”
  14. ETCD_DISCOVERY_SRV=“k8s.com”
  15. ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
  16. ETCD_INITIAL_CLUSTER_STATE=“new”
  17.  
  18. ETCD_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  19. ETCD_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  20. ETCD_CLIENT_CERT_AUTH=“true”
  21. ETCD_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  22. ETCD_AUTO_TLS=“true”
  23. ETCD_PEER_CERT_FILE=“/etc/etcd/cert/etcd.pem”
  24. ETCD_PEER_KEY_FILE=“/etc/etcd/cert/etcd-key.pem”
  25. ETCD_PEER_CLIENT_CERT_AUTH=“true”
  26. ETCD_PEER_TRUSTED_CA_FILE=“/etc/kubernetes/cert/ca.pem”
  27. ETCD_PEER_AUTO_TLS=“true”

启动测试

启动

  1. [root@node03 etcd]# systemctl restart etcd
  2. [root@node03 etcd]# systemctl status etcd
  3. ● etcd.service – Etcd Server
  4. Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
  5. Active: active (running) since Thu 2019–11–07 12:38:37 CST; 4s ago
  6. Main PID: 13460 (etcd)
  7. Tasks: 8
  8. Memory: 16.6M
  9. CGroup: /system.slice/etcd.service
  10. └─13460 /usr/bin/etcd —name=etcd3 —data–dir=/data/k8s/etcd/data —listen–client–urls=https://192.168.1.93:2379
  11.  
  12. Nov 07 12:38:36 node03.k8s.com etcd[13460]: established a TCP streaming connection with peer 40a8f19a5db99534 (stream Message writer)
  13. Nov 07 12:38:36 node03.k8s.com etcd[13460]: established a TCP streaming connection with peer 40a8f19a5db99534 (stream MsgApp v2 writer)
  14. Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 [term: 92] received a MsgVote message with higher term from a0d541999e9eb3b3 [term: 98]
  15. Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 became follower at term 98
  16. Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 [logterm: 92, index: 9, vote: 0] cast MsgVote for a0d541999e9eb3b3 [logterm: 92, index: 9] at term 98
  17. Nov 07 12:38:37 node03.k8s.com etcd[13460]: raft.node: 9888555207dbf0e0 elected leader a0d541999e9eb3b3 at term 98
  18. Nov 07 12:38:37 node03.k8s.com etcd[13460]: published {Name:etcd3 ClientURLs:[https://etcd3.k8s.com:2379]} to cluster f445a02ce3dc6a02
  19. Nov 07 12:38:37 node03.k8s.com etcd[13460]: ready to serve client requests
  20. Nov 07 12:38:37 node03.k8s.com etcd[13460]: serving client requests on 192.168.1.93:2379
  21. Nov 07 12:38:37 node03.k8s.com systemd[1]: Started Etcd Server.

日志体现

  1. Nov 7 12:38:36 node01 etcd: added member 40a8f19a5db99534 [https://etcd2.k8s.com:2380] to cluster f445a02ce3dc6a02
  2. Nov 7 12:38:36 node01 etcd: starting peer 40a8f19a5db99534...
  3. Nov 7 12:38:36 node01 etcd: started HTTP pipelining with peer 40a8f19a5db99534
  4. Nov 7 12:38:36 node01 etcd: started streaming with peer 40a8f19a5db99534 (writer)
  5. Nov 7 12:38:36 node01 etcd: started peer 40a8f19a5db99534
  6. Nov 7 12:38:36 node01 etcd: added peer 40a8f19a5db99534
  7. Nov 7 12:38:36 node01 etcd: added member 9888555207dbf0e0 [https://etcd3.k8s.com:2380] to cluster f445a02ce3dc6a02
  8. Nov 7 12:38:36 node01 etcd: starting peer 9888555207dbf0e0…
  9. Nov 7 12:38:36 node01 etcd: started HTTP pipelining with peer 9888555207dbf0e0
  10. Nov 7 12:38:36 node01 etcd: started peer 9888555207dbf0e0
  11. Nov 7 12:38:36 node01 etcd: added peer 9888555207dbf0e0
  12. Nov 7 12:38:36 node01 etcd: added member a0d541999e9eb3b3 [https://etcd1.k8s.com:2380] to cluster f445a02ce3dc6a02

测试集群状态：

  1. ETCDCTL_API=3 etcdctl —endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \
  2. —cacert=/etc/kubernetes/cert/ca.pem \
  3. —cert=/etc/etcd/cert/etcd.pem \
  4. —key=/etc/etcd/cert/etcd–key.pem endpoint health
  5.  
  6. # 输出
  7. https://etcd1.k8s.com:2379 is healthy: successfully committed proposal: took = 4.269468ms
  8. https://etcd3.k8s.com:2379 is healthy: successfully committed proposal: took = 1.58797ms
  9. https://etcd2.k8s.com:2379 is healthy: successfully committed proposal: took = 1.622151ms
  10. etcdctl \
  11. —endpoint=https://etcd1.k8s.com:2379 \
  12. —ca–file=/etc/kubernetes/cert/ca.pem \
  13. —cert–file=/etc/etcd/cert/etcd.pem \
  14. —key–file=/etc/etcd/cert/etcd–key.pem cluster–health
  15.  
  16. # 输出
  17. member 40a8f19a5db99534 is healthy: got healthy result from https://etcd2.k8s.com:2379
  18. member 9888555207dbf0e0 is healthy: got healthy result from https://etcd3.k8s.com:2379
  19. member a0d541999e9eb3b3 is healthy: got healthy result from https://etcd1.k8s.com:2379
  20. cluster is healthy

报错解决

1. 证书报错 bad certificate

日志中报错:

  1. Nov 7 12:37:03 node01 etcd: rejected connection from “192.168.1.93:46294” (error “remote error: tls: bad certificate”, ServerName “k8s.com”)

解决

报错的意思是在生成ETCD的TLS证书的时候，没有把对应的域名加进去

在创建ETCD的TLS证书请求的文件中加入对应的域名

  1. cd /opt/k8s/work
  2. cat > etcd–csr.json <<EOF
  3. {
  4. “CN”: “etcd”,
  5. “hosts”: [
  6. “127.0.0.1”,
  7. “192.168.1.91”,
  8. “192.168.1.92”,
  9. “192.168.1.93”,
  10. “k8s.com”, # 这里的域名查看是否正确
  11. “etcd1.k8s.com”,
  12. “etcd2.k8s.com”,
  13. “etcd3.k8s.com”
  14. ],
  15. “key”: {
  16. “algo”: “rsa”,
  17. “size”: 2048
  18. },
  19. “names”: [
  20. {
  21. “C”: “CN”,
  22. “ST”: “BeiJing”,
  23. “L”: “BeiJing”,
  24. “O”: “k8s”,
  25. “OU”: “4Paradigm”
  26. }
  27. ]
  28. }
  29. EOF

2. DNS 的 SRV 解析报错 cannot find local etcd member “etcd1” in SRV records

DNS 如果配置有问题，会有如下报错:

  1. etcd: error setting up initial cluster: cannot find local etcd member “etcd1” in SRV records

这里是表示DNS在配置SRV解析的时候报错，请仔细查看解析配置:

SRV解析分为两种，一种是http不带证书的解析，一种是https带证书的解析，是有区别的，如果配置错误就会包上述错误

http不带证书解析如下

bind 的解析

编辑 /var/named/k8s.com.zone 文件

  1. etcd1 IN A 192.168.1.91
  2. etcd2 IN A 192.168.1.92
  3. etcd3 IN A 192.168.1.93
  4. _etcd–server._tcp.k8s.com. IN SRV 10 10 2380 etcd1
  5. _etcd–server._tcp.k8s.com. IN SRV 10 10 2380 etcd2
  6. _etcd–server._tcp.k8s.com. IN SRV 10 10 2380 etcd3
  7. _etcd–client._tcp.k8s.com. IN SRV 10 10 2379 etcd1
  8. _etcd–client._tcp.k8s.com. IN SRV 10 10 2379 etcd2
  9. _etcd–client._tcp.k8s.com. IN SRV 10 10 2379 etcd3

dnsmasq 的解析

在/etc/dnsmasq_hosts新增下面内容

  1. 192.168.1.91 etcd1 etcd1.k8s.com
  2. 192.168.1.92 etcd2 etcd2.k8s.com
  3. 192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析内容

  1. srv–host=_etcd–server._tcp.k8s.com,etcd1.k8s.com,2380,0,100
  2. srv–host=_etcd–server._tcp.k8s.com,etcd2.k8s.com,2380,0,100
  3. srv–host=_etcd–server._tcp.k8s.com,etcd3.k8s.com,2380,0,100
  4. srv–host=_etcd–client._tcp.k8s.com,etcd1.k8s.com,2380,0,100
  5. srv–host=_etcd–client._tcp.k8s.com,etcd2.k8s.com,2380,0,100
  6. srv–host=_etcd–client._tcp.k8s.com,etcd3.k8s.com,2380,0,100

https带证书解析如下

bind 的解析

编辑 /var/named/k8s.com.zone 文件

  1. etcd1 IN A 192.168.1.91
  2. etcd2 IN A 192.168.1.92
  3. etcd3 IN A 192.168.1.93
  4. _etcd–server–ssl._tcp.k8s.com. IN SRV 10 10 2380 etcd1
  5. _etcd–server–ssl._tcp.k8s.com. IN SRV 10 10 2380 etcd2
  6. _etcd–server–ssl._tcp.k8s.com. IN SRV 10 10 2380 etcd3
  7. _etcd–client–ssl._tcp.k8s.com. IN SRV 10 10 2379 etcd1
  8. _etcd–client–ssl._tcp.k8s.com. IN SRV 10 10 2379 etcd2
  9. _etcd–client–ssl._tcp.k8s.com. IN SRV 10 10 2379 etcd3

dnsmasq 的解析

在/etc/dnsmasq_hosts新增下面内容

  1. 192.168.1.91 etcd1 etcd1.k8s.com
  2. 192.168.1.92 etcd2 etcd2.k8s.com
  3. 192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析内容

  1. srv–host=_etcd–server–ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100
  2. srv–host=_etcd–server–ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100
  3. srv–host=_etcd–server–ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100
  4. srv–host=_etcd–client–ssl._tcp.k8s.com,etcd1.k8s.com,2379,0,100
  5. srv–host=_etcd–client–ssl._tcp.k8s.com,etcd2.k8s.com,2379,0,100
  6. srv–host=_etcd–client–ssl._tcp.k8s.com,etcd3.k8s.com,2379,0,100

Related Posts:

  1. Kubernetes 1.14 二进制集群安装
  2. Kuerbernetes 1.11 集群二进制安装
  3. Kubenetes 1.13.5 集群二进制安装
  4. kubeadm 搭建Kubernetes 1.18集群

Related Post

群晖Synology专栏

2022年1月3日

群晖/极空间无公网IP搭建Cloudflared-阿里云免费隧道穿透教程

2024年4月6日

Unraid安装Jellyfin并挂载群晖NAS

2022年9月6日

发表回复