标签：-o ane 访问 google ons rsa mamicode lin uber
本文档介绍部署高可用 kube-controller-manager 集群的步骤。
该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用后，剩余节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。
为保证通信安全，本文档先生成 x509 证书和私钥，kube-controller-manager 在如下两种情况下使用该证书：
注意：如果没有特殊指明，本文档的所有操作均在 k8s-master1 节点上执行，然后远程分发文件和执行命令。
创建证书签名请求：
cd /opt/k8s/work cat > kube-controller-manager-csr.json <<EOF { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.161.150", "192.168.161.151", "192.168.161.152" ], "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-controller-manager", "OU": "4Paradigm" } ] } EOF
生成证书和私钥：
cd /opt/k8s/work cfssl gencert -ca=/opt/k8s/work/ca.pem -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager ls kube-controller-manager*pem
将生成的证书和私钥分发到所有 master 节点：
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" scp kube-controller-manager*.pem root@${node_ip}:/etc/kubernetes/cert/ done
kubeconfig 文件包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书；
cd /opt/k8s/work source /opt/k8s/bin/environment.sh kubectl config set-cluster kubernetes --certificate-authority=/opt/k8s/work/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
分发 kubeconfig 到所有 master 节点：
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" scp kube-controller-manager.kubeconfig root@${node_ip}:/etc/kubernetes/ done
cd /opt/k8s/work source /opt/k8s/bin/environment.sh cat > kube-controller-manager.service <<EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] WorkingDirectory=${K8S_DIR}/kube-controller-manager ExecStart=/opt/k8s/bin/kube-controller-manager \ --port=0 \ --secure-port=10252 \ --bind-address=127.0.0.1 \ --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ --service-cluster-ip-range=${SERVICE_CIDR} \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \ --experimental-cluster-signing-duration=8760h \ --root-ca-file=/etc/kubernetes/cert/ca.pem \ --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \ --leader-elect=true \ --controllers=*,bootstrapsigner,tokencleaner \ --horizontal-pod-autoscaler-use-rest-clients=true \ --horizontal-pod-autoscaler-sync-period=10s \ --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \ --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \ --use-service-account-credentials=true \ --kube-api-qps=1000 \ --kube-api-burst=2000 \ --logtostderr=true \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
--port=0：关闭监听 http /metrics 的请求，同时 --address 参数无效，--bind-address 参数有效；--secure-port=10252、--bind-address=0.0.0.0: 在所有网络接口监听 10252 端口的 https /metrics 请求；--kubeconfig：指定 kubeconfig 文件路径，kube-controller-manager 使用它连接和验证 kube-apiserver；--authentication-kubeconfig 和 --authorization-kubeconfig：kube-controller-manager 使用它连接 apiserver，对 client 的请求进行认证和授权。kube-controller-manager 不再使用 --tls-ca-file 对请求 https metrics 的 Client 证书进行校验。如果没有配置这两个 kubeconfig 参数，则 client 连接 kube-controller-manager https 端口的请求会被拒绝(提示权限不足)。--cluster-signing-*-file：签名 TLS Bootstrap 创建的证书；--experimental-cluster-signing-duration：指定 TLS Bootstrap 证书的有效期；--root-ca-file：放置到容器 ServiceAccount 中的 CA 证书，用来对 kube-apiserver 的证书进行校验；--service-account-private-key-file：签名 ServiceAccount 中 Token 的私钥文件，必须和 kube-apiserver 的 --service-account-key-file 指定的公钥文件配对使用；--service-cluster-ip-range ：指定 Service Cluster IP 网段，必须和 kube-apiserver 中的同名参数一致；--leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态；--controllers=*,bootstrapsigner,tokencleaner：启用的控制器列表，tokencleaner 用于自动清理过期的 Bootstrap token；--horizontal-pod-autoscaler-*：custom metrics 相关参数，支持 autoscaling/v2alpha1；--tls-cert-file、--tls-private-key-file：使用 https 输出 metrics 时使用的 Server 证书和秘钥；--use-service-account-credentials=true: kube-controller-manager 中各 controller 使用 serviceaccount 访问 kube-apiserver；完整 unit 见 kube-controller-manager.service
分发 systemd unit 文件到所有 master 节点：
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" scp kube-controller-manager.service root@${node_ip}:/etc/systemd/system/ done
ClusteRole: system:kube-controller-manager 的权限很小，只能创建 secret、serviceaccount 等资源对象，各 controller 的权限分散到 ClusterRole system:controller:XXX 中。
需要在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数，这样 main controller 会为各 controller 创建对应的 ServiceAccount XXX-controller。
内置的 ClusterRoleBinding system:controller:XXX 将赋予各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。
source /opt/k8s/bin/environment.sh
for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager"
ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
done
source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" ssh k8s@${node_ip} "systemctl status kube-controller-manager|grep Active" done
确保状态为 active (running)，否则查看日志，确认原因：
$ journalctl -u kube-controller-manager
注意：以下命令在 kube-controller-manager 节点上执行。
kube-controller-manager 监听 10252 端口，接收 https 请求：
[root@k8s-master1 work]# sudo netstat -lnpt|grep kube-controll tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 19176/kube-controll
停掉一个或两个节点的 kube-controller-manager 服务，观察其它节点的日志，看是否获取了 leader 权限。
kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
[root@k8s-master1 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: ‘{"holderIdentity":"k8s-master3_60147cc2-758e-11e9-8fcb-000c2956b313","leaseDurationSeconds":15,"acquireTime":"2019-05-13T14:49:56Z","renewTime":"2019-05-13T14:53:10Z","leaderTransitions":0}‘ creationTimestamp: 2019-05-13T14:49:56Z name: kube-controller-manager namespace: kube-system resourceVersion: "5106" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: 6015837d-758e-11e9-89c6-000c2956b313 [root@k8s-master1 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: ‘{"holderIdentity":"k8s-master1_60ab86f0-758e-11e9-931c-000c29aa0a68","leaseDurationSeconds":15,"acquireTime":"2019-05-13T14:53:37Z","renewTime":"2019-05-13T14:53:39Z","leaderTransitions":1}‘ creationTimestamp: 2019-05-13T14:49:56Z name: kube-controller-manager namespace: kube-system resourceVersion: "5116" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: 6015837d-758e-11e9-89c6-000c2956b313
可见，当前的 leader 从master3切到了master1 节点。
部署高可用 kube-controller-manager 集群
标签：-o ane 访问 google ons rsa mamicode lin uber
原文地址：https://www.cnblogs.com/gytangyao/p/10859361.html
