标签：erb https specified 使用 cts ppi which 版本 tco
上一篇我们部署了kubernetes的master集群，参考：二进制搭建kubernetes多master集群【三、配置k8s master及高可用】
本文在以下主机上操作部署k8s node
k8s-node1：192.168.80.10
k8s-node2：192.168.80.11
k8s-node3：192.168.80.12
以下kubeadm和kubectl命令操作都是在k8s-master1上执行的。
kubernetes work 节点运行如下组件：
docker和flannel部署参考：二进制搭建kubernetes多master集群【二、配置flannel网络】 、 docker-ce安装
一、安装依赖包
yum install -y epel-release wget conntrack ipvsadm ipset jq iptables curl sysstat libseccomp && /usr/sbin/modprobe ip_vs
二、部署kubelet组件
kublet 运行在每个 worker 节点上，接收 kube-apiserver 发送的请求，管理 Pod 容器，执行交互式命令，如 exec、run、logs 等。
kublet 启动时自动向 kube-apiserver 注册节点信息，内置的 cadvisor 统计和监控节点的资源使用情况。
为确保安全，本文档只开启接收 https 请求的安全端口，对请求进行认证和授权，拒绝未授权的访问(如 apiserver、heapster)。
1、下载和分发kubelet二进制文件
wget https://dl.k8s.io/v1.12.3/kubernetes-server-linux-amd64.tar.gz tar -xzvf kubernetes-server-linux-amd64.tar.gz cp kubernetes/server/bin/ cp kubelet kube-proxy /usr/local/bin scp kubelet kube-proxy k8s-node2:/usr/local/bin scp kubelet kube-proxy k8s-node3:/usr/local/bin
2、创建kubelet bootstrap kubeconfig文件 （k8s-master1上执行）
#创建 token export BOOTSTRAP_TOKEN=$(kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:k8s-master1 --kubeconfig ~/.kube/config) # 设置集群参数 kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://114.67.81.105:8443 \ --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig # 设置客户端认证参数 kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig # 设置上下文参数 kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig # 设置默认上下文 kubectl config use-context default --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig
3、查看 kubeadm 为各节点创建的 token：
[root@k8s-master1 ~]# kubeadm token list --kubeconfig ~/.kube/config TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS 8w6j3n.ruh4ne95icbae4ie 23h 2018-12-21T20:42:29+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-master3 e7n0o5.1y8sjblh43z8ftz1 23h 2018-12-21T20:41:53+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-master2 ydbwyk.yz8e97df5d5u2o70 22h 2018-12-21T19:28:43+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-master1
查看各 token 关联的 Secret：（红色的为创建生成的token）
[root@k8s-master1 ~]# kubectl get secrets -n kube-system NAME TYPE DATA AGE attachdetach-controller-token-z2w72 kubernetes.io/service-account-token 3 119m bootstrap-signer-token-hz8dr kubernetes.io/service-account-token 3 119m bootstrap-token-8w6j3n bootstrap.kubernetes.io/token 7 20m bootstrap-token-e7n0o5 bootstrap.kubernetes.io/token 7 20m bootstrap-token-ydbwyk bootstrap.kubernetes.io/token 7 93m certificate-controller-token-bjhbq kubernetes.io/service-account-token 3 119m clusterrole-aggregation-controller-token-qkqxg kubernetes.io/service-account-token 3 119m cronjob-controller-token-v7vz5 kubernetes.io/service-account-token 3 119m daemon-set-controller-token-7khdh kubernetes.io/service-account-token 3 119m default-token-nwqsr kubernetes.io/service-account-token 3 119m
4、分发bootstrap kubeconfig文件
[root@k8s-master1 ~]# scp kubelet-bootstrap-k8s-master1.kubeconfig k8s-node1:/etc/kubernetes/cert/kubelet-bootstrap.kubeconfig [root@k8s-master1 ~]# scp kubelet-bootstrap-k8s-master2.kubeconfig k8s-node2:/etc/kubernetes/cert/kubelet-bootstrap.kubeconfig [root@k8s-master1 ~]# scp kubelet-bootstrap-k8s-master3.kubeconfig k8s-node3:/etc/kubernetes/cert/kubelet-bootstrap.kubeconfig
5、创建和分发kubelet参数配置文件
从 v1.10 开始，kubelet 部分参数需在配置文件中配置，kubelet --help 会提示：
DEPRECATED: This parameter should be set via the config file specified by the Kubelet‘s --config flag
创建 kubelet 参数配置模板文件：（红色字体改成对应node主机ip）
cat > kubelet.config.json <<EOF { "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1", "authentication": { "x509": { "clientCAFile": "/etc/kubernetes/cert/ca.pem" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": false } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "address": "192.168.80.10", "port": 10250, "readOnlyPort": 0, "cgroupDriver": "cgroupfs", "hairpinMode": "promiscuous-bridge", "serializeImagePulls": false, "featureGates": { "RotateKubeletClientCertificate": true, "RotateKubeletServerCertificate": true }, "clusterDomain": "cluster.local."", "clusterDNS": ["10.254.0.2"] } EOF
为各节点创建和分发 kubelet 配置文件：
scp kubelet.config.json k8s-node1:/etc/kubernetes/cert/kubelet.config.json scp kubelet.config.json k8s-node2:/etc/kubernetes/cert/kubelet.config.json scp kubelet.config.json k8s-node3:/etc/kubernetes/cert/kubelet.config.json
6、创建和分发kubelet systemd unit文件 （红色字体改成对应node主机ip）
[root@k8s-node1 ~]# cat /etc/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/usr/local/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/cert/kubelet-bootstrap.kubeconfig --cert-dir=/etc/kubernetes/cert --kubeconfig=/etc/kubernetes/cert/kubelet.kubeconfig --config=/etc/kubernetes/cert/kubelet.config.json --hostname-override=192.168.80.10 --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1 --allow-privileged=true --alsologtostderr=true --logtostderr=false --log-dir=/var/log/kubernetes --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
--hostname-override 选项，则 kube-proxy 也需要设置该选项，否则会出现找不到 Node 的情况；--bootstrap-kubeconfig：指向 bootstrap kubeconfig 文件，kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求；--cert-dir 目录创建证书和私钥文件，然后写入 --kubeconfig 文件；为各节点创建和分发 kubelet systemd unit 文件：
scp /etc/systemd/system/kubelet.service k8s-node2:/etc/systemd/system/kubelet.service
scp /etc/systemd/system/kubelet.service k8s-node3:/etc/systemd/system/kubelet.service
7、Bootstrap Token Auth和授予权限
kublet 启动时查找配置的 --kubeletconfig 文件是否存在，如果不存在则使用 --bootstrap-kubeconfig 向 kube-apiserver 发送证书签名请求 (CSR)。
kube-apiserver 收到 CSR 请求后，对其中的 Token 进行认证（事先使用 kubeadm 创建的 token），认证通过后将请求的 user 设置为 system:bootstrap:，group 设置为 system:bootstrappers，这一过程称为 Bootstrap Token Auth。
默认情况下，这个 user 和 group 没有创建 CSR 的权限，kubelet 启动失败，错误日志如下：
sudo journalctl -u kubelet -a |grep -A 2 ‘certificatesigningrequests‘ May 06 06:42:36 kube-node1 kubelet[26986]: F0506 06:42:36.314378 26986 server.go:233] failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:lemy40" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope May 06 06:42:36 kube-node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a May 06 06:42:36 kube-node1 systemd[1]: kubelet.service: Failed with result ‘exit-code‘.
解决办法是：创建一个 clusterrolebinding，将 group system:bootstrappers 和 clusterrole system:node-bootstrapper 绑定：
[root@k8s-master1 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
8、启动kubelet服务
mkdir -p /var/log/kubernetes && mkdir -p /var/lib/kubelet systemctl daemon-reload systemctl enable kubelet systemctl restart kubelet
kubelet 启动后使用 --bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求，当这个 CSR 被 approve 后，kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥和 --kubeletconfig 文件。
注意：kube-controller-manager 需要配置 --cluster-signing-cert-file 和 --cluster-signing-key-file 参数，才会为 TLS Bootstrap 创建证书和私钥。
此时kubelet的进程有，但是监听端口还未启动，需要进行下面步骤！
9、approve kubelet csr请求
可以手动或自动 approve CSR 请求。推荐使用自动的方式，因为从 v1.8 版本开始，可以自动轮转approve csr 后生成的证书。
手动approve csr请求
查看 CSR 列表：
[root@k8s-master1 ~]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-P7XcQAc2yNlXn1pUmQFxXNCdGyyt8ccVuW3bmoUZiK4 30m system:bootstrap:e7n0o5 Pending node-csr-gD18nmcyPUNWNyDQvCo2BMYiiA4K59BNkclFRWv1SAM 79m system:bootstrap:ydbwyk Pending node-csr-u2sVzVkFYnMxPIYWjXHbqRJROtTZBYzA1s2vATPLzyo 30m system:bootstrap:8w6j3n Pending
approve CSR
[root@k8s-master1 ~]# kubectl certificate approve node-csr-gD18nmcyPUNWNyDQvCo2BMYiiA4K59BNkclFRWv1SAM certificatesigningrequest.certificates.k8s.io "node-csr gD18nmcyPUNWNyDQvCo2BMYiiA4K59BNkclFRWv1SAM" approved
查看 Approve 结果：
[root@k8s-master1 ~]# kubectl describe csr node-csr-gD18nmcyPUNWNyDQvCo2BMYiiA4K59BNkclFRWv1SAM Name: node-csr-gD18nmcyPUNWNyDQvCo2BMYiiA4K59BNkclFRWv1SAM Labels: <none> Annotations: <none> CreationTimestamp: Thu, 20 Dec 2018 19:55:39 +0800 Requesting User: system:bootstrap:ydbwyk Status: Approved,Issued Subject: Common Name: system:node:192.168.80.10 Serial Number: Organization: system:nodes Events: <none>
Requesting User：请求 CSR 的用户，kube-apiserver 对它进行认证和授权；Subject：请求签名的证书信息；自动approve csr请求
创建三个 ClusterRoleBinding，分别用于自动 approve client、renew client、renew server 证书：
[root@k8s-master1 ~]# cat > csr-crb.yaml <<EOF # Approve all CSRs for the group "system:bootstrappers" kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: auto-approve-csrs-for-group subjects: - kind: Group name: system:bootstrappers apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:nodeclient apiGroup: rbac.authorization.k8s.io --- # To let a node of the group "system:nodes" renew its own credentials kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: node-client-cert-renewal subjects: - kind: Group name: system:nodes apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient apiGroup: rbac.authorization.k8s.io --- # A ClusterRole which instructs the CSR approver to approve a node requesting a # serving cert matching its client cert. kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: approve-node-server-renewal-csr rules: - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests/selfnodeserver"] verbs: ["create"] --- # To let a node of the group "system:nodes" renew its own server credentials kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: node-server-cert-renewal subjects: - kind: Group name: system:nodes apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: approve-node-server-renewal-csr apiGroup: rbac.authorization.k8s.io EOF
生效配置：
[root@k8s-master1 ~]# kubectl apply -f csr-crb.yaml
查看kubelet情况
等待一段时间(1-10 分钟)，三个节点的 CSR 都被自动 approve：
[root@k8s-master1 ~]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-P7XcQAc2yNlXn1pUmQFxXNCdGyyt8ccVuW3bmoUZiK4 35m system:bootstrap:e7n0o5 Approved,Issued node-csr-gD18nmcyPUNWNyDQvCo2BMYiiA4K59BNkclFRWv1SAM 84m system:bootstrap:ydbwyk Approved,Issued node-csr-u2sVzVkFYnMxPIYWjXHbqRJROtTZBYzA1s2vATPLzyo 35m system:bootstrap:8w6j3n Approved,Issued
所有节点均 ready：
[root@k8s-master1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.80.10 Ready <none> 69m v1.12.3 192.168.80.11 Ready <none> 36m v1.12.3 192.168.80.12 Ready <none> 36m v1.12.3
二进制搭建kubernetes多master集群【四、配置k8s node】
标签：erb https specified 使用 cts ppi which 版本 tco
原文地址：https://www.cnblogs.com/harlanzhang/p/10152508.html
