码迷，mamicode.com
首页 > 其他好文 > 详细

kubeadm部署k8s1.9高可用集群--4部署master节点

时间：2017-12-27 22:34:44      阅读：4567      评论：0      收藏：0      [点我收藏+]

标签：vim   lin   命名   cloud   instance   iss   volume   enabled   apt   

部署master节点

kubernetes master 节点包含的组件：

  • kube-apiserver
  • kube-scheduler
  • kube-controller-manager

本文档介绍部署一个三节点高可用 master 集群的步骤，分别命名为k8s-host1、k8s-host2、k8s-host3：

  • k8s-host1：172.16.120.154
  • k8s-host2：172.16.120.155
  • k8s-host3：172.16.120.156

安装docker

在每台主机安装docker。建议使用docker v1.12 , 但是 v1.11, v1.13 and 17.03 也可以正常工作. 版本 17.06+ 可能工作, 但是没有被k8s团队测试过。
PS：kubelet设置的cgroup driver应该和Docker设置的cgroup driver一致。这边统一为systemd。

yum install -y docker
systemctl enable docker && systemctl start docker

安装kubeadm, kubelet 和 kubectl

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
  • 通过运行命令setenforce 0 禁用selinux，主要为了允许容器可以访问主机文件系统和pod networks的需要。
  • Some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed。所以需要执行下面命令：
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

安装master

kubeadm init

kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16
  • apiserver-advertise-address该参数一般指定为haproxy+keepalived 的vip。
  • pod-network-cidr 主要是在搭建pod network（calico）时候需要在init时候指定。

出现如下错误：

[root@k8s-node04 ~]# kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16

[init] Using Kubernetes version: v1.9.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Some fatal errors occurred:
        [ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

解决办法：
1：kubelet启动参数增加--fail-swap-on=false

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
Environment="KUBELET_SWAP_ARGS=--fail-swap-on=false"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS $KUBELET_SWAP_ARGS

2：kubeadm init 参数增加ignore-preflight-errors

 kubeadm init --apiserver-advertise-address=172.16.120.200  --pod-network-cidr=192.168.0.0/16 www.bajieyy.org--ignore-preflight-errors ‘Swap‘

3：在物理机上可能并不会出现该问题。

最终安装成功

[root@k8s-host1 manifests]# kubeadm init --apiserver-advertise-address=172.16.120.200  --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors ‘Swap‘
[init] Using Kubernetes version: v1.9.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING Swap]: running with swap on is not supported. Please disable swap
        [WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-host1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.120.200]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 30.002346 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node k8s-host1 as master by adding a label and a taint
[markmaster] Master k8s-host1 tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 26ea24.cb987cb5cf6f29a6
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join www.dijiuyy.com--token 26ea24.cb987cb5cf6f29a6 172.16.120.200:6443 --discovery-token-ca-cert-hash sha256:a2d5e0391695bb122d4a43dc951f93934a2aaa1360e00556401a49ea0ec2bdab

PS 需要记下 kubeadm join --token 26ea24.cb987cb5cf6f29a6 172.16.120.200:6443 --discovery-token-ca-cert-hash sha256:a2d5e0391695bb122d4a43dc951f93934a2aaa1360e00556401a49ea0ec2bdab，便于以后node加入到k8s集群中。

将kube-apiserver指向上一节部署的etcd集群

首先需要将上节生成的etcd ca等证书cp到mater主机的/etc/etcd/ssl/下

默认kubeadm安装了一个etcd节点，非集群，非加密，所以我们必须对kube-apiserver.yaml做修改。
修改后的kube-apiserver.yaml：

apiVersion: v1
kind: Pod
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --allow-privileged=true
    - --requestheader-username-headers=X-Remote-User
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --service-cluster-ip-range=10.96.0.0/12
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --enable-bootstrap-token-auth=true
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --insecure-port=0
    - --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota
    - --requestheader-allowed-names=front-proxy-client
    - --advertise-address=172.16.120.200
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --secure-port=6443
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --authorization-mode=Node,RBAC
    - --etcd-servers=https://172.16.120.151:2379,https://172.16.120.152:2379,https://172.16.120.153:2379
    - --etcd-cafile=/etc/etcd/ssl/ca.pem
    - --etcd-certfile=/etc/etcd/ssl/etcd.pem
    - --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem
    image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.0
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 172.16.120.200
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: ca-certs-etc-pki
      readOnly: true
    - mountPath: /etc/etcd/ssl
      name: ca-certs-etc-etcd-ssl
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: ca-certs-etc-pki
  - hostPath:
      path: /etc/etcd/ssl
      type: DirectoryOrCreate
    name: ca-certs-etc-etcd-ssl
status: {}

注意添加了以下四项：

--etcd-cafile=/etc/etcd/ssl/ca.pem  
--etcd-certfile=/etc/etcd/ssl/etcd.pem  
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem    
--etcd-servers=https://172.16.120.151:2379,https://172.16.120.152:2379,https://172.16.120.153:2379
# 以及挂载
 - mountPath: /etc/etcd/ssl
      name: ca-certs-etc-etcd-ssl
      readOnly: true
      
 - hostPath:
      path: /etc/etcd/ssl
      type: DirectoryOrCreate
    name: ca-certs-etc-etcd-ssl

确认第一台master三大组件都成功启动

kubectl get componentstatuses
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-1               Healthy   {"health": "true"}
etcd-0               Healthy   {"health": "true"}
etcd-2               Healthy   {"health": "true"}

安装其他主机

我一般的做法是，依旧使用kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors ‘Swap‘
主要目的是下载k8s相关组件docker镜像等，其实就是让kubeadm把容易忽略的工作都做了。

然后将第一台/etc/kubernetes/下所有文件拷贝到其他master主机对应目录下，直至其他机器master三大组件启动成功。

至此一个完整的master高可用集群搭建完毕。

kubeadm部署k8s1.9高可用集群--4部署master节点

标签：vim   lin   命名   cloud   instance   iss   volume   enabled   apt   

原文地址：https://www.cnblogs.com/tianshifu/p/8127831.html
踩
(0)
赞
(0)
   
举报
评论 一句话评论（0）
登录后才能评论！
分享档案
更多>
2021年07月29日 (22)
2021年07月28日 (40)
2021年07月27日 (32)
2021年07月26日 (79)
2021年07月23日 (29)
2021年07月22日 (30)
2021年07月21日 (42)
2021年07月20日 (16)
2021年07月19日 (90)
2021年07月16日 (35)
周排行
mamicode.com排行更多图片
更多
友情链接
兰亭集智   国之画   百度统计   站长统计   阿里云   chrome插件   新版天听网
关于我们 - 联系我们 - 留言反馈
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码！