标签：state -o override structure restore ellipsize tde pil 遇到
1.[root@k8s-master1 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}一个疑问:
kubectl version获取的client版本和server版本数据来自哪里呢?
client版本是kubectl执行文件的版本.
server版本是kube-apiserver执行文件的版本.
如果把kubectl的配置文件,~/.kube/config文件移走,无法识别server地址,读取不到server版本参数.报错见下:
[root@k8s-master1 .kube]# kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Error from server (NotFound): the server could not find the requested resource2.
细节准备
集群三个节点,三个节点都是master节点同时也是node节点.
二进制部署,配置文件不变,替换执行文件.
需要升级的组件:
master节点组件:
kubectl
kube-apiserver
kube-controller-manager
kube-scheduler
node节点组件:
kube-proxy
kubelet
暂不升级的组件,需要升级后面再单独升级
数据库组件:
etcd
网络组件:
flannel
升级前,执行文件的存放路径和权限检索记录:
[root@k8s-master1 bin]# pwd
/opt/k8s/bin
[root@k8s-master1 bin]# ll
total 819520
-rwxr-xr-x 1 k8s? root? 10376657 Feb 26 13:27 cfssl
-rwxr-xr-x 1 k8s? root? 6595195 Feb 26 13:27 cfssl-certinfo
-rwxr-xr-x 1 k8s? root? 2277873 Feb 26 13:27 cfssljson
-rwxr-xr-x 1 k8s? root? ? ? 1752 Feb 26 13:21 environment.sh
-rwxr-xr-x 1 k8s? root? 19266976 Feb 26 14:10 etcd
-rwxr-xr-x 1 k8s? root? 16018720 Feb 26 14:10 etcdctl
-rwxr-xr-x 1 k8s? root? 36327752 Feb 26 15:43 flanneld
-rwxr-xr-x 1 root root? 54038482 Feb 28 10:17 kubeadm
-rwxr-xr-x 1 k8s? root 192793815 Feb 27 14:40 kube-apiserver
-rwxr-xr-x 1 k8s? root 162973612 Feb 27 14:40 kube-controller-manager
-rwxr-xr-x 1 k8s? root? 54308597 Feb 26 13:39 kubectl
-rwxr-xr-x 1 root root 176661512 Feb 28 10:17 kubelet
-rwxr-xr-x 1 root root? 50330867 Feb 28 10:17 kube-proxy
-rwxr-xr-x 1 k8s? root? 57184656 Feb 27 14:40 kube-scheduler
-rwxr-xr-x 1 k8s? root? ? ? 2139 Feb 26 15:43 mk-docker-opts.sh
[root@k8s-master1 bin]#[root@k8s-master1 bin]# echo $PATH
/opt/k8s/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin
[root@k8s-master1 bin]#[root@k8s-master1 ~]# cat .bashrc
# .bashrc
# User specific aliases and functions
alias rm=‘rm -i‘
alias cp=‘cp -i‘
alias mv=‘mv -i‘
# Source global definitions
if [ -f /etc/bashrc ]; then
? ? ? ? . /etc/bashrc
fi
export PATH=/opt/k8s/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
[root@k8s-master1 ~]#3.
升级文件
下载地址参考:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md
下载的文件,参考见下:
[root@k8s-master1 v1.15.0]# pwd
/root/k8s/v1.15.0
[root@k8s-master1 v1.15.0]# ls
?kubernetes-client-linux-amd64.tar.gz? kubernetes-server-linux-amd64.tar.gz? kubernetes-src.tar.gz? kubernetes.tar.gz
[root@k8s-master1 v1.15.0]#一般只需要
?kubernetes-client-linux-amd64.tar.gz? kubernetes-server-linux-amd64.tar.gz4.
升级最简单的kubectl
先备份kubectl执行文件,如果有问题可以回滚.三个节点都一样操作.
[root@k8s-master1 bin]# pwd
/opt/k8s/bin
[root@k8s-master1 bin]# cp kubectl kubectl.bk.v1.12
[root@k8s-master1 bin]# ll
total 872556
-rwxr-xr-x 1 k8s? root? 10376657 Feb 26 13:27 cfssl
-rwxr-xr-x 1 k8s? root? 6595195 Feb 26 13:27 cfssl-certinfo
-rwxr-xr-x 1 k8s? root? 2277873 Feb 26 13:27 cfssljson
-rwxr-xr-x 1 k8s? root? ? ? 1752 Feb 26 13:21 environment.sh
-rwxr-xr-x 1 k8s? root? 19266976 Feb 26 14:10 etcd
-rwxr-xr-x 1 k8s? root? 16018720 Feb 26 14:10 etcdctl
-rwxr-xr-x 1 k8s? root? 36327752 Feb 26 15:43 flanneld
-rwxr-xr-x 1 root root? 54038482 Feb 28 10:17 kubeadm
-rwxr-xr-x 1 k8s? root 192793815 Feb 27 14:40 kube-apiserver
-rwxr-xr-x 1 k8s? root 162973612 Feb 27 14:40 kube-controller-manager
-rwxr-xr-x 1 k8s? root? 54308597 Feb 26 13:39 kubectl
-rwxr-xr-x 1 root root? 54308597 Jul 30 15:10 kubectl.bk.v1.12
-rwxr-xr-x 1 root root 176661512 Feb 28 10:17 kubelet
-rwxr-xr-x 1 root root? 50330867 Feb 28 10:17 kube-proxy
-rwxr-xr-x 1 k8s? root? 57184656 Feb 27 14:40 kube-scheduler
-rwxr-xr-x 1 k8s? root? ? ? 2139 Feb 26 15:43 mk-docker-opts.sh删除kubectl文件
[root@k8s-master1 bin]# rm -rf kubectl复制新版本的kubectl到目录/opt/k8s/bin
[root@k8s-master1 v1.15.0]# pwd
/root/k8s/v1.15.0
[root@k8s-master1 v1.15.0]# cp kubernetes/client/bin/kubectl /opt/k8s/bin
[root@k8s-master1 v1.15.0]# scp kubernetes/client/bin/kubectl k8s-master2:/opt/k8s/bin
kubectl? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 41MB? 10.3MB/s? 00:04? ?
[root@k8s-master1 v1.15.0]# scp kubernetes/client/bin/kubectl k8s-master3:/opt/k8s/bin
kubectl ? ? ??执行命令看看
[root@k8s-master1 v1.15.0]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}可以看到kubectl client已经升级到v1.15.0
5.
升级
kube-apiserver
kube-controller-manager
kube-scheduler
5.1
这里是测试环境,随便搞都可以,如果是正式环境,慎重操作,最好一个一个master节点去升级操作.
备份操作忽略,如果没有旧版本安装包的存档,请自行备份.
停止kube-scheduler,kube-controller-manager,kube-apiserver服务,三节点同时操作:
[root@k8s-master1 bin]# systemctl stop kube-scheduler
[root@k8s-master2 bin]# systemctl stop kube-scheduler
[root@k8s-master3 bin]# systemctl stop kube-scheduler
[root@k8s-master1 bin]# systemctl stop kube-controller-manager
[root@k8s-master2 bin]# systemctl stop kube-controller-manager
[root@k8s-master3 bin]# systemctl stop kube-controller-manager
[root@k8s-master1 bin]# systemctl stop kube-apiserver
[root@k8s-master2 bin]# systemctl stop kube-apiserver
[root@k8s-master3 bin]# systemctl stop kube-apiserver5.2
删除旧版本执行文件
[root@k8s-master1 bin]# rm -rf kube-apiserver kube-scheduler kube-controller-manager
[root@k8s-master2 bin]# rm -rf kube-apiserver kube-scheduler kube-controller-manager
[root@k8s-master3 bin]# rm -rf kube-apiserver kube-scheduler kube-controller-manager5.3
复制新版本执行文件到/opt/k8s/bin目录
[root@k8s-master1 bin]# cp kube-apiserver kube-controller-manager kube-scheduler /opt/k8s/bin
[root@k8s-master1 bin]# scp kube-apiserver kube-controller-manager kube-scheduler k8s-master2:/opt/k8s/bin
kube-apiserver? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 157MB? 9.8MB/s? 00:16? ?
kube-controller-manager? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 111MB? 12.3MB/s? 00:09? ?
kube-scheduler? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 37MB? 9.3MB/s? 00:04? ?
[root@k8s-master1 bin]# scp kube-apiserver kube-controller-manager kube-scheduler k8s-master3:/opt/k8s/bin
kube-apiserver? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 157MB? 9.8MB/s? 00:16? ?
kube-controller-manager? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 111MB? 12.3MB/s? 00:09? ?
kube-scheduler? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 37MB? 12.3MB/s? 00:03? ?
[root@k8s-master1 bin]# pwd
/root/k8s/v1.15.0/kubernetes/server/bin
[root@k8s-master1 bin]#5.4
修改权限,参考命令
chown -R k8s:root /opt/k8s/bin5.5
遇到的报错
启动服务
kube-apiserver起不来,报错:
Jul 30 16:04:47 k8s-master1 kube-apiserver[108963]: error: enable-admission-plugins plugin "Initializers" is unknown
[root@k8s-master1 kubernetes]# cat /etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
ExecStart=/opt/k8s/bin/kube-apiserver --enable-admission-plugins=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota --anonymous-auth=false --experimental-encryption-provider-config=/etc/kubernetes/encryption-config.yaml --advertise-address=192.168.32.128 --bind-address=192.168.32.128 --insecure-port=0 --authorization-mode=Node,RBAC --runtime-config=api/all --enable-bootstrap-token-auth --service-cluster-ip-range=10.254.0.0/16 --service-node-port-range=8400-9000 --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem --client-ca-file=/etc/kubernetes/cert/ca.pem --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem --service-account-key-file=/etc/kubernetes/cert/ca-key.pem --etcd-cafile=/etc/kubernetes/cert/ca.pem --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem --etcd-servers=https://192.168.32.128:2379,https://192.168.32.129:2379,https://192.168.32.130:2379 --enable-swagger-ui=true --allow-privileged=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/kube-apiserver-audit.log --event-ttl=1h --alsologtostderr=true --logtostderr=false --log-dir=/var/log/kubernetes --v=2
Restart=on-failure
RestartSec=5
Type=notify
User=k8s
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
[root@k8s-master1 kubernetes]#--enable-admission-plugins=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota删除Initializers,再启动正常.
[root@k8s-master1 kubernetes]# systemctl daemon-reload &&systemctl start kube-apiserver
● kube-apiserver.service - Kubernetes API Server
? Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
? Active: active (running) since Tue 2019-07-30 16:09:35 CST; 2min 38s ago
? ? Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 110572 (kube-apiserver)
? Memory: 309.6M
? CGroup: /system.slice/kube-apiserver.service
? ? ? ? ? └─110572 /opt/k8s/bin/kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota --anonymous-auth=false --experimental-encryption-provider-config=/etc/kubernetes/encryption-config.yaml --advertise-address=192.168.32.128 --bind-address=192.168.32.128 --insecure-port=0 --authorization-mode=Node,RBAC --runtime-config=api/all --enable-bootstrap-token-auth --service-cluster-ip-range=10.254.0.0/16 --service-node-port-range=8400-9000 --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem --client-ca-file=/etc/kubernetes/cert/ca.pem --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem --service-account-key-file=/etc/kubernetes/cert/ca-key.pem --etcd-cafile=/etc/kubernetes/cert/ca.pem --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem --etcd-servers=https://192.168.32.128:2379,https://192.168.32.129:2379,https://192.168.32.130:2379 --enable-swagger-ui=true --allow-privileged=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/kube-apiserver-audit.log --event-ttl=1h --alsologtostderr=true --logtostderr=false --log-dir=/var/log/kubernetes --v=2其他两个节点也这样操作即可.
5.6
启动scheduler和controller-manager服务,参考命令:
systemctl start kube-controller-manager &&systemctl start kube-scheduler5.7
执行命令检索版本
[root@k8s-master1 kubernetes]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}6.
升级
kube-proxy
kubelet
6.1停止服务
[root@k8s-master1 kubernetes]# systemctl stop kubelet && systemctl stop kube-proxy
[root@k8s-master2 bin]# systemctl stop kubelet && systemctl stop kube-proxy
[root@k8s-master3 bin]# systemctl stop kubelet && systemctl stop kube-proxy6.2
删除旧版本执行文件
[root@k8s-master1 bin]# rm -rf kubelet kube-proxy
[root@k8s-master2 bin]#? rm -rf kubelet kube-proxy
[root@k8s-master3 bin]#? rm -rf kubelet kube-proxy
[root@k8s-master3 bin]# pwd
/opt/k8s/bin6.3
复制新版本执行文件到目录
[root@k8s-master1 bin]# pwd
/root/k8s/v1.15.0/kubernetes/server/bin
[root@k8s-master1 bin]# cp kubelet kube-proxy /opt/k8s/bin
[root@k8s-master1 bin]# scp kubelet kube-proxy k8s-master2:/opt/k8s/bin
kubelet? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 114MB? 11.4MB/s? 00:10? ?
kube-proxy? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 35MB? 11.8MB/s? 00:03? ?
[root@k8s-master1 bin]# scp kubelet kube-proxy k8s-master3:/opt/k8s/bin
kubelet? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 114MB? 12.7MB/s? 00:09? ?
kube-proxy?6.4
修改下权限
[root@k8s-master1 bin]# chown -R k8s:root /opt/k8s/bin
[root@k8s-master1 bin]# ssh root@k8s-master2 "chown -R k8s:root /opt/k8s/bin"
[root@k8s-master1 bin]# ssh root@k8s-master3 "chown -R k8s:root /opt/k8s/bin"?6.5
启动服务
kube-proxy启动正常
6.6
kubelet服务启动不成功
[root@k8s-master1 bin]# systemctl status kubelet -l
● kubelet.service - Kubernetes Kubelet
? Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
? Active: activating (auto-restart) (Result: exit-code) since Tue 2019-07-30 16:53:57 CST; 1s ago
? ? Docs: https://github.com/GoogleCloudPlatform/kubernetes
? Process: 122129 ExecStart=/opt/k8s/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig --cert-dir=/etc/kubernetes/cert --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kubelet.config.json --hostname-override=k8s-master1 --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest --allow-privileged=true --alsologtostderr=true --logtostderr=false --log-dir=/var/log/kubernetes --v=2 (code=exited, status=255)
Main PID: 122129 (code=exited, status=255)
Jul 30 16:53:57 k8s-master1 kubelet[122129]: --tls-cert-file string? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet‘s --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jul 30 16:53:57 k8s-master1 kubelet[122129]: --tls-cipher-suites strings? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used. Possible values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_SHA (DEPRECATED: This parameter should be set via the config file specified by the Kubelet‘s --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jul 30 16:53:57 k8s-master1 kubelet[122129]: --tls-min-version string? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet‘s --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jul 30 16:53:57 k8s-master1 kubelet[122129]: --tls-private-key-file string? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet‘s --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jul 30 16:53:57 k8s-master1 kubelet[122129]: -v, --v Level? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? number for the log level verbosity
Jul 30 16:53:57 k8s-master1 kubelet[122129]: --version version[=true]? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Print version information and quit
Jul 30 16:53:57 k8s-master1 kubelet[122129]: --vmodule moduleSpec? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? comma-separated list of pattern=N settings for file-filtered logging
Jul 30 16:53:57 k8s-master1 kubelet[122129]: --volume-plugin-dir string? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")
Jul 30 16:53:57 k8s-master1 kubelet[122129]: --volume-stats-agg-period duration? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.? To disable volume calculations, set to 0. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet‘s --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jul 30 16:53:57 k8s-master1 kubelet[122129]: F0730 16:53:57.422346? 122129 server.go:156] unknown flag: --allow-privileged
[root@k8s-master1 bin]#这是一个bug报错:
参考下面解决两篇文章解决的
https://github.com/microsoft/SDN/issues/379
https://github.com/wk8/SDN/commit/8db8f06cd5ccdc91eb74ce1d00041597881cd0c1
[root@k8s-master1 bin]# cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/k8s/bin/kubelet ? --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig ? --cert-dir=/etc/kubernetes/cert ? --kubeconfig=/etc/kubernetes/kubelet.kubeconfig ? --config=/etc/kubernetes/kubelet.config.json ? --hostname-override=k8s-master1 ? --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest ? --allow-privileged=true ? --alsologtostderr=true ? --logtostderr=false ? --log-dir=/var/log/kubernetes ? --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target把? --allow-privileged=true \这个参数删除后,启动正常.
[root@k8s-master1 bin]# systemctl daemon-reload &&systemctl restart kubelet
[root@k8s-master1 bin]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
? Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
? Active: active (running) since Tue 2019-07-30 17:11:19 CST; 9s ago
? ? Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 127177 (kubelet)
? Memory: 61.0M
? CGroup: /system.slice/kubelet.service
? ? ? ? ? └─127177 /opt/k8s/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig --cert-dir=/etc/kubernetes/cert --kubec...
Jul 30 17:11:20 k8s-master1 kubelet[127177]: I0730 17:11:20.658288? 127177 state_mem.go:84] [cpumanager] updated default cpuset: ""
Jul 30 17:11:20 k8s-master1 kubelet[127177]: I0730 17:11:20.658314? 127177 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
Jul 30 17:11:20 k8s-master1 kubelet[127177]: I0730 17:11:20.658337? 127177 state_checkpoint.go:100] [cpumanager] state checkpoint: restored ...eckpoint
Jul 30 17:11:20 k8s-master1 kubelet[127177]: I0730 17:11:20.658345? 127177 state_checkpoint.go:101] [cpumanager] state checkpoint: defaultCPUSet:
Jul 30 17:11:20 k8s-master1 kubelet[127177]: I0730 17:11:20.658427? 127177 server.go:1025] Using root directory: /var/lib/kubelet
Jul 30 17:11:20 k8s-master1 kubelet[127177]: I0730 17:11:20.658452? 127177 kubelet.go:306] Watching apiserver
Jul 30 17:11:20 k8s-master1 kubelet[127177]: I0730 17:11:20.688516? 127177 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Jul 30 17:11:20 k8s-master1 kubelet[127177]: I0730 17:11:20.688542? 127177 client.go:104] Start docker client with request timeout=2m0s
Jul 30 17:11:20 k8s-master1 kubelet[127177]: W0730 17:11:20.751096? 127177 docker_service.go:561] Hairpin mode set to "promiscuous-bridge" b...in-veth"
Jul 30 17:11:20 k8s-master1 kubelet[127177]: I0730 17:11:20.753080? 127177 docker_service.go:238] Hairpin mode set to "hairpin-veth"
Hint: Some lines were ellipsized, use -l to show in full.7.
升级完成
[root@k8s-master1 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master1 ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.32.127:8443
CoreDNS is running at https://192.168.32.127:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump‘.
[root@k8s-master1 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
[root@k8s-master1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready <none> 98d v1.15.0
k8s-master2 Ready <none> 98d v1.15.0
k8s-master3 Ready <none> 98d v1.15.0
[root@k8s-master1 ~]# k8s实践19:kubernetes二进制部署集群v1.12升级v1.15
标签：state -o override structure restore ellipsize tde pil 遇到
原文地址：https://blog.51cto.com/goome/2424921
