最近折腾k8s，使用kubeadm部署集群，遇到镜像下载的问题

$ kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.14.1
k8s.gcr.io/kube-controller-manager:v1.14.1
k8s.gcr.io/kube-scheduler:v1.14.1
k8s.gcr.io/kube-proxy:v1.14.1
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1

由于一些众所周知的原因，国内是不太容易下载k8s.gcr.io站点的镜像的，偶然发现微软做了一个k8s的镜像站点，真香

注意，因为k8s.gcr.io会重定向到gcr.io/google-containers，因此比如安装k8s所必须的镜像如 k8s.gcr.io/pause:3.1
需要替换成 gcr.azk8s.cn/google-containers/pause:3.1
已经使用超过一周，非常稳定。

• https://github.com/Azure/container-service-for-azure-china/blob/master/aks/README.md
• 官网介绍

•  Container Registry

  2.1 Azure Container Registry(ACR)

  Azure Container Registry(ACR) provides storage of private Docker container images, enabling fast, scalable retrieval, and network-close deployment of container workloads on Azure.

  • ACR does not provide public anonymous access functionality on Azure China, this feature is in public preview on global Azure.

  • AKS has good integration with ACR, container image stored in ACR could be pulled in AKS after Configure ACR authentication.

  2.2 Container Registry Proxy

  Since some well known container registries like docker.io, gcr.io are not accessible or very slow in China, we have set up container registry proxies on Azure China for public anonymous access:

  The first docker pull of new image will be still slow, and then image would be cached, would be much faster in the next docker pull action.

  global	proxy in China	format	example
  dockerhub(docker.io)	dockerhub.azk8s.cn	dockerhub.azk8s.cn/<repo-name>/<image-name>:<version>	dockerhub.azk8s.cn/microsoft/azure-cli:2.0.61dockerhub.azk8s.cn/library/nginx:1.15
  gcr.io	gcr.azk8s.cn	gcr.azk8s.cn/<repo-name>/<image-name>:<version>	gcr.azk8s.cn/google_containers/hyperkube-amd64:v1.13.5
  quay.io	quay.azk8s.cn	quay.azk8s.cn/<repo-name>/<image-name>:<version>	quay.azk8s.cn/deis/go-dev:v1.10.0

  Note: k8s.gcr.io would redirect to gcr.io/google-containers, following image urls are identical:

  k8s.gcr.io/pause-amd64:3.1
  gcr.io/google_containers/pause-amd64:3.1
  

  • Container Registry Proxy Example

    specify defaultBackend.image.repository as gcr.azk8s.cn/google_containers/defaultbackend in nginx-ingress chart since original k8s.gcr.io does not work in Azure China:

    helm install stable/nginx-ingress --set defaultBackend.image.repository=gcr.azk8s.cn/google_containers/defaultbackend --set defaultBackend.image.tag=1.4
    

  3. Install kubectl

  az aks install-cli command is used to download kubectl binary, it works on Azure China from version 2.0.61 or later, another alternative is use following command to download kubectl if don‘t have azure-cli:

  # docker run -v ${HOME}:/root -v /usr/local/bin/:/kube -it dockerhub.azk8s.cn/microsoft/azure-cli:2.0.61
  root@09feb993f352:/# az cloud set --name AzureChinaCloud
  root@09feb993f352:/# az aks install-cli --install-location /kube/kubectl

  run sudo az aks install-cli if hit following permission error

  Connection error while attempting to download client ([Errno 13] Permission denied: ‘/usr/local/bin/kubectl‘
  

  4. Install helm

  Follow detailed installation steps here.

  • Example:

  # Install wordpress
  helm repo add bitnami https://charts.bitnami.com/bitnami
  helm install bitnami/wordpress --set global.imageRegistry=dockerhub.azk8s.cn
  
  # Install nginx-ingress
  helm repo add stable https://mirror.azure.cn/kubernetes/charts/
  helm install stable/nginx-ingress  --set defaultBackend.image.repository=gcr.azk8s.cn/google_containers/defaultbackend
  

  Note: All kubernetes related binaries on github could be found under https://mirror.azk8s.cn/kubernetes, e.g. helm, charts, etc.

  5. Cluster autoscaler

  Note: AKS integrated Cluster-autoscaler is not availalbe on Azure China now since it‘s still in Preview on Global Azure, instead following autoscaler is supported on Azure China now, it supports both VMAS and VMSS: Follow detailed steps in Cluster Autoscaler on Azure and in Deployment config of aks-cluster-autoscaler.yaml:

  • use gcr.azk8s.cn/google-containers/cluster-autoscaler:version instead of gcr.io/google-containers/cluster-autoscaler:version

  • add following environment variable:

    - name: ARM_CLOUD
      value: AzureChinaCloud
    

    Here is the complete Deployment config example.

  Hands on

  • run a simple web application on AKS cluster
  • AKS workshop

  Known issues

  • RBAC related issues(RABC is enabled on AKS cluster): https://github.com/andyzhangx/demo/blob/master/issues/rbac-issues.md

  Tips

  • For production usage:

    • agent VM size should have at least 8 CPU cores(e.g. D4_v2) since k8s components would also occupy CPU, memory resources on the node, details about AKS resource reservation.
    • it‘s better set a bigger os disk size on agent VM in AKS cluster creation, e.g. set --node-osdisk-size 128, original 30GB os disk size is not enough since all images are stored on os disk.

  • GPU workload support best practices on Azure China

  Links

  • AKS doc: https://docs.azure.cn/zh-cn/aks/

  • Azure Kubernetes Service Issues

  • Frequently asked questions about Azure Container Service (AKS)