码迷,mamicode.com
首页 > Windows程序 > 详细

heapster源码分析——kubelet的api调用分析

时间:2017-11-11 14:10:37      阅读:452      评论:0      收藏:0      [点我收藏+]

标签:star   agg   目录   自己   hid   就会   hub   post方法   led   

一、heapster简介

什么是Heapster?

Heapster是容器集群监控和性能分析工具,天然的支持Kubernetes和CoreOS。
Kubernetes有个出名的监控agent---cAdvisor。在每个kubernetes Node上都会运行cAdvisor,它会收集本机以及容器的监控数据(cpu,memory,filesystem,network,uptime)。
在较新的版本中,K8S已经将cAdvisor功能集成到kubelet组件中。每个Node节点可以直接进行web访问。

cAdvisor web界面访问: http://< Node-IP >:4194
cAdvisor也提供Restful API: https://github.com/google/cadv ... pi.md

Heapster是一个收集者,将每个Node上的cAdvisor的数据进行汇总,然后导到第三方工具(如InfluxDB)。

技术分享

 

 

二、heapster调用kubelet源码分析

1、整体源码

heapster整体的源码分析可以参考文章:https://segmentfault.com/a/1190000008863353#articleHeader2

上面链接中的文章中会对heapster从启动开始进行分析,主要讲述了下面内容:

  • main()
  • 创建数据源
  • 创建后端服务
  • 创建数据processors
  • 获取源数据并存储
  • heapster API创建

给上述文章的作者点个赞,思路真的非常清晰。不过笔者最关注的点是heapster如何进行的kubelet调用。

2、调用kubelet源码分析

从上述文章提到的 NewKubeletProvider 函数中,我们找到了创建kubeClient和kubeletClient的地方,见下面代码的飘黄部分:

func NewKubeletProvider(uri *url.URL) (MetricsSourceProvider, error) {
    // 创建kubernetes master及kubelet client相关的配置
    kubeConfig, kubeletConfig, err := GetKubeConfigs(uri)
    if err != nil {
        return nil, err
    }
    // 创建kubeClient及kubeletClient
    kubeClient := kube_client.NewOrDie(kubeConfig)
    kubeletClient, err := NewKubeletClient(kubeletConfig)
    if err != nil {
        return nil, err
    }

    // 获取下所有的Nodes,测试下创建的client是否能正常通讯
    if _, err := kubeClient.Nodes().List(kube_api.ListOptions{
        LabelSelector: labels.Everything(),
        FieldSelector: fields.Everything()}); err != nil {
        glog.Errorf("Failed to load nodes: %v", err)
    }

    // 监控k8s的nodes变更
    // 这里会创建协程进行watch,便于后面调用nodeLister.List()列出所有的nodes。
    // 该Watch的实现,需要看下apiServer中的实现,后面会进行讲解
    lw := cache.NewListWatchFromClient(kubeClient, "nodes", kube_api.NamespaceAll, fields.Everything())
    nodeLister := &cache.StoreToNodeLister{Store: cache.NewStore(cache.MetaNamespaceKeyFunc)}
    reflector := cache.NewReflector(lw, &kube_api.Node{}, nodeLister.Store, time.Hour)
    reflector.Run()
    // 结构在前面介绍过
    return &kubeletProvider{
        nodeLister:    nodeLister,
        reflector:     reflector,
        kubeletClient: kubeletClient,
    }, nil
}

1)heapster源码目录

我们发现这个函数所在的文件为kubelet.go,所在的包为metrics/sources/kubelet

我们可以看到,kubelet调用相关的代码都在这个里面:

技术分享

2)heapster调用kubelet的API

我们进入到NewKubeletClient函数看一下:

// 传入的是KubeletClientConfig,通过读取相关配置,来初始化client
func NewKubeletClient(kubeletConfig *kubelet_client.KubeletClientConfig) (*KubeletClient, error) {
    transport, err := kubelet_client.MakeTransport(kubeletConfig)
    if err != nil {
        return nil, err
    }
    c := &http.Client{ // 此处可以看到,是http调用
        Transport: transport,
        Timeout:   kubeletConfig.HTTPTimeout,
    }
    return &KubeletClient{
        config: kubeletConfig,
        client: c,
    }, nil
}

接下来我们看一下kubelet_client.go中KubeletClient有哪些方法:

(i)获取所有的containersInfo

第一个,获取所有的container统计信息,如果对cAdvisor比较了解的话,可以进入到该方法的返回值的ContainerInfo里面看下,是非常全面的容器监控信息(后续会有cAdvisor的源码分析,敬请期待)

func (self *KubeletClient) getAllContainers(url string, start, end time.Time) ([]cadvisor.ContainerInfo, error) {
    // Request data from all subcontainers.此处是构造requestBody
    request := statsRequest{
        ContainerName: "/",
        NumStats:      1,
        Start:         start, // 2017-11-10T06:46:17Z 这种utc格式的时间戳
        End:           end,
        Subcontainers: true,
    }
    body, err := json.Marshal(request)
    if err != nil {
        return nil, err
    }
    req, err := http.NewRequest("POST", url, bytes.NewBuffer(body))
    if err != nil {
        return nil, err
    }
// 设置请求头 req.Header.Set(
"Content-Type", "application/json") var containers map[string]cadvisor.ContainerInfo client := self.client if client == nil { client = http.DefaultClient } err = self.postRequestAndGetValue(client, req, &containers) if err != nil { return nil, fmt.Errorf("failed to get all container stats from Kubelet URL %q: %v", url, err) } result := make([]cadvisor.ContainerInfo, 0, len(containers)) for _, containerInfo := range containers { cont := self.parseStat(&containerInfo) if cont != nil { result = append(result, *cont) } } return result, nil }

我们可以看到上面的源码其实是构造了一个http的post请求。那么我们在本地通过restClient模拟一下:

http://ip:10255/stats/container/(为什么用10255端口呢?在上面的源码中初始化NewKubeletClient的地方,我们进入到GetKubeClientConfig看一下就会发现,用的是kubelet的默认开放端口,在configs.go文件中)

技术分享

返回值部分摘录如下(内容太多,折叠一下):

技术分享
 "/": {
        "name": "/",
        "subcontainers": [
            {
                "name": "/docker"
            },
            {
                "name": "/init.scope"
            },
            {
                "name": "/kube-proxy"
            },
            {
                "name": "/kubepods"
            },
            {
                "name": "/system.slice"
            },
            {
                "name": "/user.slice"
            }
        ],
        "spec": {
            "creation_time": "2017-10-14T17:16:59.086488213+08:00",
            "has_cpu": true,
            "cpu": {
                "limit": 1024,
                "max_limit": 0,
                "mask": "0-3",
                "period": 100000
            },
            "has_memory": true,
            "memory": {
                "limit": 33604153344,
                "reservation": 9223372036854771712
            },
            "has_network": true,
            "has_filesystem": true,
            "has_diskio": true,
            "has_custom_metrics": false
        },
        "stats": [
            {
                "timestamp": "2017-11-10T14:29:25.105117933+08:00",
                "cpu": {
                    "usage": {
                        "total": 916886132160774,
                        "per_cpu_usage": [
                            225713500943530,
                            230866928282148,
                            229896967301966,
                            230408735633130
                        ],
                        "user": 443962220000000,
                        "system": 438380440000000
                    },
                    "cfs": {
                        "periods": 0,
                        "throttled_periods": 0,
                        "throttled_time": 0
                    },
                    "load_average": 0
                },
                "diskio": {
                    "io_service_bytes": [
                        {
                            "major": 7,
                            "minor": 4,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 15,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 14,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 12,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 0,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 0,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 11,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 4,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 253,
                            "minor": 0,
                            "stats": {
                                "Async": 61989669888,
                                "Read": 23649280,
                                "Sync": 17028616192,
                                "Total": 79018286080,
                                "Write": 78994636800
                            }
                        },
                        {
                            "major": 7,
                            "minor": 7,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 9,
                            "minor": 0,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 13,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 6,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 6,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 5,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 5,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 3,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 7,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 253,
                            "minor": 16,
                            "stats": {
                                "Async": 258048,
                                "Read": 21504,
                                "Sync": 4773888,
                                "Total": 5031936,
                                "Write": 5010432
                            }
                        },
                        {
                            "major": 7,
                            "minor": 3,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 2,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 1,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 10,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 9,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 8,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 2,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 1,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        }
                    ],
                    "io_serviced": [
                        {
                            "major": 7,
                            "minor": 4,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 9,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 8,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 9,
                            "minor": 0,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 1,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 15,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 7,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 1,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 4,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 0,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 253,
                            "minor": 0,
                            "stats": {
                                "Async": 2738092,
                                "Read": 2965,
                                "Sync": 4157380,
                                "Total": 6895472,
                                "Write": 6892507
                            }
                        },
                        {
                            "major": 1,
                            "minor": 13,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 5,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 14,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 5,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 253,
                            "minor": 16,
                            "stats": {
                                "Async": 57,
                                "Read": 7,
                                "Sync": 32,
                                "Total": 89,
                                "Write": 82
                            }
                        },
                        {
                            "major": 1,
                            "minor": 3,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 2,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 6,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 2,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 11,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 10,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 6,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 12,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 7,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 3,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 0,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        }
                    ]
                },
                "memory": {
                    "usage": 12189630464,
                    "cache": 288108544,
                    "rss": 3631652864,
                    "swap": 0,
                    "working_set": 7756312576,
                    "failcnt": 0,
                    "container_data": {
                        "pgfault": 8947531,
                        "pgmajfault": 87
                    },
                    "hierarchical_data": {
                        "pgfault": 8947531,
                        "pgmajfault": 87
                    }
                },
                "network": {
                    "name": "cbr0",
                    "rx_bytes": 1885776219,
                    "rx_packets": 8199780,
                    "rx_errors": 0,
                    "rx_dropped": 0,
                    "tx_bytes": 1854690043,
                    "tx_packets": 8765796,
                    "tx_errors": 0,
                    "tx_dropped": 0,
                    "interfaces": [
                        {
                            "name": "cbr0",
                            "rx_bytes": 1885776219,
                            "rx_packets": 8199780,
                            "rx_errors": 0,
                            "rx_dropped": 0,
                            "tx_bytes": 1854690043,
                            "tx_packets": 8765796,
                            "tx_errors": 0,
                            "tx_dropped": 0
                        },
                        {
                            "name": "ens3",
                            "rx_bytes": 25108832715,
                            "rx_packets": 74352449,
                            "rx_errors": 0,
                            "rx_dropped": 0,
                            "tx_bytes": 93157559917,
                            "tx_packets": 75983196,
                            "tx_errors": 0,
                            "tx_dropped": 0
                        }
                    ],
                    "tcp": {
                        "Established": 0,
                        "SynSent": 0,
                        "SynRecv": 0,
                        "FinWait1": 0,
                        "FinWait2": 0,
                        "TimeWait": 0,
                        "Close": 0,
                        "CloseWait": 0,
                        "LastAck": 0,
                        "Listen": 0,
                        "Closing": 0
                    },
                    "tcp6": {
                        "Established": 0,
                        "SynSent": 0,
                        "SynRecv": 0,
                        "FinWait1": 0,
                        "FinWait2": 0,
                        "TimeWait": 0,
                        "Close": 0,
                        "CloseWait": 0,
                        "LastAck": 0,
                        "Listen": 0,
                        "Closing": 0
                    },
                    "udp": {
                        "Listen": 0,
                        "Dropped": 0,
                        "RxQueued": 0,
                        "TxQueued": 0
                    },
                    "udp6": {
                        "Listen": 0,
                        "Dropped": 0,
                        "RxQueued": 0,
                        "TxQueued": 0
                    }
                },
                "filesystem": [
                    {
                        "device": "/dev/root",
                        "type": "vfs",
                        "capacity": 20749852672,
                        "usage": 9800732672,
                        "base_usage": 0,
                        "available": 10932342784,
                        "has_inodes": true,
                        "inodes": 2560000,
                        "inodes_free": 2340243,
                        "reads_completed": 0,
                        "reads_merged": 0,
                        "sectors_read": 0,
                        "read_time": 0,
                        "writes_completed": 0,
                        "writes_merged": 0,
                        "sectors_written": 0,
                        "write_time": 0,
                        "io_in_progress": 0,
                        "io_time": 0,
                        "weighted_io_time": 0
                    },
                    {
                        "device": "tmpfs",
                        "type": "vfs",
                        "capacity": 16802074624,
                        "usage": 2224128,
                        "base_usage": 0,
                        "available": 16799850496,
                        "has_inodes": true,
                        "inodes": 4102069,
                        "inodes_free": 4101999,
                        "reads_completed": 0,
                        "reads_merged": 0,
                        "sectors_read": 0,
                        "read_time": 0,
                        "writes_completed": 0,
                        "writes_merged": 0,
                        "sectors_written": 0,
                        "write_time": 0,
                        "io_in_progress": 0,
                        "io_time": 0,
                        "weighted_io_time": 0
                    },
                    {
                        "device": "shm",
                        "type": "vfs",
                        "capacity": 67108864,
                        "usage": 0,
                        "base_usage": 0,
                        "available": 67108864,
                        "has_inodes": true,
                        "inodes": 4102069,
                        "inodes_free": 4102068,
                        "reads_completed": 0,
                        "reads_merged": 0,
                        "sectors_read": 0,
                        "read_time": 0,
                        "writes_completed": 0,
                        "writes_merged": 0,
                        "sectors_written": 0,
                        "write_time": 0,
                        "io_in_progress": 0,
                        "io_time": 0,
                        "weighted_io_time": 0
                    },
                    {
                        "device": "/dev/vdb",
                        "type": "vfs",
                        "capacity": 53660876800,
                        "usage": 35852288,
                        "base_usage": 0,
                        "available": 53625024512,
                        "has_inodes": true,
                        "inodes": 26214400,
                        "inodes_free": 26214393,
                        "reads_completed": 457,
                        "reads_merged": 0,
                        "sectors_read": 13162,
                        "read_time": 3428,
                        "writes_completed": 231,
                        "writes_merged": 20,
                        "sectors_written": 72963,
                        "write_time": 1756,
                        "io_in_progress": 0,
                        "io_time": 3584,
                        "weighted_io_time": 5184
                    }
                ],
                "task_stats": {
                    "nr_sleeping": 0,
                    "nr_running": 0,
                    "nr_stopped": 0,
                    "nr_uninterruptible": 0,
                    "nr_io_wait": 0
                }
            },
            {
                "timestamp": "2017-11-10T14:29:36.487311961+08:00",
                "cpu": {
                    "usage": {
                        "total": 916888682985895,
                        "per_cpu_usage": [
                            225714036142385,
                            230867670611465,
                            229897633996290,
                            230409342235755
                        ],
                        "user": 443963990000000,
                        "system": 438381050000000
                    },
                    "cfs": {
                        "periods": 0,
                        "throttled_periods": 0,
                        "throttled_time": 0
                    },
                    "load_average": 0
                },
                "diskio": {
                    "io_service_bytes": [
                        {
                            "major": 9,
                            "minor": 0,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 253,
                            "minor": 0,
                            "stats": {
                                "Async": 61989710848,
                                "Read": 23649280,
                                "Sync": 17028677632,
                                "Total": 79018388480,
                                "Write": 78994739200
                            }
                        },
                        {
                            "major": 7,
                            "minor": 6,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 1,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 10,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 8,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 4,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 2,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 13,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 1,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 7,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 0,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 253,
                            "minor": 16,
                            "stats": {
                                "Async": 258048,
                                "Read": 21504,
                                "Sync": 4773888,
                                "Total": 5031936,
                                "Write": 5010432
                            }
                        },
                        {
                            "major": 1,
                            "minor": 14,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 12,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 9,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 6,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 7,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 15,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 5,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 5,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 4,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 3,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 2,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 3,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 0,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 11,
                            "stats": {
                                "Async": 12288,
                                "Read": 12288,
                                "Sync": 0,
                                "Total": 12288,
                                "Write": 0
                            }
                        }
                    ],
                    "io_serviced": [
                        {
                            "major": 1,
                            "minor": 11,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 9,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 4,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 3,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 14,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 10,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 5,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 2,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 1,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 0,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 9,
                            "minor": 0,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 253,
                            "minor": 0,
                            "stats": {
                                "Async": 2738102,
                                "Read": 2965,
                                "Sync": 4157395,
                                "Total": 6895497,
                                "Write": 6892532
                            }
                        },
                        {
                            "major": 7,
                            "minor": 1,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 15,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 3,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 5,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 13,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 12,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 6,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 4,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 253,
                            "minor": 16,
                            "stats": {
                                "Async": 57,
                                "Read": 7,
                                "Sync": 32,
                                "Total": 89,
                                "Write": 82
                            }
                        },
                        {
                            "major": 7,
                            "minor": 6,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 2,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 0,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 7,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        },
                        {
                            "major": 7,
                            "minor": 7,
                            "stats": {
                                "Async": 0,
                                "Read": 0,
                                "Sync": 0,
                                "Total": 0,
                                "Write": 0
                            }
                        },
                        {
                            "major": 1,
                            "minor": 8,
                            "stats": {
                                "Async": 3,
                                "Read": 3,
                                "Sync": 0,
                                "Total": 3,
                                "Write": 0
                            }
                        }
                    ]
                },
                "memory": {
                    "usage": 12189343744,
                    "cache": 288108544,
                    "rss": 3631366144,
                    "swap": 0,
                    "working_set": 7756025856,
                    "failcnt": 0,
                    "container_data": {
                        "pgfault": 8947531,
                        "pgmajfault": 87
                    },
                    "hierarchical_data": {
                        "pgfault": 8947531,
                        "pgmajfault": 87
                    }
                },
                "network": {
                    "name": "cbr0",
                    "rx_bytes": 1885783888,
                    "rx_packets": 8199819,
                    "rx_errors": 0,
                    "rx_dropped": 0,
                    "tx_bytes": 1854698778,
                    "tx_packets": 8765835,
                    "tx_errors": 0,
                    "tx_dropped": 0,
                    "interfaces": [
                        {
                            "name": "cbr0",
                            "rx_bytes": 1885783888,
                            "rx_packets": 8199819,
                            "rx_errors": 0,
                            "rx_dropped": 0,
                            "tx_bytes": 1854698778,
                            "tx_packets": 8765835,
                            "tx_errors": 0,
                            "tx_dropped": 0
                        },
                        {
                            "name": "ens3",
                            "rx_bytes": 25108971486,
                            "rx_packets": 74353959,
                            "rx_errors": 0,
                            "rx_dropped": 0,
                            "tx_bytes": 93160391886,
                            "tx_packets": 75985129,
                            "tx_errors": 0,
                            "tx_dropped": 0
                        }
                    ],
                    "tcp": {
                        "Established": 0,
                        "SynSent": 0,
                        "SynRecv": 0,
                        "FinWait1": 0,
                        "FinWait2": 0,
                        "TimeWait": 0,
                        "Close": 0,
                        "CloseWait": 0,
                        "LastAck": 0,
                        "Listen": 0,
                        "Closing": 0
                    },
                    "tcp6": {
                        "Established": 0,
                        "SynSent": 0,
                        "SynRecv": 0,
                        "FinWait1": 0,
                        "FinWait2": 0,
                        "TimeWait": 0,
                        "Close": 0,
                        "CloseWait": 0,
                        "LastAck": 0,
                        "Listen": 0,
                        "Closing": 0
                    },
                    "udp": {
                        "Listen": 0,
                        "Dropped": 0,
                        "RxQueued": 0,
                        "TxQueued": 0
                    },
                    "udp6": {
                        "Listen": 0,
                        "Dropped": 0,
                        "RxQueued": 0,
                        "TxQueued": 0
                    }
                },
                "filesystem": [
                    {
                        "device": "/dev/root",
                        "type": "vfs",
                        "capacity": 20749852672,
                        "usage": 9800732672,
                        "base_usage": 0,
                        "available": 10932342784,
                        "has_inodes": true,
                        "inodes": 2560000,
                        "inodes_free": 2340243,
                        "reads_completed": 0,
                        "reads_merged": 0,
                        "sectors_read": 0,
                        "read_time": 0,
                        "writes_completed": 0,
                        "writes_merged": 0,
                        "sectors_written": 0,
                        "write_time": 0,
                        "io_in_progress": 0,
                        "io_time": 0,
                        "weighted_io_time": 0
                    },
                    {
                        "device": "tmpfs",
                        "type": "vfs",
                        "capacity": 16802074624,
                        "usage": 2224128,
                        "base_usage": 0,
                        "available": 16799850496,
                        "has_inodes": true,
                        "inodes": 4102069,
                        "inodes_free": 4101999,
                        "reads_completed": 0,
                        "reads_merged": 0,
                        "sectors_read": 0,
                        "read_time": 0,
                        "writes_completed": 0,
                        "writes_merged": 0,
                        "sectors_written": 0,
                        "write_time": 0,
                        "io_in_progress": 0,
                        "io_time": 0,
                        "weighted_io_time": 0
                    },
                    {
                        "device": "shm",
                        "type": "vfs",
                        "capacity": 67108864,
                        "usage": 0,
                        "base_usage": 0,
                        "available": 67108864,
                        "has_inodes": true,
                        "inodes": 4102069,
                        "inodes_free": 4102068,
                        "reads_completed": 0,
                        "reads_merged": 0,
                        "sectors_read": 0,
                        "read_time": 0,
                        "writes_completed": 0,
                        "writes_merged": 0,
                        "sectors_written": 0,
                        "write_time": 0,
                        "io_in_progress": 0,
                        "io_time": 0,
                        "weighted_io_time": 0
                    },
                    {
                        "device": "/dev/vdb",
                        "type": "vfs",
                        "capacity": 53660876800,
                        "usage": 35852288,
                        "base_usage": 0,
                        "available": 53625024512,
                        "has_inodes": true,
                        "inodes": 26214400,
                        "inodes_free": 26214393,
                        "reads_completed": 457,
                        "reads_merged": 0,
                        "sectors_read": 13162,
                        "read_time": 3428,
                        "writes_completed": 231,
                        "writes_merged": 20,
                        "sectors_written": 72963,
                        "write_time": 1756,
                        "io_in_progress": 0,
                        "io_time": 3584,
                        "weighted_io_time": 5184
                    }
                ],
                "task_stats": {
                    "nr_sleeping": 0,
                    "nr_running": 0,
                    "nr_stopped": 0,
                    "nr_uninterruptible": 0,
                    "nr_io_wait": 0
                }
            }
        ]
    }
View Code

我们可以看到这一段json数据主要分了4部分:

  • reference——container本身的引用
  • subcontainers——子容器的信息
  • spec——说明信息
  • stats——统计信息

与cAdvisor提供的统计数据一致:

type ContainerInfo struct {
    ContainerReference

    // The direct subcontainers of the current container.
    Subcontainers []ContainerReference `json:"subcontainers,omitempty"`

    // The isolation used in the container.
    Spec ContainerSpec `json:"spec,omitempty"`

    // Historical statistics gathered from the container.
    Stats []*ContainerStats `json:"stats,omitempty"`
}

太棒了。

那么我就可以知道,获取allcontainers的逻辑,其实就是向kubelet发送了一个http的post请求,然后获取到返回的json数据。

(ii) 获取pod相关的summary

在kubelet_client.go文件中还有一个获取summary的方法:

func (self *KubeletClient) GetSummary(host Host) (*stats.Summary, error) {
    url := url.URL{
        Scheme: "http",
        Host:   host.String(),
        Path:   "/stats/summary/",
    }
    if self.config != nil && self.config.EnableHttps {
        url.Scheme = "https"
    }

    req, err := http.NewRequest("GET", url.String(), nil)
    if err != nil {
        return nil, err
    }
    summary := &stats.Summary{}
    client := self.client
    if client == nil {
        client = http.DefaultClient
    }
    err = self.postRequestAndGetValue(client, req, summary)
    return summary, err
}

恩,分析一下,很简单,就是一个http  get方法的调用。那我们再来模拟一下:

http://ip:10255/stats/summary

直接调用这个url即可,返回值太多,就不一一分析了。返回的数据结构如下:

type Summary struct {
    // Overall node stats.
    Node NodeStats `json:"node"`
    // Per-pod stats.
    Pods []PodStats `json:"pods"`
}

主要就是当前node的统计信息,还有pod的聚合信息。可以自己追进NodeStats和PodeStats的源码中看一下结构,在此就不列了。

值得注意的是,summary里面的stats和ContainerInfo是有些不一样的。比如ContainerInfo里面会有DiskInfo相关的信息。

3)小结

heapster调用kubelet,其实就是发送了两个http请求,一个是get方法,获取所有pod的summary信息,另一个是post方法,获取cAdvisor提供的所有ContainerInfo。然后在pod_aggregator.go方法中进行的数据相关的聚合。那我们对kubelet调用的源码分析就到此为止了。接下来是笔者自己做的一个小实验。

三、实验

1、禁掉节点的4194端口

禁掉node的4194端口后,就无法通过ip:4194获取到cAdvisor的stats数据了。那我们调用一下kubelet的api:调用kubelet的api:http://ip:10255/stats/container/

我们会发现,返回的数据项里缺少了stats这一项,而这一项恰恰是cAdvisor提供的最重要的监控统计数据。

2、重新开启节点的4194端口

重新开启node的4194端口,再重新调用一下kubelet的api,这下返回的结果里面又重新出现了stats这一项。

3、总结

结合kubelet的源码可以发现,在kubelet启动的时候会去启动cAdvisor的4194端口,获取基础的统计数据。

heapster源码分析——kubelet的api调用分析

标签:star   agg   目录   自己   hid   就会   hub   post方法   led   

原文地址:http://www.cnblogs.com/sonofelice/p/7818706.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!