根据Prometheus收集的指标，实现HPA和kubectl top命令的功能

2 年 ago
宇, 华
3 minutes

首先

在执行Horizontal Pod Autoscaler(HPA)或kubectl top命令时，会从metrics.k8s.io的APIService获取指标数据。
如果环境中未默认部署Metrics Server，则执行kubectl top命令会出现下述错误。

$ kubectl top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

常见的方式是使用Metrics Server，但这次我们将介绍如何使用Prometheus Adapter。

环境

macOS Mojave 10.14.6
EKS 1.16
Helm 2.14.3
Helmfile 0.122.0

macOS Mojave 10.14.6
EKS 1.16
Helm 2.14.3
Helmfile 0.122.0

引入

假设 EKS 已经建立并已经部署 Helm Tiller。
我们使用 Helmfile 来部署 Prometheus 和 Prometheus-Adapter。

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com

releases:
  # https://github.com/helm/charts/tree/master/stable/prometheus
  - name: prometheus
    namespace: kube-system
    chart: stable/prometheus
    version: 11.7.0
  # https://github.com/helm/charts/tree/master/stable/prometheus-adapter
  - name: prometheus-adapter
    namespace: kube-system
    chart: stable/prometheus-adapter
    version: 2.5.0
    values:
      - prometheus:
          url: http://prometheus-server.kube-system.svc
          port: 80
        rules:
          resource: 
            cpu:
              containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[3m])) by (<<.GroupBy>>)
              nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[3m])) by (<<.GroupBy>>)
              resources:
                overrides:
                  instance:
                    resource: node
                  namespace:
                    resource: namespace
                  pod:
                    resource: pod
              containerLabel: container
            memory:
              containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>}) by (<<.GroupBy>>)
              nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>)
              resources:
                overrides:
                  instance:
                    resource: node
                  namespace:
                    resource: namespace
                  pod:
                    resource: pod
              containerLabel: container
            window: 3m

$ helmfile -f helmfile.yaml apply

关于Prometheus的Helm Chart的values，这次我们不会进行任何设置。
如果需要从外部访问Prometheus，则需要注意进行Ingress设置。

在Helm Chart的 values 中解释了 Prometheus Adapter。首先是.prometheus。这里指定了 Prometheus Service，这样就可以访问 Prometheus Pod。

然后就是.rules.resource。在这里定义的内容可以在/apis/metrics.k8s.io/v1beta1中获取。这样就可以使用HPA和kubectl top命令了。

可以通过以下命令看出APIService已经被创建。

$ kubectl describe apiservice v1beta1.metrics.k8s.io   
Name:         v1beta1.metrics.k8s.io
Namespace:    
Labels:       app=prometheus-adapter
              chart=prometheus-adapter-2.5.0
              heritage=Tiller
              release=prometheus-adapter
Annotations:  <none>
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2020-07-19T09:24:54Z
  Resource Version:    27240
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  UID:                 e3c2e4f4-a58b-4294-8076-5f2cb51d6886
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            prometheus-adapter
    Namespace:       kube-system
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2020-07-19T09:25:35Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

确认

为了确认HPA，请部署以下清单文件。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hpa-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hpa-test
  template:
    metadata:
      labels:
        app: hpa-test
    spec:
      containers:
        - image: busybox
          name: hpa-test
          command: ["dd", "if=/dev/zero", "of=/dev/null"]
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hpa-test
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 50

$ kubectl apply -f hpa-test.yaml

过了一会儿，我们可以确认有5个Pod已经达到最大数量。

$ kubectl get pod
NAME                       READY   STATUS    RESTARTS   AGE
hpa-test-596d78d9d-2gwr5   1/1     Running   0          3m39s
hpa-test-596d78d9d-6l9jj   1/1     Running   0          21s
hpa-test-596d78d9d-hg7vc   1/1     Running   0          21s
hpa-test-596d78d9d-sxghk   1/1     Running   0          6s
hpa-test-596d78d9d-vntkf   1/1     Running   0          21s

kubectl top命令也能正常运行无误。

$ kubectl top node
NAME                                              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-10-0-109-39.ap-northeast-1.compute.internal    1523m        78%    516Mi           15%       
ip-10-0-132-197.ap-northeast-1.compute.internal   1654m        85%    529Mi           15%       
ip-10-0-46-89.ap-northeast-1.compute.internal     833m         43%    741Mi           22%

$ kubectl top pod
NAME                       CPU(cores)   MEMORY(bytes)   
hpa-test-596d78d9d-2gwr5   1941m        1Mi             
hpa-test-596d78d9d-6l9jj   1362m        1Mi             
hpa-test-596d78d9d-hg7vc   1300m        1Mi             
hpa-test-596d78d9d-sxghk   1145m        1Mi             
hpa-test-596d78d9d-vntkf   1294m        2Mi

总结

我们已经确认了在不使用Metrics Server的情况下，可以通过Prometheus Adapter和Prometheus来执行HPA和kubectl top命令。
在部署了Prometheus的集群环境中，可能还可以考虑使用Prometheus Adapter。

✖
广告
将在 10 秒后关闭
bannerAds