在本地环境中尝试使用 Kubernetes HPA 的 Alpha 功能 HPAScaleToZero

1 年 ago
逸, 科
4 minutes

背景

在HPA中无法进行ScaleIn到零。

kubectl explain HorizontalPodAutoscaler.spec.minReplicas
KIND:     HorizontalPodAutoscaler
VERSION:  autoscaling/v1

FIELD:    minReplicas <integer>

DESCRIPTION:
     minReplicas is the lower limit for the number of replicas to which the
     autoscaler can scale down. It defaults to 1 pod. minReplicas is allowed to
     be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one
     Object or External metric is configured. Scaling is active as long as at
     least one metric value is available.

如果启用了HPAScaleToZero的alpha功能，并且配置了至少一个对象或外部指标，那么minReplicas可以为0。

    • alpha feature gateを enableする必要がある

 

    Object or External metric少なくとも1つは必要

在之前介绍的使用RabbitMQ和Prometheus来试验Kubernetes的HPA自定义指标的案例中，当队列消息为零时，有时我们希望完全进行缩容。本次实验将尝试使用HPA将缩容至零。

Feature Gate是什么？

特性开关：表中列出了每个 Kubernetes 特性的阶段和默认启用状态，以及从哪个版本开始（到哪个版本结束）可用。

要启用 Enable，请在启动命令中列出要启用的 Feature 键值对（Feature 名称和 true/false）：-feature-gates=”…,DynamicKubeletConfig=true”。

当查看要使用的HPAScaleToZero时，发现HPAScaleToZero为false，Alpha为1.16，使用它需要明确地启用。

创建一个在本地启用了HPAScaleToZero的Kubernetes集群。

为了确认本地行为，我想使用容易在本地进行指定的种类。

先决条件

    • go (1.11+)

 

    Docker

安装kind

对于Mac用户来说，可以使用brew进行安装（有关详细信息，请参阅https://kind.sigs.k8s.io/docs/user/quick-start/#installation）。

brew install kind

为kind集群准备Yaml格式文件。

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  # any feature gate can be enabled here with "Name": true
  # or disabled here with "Name": false
  # not all feature gates are tested, however
  "HPAScaleToZero": true

使用kind创建Kubernetes集群。

kind create cluster --config cluster-with-alpha-feature.yaml

kind create cluster --config cluster-with-alpha-feature.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) ? 
 ✓ Preparing nodes ?  
 ✓ Writing configuration ? 
 ✓ Starting control-plane ?️ 
 ✓ Installing CNI ? 
 ✓ Installing StorageClass ? 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! ?

为了HPA做准备

部署与上次相同的内容。

image.png

这次省略了Grafana等内容。

    1. 将以下内容用中文进行释义：

克隆 Code：

将代码克隆到本地：

git clone https://github.com/nakamasato/kubernetes-training && cd kubernetes-training/autoscaler/hpa/custom-metrics

使用 Prometheus Operator：

部署 Prometheus Operator：

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
kubectl create ns monitoring
kubectl apply -k ../../../prometheus-operator -n monitoring

使用 RabbitMQ Operator：

部署 RabbitMQ Operator：

kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
kubectl apply -f rabbitmq/rabbitmq-cluster.yaml
kubectl apply -f rabbitmq/pod-monitor-rabbitmq.yaml

部署 RabbitMQConsumer 和 RabbitMQ Producer：

部署 RabbitMQProducer CronJob 和 RabbitMQConsumer Deployment：

kubectl apply -f rabbitmq-producer-cronjob.yaml
kubectl apply -f rabbitmq-consumer-deployment.yaml

部署 Prometheus Adapter (https://github.com/stefanprodan/k8s-prom-hpa/)：

克隆 Prometheus Adapter 代码并部署：

git clone git@github.com:stefanprodan/k8s-prom-hpa.git && cd k8s-prom-hpa
touch metrics-ca.key metrics-ca.crt metrics-ca-config.json
make certs
kubectl create -f ./custom-metrics-api

确认可以通过 custom.metrics.k8s.io 获取到 rabbitmq_queue_messages_ready 的指标数据：

kubectl get –raw “/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/rabbitmq_queue_messages_ready”| jq .
{
“kind”: “MetricValueList”,
“apiVersion”: “custom.metrics.k8s.io/v1beta1”,
“metadata”: {
“selfLink”: “/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/rabbitmq_queue_messages_ready”
},
“items”: [
{
“describedObject”: {
“kind”: “Pod”,
“namespace”: “default”,
“name”: “rabbitmq-server-0”,
“apiVersion”: “/v1”
},
“metricName”: “rabbitmq_queue_messages_ready”,
“timestamp”: “2021-05-19T01:55:02Z”,
“value”: “0”
}
]
}

以(minReplicas: 0)配置HPA

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: rabbitmq-consumer
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rabbitmq-consumer
  minReplicas: 0
  maxReplicas: 20
  metrics:
    - type: Object
      object:
        metric:
          name: rabbitmq_queue_messages_ready
        describedObject:
          kind: Pod
          name: rabbitmq-server-0
          apiVersion: v1
        target:
          type: Value
          averageValue: 1

kubectl apply -f rabbitmq-consumer-hpa-with-min-zero.yaml

确认确认新尺寸为0！

Events:
  Type    Reason             Age                  From                       Message
  ----    ------             ----                 ----                       -------
  Normal  SuccessfulRescale  35m                  horizontal-pod-autoscaler  New size: 7; reason: All metrics below target
  Normal  SuccessfulRescale  30m (x2 over 40m)    horizontal-pod-autoscaler  New size: 16; reason: All metrics below target
  Normal  SuccessfulRescale  30m (x2 over 39m)    horizontal-pod-autoscaler  New size: 18; reason: All metrics below target
  Normal  SuccessfulRescale  25m (x2 over 35m)    horizontal-pod-autoscaler  New size: 14; reason: All metrics below target
  Normal  SuccessfulRescale  25m                  horizontal-pod-autoscaler  New size: 4; reason: All metrics below target
  Normal  SuccessfulRescale  20m                  horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
  Normal  SuccessfulRescale  10m (x4 over 40m)    horizontal-pod-autoscaler  New size: 4; reason: external metric rabbitmq_queue_messages_ready(nil) above target
  Normal  SuccessfulRescale  5m13s (x4 over 35m)  horizontal-pod-autoscaler  New size: 2; reason: All metrics below target
  Normal  SuccessfulRescale  4m58s                horizontal-pod-autoscaler  New size: 0; reason: All metrics below target
  Normal  SuccessfulRescale  37s (x9 over 20m)    horizontal-pod-autoscaler  (combined from similar events): New size: 4; reason: external metric rabbitmq_queue_messages_ready(nil) above target
  Normal  SuccessfulRescale  22s (x4 over 40m)    horizontal-pod-autoscaler  New size: 8; reason: external metric rabbitmq_queue_messages_ready(nil) above target
  Normal  SuccessfulRescale  6s                   horizontal-pod-autoscaler  New size: 16; reason: external metric rabbitmq_queue_messages_ready(nil) above targe

整理

    1. 清理Kubernetes资源

kubectl delete -f rabbitmq-consumer-hpa-with-min-zero.yaml
kubectl delete -f rabbitmq-producer-cronjob.yaml
kubectl delete -f rabbitmq-consumer-deployment.yaml
kubectl delete -f https://k8s.io/examples/application/php-apache.yaml
kubectl delete -f rabbitmq/rabbitmq-cluster.yaml
kubectl delete -k ../../../prometheus-operator -n monitoring
kubectl delete -f ./k8s-prom-hpa/custom-metrics-api
kubectl delete -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
kubectl delete -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml

删除kind集群

kind delete cluster –name kind

总结

我在本地Kubernetes集群kind上启用了Alpha Feature的HPAScaleToZero，并通过自定义指标的HPA验证了minReplicas: 0的正常工作。

请提供相关链接

    • https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

 

    • https://github.com/nakamasato/kubernetes-training/tree/master/autoscaler/hpa/custom-metrics

 

    • https://qiita.com/gymnstcs/items/ae375899d254b7b36c80

 

    https://kind.sigs.k8s.io/docs/user/quick-start/#installation
✖
广告
将在 10 秒后关闭
bannerAds