在Kubernetes中，基于Prometheus的指标自动扩展
首先
使用Kubernetes作为容器编排的主要优点之一是可以通过水平扩展应用程序，以便轻松应对负载增加。通常，水平Pod自动伸缩器可以根据CPU和内存使用量来进行扩展部署，但在更复杂的情况下，需要考虑其他指标才能确定扩展策略。
Prometheus是一种推荐的工具，用于监控部署的工作负载和Kubernetes集群本身。Prometheus适配器利用由Prometheus收集的指标，并利用它们进行扩展决策的确定。这些指标可以由API服务公开，并可以在Horizontal PodAutoscaling对象中轻松使用。
部署
建筑设计的概要
使用Prometheus Adapter，通过从Prometheus安装中拉取自定义指标，使得水平Pod自动扩缩器可以使用这些指标来扩大或缩小Pod。 Prometheus Adapter将作为一个部署，使用集群内的服务进行公开。通常情况下，一个适配器的单个复制品对于中小规模的集群来说已经足够。然而，如果集群非常大，可以使用Node Affinities属性和Pod-AntiAffinity属性来分散Prometheus Adapter的多个复制品在整个节点上运行。
前提条件 (Paraphrase: 初始条件)
-
- 少なくとも3つのノードでKubernetesクラスターを実行すること。 ちなみに、このチュートリアルでは、GKEクラスターを使用します。
-
- 水平ポッドの自動スケーリングに関する基本的な知識。
- Prometheusはクラスター内にデプロイされているか、エンドポイントを使用してアクセスできること。
我们将使用Prometheus-Thanos的高可用性部署。请查看此处以获取更多详细信息。
部署样例应用
首先，让我们部署一个用于测试 Prometheus 指标自动扩缩容的示例应用程序。您可以使用以下清单来完成这个操作。
apiVersion: v1
kind: Namespace
metadata:
name: nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: nginx
name: nginx-deployment
spec:
replicas: 1
template:
metadata:
annotations:
prometheus.io/path: "/status/format/prometheus"
prometheus.io/scrape: "true"
prometheus.io/port: "80"
labels:
app: nginx-server
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nginx-server
topologyKey: kubernetes.io/hostname
containers:
- name: nginx-demo
image: vaibhavthakur/nginx-vts:1.0
imagePullPolicy: Always
resources:
limits:
cpu: 2500m
requests:
cpu: 2000m
ports:
- containerPort: 80
name: http
---
apiVersion: v1
kind: Service
metadata:
namespace: nginx
name: nginx-service
spec:
ports:
- port: 80
targetPort: 80
name: http
selector:
app: nginx-server
type: LoadBalancer
通过这个步骤，将创建一个名为nginx的命名空间，并在其中部署一个示例的nginx应用程序。该应用程序可以通过服务进行访问，并通过端口80暴露nginxvts指标的/ status / format / prometheus端点。为了设置，我们创建了一个与ExternalIP映射到nginx.gotham.com的DNS条目。
root$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 1/1 1 1 43d
root$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-65d8df7488-c578v 1/1 Running 0 9h
root$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-service ClusterIP 10.63.253.154 35.232.67.34 80/TCP 43d
root$ kubectl describe deploy nginx-deployment
Name: nginx-deployment
Namespace: nginx
CreationTimestamp: Tue, 08 Oct 2019 11:47:36 -0700
Labels: app=nginx-server
Annotations: deployment.kubernetes.io/revision: 1
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"name":"nginx-deployment","namespace":"nginx"},"spec":...
Selector: app=nginx-server
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: app=nginx-server
Annotations: prometheus.io/path: /status/format/prometheus
prometheus.io/port: 80
prometheus.io/scrape: true
Containers:
nginx-demo:
Image: vaibhavthakur/nginx-vts:v1.0
Port: 80/TCP
Host Port: 0/TCP
Limits:
cpu: 250m
Requests:
cpu: 200m
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-65d8df7488 (1/1 replicas created)
Events: <none>
root$ curl nginx.gotham.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
这些是目前所有应用程序公开的指标。
$ curl nginx.gotham.com/status/format/prometheus
# HELP nginx_vts_info Nginx info
# TYPE nginx_vts_info gauge
nginx_vts_info{hostname="nginx-deployment-65d8df7488-c578v",version="1.13.12"} 1
# HELP nginx_vts_start_time_seconds Nginx start time
# TYPE nginx_vts_start_time_seconds gauge
nginx_vts_start_time_seconds 1574283147.043
# HELP nginx_vts_main_connections Nginx connections
# TYPE nginx_vts_main_connections gauge
nginx_vts_main_connections{status="accepted"} 215
nginx_vts_main_connections{status="active"} 4
nginx_vts_main_connections{status="handled"} 215
nginx_vts_main_connections{status="reading"} 0
nginx_vts_main_connections{status="requests"} 15577
nginx_vts_main_connections{status="waiting"} 3
nginx_vts_main_connections{status="writing"} 1
# HELP nginx_vts_main_shm_usage_bytes Shared memory [ngx_http_vhost_traffic_status] info
# TYPE nginx_vts_main_shm_usage_bytes gauge
nginx_vts_main_shm_usage_bytes{shared="max_size"} 1048575
nginx_vts_main_shm_usage_bytes{shared="used_size"} 3510
nginx_vts_main_shm_usage_bytes{shared="used_node"} 1
# HELP nginx_vts_server_bytes_total The request/response bytes
# TYPE nginx_vts_server_bytes_total counter
# HELP nginx_vts_server_requests_total The requests counter
# TYPE nginx_vts_server_requests_total counter
# HELP nginx_vts_server_request_seconds_total The request processing time in seconds
# TYPE nginx_vts_server_request_seconds_total counter
# HELP nginx_vts_server_request_seconds The average of request processing times in seconds
# TYPE nginx_vts_server_request_seconds gauge
# HELP nginx_vts_server_request_duration_seconds The histogram of request processing time
# TYPE nginx_vts_server_request_duration_seconds histogram
# HELP nginx_vts_server_cache_total The requests cache counter
# TYPE nginx_vts_server_cache_total counter
nginx_vts_server_bytes_total{host="_",direction="in"} 3303449
nginx_vts_server_bytes_total{host="_",direction="out"} 61641572
nginx_vts_server_requests_total{host="_",code="1xx"} 0
nginx_vts_server_requests_total{host="_",code="2xx"} 15574
nginx_vts_server_requests_total{host="_",code="3xx"} 0
nginx_vts_server_requests_total{host="_",code="4xx"} 2
nginx_vts_server_requests_total{host="_",code="5xx"} 0
nginx_vts_server_requests_total{host="_",code="total"} 15576
nginx_vts_server_request_seconds_total{host="_"} 0.000
nginx_vts_server_request_seconds{host="_"} 0.000
nginx_vts_server_cache_total{host="_",status="miss"} 0
nginx_vts_server_cache_total{host="_",status="bypass"} 0
nginx_vts_server_cache_total{host="_",status="expired"} 0
nginx_vts_server_cache_total{host="_",status="stale"} 0
nginx_vts_server_cache_total{host="_",status="updating"} 0
nginx_vts_server_cache_total{host="_",status="revalidated"} 0
nginx_vts_server_cache_total{host="_",status="hit"} 0
nginx_vts_server_cache_total{host="_",status="scarce"} 0
nginx_vts_server_bytes_total{host="*",direction="in"} 3303449
nginx_vts_server_bytes_total{host="*",direction="out"} 61641572
nginx_vts_server_requests_total{host="*",code="1xx"} 0
nginx_vts_server_requests_total{host="*",code="2xx"} 15574
nginx_vts_server_requests_total{host="*",code="3xx"} 0
nginx_vts_server_requests_total{host="*",code="4xx"} 2
nginx_vts_server_requests_total{host="*",code="5xx"} 0
nginx_vts_server_requests_total{host="*",code="total"} 15576
nginx_vts_server_request_seconds_total{host="*"} 0.000
nginx_vts_server_request_seconds{host="*"} 0.000
nginx_vts_server_cache_total{host="*",status="miss"} 0
nginx_vts_server_cache_total{host="*",status="bypass"} 0
nginx_vts_server_cache_total{host="*",status="expired"} 0
nginx_vts_server_cache_total{host="*",status="stale"} 0
nginx_vts_server_cache_total{host="*",status="updating"} 0
nginx_vts_server_cache_total{host="*",status="revalidated"} 0
nginx_vts_server_cache_total{host="*",status="hit"} 0
nginx_vts_server_cache_total{host="*",status="scarce"} 0
在这些中，尤其是nginx_vts_server_requests_total是非常有趣的。通过使用这个指标的值，决定是否要扩展nginx部署。
创建 Prometheus 适配器的 SSL 证书和 Kubernetes 密钥。
使用以下的Makefile生成openssl证书和相应的Kubernetes密钥：
# Makefile for generating TLS certs for the Prometheus custom metrics API adapter
SHELL=bash
UNAME := $(shell uname)
PURPOSE:=metrics
SERVICE_NAME:=custom-metrics-apiserver
ALT_NAMES:="custom-metrics-apiserver.monitoring","custom-metrics-apiserver.monitoring.svc"
SECRET_FILE:=custom-metrics-api/cm-adapter-serving-certs.yaml
certs: gensecret rmcerts
.PHONY: gencerts
gencerts:
@echo Generating TLS certs
@docker pull cfssl/cfssl
@mkdir -p output
@touch output/apiserver.pem
@touch output/apiserver-key.pem
@openssl req -x509 -sha256 -new -nodes -days 365 -newkey rsa:2048 -keyout $(PURPOSE)-ca.key -out $(PURPOSE)-ca.crt -subj "/CN=ca"
@echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'$(PURPOSE)'"]}}}' > "$(PURPOSE)-ca-config.json"
@echo '{"CN":"'$(SERVICE_NAME)'","hosts":[$(ALT_NAMES)],"key":{"algo":"rsa","size":2048}}' | docker run -v ${HOME}:${HOME} -v ${PWD}/metrics-ca.key:/go/src/github.com/cloudflare/cfssl/metrics-ca.key -v ${PWD}/metrics-ca.crt:/go/src/github.com/cloudflare/cfssl/metrics-ca.crt -v ${PWD}/metrics-ca-config.json:/go/src/github.com/cloudflare/cfssl/metrics-ca-config.json -i cfssl/cfssl gencert -ca=metrics-ca.crt -ca-key=metrics-ca.key -config=metrics-ca-config.json - | docker run --entrypoint=cfssljson -v ${HOME}:${HOME} -v ${PWD}/output:/go/src/github.com/cloudflare/cfssl/output -i cfssl/cfssl -bare output/apiserver
.PHONY: gensecret
gensecret: gencerts
@echo Generating $(SECRET_FILE)
@echo "apiVersion: v1" > $(SECRET_FILE)
@echo "kind: Secret" >> $(SECRET_FILE)
@echo "metadata:" >> $(SECRET_FILE)
@echo " name: cm-adapter-serving-certs" >> $(SECRET_FILE)
@echo " namespace: monitoring" >> $(SECRET_FILE)
@echo "data:" >> $(SECRET_FILE)
ifeq ($(UNAME), Darwin)
@echo " serving.crt: $$(cat output/apiserver.pem | base64)" >> $(SECRET_FILE)
@echo " serving.key: $$(cat output/apiserver-key.pem | base64)" >> $(SECRET_FILE)
endif
ifeq ($(UNAME), Linux)
@echo " serving.crt: $$(cat output/apiserver.pem | base64 -w 0)" >> $(SECRET_FILE)
@echo " serving.key: $$(cat output/apiserver-key.pem | base64 -w 0)" >> $(SECRET_FILE)
endif
.PHONY: rmcerts
rmcerts:
@rm -f apiserver-key.pem apiserver.csr apiserver.pem
@rm -f metrics-ca-config.json metrics-ca.crt metrics-ca.key
.PHONY: deploy
deploy:
kubectl create -f ./custom-metrics-api
只需创建一个make文件，然后执行以下命令。
make certs
然后，创建SSL证书和相应的Kubernetes密钥。在创建密钥之前，请确保监视命名空间存在。这个密钥是为了使用后续部署的Prometheus适配器。
创建 Prometheus 适配器配置地图
使用以下清单创建Prometheus适配器配置映射。
apiVersion: v1
kind: ConfigMap
metadata:
name: adapter-config
namespace: monitoring
data:
config.yaml: |
rules:
- seriesQuery: 'nginx_vts_server_requests_total'
resources:
overrides:
kubernetes_namespace:
resource: namespace
kubernetes_pod_name:
resource: pod
name:
matches: "^(.*)_total"
as: "${1}_per_second"
metricsQuery: (sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>))
这张设定图只能指定单一的指标，但您随时可以添加指标。您可以参考此链接以添加指标。强烈建议只获取水平Pod自动扩展器所需的指标。这对于调试很有帮助，这些附加组件生成非常详细的日志，并由日志后端进行收集。如果获取不必要的指标，将会导致服务的负载增加，并且会向日志后端发送垃圾邮件。如果您想了解更多关于配置图的信息，请阅读此处。
创建Prometheus适配器的部署
使用下面的清单来部署Prometheus适配器。
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: custom-metrics-apiserver
name: custom-metrics-apiserver
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: custom-metrics-apiserver
template:
metadata:
labels:
app: custom-metrics-apiserver
name: custom-metrics-apiserver
spec:
serviceAccountName: monitoring
containers:
- name: custom-metrics-apiserver
image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.4.1
args:
- /adapter
- --secure-port=6443
- --tls-cert-file=/var/run/serving-cert/serving.crt
- --tls-private-key-file=/var/run/serving-cert/serving.key
- --logtostderr=true
- --prometheus-url=http://thanos-querier.monitoring:9090/
- --metrics-relist-interval=30s
- --v=10
- --config=/etc/adapter/config.yaml
ports:
- containerPort: 6443
volumeMounts:
- mountPath: /var/run/serving-cert
name: volume-serving-cert
readOnly: true
- mountPath: /etc/adapter/
name: config
readOnly: true
volumes:
- name: volume-serving-cert
secret:
secretName: cm-adapter-serving-certs
- name: config
configMap:
name: adapter-config
通过这样做，将创建一个部署，它生成 Prometheus 适配器 Pods 并从 Prometheus 拉取指标。请注意，–prometheus-url = http：//thanos-querier.monitoring：9090 / 已被配置。这是因为在与 PrometheusAdapter 相同的 Kubernetes 集群监视命名空间中部署了 Thanos 支持的 Prometheus 集群。您可以通过更改这个参数来指定 Prometheus 部署。
当你注意到这个容器的日志时，你会发现它正在获取在配置文件中定义的指标。
I1122 00:26:53.228394 1 api.go:74] GET http://thanos-querier.monitoring:9090/api/v1/series?match%5B%5D=nginx_vts_server_requests_total&start=1574381213.217 200 OK
I1122 00:26:53.234234 1 api.go:93] Response Body: {"status":"success","data":[{"__name__":"nginx_vts_server_requests_
total","app":"nginx-server","cluster":"prometheus-ha","code":"1xx","host":"*","instance":"10.60.64.39:80","job":"kubernetes
-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df
7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"1xx","host":"*"
,"instance":"10.60.64.8:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-
65d8df7488-mwzxg","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server",
"cluster":"prometheus-ha","code":"1xx","host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace
":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_
vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"1xx","host":"_","instance":"10.60.64.8:80
","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg",
"pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-
ha","code":"2xx","host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_
pod_name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_
total","app":"nginx-server","cluster":"prometheus-ha","code":"2xx","host":"*","instance":"10.60.64.8:80","job":"kubernetes-
pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"2xx",
"host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"2xx","host":"_","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"3xx",
"host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"3xx","host":"*","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"3xx",
"host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"3xx","host":"_","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"4xx",
"host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"4xx","host":"*","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"4xx",
"host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"4xx","host":"_","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"5xx",
"host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"5xx","host":"*","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"5xx",
"host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"5xx","host":"_","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":
"total","host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_
name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total"
,"app":"nginx-server","cluster":"prometheus-ha","code":"total","host":"*","instance":"10.60.64.8:80","job":
"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg",
"pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":
"prometheus-ha","code":"total","host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":
"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":
"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"total","host":"_","instance":
"10.60.64.8:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-
65d8df7488-mwzxg","pod_template_hash":"65d8df7488"}]}
创建 Prometheus AdapterAPI 服务
以下的说明将会创建API服务，并使其能够通过Kubernetes API访问Prometheus适配器。这将使得水平Pod Autoscaler能够获取到指标信息。
apiVersion: v1
kind: Service
metadata:
name: custom-metrics-apiserver
namespace: monitoring
spec:
ports:
- port: 443
targetPort: 6443
selector:
app: custom-metrics-apiserver
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.custom.metrics.k8s.io
spec:
service:
name: custom-metrics-apiserver
namespace: monitoring
group: custom.metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100
安装测试
我们来查看所有可用的自定义指标。
root$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "pods/nginx_vts_server_requests_per_second",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "namespaces/nginx_vts_server_requests_per_second",
"singularName": "",
"namespaced": false,
"kind": "MetricValueList",
"verbs": [
"get"
]
}
]
}
可以看到 nginx_vts_server_requests_per_second 指标是可用的。
那么，我们来确认一下目前这个度量的值吧。
root$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/nginx/pods/*/nginx_vts_server_requests_per_second" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/nginx/pods/%2A/nginx_vts_server_requests_per_second"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "nginx",
"name": "nginx-deployment-65d8df7488-v575j",
"apiVersion": "/v1"
},
"metricName": "nginx_vts_server_requests_per_second",
"timestamp": "2019-11-19T18:38:21Z",
"value": "1236m"
}
]
}
可以使用以下清单创建一个使用这些度量标准的HPA。
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-custom-hpa
namespace: nginx
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx-deployment
minReplicas: 2
maxReplicas: 10
metrics:
- type: Pods
pods:
metricName: nginx_vts_server_requests_per_second
targetAverageValue: 4000m
使用此清单，您可以查看HPA的当前状态如下。
root$ kubectl describe hpa
Name: nginx-custom-hpa
Namespace: nginx
Labels: <none>
Annotations: autoscaling.alpha.kubernetes.io/metrics:
[{"type":"Pods","pods":{"metricName":"nginx_vts_server_requests_per_second","targetAverageValue":"4"}}]
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-custom-hpa","namespace":"n...
CreationTimestamp: Thu, 21 Nov 2019 11:11:05 -0800
Reference: Deployment/nginx-deployment
Min replicas: 2
Max replicas: 10
Deployment pods: 0 current / 0 desired
Events: <none>
那么，我们开始给服务增加负载吧。我们将使用一个名为vegeta的实用程序。
请在另一个终端中执行以下命令。
echo "GET http://nginx.gotham.com/" | vegeta attack -rate=5 -duration=0 | vegeta report
同时监视nginx Pod和水平Pod自动扩缩器，会显示如下内容。
root$ kubectl get -w pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-65d8df7488-mwzxg 1/1 Running 0 9h
nginx-deployment-65d8df7488-sbp95 1/1 Running 0 4m9s
NAME AGE
nginx-deployment-65d8df7488-pwjzm 0s
nginx-deployment-65d8df7488-pwjzm 0s
nginx-deployment-65d8df7488-pwjzm 0s
nginx-deployment-65d8df7488-pwjzm 2s
nginx-deployment-65d8df7488-pwjzm 4s
nginx-deployment-65d8df7488-jvbvp 0s
nginx-deployment-65d8df7488-jvbvp 0s
nginx-deployment-65d8df7488-jvbvp 1s
nginx-deployment-65d8df7488-jvbvp 4s
nginx-deployment-65d8df7488-jvbvp 7s
nginx-deployment-65d8df7488-skjkm 0s
nginx-deployment-65d8df7488-skjkm 0s
nginx-deployment-65d8df7488-jh5vw 0s
nginx-deployment-65d8df7488-skjkm 0s
nginx-deployment-65d8df7488-jh5vw 0s
nginx-deployment-65d8df7488-jh5vw 1s
nginx-deployment-65d8df7488-skjkm 2s
nginx-deployment-65d8df7488-jh5vw 2s
nginx-deployment-65d8df7488-skjkm 3s
nginx-deployment-65d8df7488-jh5vw 4s
root$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-custom-hpa Deployment/nginx-deployment 5223m/4 2 10 3 5m5s
明显地看到水平Pod自动缩放根据需求扩展了Pod。如果中断Vegeta命令，将显示Vegeta报告。这清楚地表明所有请求已被应用程序处理。
root$ echo "GET http://nginx.gotham.com/" | vegeta attack -rate=5 -duration=0 | vegeta report
^CRequests [total, rate, throughput] 224, 5.02, 5.02
Duration [total, attack, wait] 44.663806863s, 44.601823883s, 61.98298ms
Latencies [mean, 50, 95, 99, max] 63.3879ms, 60.867241ms, 79.414139ms, 111.981619ms, 229.310088ms
Bytes In [total, mean] 137088, 612.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:224
Error Set:
总结
这个设置演示了如何使用Prometheus适配器根据一些自定义指标自动扩展部署。为了简化，只从Prometheus服务器提取了一个指标。但是，可以通过扩展适配器配置映射，提取部分或全部可用指标，并将它们用于自动扩展。
如果Prometheus的安装在Kubernetes集群之外，需要确保查询终点可从集群访问，并且需要在适配器部署清单中更新。可以在多个指标中获取并组合使用，可能会有复杂的情景来决定扩展。
如果对Prometheus的使用感兴趣，但担心它的设置和管理会消耗太多资源，可以预订一个演示，询问MetricFire提供的HostedPrometheus如何适应监控环境。
这篇文章是由客座博客作家Vaibhav Thakur撰写的。他是原文的作者。
