通过Operator简单地将Prometheus和Grafana安装到Kubernetes集群上
请参考官方Quickstart文档，该资料内容与最新版本不同。
简而言之或总的来说
介绍在Kubernetes集群上安装Prometheus的方法。
在这里，我们使用Prometheus的Kubernetes运算符。
由于prometheus-operator/prometheus-operator仍处于beta阶段，所以我们将使用prometheus-operator/kube-prometheus。
环境
-
- Docker for Desktop 4.1.1 with K8s
vCPU: 4
Memory: 5GB
Swap: 1GB
Disk Image size: 59.6GB(21.5GB used)
Kubernetes v1.21.5
M1 Macbook Air 2021
$ kubectl version (git)[main]
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/arm64"}
操作步骤
边阅读这个URL上的快速入门指南，同时进行操作。
# リポジトリを取得
git clone --depth=1 git@github.com:prometheus-operator/kube-prometheus.git
cd kube-prometheus/
# セットアップを実行
kubectl create -f manifests/setup
# 待つ
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
# マニフェストを適用（リソースを追加）
kubectl create -f manifests/
如果资源不足，Pod将处于等待状态，需要增加资源。原本应该先阅读要求并查看适当节点的规格，然后进行操作。
kubectl get pod -n monitoring -l prometheus=k8s (git)[main]
NAME READY STATUS RESTARTS AGE
prometheus-k8s-0 0/2 Pending 0 3m38s
prometheus-k8s-1 0/2 Pending 0 3m38s
调查原因后发现，有0/1个可用节点：1内存不足，因此需要增加集群的内存。
kubectl describe pod/prometheus-k8s-0 -n monitoring (git)[main]
Name: prometheus-k8s-0
Namespace: monitoring
Priority: 0
Node: docker-desktop/192.168.65.4
Start Time: Wed, 10 Nov 2021 16:56:14 +0900
Labels: app.kubernetes.io/component=prometheus
app.kubernetes.io/instance=k8s
app.kubernetes.io/managed-by=prometheus-operator
app.kubernetes.io/name=prometheus
app.kubernetes.io/part-of=kube-prometheus
app.kubernetes.io/version=2.31.1
controller-revision-hash=prometheus-k8s-7d994f449b
operator.prometheus.io/name=k8s
operator.prometheus.io/shard=0
prometheus=k8s
statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations: kubectl.kubernetes.io/default-container: prometheus
Status: Pending
IP: 10.1.0.75
IPs:
IP: 10.1.0.75
Controlled By: StatefulSet/prometheus-k8s
Init Containers:
init-config-reloader:
Container ID: docker://d159301da6826148013e624a8e050f64ad4c8fcfd119a3ca7c2e7c19585edf6c
Image: quay.io/prometheus-operator/prometheus-config-reloader:v0.52.0
Image ID: docker-pullable://quay.io/prometheus-operator/prometheus-config-reloader@sha256:d8ceaa9878215db2c2e06e79ee8593a48186888ea48179793b9cc34801b6b486
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/prometheus-config-reloader
Args:
--watch-interval=0
--listen-address=:8080
--config-file=/etc/prometheus/config/prometheus.yaml.gz
--config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
--watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 10 Nov 2021 16:56:15 +0900
Finished: Wed, 10 Nov 2021 16:56:15 +0900
Ready: True
Restart Count: 0
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 100m
memory: 50Mi
Environment:
POD_NAME: prometheus-k8s-0 (v1:metadata.name)
SHARD: 0
Mounts:
/etc/prometheus/config from config (rw)
/etc/prometheus/config_out from config-out (rw)
/etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
Containers:
prometheus:
Container ID:
Image: quay.io/prometheus/prometheus:v2.31.1
Image ID:
Port: 9090/TCP
Host Port: 0/TCP
Args:
--web.console.templates=/etc/prometheus/consoles
--web.console.libraries=/etc/prometheus/console_libraries
--config.file=/etc/prometheus/config_out/prometheus.env.yaml
--storage.tsdb.path=/prometheus
--storage.tsdb.retention.time=24h
--web.enable-lifecycle
--web.route-prefix=/
--web.config.file=/etc/prometheus/web_config/web-config.yaml
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
memory: 400Mi
Readiness: http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
Environment: <none>
Mounts:
/etc/prometheus/certs from tls-assets (ro)
/etc/prometheus/config_out from config-out (ro)
/etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
/etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
/prometheus from prometheus-k8s-db (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
config-reloader:
Container ID:
Image: quay.io/prometheus-operator/prometheus-config-reloader:v0.52.0
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/prometheus-config-reloader
Args:
--listen-address=:8080
--reload-url=http://localhost:9090/-/reload
--config-file=/etc/prometheus/config/prometheus.yaml.gz
--config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
--watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 100m
memory: 50Mi
Environment:
POD_NAME: prometheus-k8s-0 (v1:metadata.name)
SHARD: 0
Mounts:
/etc/prometheus/config from config (rw)
/etc/prometheus/config_out from config-out (rw)
/etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: Secret (a volume populated by a Secret)
SecretName: prometheus-k8s
Optional: false
tls-assets:
Type: Secret (a volume populated by a Secret)
SecretName: prometheus-k8s-tls-assets
Optional: false
config-out:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
prometheus-k8s-rulefiles-0:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: prometheus-k8s-rulefiles-0
Optional: false
web-config:
Type: Secret (a volume populated by a Secret)
SecretName: prometheus-k8s-web-config
Optional: false
prometheus-k8s-db:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-6jj68:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m57s default-scheduler 0/1 nodes are available: 1 Insufficient memory.
Warning FailedScheduling 5m55s default-scheduler 0/1 nodes are available: 1 Insufficient memory.
Normal Scheduled 12s default-scheduler Successfully assigned monitoring/prometheus-k8s-0 to docker-desktop
网络访问
解释两种从网络上访问的方法。
- 创建Ingress并进行kubectl port-forward & SSH端口转发。
(1) kubectl端口转发和SSH端口转发
使用kubectl port-forward将localhost:9090与Service关联起来。
kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
通过浏览器访问 http://localhost:9090/。
如果您正在虚拟机中执行 kubectl 命令，可以使用 ssh 选项进行端口转发。
以下是将本地计算机的 9000 端口映射到 SSH 目标主机的 9090 端口的示例。
在这种情况下，可以通过浏览器访问 http://localhost:9000/。
ssh -L 9000:localhost:9090 <YOUR_HOST_NAME>
在Grafana的情况下，将svc/prometheus-k8s改为svc/grafana。
(2) 创造Ingress
有一种方法可以实现在kubectl port-forward之外进行访问，即使用ingress。通过在ingress中设置使其对外部进行公开，从而实现简单的外部访问。
这里需要使用Ingress来使得Grafana可以从集群外部访问。
将以下内容保存为grafana-publish.yml，并保存在/opt/prometheus文件夹中。
将替换为您的主机名。
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-grafana
namespace: monitoring
spec:
ingressClassName: nginx
rules:
- host: <YOUR_HOSTNAME>
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grafana
port:
number: 3000
使用kubectl命令创建资源。
kubectl apply -f /opt/prometheus/grafana-publish.yml
使用浏览器访问以下网址。
http://<YOUR_HOSTNAME>/
下面是一个从Grafana上查看Pod的CPU和内存使用情况的示例。
