使用 Prometheus Metrics（按节点进行 CPU ）
在Kubernetes中，通常使用Prometheus Metrics获取运行统计信息。
现在可以通过使用 Grafana UI 等工具轻松地查看 Prometheus Metrics 的信息，但在想要查看特定时间段的运行统计信息时可能会不方便。
在这里，我们将介绍如何通过Red Hat OpenShift Container Platform (OCP) 4.6的Cluster Monitoring来获取来自Prometheus Metrics的类似Linux vmstat命令的节点CPU使用率的示例。
Linux的vmstat命令。
Linux 的 vmstat 命令可以获取以下信息。
$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 14748572 3236 659080 0 0 35 4 31 31 0 1 99 0 0
在这里，我们将尝试从Prometheus指标获取与vmstat命令输出的us、sy、id和wa相对应的信息。
$ man vmstat
:
FIELD DESCRIPTION FOR VM MODE
CPU
These are percentages of total CPU time.
us: Time spent running non-kernel code. (user time, including nice time)
sy: Time spent running kernel code. (system time)
id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.
:
在假设已经事先执行过oc login的前提下，按照以下步骤进行。
选择要使用的度量标准
选取 Prometheus 从抓取的数据中使用的度量标准。
OpenShift 容器平台的 Prometheus 可以通过以下方法确认其所抓取的数据。
$ TOKEN=$(oc whoami -t)
$ ROUTE=$(oc get route prometheus-k8s -n openshift-monitoring -o jsonpath='{.spec.host}')
$ curl -v -ks -G -L -H "Authorization: Bearer ${TOKEN}" "https://${ROUTE}/api/v1/metadata" > METADATA.json
在这里，我们将使用最常见的 node_cpu_seconds_total。
$ jq -r '.data | with_entries(select(.key == "node_cpu_seconds_total"))' METADATA.json
{
"node_cpu_seconds_total": [
{
"type": "counter",
"help": "Seconds the cpus spent in each mode.",
"unit": ""
}
]
}
获取CPU使用率
为了获取特定时间范围内的运行统计信息，我们将使用查询范围 API (query_range)。在这个例子中，我们将尝试获取从日本时间 2021/10/03 00:00:00 到 23:59:59 的每60秒的全局 CPU 平均使用率。
$ TOKEN=$(oc whoami -t)
$ ROUTE=$(oc get route prometheus-k8s -n openshift-monitoring -o jsonpath='{.spec.host}')
$ START=$(date --date="2021-10-03 00:00:00+09:00" '+%s')
$ END=$(date --date="2021-10-03 23:59:59+09:00" '+%s')
$ curl -v -ks -G -L -H "Authorization: Bearer ${TOKEN}" "https://${ROUTE}/api/v1/query_range" \
--data-urlencode "query=avg without(cpu) (rate(node_cpu_seconds_total[60s]))" \
--data-urlencode "start=${START}" \
--data-urlencode "end=${END}" \
--data-urlencode "step=60" \
> CPU.json
如果HTTP状态代码是200，并且JSON的.status如下所示，则表示成功获取。
$ jq -r '.status' CPU.json
success
整理 CPU 使用率（CPU.json）的数据。
为了增强对获得的CPU使用率的可视性，我们将对其进行格式化。在这里，我们将尝试将其转换为广泛使用的Microsoft Excel可用的CSV格式。转换为CSV有许多不同的实现方式，但在这里我们将尝试使用jq命令和awk命令。
将JSON转换为TMP
使用jq命令，将CPU.json转换为临时文件（CPU.tmp）。
$ jq -r '.data.result[] |
.values[] += [(.metric.instance + " " + .metric.mode)] |
.metric.mode as $M | select($M == "user" or $M == "system" or $M == "iowait" or $M == "idle") |
.values[] | @csv' CPU.json > CPU.tmp
$ head CPU.tmp
1633186800,"0.3987222222300867","infra01 idle"
1633186860,"0.30188888889840904","infra01 idle"
1633186920,"0.5176840654579138","infra01 idle"
1633186980,"0.4833888888876471","infra01 idle"
1633187040,"0.4952777777694993","infra01 idle"
1633187100,"0.5275000000062088","infra01 idle"
1633187160,"0.4790342651359072","infra01 idle"
1633187220,"0.5096666666575604","infra01 idle"
1633187280,"0.49066666666832226","infra01 idle"
1633187340,"0.5019665355506373","infra01 idle"
将TMP转为CSV格式
使用awk命令将CPU.tmp转换为CSV文件（CPU.csv）。
$ SEP=" " # Tab Character
$ TS="$(awk -F "," '{print $1}' CPU.tmp | sort -n | uniq | tr "\n" "$SEP")"
$ ES="$(awk -F "," '{print $3}' CPU.tmp | sort | uniq | tr "\n" "$SEP")"
$ awk -v FS="," -v SEP="${SEP}" -v TS="${TS}" -v ES="${ES}" '
BEGIN {
OA[1][1]=""
split(TS, TA, SEP)
split(ES, EA, SEP)
}
{
for (i=1; i<=length(TA); i++){
if ($1 == TA[i]){
for(j=1; j<=length(EA); j++){ if($3 == EA[j]) { break } }
OA[i][j]=$2
}
}
}
END {
printf "%s,", "TIME"
for (i=1; i<=length(EA); i++){
if (i == length(EA)){ S="\n" }else{ S="," }
printf "%s%s", EA[i], S
}
for (i=1; i<=length(TA); i++){
printf "\"%s\",", strftime("%y/%m/%d %H:%M:%S %Z", TA[i])
for (j=1; j<=length(OA[i]); j++){
if (j == length(OA[i])){ S="\n" }else{ S="," }
printf "%s%s", OA[i][j], S
}
}
}
' CPU.tmp > CPU.csv
$ head CPU.csv
TIME,"infra01 idle","infra01 iowait","infra01 system","infra01 user","infra02 idle","infra02 iowait","infra02 system","infra02 user","infra03 idle","infra03 iowait","infra03 system","infra03 user","master01 idle","master01 iowait","master01 system","master01 user","master02 idle","master02 iowait","master02 system","master02 user","master03 idle","master03 iowait","master03 system","master03 user","worker01 idle","worker01 iowait","worker01 system","worker01 user","worker02 idle","worker02 iowait","worker02 system","worker02 user"
"21/10/03 00:00:00 JST","0.3987222222300867","0.0008333333333212067","0.08261111111108523","0.490222222223464","0.5200555555522441","0.0005555555555575765","0.04905555555532272","0.4120555555551416","0.7921666666658388","0.0004444444444440402","0.0662222222218083","0.11905555555519337","0.6671666666658388","0.0003888888888923247","0.06416666666692536","0.22533333333299702","0.8691111111133877","0.00044444444444656663","0.03433333333313284","0.07788888888866899","0.8824444444487906","0.0004999999999982821","0.03211111111110464","0.06711111111152503","0.921444444461829","0","0.0281111111107748","0.03888888889018239","0.9005555555638339","0.00011111111111353642","0.043444444443513114","0.04299999999994826"
"21/10/03 00:01:00 JST","0.30188888889840904","0.0013888888888888887","0.07783333333273831","0.5917777777804683","0.34643341410692885","0.004945873252269644","0.06990908484779448","0.5568275279535547","0.7985555555257531","0.0009444444444347431","0.06016666666708058","0.11938888889013065","0.7134205964717821","0.0025556691408496436","0.05789146184265802","0.18584159296007516","0.8725891031912415","0.003944093858323908","0.031330548395290744","0.0742156252776755","0.8868171918758154","0.0039446197608784634","0.027501222276448606","0.06550291124070655","0.9221111111135946","0.0002222222222220201","0.027555555555348596","0.03933333333245374","0.7872222222180829","0.0006666666666660603","0.06077777777746733","0.13211111111126633"
"21/10/03 00:02:00 JST","0.5176840654579138","0.0007224791036869701","0.06107727191885274","0.4015316556998044","0.5281111111243565","0.0005000000000109139","0.045722222221471986","0.4098333333334368","0.8037222222218082","0.00038888888889737747","0.06111111111175785","0.11461111110919672","0.7263888888888888","0.000444444444442777","0.05755555555547795","0.17916666666666667","0.8828333333341611","0.0005000000000008084","0.031722222222338635","0.06722222222209287","0.8913333333304359","0.0005555555555550502","0.028722222222293363","0.0634444444443539","0.9228888888739877","0.00011111111111101006","0.027444444444780752","0.038555555555245105","0.9017777777794335","0","0.04222222222273962","0.04388888888837149"
"21/10/03 00:03:00 JST","0.4833888888876471","0.0007222222222177759","0.061055555555503814","0.43138888888837146","0.5196111111073858","0.0005000000000008084","0.04583333333333334","0.41738888888826803","0.7861111111111111","0.000388888888887272","0.06411111111131806","0.1273333333342129","0.6887777777802613","0.00033333333333429333","0.06394444444449618","0.2056666666667701","0.8672777777744665","0.0005000000000008084","0.03327777777772604","0.08055555555555555","0.88966666665963","0.0005555555555550503","0.02944444444421808","0.06344444444435389","0.9211111110945542","0.00011111111111101006","0.027555555555348592","0.03988888888982021","0.8808888888814382","0.00011111111111353642","0.04433333333322985","0.06044444444382356"
"21/10/03 00:04:00 JST","0.4952777777694993","0.0008333333333414177","0.058222222222118744","0.4263888888888889","0.5219444444506532","0.0006111111111143448","0.04877777777793299","0.41172222222408483","0.7915555555476911","0.00027777777778384107","0.06449999999992238","0.12183333333426466","0.7075555555625923","0.0002777777777800515","0.0605000000000776","0.19288888888873368","0.875833333325055","0.0006111111111118184","0.032500000000032336","0.07088888888861725","0.8841666666635623","0.0005000000000033347","0.03172222222217695","0.06511111111127926","0.921847917877767","0.00022221234611774818","0.027665437090934615","0.03955379760832256","0.9019999999966886","0.00011111111110848367","0.042222222221446115","0.04311111111116285"
"21/10/03 00:05:00 JST","0.5275000000062088","0.0007777777777644385","0.05744444444426335","0.39566666666521794","0.5479444444489976","0.0005555555555575766","0.043166666666770145","0.39377777777715683","0.8363888888930281","0.00044444444443393464","0.05683333333387659","0.0888333333332816","0.7460000000066227","0.00027777777777752516","0.053444444444741945","0.16722222222209288","0.8853888888895096","0.0003333333333330301","0.029333333333488552","0.06727777777802355","0.9012777777802613","0.0003333333333330301","0.02677777777781658","0.05727777777776484","0.9241111111102832","0.00022222222222454646","0.02722222222170482","0.03844444444403052","0.9011111111069718","0","0.041777777777881255","0.04322222222237744"
"21/10/03 00:06:00 JST","0.4790342651359072","0.0006110839518275969","0.06105284209585511","0.43731389716126445","0.5289444444494115","0.0006111111111042393","0.04483333333304876","0.4102777777782952","0.7868333333378865","0.00033333333334060924","0.06505555555599535","0.12544444444356487","0.7128333333414048","0.000333333333331767","0.05872222222161427","0.1900555555553486","0.8783333333364376","0.0005000000000008084","0.03216666666671194","0.06933333333322984","0.8877222222172552","0.0005555555555525238","0.029499999999987064","0.06611111111091708","0.9258888888897168","0","0.025888888888423226","0.037777777778036474","0.8784444444295432","0.00011111111110848367","0.045555555555296855","0.06077777777746733"
"21/10/03 00:07:00 JST","0.5096666666575604","0.001055555555558385","0.0599444444446514","0.4101666666670805","0.5377222222172551","0.0005555555555575766","0.04461111111191308","0.40133333333198806","0.8203333333238131","0.00038888888888727194","0.05938888888857844","0.10088888888971674","0.7105555555627991","0.00033333333333429333","0.058944444444690214","0.19400000000023282","0.8642222222199456","0.00044444444444151387","0.03427777777784892","0.08083333333326866","0.8942222222220153","0.00044444444444404017","0.029000000000006465","0.06072222222250679","0.9253333333258826","0.00011111111111101006","0.02644444444449618","0.037999999999172154","0.857333333314293","0","0.05233333333421292","0.07522222222372268"
"21/10/03 00:08:00 JST","0.49066666666832226","0.0006666666666711131","0.06177777777839866","0.4258888888897167","0.5351111111024188","0.0005000000000008085","0.048722222222972456","0.40133333333716203","0.8003888888905446","0.0003333333333406092","0.05966666666661492","0.11811111111245637","0.6932777777779847","0.0002777777777787883","0.06127777777728625","0.20516666666630448","0.875","0.00038888888889232476","0.032611111111085236","0.07238888888839734","0.8914444444442374","0.0004444444444465666","0.029222222221950582","0.06322222222192471","0.9244444444361659","0.00011111111111101006","0.026888888888707798","0.038333333332815925","0.8721111111135946","0","0.049555555556435135","0.06355555555524511"
