使用jq工具将嵌套的数组展开，并转换为多个对象

我因为想在jq中做以下事情而纠结了一下，最终想出了一个方法，做个备忘录。

我想做的事情

使用中文来重新表达这个含有以下数组的对象

{
  "pod": {
    "name": "mypod"
  },
  "containers": [
    {
      "name": "mycontainer1",
      "usage": {
            "cpu": "50"
          }
    },
    {
      "name": "mycontainer2",
      "usage": {
            "cpu": "100"
          }
    }
  ]
}

我想把以下这样的多个对象实现。

{
  "pod_name": "mypod",
  "container_name": "mycontainer1",
  "container_cpu": "50"
}
{
  "pod_name": "mypod",
  "container_name": "mycontainer2",
  "container_cpu": "100"
}

办法

在jq手册的对象构造部分有解释，可以创建一个新的对象，并通过对containers数组进行解包（[]）来将其展开为不同的对象。

嵌套数组的展开

cat sample.json | \
  jq '{
        pod,
        container: .containers[]
      }'

{
  "pod": {
    "name": "mypod"
  },
  "container": {
    "name": "mycontainer1",
    "usage": {
      "cpu": "50"
    }
  }
}
{
  "pod": {
    "name": "mypod"
  },
  "container": {
    "name": "mycontainer2",
    "usage": {
      "cpu": "100"
    }
  }
}

（补充）pod的部分采用省略形式，直接传递原始字段，但也可以适当地进行详细描述。

cat sample.json | \
  jq '{
        pod: { name: .pod.name },
        container: .containers[]
      }'

创建对象

从展开的结果中挑选出所需信息，并创建一个新的对象。

cat sample.json | \
  jq '{
        pod,
        container: .containers[]
      } |
      {
        pod_name: .pod.name,
        container_name: .container.name,
        container_cpu: .container.usage.cpu
      }'

{
  "pod_name": "mypod",
  "container_name": "mycontainer1",
  "container_cpu": "50"
}
{
  "pod_name": "mypod",
  "container_name": "mycontainer2",
  "container_cpu": "100"
}

注意事项

直接创建对象而不通过中间对象会导致结果变成乘法，并且对象的数量会增加。

cat sample.json | \
  jq '{
        pod_name: .pod.name,
        container_name: .containers[].name,
        container_cpu: .containers[].usage.cpu
       }'

{
  "pod_name": "mypod",
  "container_name": "mycontainer1",
  "container_cpu": "50"
}
{
  "pod_name": "mypod",
  "container_name": "mycontainer1",
  "container_cpu": "100"
}
{
  "pod_name": "mypod",
  "container_name": "mycontainer2",
  "container_cpu": "50"
}
{
  "pod_name": "mypod",
  "container_name": "mycontainer2",
  "container_cpu": "100"
}

不经过中间对象，直接创建数组会导致原始数组的元素按照原样排列。

cat sample.json | \
  jq '[
        .pod.name,
        .containers[].name,
        .containers[].usage.cpu
       ]'

[
  "mypod",
  "mycontainer1",
  "mycontainer2",
  "50",
  "100"
]

給你额外的东西。

原本想做的事情是从Kubernetes的Metric API获取类似以下的JSON，然后将其转换为CSV格式。以下是IBM Cloud Private v3.1.2的情况。

{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/pods"
  },
  "items": [
    {
      "metadata": {
        "name": "audit-logging-fluentd-ds-pgp88",
        "namespace": "kube-system",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/audit-logging-fluentd-ds-pgp88",
        "creationTimestamp": "2019-04-10T01:22:42Z"
      },
      "timestamp": "2019-04-10T01:22:17Z",
      "window": "30s",
      "containers": [
        {
          "name": "fluentd",
          "usage": {
            "cpu": "2513099n",
            "memory": "62276Ki"
          }
        }
      ]
    },

（以下略）

获取令牌

在使用cloudctl login进行登陆后，使用cloudctl tokens获取令牌。

ID_TOKEN=$(LANG=C cloudctl tokens | grep "ID token:" | awk '{print $3}')

通过度量API获取数据

用 curl 获取 JSON 数据并将其存入变量中。

CLUSTER="mycluster.icp"
JSON=$(curl -s -k -H "Authorization: Bearer ${ID_TOKEN}" \
  "https://${CLUSTER}:8001/apis/metrics.k8s.io/v1beta1/pods" \
  | jq -c '.')

使用jq将JSON转换为CSV。

使用jq将数组展开为多个对象并创建中间数据，然后从中选择所需的数据并将其分别组成数组，再使用@csv将其转换为csv格式。最后，使用rtrimstr(str)函数去除单位。

echo $JSON | \
  jq -r '.items[] |
          {
            metadata,
            timestamp,
            container: .containers[]
          } |
          [
            .timestamp,
            .metadata.namespace,
            .metadata.name,
            .container.name,
            ( .container.usage.cpu | rtrimstr("n") ),
            ( .container.usage.memory | rtrimstr("Ki") )
          ] | @csv'

"2019-04-10T02:35:25Z","kube-system","monitoring-prometheus-collectdexporter-57b8c6ff5c-dclw4","collectd-exporter","1488183","11416"
"2019-04-10T02:35:25Z","kube-system","monitoring-prometheus-collectdexporter-57b8c6ff5c-dclw4","router","54112","4532"
"2019-04-10T02:35:09Z","kube-system","logging-elk-filebeat-ds-2x49r","filebeat","4745057","28692"
"2019-04-10T02:35:20Z","kube-system","monitoring-prometheus-elasticsearchexporter-7f5967f56-d68sd","router","12534","8312"
"2019-04-10T02:35:20Z","kube-system","monitoring-prometheus-elasticsearchexporter-7f5967f56-d68sd","elasticsearchexporter","3207930","13132"
"2019-04-10T02:35:10Z","kube-system","k8s-etcd-9.188.124.130","etcd","50751515","261432"

（以下略）