我在OpenShift上尝试了渐进式交付!(Argo Rollouts应用篇-与Istio+Prometheus协作)

不变

因为这篇文章是上一篇文章的延续,请参考以下文章来了解背景信息。
在OpenShift上尝试进行渐进式交付!(Argo Rollouts基础版本-CLI)
在OpenShift上尝试进行渐进式交付!(Argo Rollouts基本版本-GUI)
在OpenShift上尝试进行渐进式交付!(Argo Rollouts应用版本-与Prometheus指标集成)

剧本

这次验证中执行的渐进交付(Progressive Delivery)场景如下所示。

    • Rolloutのコンテナ・イメージを新しいバージョンに更新

新しいバージョンはアクセスすると500 InternalServerErrorが発生する

Argo Rolloutsによりトラフィックの50%がCanaryに割り当てた後一時停止
アプリケーションの検証実施

50%の確率でCanaryに割り当てられるのでInternalServerErrorが応答される

Argo Rolloutsで一時停止しているRolloutを再開(Promote)
Argo Rolloutsの分析が実行される

Canaryのサービスへのトラフィック上で過去5分の間の500 InternalServerErrorの回数をPrometheusから取得

5回以下の場合はRolloutが継続し完了する
5回より多く出力されている場合はRolloutがロールバックされる

构成步骤

样本应用的扩展

我们将使用下面在OpenShift v4.6手册中使用的样本应用程序,创建一个新版本,该版本会输出500 InternalServerError。

    • 前述したリポジトリをFORKして以下の箇所を編集

コンテキストルート(/)へのアクセスを200 正常応答から500 InternalServerErrorに変更

image.png
    コンテナ・イメージをビルドするためOpenShiftでアプリケーションを作成
PS D:\git> oc new-app --as-deployment-config --name prometheus-example-app golang~https://github.com/strada501/prometheus-example-app.git
--> Found image ca0cba7 (11 months old) in image stream "openshift/golang" under tag "1.13.4-ubi8" for "golang"

    Go 1.13.4
    ---------
    Go Toolset available as a container is a base platform for building and running various Go applications and frameworks. Go is an easy to learn, powerful, statically typed language in the C/C++ tradition with garbage collection, concurrent programming support, and memory safety features.

    Tags: builder, golang, golang113, rh-golang113, go

    * A source build using source code from https://github.com/strada501/prometheus-example-app.git will be created
      * The resulting image will be pushed to image stream tag "prometheus-example-app:latest"
      * Use 'oc start-build' to trigger a new build
    * This image will be deployed in deployment config "prometheus-example-app"
    * The image does not expose any ports - if you want to load balance or send traffic to this component
      you will need to create a service with 'oc expose dc/prometheus-example-app --port=[port]' later

--> Creating resources ...
    imagestream.image.openshift.io "prometheus-example-app" created
    buildconfig.build.openshift.io "prometheus-example-app" created
    deploymentconfig.apps.openshift.io "prometheus-example-app" created
--> Success
    Build scheduled, use 'oc logs -f buildconfig/prometheus-example-app' to track its progress.
    Run 'oc status' to view your app.
    ビルド・ログを確認し、Push successfulを確認
PS D:\git> oc logs -f buildconfig/prometheus-example-app
Cloning "https://github.com/strada501/prometheus-example-app.git" ...
        Commit: c65b7cdbd27e27ad4e3c22c1e94de67c83a8a4fa (Update main.go)
        Author: Yasuyuki Kubota <strada@jp.ibm.com>
        Date:   Thu Jul 15 14:13:08 2021 +0900
Caching blobs under "/var/cache/blobs".
Getting image source signatures
Copying blob sha256:47db82df7f3f4393c1f19c362a2db2c47ca049b6fb20bef041dfc9bdb12a4504
Copying blob sha256:71391dc11a78542160544b68e45bc123ff55a2e84aeb6fa99b672d75765bc2f8
Copying blob sha256:6cc5487eb078291c64e0539c6d57546606b3936b71eabc7eec7960d069927ba4
Copying blob sha256:77c58f19bd6e67185938abb6bbb6ec229e07a5e607453904294d982de141d2f0
Copying blob sha256:0b1d2fb42752674d5c08548ef22a1c25885dd8588c0df75278a6a17fa1362e80
Warning: Pull failed, retrying in 5s ...
Getting image source signatures
Copying blob sha256:47db82df7f3f4393c1f19c362a2db2c47ca049b6fb20bef041dfc9bdb12a4504
Copying blob sha256:0b1d2fb42752674d5c08548ef22a1c25885dd8588c0df75278a6a17fa1362e80
Copying blob sha256:71391dc11a78542160544b68e45bc123ff55a2e84aeb6fa99b672d75765bc2f8
Copying blob sha256:77c58f19bd6e67185938abb6bbb6ec229e07a5e607453904294d982de141d2f0
Copying blob sha256:6cc5487eb078291c64e0539c6d57546606b3936b71eabc7eec7960d069927ba4
Copying config sha256:ca0cba75b144d9ec831c913b2146a6e9746c6cf43a4f6314047915f934c0b2f0
Writing manifest to image destination
Storing signatures
Generating dockerfile with builder image image-registry.openshift-image-registry.svc:5000/openshift/golang@sha256:ff3b2217e9a0bbb05536713606bdf49d0c9ef7754bd323482c41317a896c5bf2
STEP 1: FROM image-registry.openshift-image-registry.svc:5000/openshift/golang@sha256:ff3b2217e9a0bbb05536713606bdf49d0c9ef7754bd323482c41317a896c5bf2
STEP 2: LABEL "io.openshift.build.commit.id"="c65b7cdbd27e27ad4e3c22c1e94de67c83a8a4fa"       "io.openshift.build.commit.ref"="master"       "io.openshift.build.commit.message"="Update main.go"       "io.openshift.build.source-location"="https://github.com/strada501/prometheus-example-app.git"       "io.openshift.build.image"="image-registry.openshift-image-registry.svc:5000/openshift/golang@sha256:ff3b2217e9a0bbb05536713606bdf49d0c9ef7754bd323482c41317a896c5bf2"       "io.openshift.build.commit.author"="Yasuyuki Kubota <strada@jp.ibm.com>"       "io.openshift.build.commit.date"="Thu Jul 15 14:13:08 2021 +0900"
STEP 3: ENV OPENSHIFT_BUILD_NAME="prometheus-example-app-1"     OPENSHIFT_BUILD_NAMESPACE="kubota-test"     OPENSHIFT_BUILD_SOURCE="https://github.com/strada501/prometheus-example-app.git"     OPENSHIFT_BUILD_COMMIT="c65b7cdbd27e27ad4e3c22c1e94de67c83a8a4fa"
STEP 4: USER root
STEP 5: COPY upload/src /tmp/src
STEP 6: RUN chown -R 1001:0 /tmp/src
STEP 7: USER 1001
STEP 8: RUN /usr/libexec/s2i/assemble
/tmp/src ~
go: downloading github.com/prometheus/client_golang v1.3.0
go: extracting github.com/prometheus/client_golang v1.3.0
go: downloading github.com/prometheus/common v0.7.0
go: downloading github.com/prometheus/client_model v0.1.0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.1
go: downloading github.com/golang/protobuf v1.3.2
go: downloading github.com/prometheus/procfs v0.0.8
go: extracting github.com/prometheus/client_model v0.1.0
go: extracting github.com/beorn7/perks v1.0.1
go: extracting github.com/cespare/xxhash/v2 v2.1.1
go: extracting github.com/prometheus/common v0.7.0
go: extracting github.com/prometheus/procfs v0.0.8
go: extracting github.com/golang/protobuf v1.3.2
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: extracting github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/prometheus/client_golang v1.3.0
go: finding github.com/prometheus/client_model v0.1.0
go: finding github.com/beorn7/perks v1.0.1
go: finding github.com/cespare/xxhash/v2 v2.1.1
go: finding github.com/prometheus/common v0.7.0
go: finding github.com/golang/protobuf v1.3.2
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/prometheus/procfs v0.0.8
~
STEP 9: CMD /usr/libexec/s2i/run
STEP 10: COMMIT temp.builder.openshift.io/kubota-test/prometheus-example-app-1:1b4719fa
Getting image source signatures
Copying blob sha256:226bfaae015f1d5712cfced3b5b628206618eaacf72f4a44d0e4084071996319
Copying blob sha256:70056249a0e202adae10aa45fef56ac4cc6497619767753515022bc9c1278251
Copying blob sha256:0bfe5b62a1ad05827328d7629f8a9a7e16642fd14f40070a395fcd16917eb4f7
Copying blob sha256:7fbe9fcfa2da3fadd05c2e7caabc3da06c1a7fc36c14f5b6dddedc5ffb42be11
Copying blob sha256:1ee5a618948c954f5939610cdea18abc9b4ea49837b2284dfcc704c107e8fdd8
Copying blob sha256:a5e344f477e23a8dcafc25df603a178a9470835a1524867bfd1db3e3de38f41c
Copying config sha256:fda5edea5118eeb68890239e18a706f07cee81550e4029540c9cfaa8834a5ee5
Writing manifest to image destination
Storing signatures
--> fda5edea511
fda5edea5118eeb68890239e18a706f07cee81550e4029540c9cfaa8834a5ee5

Pushing image image-registry.openshift-image-registry.svc:5000/kubota-test/prometheus-example-app:latest ...
Getting image source signatures
Copying blob sha256:a5e344f477e23a8dcafc25df603a178a9470835a1524867bfd1db3e3de38f41c
Copying blob sha256:47db82df7f3f4393c1f19c362a2db2c47ca049b6fb20bef041dfc9bdb12a4504
Copying blob sha256:6cc5487eb078291c64e0539c6d57546606b3936b71eabc7eec7960d069927ba4
Copying blob sha256:71391dc11a78542160544b68e45bc123ff55a2e84aeb6fa99b672d75765bc2f8
Copying blob sha256:77c58f19bd6e67185938abb6bbb6ec229e07a5e607453904294d982de141d2f0
Copying blob sha256:0b1d2fb42752674d5c08548ef22a1c25885dd8588c0df75278a6a17fa1362e80
Copying config sha256:fda5edea5118eeb68890239e18a706f07cee81550e4029540c9cfaa8834a5ee5
Writing manifest to image destination
Storing signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/kubota-test/prometheus-example-app@sha256:5613eb8474ea6ab372ec6c1c98a9517be61cb39ccaf20214c348c4ab04c3b227
Push successful
PS D:\git>

Successfully pushed以降のコンテナ・イメージのURIをメモしておく

image-registry.openshift-image-registry.svc:5000/kubota-test/prometheus-example-app@sha256:5613eb8474ea6ab372ec6c1c98a9517be61cb39ccaf20214c348c4ab04c3b227
    PodがデプロイされてRunningとなっていることを確認
PS D:\git> oc get po
NAME                                  READY   STATUS      RESTARTS   AGE
prometheus-example-app-1-56mtd        1/1     Running     0          18s
prometheus-example-app-1-build        0/1     Completed   0          94s
prometheus-example-app-1-deploy       0/1     Completed   0          23s
    Podに接続して稼働確認
PS D:\git> oc rsh prometheus-example-app-1-56mtd
sh-4.4$ curl -si http://localhost:8080/
HTTP/1.1 500 Internal Server Error
Date: Thu, 15 Jul 2021 05:16:11 GMT
Content-Length: 0

sh-4.4$ exit
exit
PS D:\git>

安装 OpenShift 服务网格

请参考以下文章了解OpenShift Service Mesh的安装步骤。

安装 Argo Rollouts

请参考以下文章了解 Argo Rollouts 的安装步骤。

Prometheus Operator的安装

请参考以下文章以获取 Prometheus Operator 的安装步骤。

创建用于 Argo Rollouts 和 OpenShift Service Mesh 的资源

参考以下手册,创建 Argo Rollouts 资源。

这次使用了Host-level Traffic Splitting的例子。

如果要与Istio集成,您需要在strategy->canary下添加以下设置。

  strategy:
    canary:
      canaryService: canary-svc  # required
      stableService: stable-svc  # required
      trafficRouting:
        istio:
          virtualService: 
            name: rollout-vsvc   # required
            routes:
            - primary            # optional if there is a single route in VirtualService, required otherwise
    istio-systemのistio-ingressgatewayのRouteを取得
PS D:\git> oc get route -n istio-system
NAME                                                 HOST/PORT                                                                                                                                                   PATH   SERVICES               PORT    TERMINATION          WILDCARD
grafana                                              grafana-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud                                                     grafana                <all>   reencrypt/Redirect   None
istio-ingressgateway                                 istio-ingressgateway-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud                                        istio-ingressgateway   8080                         None
jaeger                                               jaeger-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud                                                      jaeger-query           <all>   reencrypt            None
kiali                                                kiali-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud                                                       kiali                  <all>   reencrypt/Redirect   None
kubota-test-bookinfo-gateway-525eca1d5089dbdc        kubota-test-bookinfo-gateway-525eca1d5089dbdc-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud               istio-ingressgateway   http2                        None
kubota-test-rollouts-demo-gateway-525eca1d5089dbdc   kubota-test-rollouts-demo-gateway-525eca1d5089dbdc-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud          istio-ingressgateway   http2                        None
prometheus                                           prometheus-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud                                                  prometheus             <all>   reencrypt/Redirect   None
PS D:\git>
    • Prometheusと連携するためのAnalysisTemplateのマニフェストファイルを作成

Prometheusのアドレスにはistio-systemのistio-ingressgatewayのRouteのアドレスを設定
Prometheusのクエリーにはcanaryで発生した直近5分の500 InternalServerErrorの回数を取得するクエリーを設定

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: failure-count
spec:
  metrics:
  - name: http_error_count
    successCondition: result[0] <= 5
    provider:
      prometheus:
        address: "http://istio-ingressgateway-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud"
        query: |
          delta(http_requests_total{code="500",namespace="kubota-test",service="rollouts-demo-canary"}[5m])

    • Rolloutのマニフェストファイルを作成

IstioのSidecarを注入するためAnnotationsにsidecar.istio.io/inject: “true”を追加
ステップは以下の通り

50%をcanaryに割り当て一時停止
再開されたらfailure-countのAnalysisTemplateを実行

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
spec:
  replicas: 1
  strategy:
    canary:
      canaryService: rollouts-demo-canary
      stableService: rollouts-demo-stable
      trafficRouting:
        istio:
          virtualService:
            name: rollouts-demo-vsvc
            routes:
            - primary # At least one route is required
      steps:
      - setWeight: 50
      - pause: {}
      - analysis:
          templates:
          - templateName: failure-count
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: rollouts-demo
    spec:
      containers:
      - name: rollouts-demo
        image: quay.io/brancz/prometheus-example-app:v0.2.0
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m
    • Serviceのマニフェストファイルを作成

OpenShift Service Meshでのトラフィックコントロールに対応するためStableとCanaryの2つのサービスを定義

apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-canary
  labels:
    app: rollouts-demo-canary
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: rollouts-demo
    # This selector will be updated with the pod-template-hash of the canary ReplicaSet. e.g.:
    # rollouts-pod-template-hash: 7bf84f9696
---
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-stable
  labels:
    app: rollouts-demo-stable
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: rollouts-demo
    # This selector will be updated with the pod-template-hash of the stable ReplicaSet. e.g.:
    # rollouts-pod-template-hash: 789746c88d
    • ServiceMonitorのマニフェストファイルを作成

2つのサービスを定義するのでこちらも2つのMonitorを定義

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    team: frontend
    k8s-app: rollouts-demo-canary
  name: rollouts-demo-canary
spec:
  endpoints:
  - interval: 30s
    port: http
    scheme: http
  selector:
    matchLabels:
      app: rollouts-demo-canary
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    team: frontend
    k8s-app: rollouts-demo-stable
  name: rollouts-demo-stable
spec:
  endpoints:
  - interval: 30s
    port: http
    scheme: http
  selector:
    matchLabels:
      app: rollouts-demo-stable
    Gatewayのマニフェストファイルを作成
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: rollouts-demo-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
    • VirtualServiceのマニフェストファイルを作成

コンテキストルート(/)と/errへのアクセスが検証用アプリケーションに割り振られるように設定
/apiへのアクセスがPrometheusに割り振られるように設定

サービスメッシュに参加しているプロジェクトではサービスのFQDN(prometheus.kubota-test.svc.cluster.local:9090)ではアクセスできないためingressgateway経由でアクセスできるようする

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: rollouts-demo-vsvc
spec:
  gateways:
  - rollouts-demo-gateway
  hosts:
  - "*"
  http:
  - name: primary
    match:
    - uri:
        exact: /
    - uri:
        exact: /err
    route:
    - destination:
        host: rollouts-demo-stable
      weight: 100
    - destination:
        host: rollouts-demo-canary
      weight: 0
  - name: prometheus
    match:
    - uri:
        prefix: /api
    route:
    - destination:
        host: prometheus
    • マニフェストファイルを以下の順番で適用

Gateway
VirtualService
AnalysisTemplate
Rollout
Service
ServiceMonitor

PS D:\git> oc apply -f .\gateway.yaml
gateway.networking.istio.io/rollouts-demo-gateway created
PS D:\git> oc apply -f .\virtualservice.yaml
virtualservice.networking.istio.io/rollouts-demo-vsvc created
PS D:\git> oc apply -f .\prometheus-anlysis-template.yaml
analysistemplate.argoproj.io/failure-count created
PS D:\git> oc apply -f .\rollout.yaml
rollout.argoproj.io/rollouts-demo created
PS D:\git> oc apply -f .\services.yaml
service/rollouts-demo-canary created
service/rollouts-demo-stable created
PS D:\git> oc apply -f .\servicemonitors.yaml
servicemonitor.monitoring.coreos.com/rollouts-demo-canary created
servicemonitor.monitoring.coreos.com/rollouts-demo-stable created
PS D:\git>
    Argo RolloutsのDashboardを起動

请参考以下文章来了解启动方法。

image.png
    • コンテナ・イメージを500 InternalServerErrorが出力される新しいバージョンに変更

なお、DashboardからだとOpenShiftの内部レジストリーを参照する形式のURIを正しく設定できないようなので以下のコマンドを実行して変更

PS D:\git> docker run --rm -v C:\Users\YASUYUKIKUBOTA\.kube\config:/.kube/config quay.io/argoproj/kubectl-argo-rollouts:master set image rollouts-demo rollouts-demo=image-registry.openshift-image-registry.svc:5000/kubota-test/prometheus-example-app@sha256:5613eb8474ea6ab372ec6c1c98a9517be61cb39ccaf20214c348c4ab04c3b227 --insecure-skip-tls-verify
rollout "rollouts-demo" image updated
PS D:\git>
image.png
PS D:\git> oc describe vs rollouts-demo-vsvc
Name:         rollouts-demo-vsvc
Namespace:    kubota-test
Labels:       <none>
Annotations:  <none>
API Version:  networking.istio.io/v1beta1
Kind:         VirtualService
Metadata:
  Creation Timestamp:  2021-07-16T09:32:47Z
  Generation:          8
  Managed Fields:
    API Version:  networking.istio.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:gateways:
        f:hosts:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-07-16T09:32:47Z
    API Version:  networking.istio.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:http:
    Manager:         rollouts-controller
    Operation:       Update
    Time:            2021-07-16T10:32:56Z
  Resource Version:  735642
  Self Link:         /apis/networking.istio.io/v1beta1/namespaces/kubota-test/virtualservices/rollouts-demo-vsvc
  UID:               ac110ed1-dc46-4797-946f-6dad62d067e0
Spec:
  Gateways:
    rollouts-demo-gateway
  Hosts:
    *
  Http:
    Match:
      Uri:
        Exact:  /
      Uri:
        Exact:  /err
    Name:       primary
    Route:
      Destination:
        Host:  rollouts-demo-stable
      Weight:  50
      Destination:
        Host:  rollouts-demo-canary
      Weight:  50
    Match:
      Uri:
        Prefix:  /api
    Name:        prometheus
    Route:
      Destination:
        Host:  prometheus
Events:        <none>
PS D:\git>

我们可以看到Stable和Canary的比例都达到了50%。

image.png
delta(http_requests_total{code="500",namespace="kubota-test",service="rollouts-demo-canary"}[5m])
image.png

我将确认结果大于5。

image.png
image.png

由于 Prometheus 查询的结果大于5,所以 Rollout 失败。

    以下のコマンドでAnalysisRunの内容を確認
PS D:\git> oc get analysisrun -o yaml
apiVersion: v1
items:
- apiVersion: argoproj.io/v1alpha1
  kind: AnalysisRun
  metadata:
    annotations:
      rollout.argoproj.io/revision: "2"
    creationTimestamp: "2021-07-16T10:15:58Z"
    generation: 2
    labels:
      rollout-type: Step
      rollouts-pod-template-hash: 59b469fdf7
      step-index: "2"
--- 中略 ---
    name: rollouts-demo-59b469fdf7-2-2
    namespace: kubota-test
    ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Rollout
      name: rollouts-demo
      uid: 5dc549be-f03e-4628-9e25-3182f7884180
    resourceVersion: "729720"
    selfLink: /apis/argoproj.io/v1alpha1/namespaces/kubota-test/analysisruns/rollouts-demo-59b469fdf7-2-2
    uid: b7932425-5502-4345-a603-94b269b0d1f0
  spec:
    metrics:
    - name: http_error_count
      provider:
        prometheus:
          address: http://istio-ingressgateway-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud
          query: |
            delta(http_requests_total{code="500",namespace="kubota-test",service="rollouts-demo-canary"}[5m])
      successCondition: result[0] <= 5
  status:
    message: metric "http_error_count" assessed Failed due to failed (1) > failureLimit (0)
    metricResults:
    - count: 1
      failed: 1
      measurements:
      - finishedAt: "2021-07-16T10:15:58Z"
        phase: Failed
        startedAt: "2021-07-16T10:15:58Z"
        value: '[16.311243770937153]'
      name: http_error_count
      phase: Failed
    phase: Failed
    startedAt: "2021-07-16T10:15:58Z"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
PS D:\git>

可以通过查询结果为 ‘[16.311243770937153]’ 来判断失败了。

image.png

请确认查询结果不超过5。

image.png

通过使用以下命令检查VirtualService的配置,可以自动确认稳定版本恢复到了100%。

PS D:\git> oc describe vs rollouts-demo-vsvc
名称:rollouts-demo-vsvc
命名空间:kubota-test
标签:<无>
注释:<无>
API版本:networking.istio.io/v1beta1
类型:VirtualService
元数据:
创建时间戳:2021-07-16T09:32:47Z
生成次数:7
管理字段:
API版本:networking.istio.io/v1alpha3
字段类型:FieldsV1
fieldsV1:
f:metadata:
f:annotations:
。:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
。:
f:gateways:
f:hosts:
管理器:kubectl-client-side-apply
操作:更新
时间:2021-07-16T09:32:47Z
API版本:networking.istio.io/v1alpha3
字段类型:FieldsV1
fieldsV1:
f:spec:
f:http:
管理器:rollouts-controller
操作:更新
时间:2021-07-16T10:24:58Z
资源版本:732846
自链接:/apis/networking.istio.io/v1beta1/namespaces/kubota-test/virtualservices/rollouts-demo-vsvc
UID:ac110ed1-dc46-4797-946f-6dad62d067e0
规范:
网关:
rollouts-demo-gateway
主机:
*
Http:
匹配:
Uri:
精确:/
Uri:
精确:/err
名称:primary
路由:
目的地:
主机:rollouts-demo-stable
权重:100
目的地:
主机:rollouts-demo-canary
权重:0
匹配:
Uri:
前缀:/api
名称:prometheus
路由:
目的地:
主机:prometheus
事件:<无>
PS D:\git>

image.png
--- 省略 ---
  spec:
    metrics:
    - name: http_error_count
      provider:
        prometheus:
          address: http://istio-ingressgateway-istio-system.dte-ocp46-8uvivd-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud
          query: |
            delta(http_requests_total{code="500",namespace="kubota-test",service="rollouts-demo-canary"}[5m])
      successCondition: result[0] <= 5
  status:
    metricResults:
    - count: 1
      measurements:
      - finishedAt: "2021-07-16T10:24:58Z"
        phase: Successful
        startedAt: "2021-07-16T10:24:58Z"
        value: '[0]'
      name: http_error_count
      phase: Successful
      successful: 1
    phase: Successful
    startedAt: "2021-07-16T10:24:58Z"
--- 省略 ---

通过查询结果是'[0]’,可以确认成功。

总结

通过验证本次文章中,我们发现通过将Argo Rollouts和OpenShift Service Mesh进行协作,可以通过与Service Mesh协同进行流量控制。
通过与Service Mesh协同,我们可以只获取对Canary的访问指标,从而可以根据更准确的信息进行分析。
此外,如果不使用Service Mesh,流量分配的方式将是将一个Pod分配给Canary,比例为总Pod数(5)的20%,这会导致副本数依赖于Rollout的步骤,这并不理想。
使用Service Mesh可以消除此类浪费。

我們將來的課題是,當Prometheus的查詢結果為空時,AnalysisRun將失敗,所以我們希望能夠找到一種不會失敗的方法,並進行調查。

广告
将在 10 秒后关闭
bannerAds