我要试用Metrics基于的发布支持工具iter8在Openshift上！
Iter8是一个工具，通过与Istio集成，支持逐步将新版本部署到生产环境中。它基于Prometheus指标进行条件设置，在每个设置的迭代结束时通过清除条件来逐步增加流量分配到新版本，从而实现逐步升级到新版本。如果在所有迭代中无法将流量完全切换到新版本，则可以轻松自动化旧版本的回滚，或在满足错误条件时中断和回滚发布。因此，结合使用Iter8和Istio可以更轻松地实施基于B/G或金丝雀发布。
前提
- Openshiftの実行環境はIBM DemoのLab環境
进行iter8安装
如果在OpenShift上进行安装，则需要使用OpenShift Service Mesh而不是istio（istio达不到要求）。
https://github.com/iter8-tools/docs/blob/v0.1.1/doc_files/platforms/redhat_openshift.md
如果要执行教程，则需要在Namespace和Project中添加bookinfo-iter8。
安装OpenShift Service Mesh。
安装Elasticsearch运算符
由于遵循了Update channel上的步骤选择最新的4.4版本进行安装时失败并处于待处理状态，所以我选择了4.3版本成功进行了安装。
安装Jaeger运算符
安装 Kiali 运算符。
按照步骤进行，没有什么特别的问题
安装红帽OpenShift服务网格运算符
部署红帽OpenShift服务网格的控制平面
从Web控制台部署控制平面
创建Red Hat OpenShift服务网格成员角色
在网页控制台上创建会员名单
apiVersion: maistra.io/v1
kind: ServiceMeshMemberRoll
metadata:
name: default
namespace: istio-system
spec:
members:
- bookinfo-iter8
创建Red Hat OpenShift服务网格的成员
只有在需要添加管理者以外的独立用户时，才会判断并跳过该步骤。
安装iter8。
-
- 以降ocコマンド、kubectlコマンドでクラスターとの接続が必要なのでOpenshiftへのログインを実施しておく
ログインコマンドはOpenshiftのコンソールから右上のユーザー名のドロップダウンからコピー出来る
安装iter8分析服务
被要求下载 Helm Chart 的 tar 文件，但是链接失效且在 Github 仓库中也不存在该 tar 文件。然而，Helm Chart 本身在以下的位置存在，因此我克隆并使用了它。
https://github.com/iter8-tools/iter8-analytics/tree/master/install/kubernetes/helm/iter8-analytics
由于该仓库中的 Helm 文件与手册内容不匹配，所以请按以下步骤进行操作以使其正常工作。
– 获取 Prometheus 的密码。
$ PROMETHEUS_PASSWORD=$(kubectl -n istio-system get secret htpasswd -o jsonpath='{.data.rawPassword}' | base64 --decode)
$ echo $PROMETHEUS_PASSWORD
p8aqeFudWKLkzYW3sBGOSo46b9+r8msPNyVG1RWfwBA5HYid74PNhdnjE9Dba3WkNt+BRUmvsSBqIjVZjrYHVfRI/VPHjgtaTyjnLK+OsT7h2ShXkOP7uakJM987zH9EQsrgd3pSYUadFyMoHUhGY7Yp7ok/jbG5xAPL4bSYJJxNX9yxRfoT3+Du2kIDQdb65Ik0+z/lavmymum6byzRLHI4MXJYvWtGjkdIkpQrP5I1IJ+aUur9VcV/MN+dN3TF//tWBkzWQroJnhtbXBJb+0YLxIc1EGQroYzUGPFgDTmlG+2sYhHO3PdNPT6Ge6PZFb4w3FfjdiAxfpQ5s0XS
- Helmに渡すその他の環境変数を設定
$ REPO=iter8/iter8-analytics
$ PROMETHEUS_SERVICE='https://prometheus.istio-system:9090'
$ PROMETHEUS_USERNAME='internal'
-
- Helmのテンプレートから生成されるyamlの内容を確認する
ConfigMapのconfig.yamlのmetricBackend配下の以下の項目が正しく設定されていること
項目名値url環境変数PROMETHEUS_SERVICEで設定した値(httpsであること)insecure_skip_verifytruepassword環境変数PROMETHEUS_PASSWORDで設定した値typebasicusername環境変数PROMETHEUS_USERNAMEで設定した値
请提供一个中国人的母语的的选项
helm template install/kubernetes/helm/iter8-analytics \
--name iter8-analytics \
--set image.repository=${REPO} \
--set image.tag=v0.1.1 \
--set metricsBackend.authentication.type=basic \
--set metricsBackend.authentication.username=${PROMETHEUS_USERNAME} \
--set metricsBackend.authentication.password=${PROMETHEUS_PASSWORD} \
--set metricsBackend.authentication.insecure_skip_verify=true \
--set metricsBackend.url=${PROMETHEUS_SERVICE} \
最终结果：
$ helm template install/kubernetes/helm/iter8-analytics \
> --name iter8-analytics \
> --set image.repository=${REPO} \
> --set image.tag=v0.1.1 \
> --set metricsBackend.authentication.type=basic \
> --set metricsBackend.authentication.username=${PROMETHEUS_USERNAME} \
> --set metricsBackend.authentication.password=${PROMETHEUS_PASSWORD} \
> --set metricsBackend.authentication.insecure_skip_verify=true \
> --set metricsBackend.url=${PROMETHEUS_SERVICE} \
>
---
# Source: iter8-analytics/templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: iter8
---
# Source: iter8-analytics/templates/config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: iter8-analytics
namespace: iter8
labels:
app: iter8-analytics
data:
config.yaml: |
port: 8080
metricsBackend:
url: https://prometheus.istio-system:9090
auth:
ca_file:
insecure_skip_verify: true
password: p8aqeFudWKLkzYW3sBGOSo46b9+r8msPNyVG1RWfwBA5HYid74PNhdnjE9Dba3WkNt+BRUmvsSBqIjVZjrYHVfRI/VPHjgtaTyjnLK+OsT7h2ShXkOP7uakJM987zH9EQsrgd3pSYUadFyMoHUhGY7Yp7ok/jbG5xAPL4bSYJJxNX9yxRfoT3+Du2kIDQdb65Ik0+z/lavmymum6byzRLHI4MXJYvWtGjkdIkpQrP5I1IJ+aUur9VcV/MN+dN3TF//tWBkzWQroJnhtbXBJb+0YLxIc1EGQroYzUGPFgDTmlG+2sYhHO3PdNPT6Ge6PZFb4w3FfjdiAxfpQ5s0XS
token:
type: basic
username: internal
---
# Source: iter8-analytics/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: iter8-analytics
namespace: iter8
labels:
app: iter8-analytics
spec:
selector:
app: iter8-analytics
type: ClusterIP
ports:
- port: 8080
targetPort: 8080
protocol: TCP
name: http
---
# Source: iter8-analytics/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: iter8-analytics
namespace: iter8
labels:
name: iter8-analytics
spec:
replicas: 1
selector:
matchLabels:
app: iter8-analytics
template:
metadata:
labels:
app: iter8-analytics
spec:
containers:
- name: iter8-analytics
image: "iter8/iter8-analytics:v0.1.1"
imagePullPolicy: Always
env:
- name: METRICS_BACKEND_CONFIGFILE
value: "config.yaml"
resources:
{}
volumeMounts:
- name: config-volume
mountPath: /python_code/src/config.yaml
subPath: config.yaml
volumes:
- name: config-volume
configMap:
name: iter8-analytics
- 問題なければ以下のコマンドでiter8 analyticsをインストール
命令:
helm template install/kubernetes/helm/iter8-analytics \
--name iter8-analytics \
--set image.repository=${REPO} \
--set image.tag=v0.1.1 \
--set metricsBackend.authentication.type=basic \
--set metricsBackend.authentication.username=${PROMETHEUS_USERNAME} \
--set metricsBackend.authentication.password=${PROMETHEUS_PASSWORD} \
--set metricsBackend.authentication.insecure_skip_verify=true \
--set metricsBackend.url=${PROMETHEUS_SERVICE} \
| kubectl -n iter8 apply -f -
结果：
$ helm template install/kubernetes/helm/iter8-analytics \
> --name iter8-analytics \
> --set image.repository=${REPO} \
> --set image.tag=v0.1.1 \
> --set metricsBackend.authentication.type=basic \
> --set metricsBackend.authentication.username=${PROMETHEUS_USERNAME} \
> --set metricsBackend.authentication.password=${PROMETHEUS_PASSWORD} \
> --set metricsBackend.authentication.insecure_skip_verify=true \
> --set metricsBackend.url=${PROMETHEUS_SERVICE} \
> | kubectl -n iter8 apply -f -
namespace/iter8 created
configmap/iter8-analytics created
service/iter8-analytics created
deployment.apps/iter8-analytics created
安装iter8控制器。
- iter8 controllerを以下のコマンドでインストール
$ kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/install/iter8-controller.yaml
namespace/iter8 unchanged
configmap/iter8config-metrics created
configmap/iter8config-notifiers created
serviceaccount/iter8-controller created
customresourcedefinition.apiextensions.k8s.io/experiments.iter8.tools created
clusterrole.rbac.authorization.k8s.io/iter8-controller-role created
clusterrolebinding.rbac.authorization.k8s.io/iter8-controller-rolebinding created
service/iter8-controller created
deployment.apps/iter8-controller created
验证安装
- インストール結果の確認
$ kubectl get pods -n iter8
NAME READY STATUS RESTARTS AGE
iter8-analytics-744489dcf-nb99p 1/1 Running 0 23m
iter8-controller-67dc866454-944pp 1/1 Running 0 23m
$ kubectl get svc -n iter8
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
iter8-analytics ClusterIP 172.21.94.199 <none> 8080/TCP 23m
iter8-controller ClusterIP 172.21.19.62 <none> 443/TCP 23m
进行教程
Part 1: 成功的金丝雀发布：reviews-v2 到 reviews-v3
部署Bookinfo应用程序
- Namespaceにistio-injectionの設定
$ kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/namespace.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
namespace/bookinfo-iter8 configured
- Bookinfoのインストール
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/bookinfo-tutorial.yaml
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v2 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created
- Podの状態を確認
$ kubectl get pods -n bookinfo-iter8
NAME READY STATUS RESTARTS AGE
details-v1-5b59996c6d-nsm9m 2/2 Running 0 116s
productpage-v1-697bf98699-qvvdw 2/2 Running 0 107s
ratings-v1-74b7bc7d7b-2rmhn 2/2 Running 0 112s
reviews-v2-784cd5b59b-n2bvh 2/2 Running 0 109s
- Gatewayの設定
由于iter8的手册设置存在问题，无法成功从外部访问。因此，根据Openshift Service Mesh手册中的步骤进行设置。
iter8的设置似乎仅限于对hosts进行限定，只有当HTTP请求标头设置为特定host时才会响应。然而，这个设置无法正常工作，导致无法访问bookinfo。所以我们将按照Openshift Service Mesh手册中未对hosts进行限定的步骤来创建Gateway。
$ oc apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/Maistra/istio/maistra-1.1/samples/bookinfo/networking/bookinfo-gateway.yaml
gateway.networking.istio.io/bookinfo-gateway created
virtualservice.networking.istio.io/bookinfo created
- 確認
$ kubectl get gateway -n bookinfo-iter8
NAME AGE
bookinfo-gateway 48s
$ kubectl get vs -n bookinfo-iter8
NAME GATEWAYS HOSTS AGE
bookinfo [bookinfo-gateway] [*] 59s
- GatewayのURL確認
在中国，也可以按照 Openshift Service Mesh 的手册中所提及的步骤进行确认。
$ export GATEWAY_URL=$(oc -n istio-system get route istio-ingressgateway -o jsonpath='{.spec.host}')
$ echo $GATEWAY_URL
istio-ingressgateway-istio-system.dte-ocp4-frbl36-915b3b336cabec458a7c7ec2aa7c625f-0000.us-south.containers.appdomain.cloud
访问Bookinfo应用程序
# curl -Is "http://${GATEWAY_URL}/productpage"
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 5183
server: istio-envoy
date: Wed, 24 Jun 2020 05:14:07 GMT
x-envoy-upstream-service-time: 89
Set-Cookie: cd10b69e39387eb7ec9ac241201ab1ab=ac85f91a351fed838e2d7b9c262839e2; path=/; HttpOnly
Cache-control: private
3. 对应用程序进行负载生成
指令：
watch -n 0.1 curl -Is http://${GATEWAY_URL}/productpage
结果为：
Every 0.1s: curl -Is http://istio-ingressgateway-istio-system.dte-ocp4-frbl36... 4acc5cf83fe6: Wed Jun 24 05:15:46 2020
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 5183
server: istio-envoy
date: Wed, 24 Jun 2020 05:15:46 GMT
x-envoy-upstream-service-time: 49
Set-Cookie: cd10b69e39387eb7ec9ac241201ab1ab=ac85f91a351fed838e2d7b9c262839e2; path=/; HttpOnly
Cache-control: private
如果屏幕上显示此画面，则每0.1秒会对Bookinfo进行访问并增加负载。
设置评论服务的金丝雀发布
- v2 から v3へのカナリア・ロールアウトを作成
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/canary_reviews-v2_to_reviews-v3.yaml
experiment.iter8.tools/reviews-v3-rollout created
-
- 確認
この段階ではv3のサービスが無いのでMissing Candidateになる
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Pause TargetsNotFound: Missing Candidate reviews-v2 100 reviews-v3 0
- v3のサービスを作成するとロールアウトが開始される
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/reviews-v3.yaml
deployment.apps/reviews-v3 created
-
- 30秒ごとにIterationが実行されて、Iterationの最後に成功基準を満たしていれば20%ずつv3にトラフィックが切り替わる。
-
- 成功基準は平均レイテンシーが0.2秒以下
- 最後のIterationの終わりに成功基準が満たされている場合にv2(0%)からv3(100%)に移行する
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Progressing IterationUpdate: Iteration 1 Started reviews-v2 80 reviews-v3 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Progressing IterationUpdate: Iteration 2 Started reviews-v2 80 reviews-v3 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Progressing IterationUpdate: Iteration 3 Started reviews-v2 60 reviews-v3 40
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Progressing IterationUpdate: Iteration 4 Started reviews-v2 40 reviews-v3 60
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Progressing IterationUpdate: Iteration 5 Started reviews-v2 20 reviews-v3 80
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Progressing IterationUpdate: Iteration 6 Started reviews-v2 20 reviews-v3 80
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Progressing IterationUpdate: Iteration 7 Started reviews-v2 20 reviews-v3 80
AzureAD+YASUYUKIKUBOTA@DESKTOP-76V3US4 MINGW64 /d/git/iter8-analytics (master)
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
6. 检查Grafana仪表板
很遗憾，尽管尝试访问Grafana的仪表板，但无法找到仪表板并显示出来。
第二部分：高延迟金丝雀发布：reviews-v3到reviews-v4
1. 金丝雀滚动配置
- v3 から v4へのカナリア・ロールアウトを作成
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/canary_reviews-v3_to_reviews-v4.yaml
experiment.iter8.tools/reviews-v4-rollout created
- 確認
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Pause TargetsNotFound: Missing Candidate reviews-v3 100 reviews-v4 0
- v4のサービスを作成するとロールアウトが開始される
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/reviews-v4.yaml
deployment.apps/reviews-v4 created
-
- 各Iterationの終わりに成功基準を達成しないために初回の80%から以降することなく最大6回のIterationを終え失敗する
Grafanaのダッシュボードが見られないので微妙だがマニュアルを見る限りv4は平均レイテンシーが5秒に張り付いているので5秒立ったら応答を返すアプリなのだろう
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Progressing IterationUpdate: Iteration 1 Started reviews-v3 80 reviews-v4 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Progressing IterationUpdate: Iteration 1 Started reviews-v3 80 reviews-v4 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Progressing IterationUpdate: Iteration 3 Started reviews-v3 80 reviews-v4 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Progressing IterationUpdate: Iteration 5 Started reviews-v3 80 reviews-v4 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Progressing IterationUpdate: Iteration 6 Started reviews-v3 80 reviews-v4 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Completed ExperimentFailed: Not All Success Criteria Were Met reviews-v3 100 reviews-v4 0
部分三：引发错误的金丝雀发布：reviews-v3 至 reviews-v5
1. 金丝雀推出配置
- v3 から v5へのカナリア・ロールアウトを作成
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/canary_reviews-v3_to_reviews-v5.yaml
experiment.iter8.tools/reviews-v5-rollout created
- v5のサービスを作成するとロールアウトが開始される
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/reviews-v5.yaml
deployment.apps/reviews-v5 created
-
- 今回は成功基準にエラー率の設定が追加されているため途中で終了する
-
- エラー率は2%以内で途中でも終了する設定になっている
マニュアルを見る限りv5は常にエラーを返すアプリなのだろう
AzureAD+YASUYUKIKUBOTA@DESKTOP-76V3US4 MINGW64 /d/git/iter8-analytics (master)
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Completed ExperimentFailed: Not All Success Criteria Were Met reviews-v3 100 reviews-v4 0
reviews-v5-rollout Progressing IterationUpdate: Iteration 1 Started reviews-v3 80 reviews-v5 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Completed ExperimentFailed: Not All Success Criteria Were Met reviews-v3 100 reviews-v4 0
reviews-v5-rollout Progressing IterationUpdate: Iteration 2 Started reviews-v3 80 reviews-v5 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Completed ExperimentFailed: Not All Success Criteria Were Met reviews-v3 100 reviews-v4 0
reviews-v5-rollout Completed ExperimentFailed: Abort reviews-v3 100 reviews-v5 0
第四部分：使用自定义指标
- 現在使用できるメトリックの確認
$ kubectl get configmap iter8config-metrics -n iter8 -oyaml
apiVersion: v1
data:
metrics: |-
- name: iter8_latency
is_counter: False
absent_value: "None"
sample_size_query_template: iter8_sample_size
- name: iter8_error_count
is_counter: True
absent_value: 0
sample_size_query_template: iter8_sample_size
- name: iter8_error_rate
is_counter: False
absent_value: 0
sample_size_query_template: iter8_sample_size
query_templates: |-
iter8_sample_size: "sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)"
iter8_latency: "(sum(increase(istio_request_duration_seconds_sum{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)) / (sum(increase(istio_request_duration_seconds_count{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels))"
iter8_error_count: "sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',response_code=~'5..',reporter='source'}[$interval]$offset_str)) by ($entity_labels)"
iter8_error_rate: "sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',response_code=~'5..',reporter='source'}[$interval]$offset_str)) by ($entity_labels) / sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)"
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"metrics":"- name: iter8_latency\n is_counter: False\n absent_value: \"None\"\n sample_size_query_template: iter8_sample_size\n- name: iter8_error_count\n is_counter: True\n absent_value: 0\n sample_size_query_template: iter8_sample_size\n- name: iter8_error_rate\n is_counter: False\n absent_value: 0\n sample_size_query_template: iter8_sample_size","query_templates":"iter8_sample_size: \"sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)\"\niter8_latency: \"(sum(increase(istio_request_duration_seconds_sum{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)) / (sum(increase(istio_request_duration_seconds_count{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels))\"\niter8_error_count: \"sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',response_code=~'5..',reporter='source'}[$interval]$offset_str)) by ($entity_labels)\"\niter8_error_rate: \"sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',response_code=~'5..',reporter='source'}[$interval]$offset_str)) by ($entity_labels) / sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)\""},"kind":"ConfigMap","metadata":{"annotations":{},"name":"iter8config-metrics","namespace":"iter8"}}
creationTimestamp: "2020-06-24T04:28:52Z"
name: iter8config-metrics
namespace: iter8
resourceVersion: "197349"
selfLink: /api/v1/namespaces/iter8/configmaps/iter8config-metrics
uid: 09b64765-5f89-43bb-b490-56aabc310d5a
- カスタム・メトリックの追加
$ kubectl apply -n iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/iter8_metrics_extended.yaml
configmap/iter8config-metrics configured
-
- 確認
メトリック名にiter8_90_perc_latencyが追加された
$ kubectl get configmap iter8config-metrics -n iter8 -oyaml
apiVersion: v1
data:
metrics: |-
- name: iter8_latency
is_counter: False
absent_value: "None"
sample_size_query_template: iter8_sample_size
- name: iter8_error_count
is_counter: True
sample_size_query_template: iter8_sample_size
- name: iter8_error_rate
is_counter: False
sample_size_query_template: iter8_sample_size
- name: iter8_90_perc_latency
is_counter: False
absent_value: "None"
sample_size_query_template: iter8_sample_size
query_templates: |-
iter8_sample_size: "sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)"
iter8_latency: "(sum(increase(istio_request_duration_seconds_sum{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)) / (sum(increase(istio_request_duration_seconds_count{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels))"
iter8_error_count: "sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',response_code=~'5..',reporter='source'}[$interval]$offset_str)) by ($entity_labels)"
iter8_error_rate: "sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',response_code=~'5..',reporter='source'}[$interval]$offset_str)) by ($entity_labels) / sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)"
iter8_90_perc_latency: "histogram_quantile(0.90, sum(increase(istio_request_duration_seconds_bucket{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels, le))"
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"metrics":"- name: iter8_latency\n is_counter: False\n absent_value: \"None\"\n sample_size_query_template: iter8_sample_size\n- name: iter8_error_count\n is_counter: True\n sample_size_query_template: iter8_sample_size\n- name: iter8_error_rate\n is_counter: False\n sample_size_query_template: iter8_sample_size\n- name: iter8_90_perc_latency\n is_counter: False\n absent_value: \"None\"\n sample_size_query_template: iter8_sample_size","query_templates":"iter8_sample_size: \"sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)\"\niter8_latency: \"(sum(increase(istio_request_duration_seconds_sum{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)) / (sum(increase(istio_request_duration_seconds_count{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels))\"\niter8_error_count: \"sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',response_code=~'5..',reporter='source'}[$interval]$offset_str)) by ($entity_labels)\"\niter8_error_rate: \"sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',response_code=~'5..',reporter='source'}[$interval]$offset_str)) by ($entity_labels) / sum(increase(istio_requests_total{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels)\"\niter8_90_perc_latency: \"histogram_quantile(0.90, sum(increase(istio_request_duration_seconds_bucket{source_workload_namespace!='knative-serving',reporter='source'}[$interval]$offset_str)) by ($entity_labels, le))\""},"kind":"ConfigMap","metadata":{"annotations":{},"name":"iter8config-metrics","namespace":"iter8"}}
creationTimestamp: "2020-06-24T04:28:52Z"
name: iter8config-metrics
namespace: iter8
resourceVersion: "251241"
selfLink: /api/v1/namespaces/iter8/configmaps/iter8config-metrics
uid: 09b64765-5f89-43bb-b490-56aabc310d5a
1. 金丝雀灰度发布配置
- v3 から v6へのカナリア・ロールアウトを作成
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/canary_reviews-v3_to_reviews-v6.yaml
experiment.iter8.tools/reviews-v6-rollout created
- v6のサービスを作成するとロールアウトが開始される
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/reviews-v6.yaml
deployment.apps/reviews-v6 created
- カスタム・メトリックでも問題なくv6へ移行した
$ kubectl get experiments -n bookinfo-iter8 NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Completed ExperimentFailed: Not All Success Criteria Were Met reviews-v3 100 reviews-v4 0
reviews-v5-rollout Completed ExperimentFailed: Abort reviews-v3 100 reviews-v5 0
reviews-v6-rollout Progressing IterationUpdate: Iteration 1 Started reviews-v3 80 reviews-v6 20
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Completed ExperimentFailed: Not All Success Criteria Were Met reviews-v3 100 reviews-v4 0
reviews-v5-rollout Completed ExperimentFailed: Abort reviews-v3 100 reviews-v5 0
reviews-v6-rollout Progressing IterationUpdate: Iteration 3 Started reviews-v3 60 reviews-v6 40
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Completed ExperimentFailed: Not All Success Criteria Were Met reviews-v3 100 reviews-v4 0
reviews-v5-rollout Completed ExperimentFailed: Abort reviews-v3 100 reviews-v5 0
reviews-v6-rollout Progressing IterationUpdate: Iteration 4 Started reviews-v3 40 reviews-v6 60
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Completed ExperimentFailed: Not All Success Criteria Were Met reviews-v3 100 reviews-v4 0
reviews-v5-rollout Completed ExperimentFailed: Abort reviews-v3 100 reviews-v5 0
reviews-v6-rollout Progressing IterationUpdate: Iteration 5 Started reviews-v3 20 reviews-v6 80
$ kubectl get experiments -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
reviews-v3-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v2 0 reviews-v3 100
reviews-v4-rollout Completed ExperimentFailed: Not All Success Criteria Were Met reviews-v3 100 reviews-v4 0
reviews-v5-rollout Completed ExperimentFailed: Abort reviews-v3 100 reviews-v5 0
reviews-v6-rollout Completed ExperimentSucceeded: All Success Criteria Were Met reviews-v3 0 reviews-v6 100
第五部分：用户面向服务的金丝雀发布
为产品页面服务配置金丝雀式发布
只需一个选项：对于外部公开的服务，可以通过指定VirtualService的名称，iter8会自动为VirtualService设置流量切换。
- productpageのv1 から v2へのカナリア・ロールアウトを作成する
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/canary_productpage-v1_to_productpage-v2.yaml
experiment.iter8.tools/productpage-v2-rollout created
- 確認
$ kubectl get experiment productpage-v2-rollout -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
productpage-v2-rollout Pause TargetsNotFound: Missing Candidate productpage-v1 100 productpage-v2 0
- productpageのv2のサービスを作成するとロールアウトが開始される
$ kubectl apply -n bookinfo-iter8 -f https://raw.githubusercontent.com/iter8-tools/iter8-controller/v0.1.1/doc/tutorials/istio/bookinfo/productpage-v2.yaml
deployment.apps/productpage-v2 created
- ロールアウトが開始される
$ kubectl get experiment productpage-v2-rollout -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
productpage-v2-rollout Progressing IterationUpdate: Iteration 1 Started productpage-v1 80 productpage-v2 20
-
- 開始されるとVirtualServiceの内容がiter8により書き換えられていることが分かる
baselineが80%、candidateが20%になっている
$ kubectl get vs bookinfo -n bookinfo-iter8 -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"bookinfo","namespace":"bookinfo-iter8"},"spec":{"gateways":["bookinfo-gateway"],"hosts":["*"],"http":[{"match":[{"uri":{"exact":"/productpage"}},{"uri":{"prefix":"/static"}},{"uri":{"exact":"/login"}},{"uri":{"exact":"/logout"}},{"uri":{"prefix":"/api/v1/products"}}],"route":[{"destination":{"host":"productpage","port":{"number":9080}}}]}]}}
creationTimestamp: "2020-06-24T05:01:54Z"
generation: 4
labels:
iter8-tools/experiment: productpage-v2-rollout
iter8-tools/host: productpage
iter8-tools/role: progressing
name: bookinfo
namespace: bookinfo-iter8
resourceVersion: "261486"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/bookinfo-iter8/virtualservices/bookinfo
uid: 6b9d2df9-a08b-464f-a6a5-5d462bb963ef
spec:
gateways:
- bookinfo-gateway
hosts:
- '*'
http:
- match:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
route:
- destination:
host: productpage
port:
number: 9080
subset: baseline
weight: 80
- destination:
host: productpage
port:
number: 9080
subset: candidate
weight: 20
- ロールアウトの状態を再び確認すると40%/60%に
$ kubectl get experiment productpage-v2-rollout -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
productpage-v2-rollout Progressing IterationUpdate: Iteration 5 Started productpage-v1 40 productpage-v2 60
- VirtualSeviceも40%/60%に
$ kubectl get vs bookinfo -n bookinfo-iter8 -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"bookinfo","namespace":"bookinfo-iter8"},"spec":{"gateways":["bookinfo-gateway"],"hosts":["*"],"http":[{"match":[{"uri":{"exact":"/productpage"}},{"uri":{"prefix":"/static"}},{"uri":{"exact":"/login"}},{"uri":{"exact":"/logout"}},{"uri":{"prefix":"/api/v1/products"}}],"route":[{"destination":{"host":"productpage","port":{"number":9080}}}]}]}}
creationTimestamp: "2020-06-24T05:01:54Z"
generation: 6
labels:
iter8-tools/experiment: productpage-v2-rollout
iter8-tools/host: productpage
iter8-tools/role: progressing
name: bookinfo
namespace: bookinfo-iter8
resourceVersion: "262507"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/bookinfo-iter8/virtualservices/bookinfo
uid: 6b9d2df9-a08b-464f-a6a5-5d462bb963ef
spec:
gateways:
- bookinfo-gateway
hosts:
- '*'
http:
- match:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
route:
- destination:
host: productpage
port:
number: 9080
subset: baseline
weight: 40
- destination:
host: productpage
port:
number: 9080
subset: candidate
weight: 60
- ロールアウト完了
$ kubectl get experiment productpage-v2-rollout -n bookinfo-iter8
NAME PHASE STATUS BASELINE PERCENTAGE CANDIDATE PERCENTAGE
productpage-v2-rollout Completed ExperimentSucceeded: All Success Criteria Were Met productpage-v1 0 productpage-v2 100
- VirtualSeviceはもとの設定に戻る
$ kubectl get vs bookinfo -n bookinfo-iter8 -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"bookinfo","namespace":"bookinfo-iter8"},"spec":{"gateways":["bookinfo-gateway"],"hosts":["*"],"http":[{"match":[{"uri":{"exact":"/productpage"}},{"uri":{"prefix":"/static"}},{"uri":{"exact":"/login"}},{"uri":{"exact":"/logout"}},{"uri":{"prefix":"/api/v1/products"}}],"route":[{"destination":{"host":"productpage","port":{"number":9080}}}]}]}}
creationTimestamp: "2020-06-24T05:01:54Z"
generation: 8
labels:
iter8-tools/host: productpage
iter8-tools/role: stable
name: bookinfo
namespace: bookinfo-iter8
resourceVersion: "263010"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/bookinfo-iter8/virtualservices/bookinfo
uid: 6b9d2df9-a08b-464f-a6a5-5d462bb963ef
spec:
gateways:
- bookinfo-gateway
hosts:
- '*'
http:
- match:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
route:
- destination:
host: productpage
port:
number: 9080
subset: stable
weight: 100
所感的意思是对某事或某人产生的感受或印象。
-
- Good
アプリケーションの非機能要件に関する確認には使えそう
ロールアウトを途中で失敗にすることでロールバックすることも出来る
今回のチュートリアルでは単に同一のリクエストで負荷をかけていたが、それに加えて機能テストも実行するようにすれば、機能テスト＋非機能テストを同時に実施するパイプラインが簡単にできそう
Bad
ROKSの環境だとGrafanaのダッシュボードが連携ができなかった
別の環境なら連携できるのかは試していないので不明
