検証 Kubernetes 的 StatefulSet 时的备忘录
验证和确认管理有状态容器的StatefulSet行为时的笔记。
有状态副本集
使用有状态集合（Stateful Sets）部署Cassandra
环境
kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-28T20:03:09Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.5-eks-6bad6d", GitCommit:"6bad6d9c768dc0864dab48a11653aa53b5a47043", GitTreeState:"clean", BuildDate:"2018-12-06T23:13:14Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
尝试部署具有Stateful Sets的Cassandra教程示例。
创建一个无头服务的Cassandra
首先创建一个服务。
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
clusterIP: None
ports:
- port: 9042
selector:
app: cassandra
# Service を作成
$kubectl create -f https://k8s.io/examples/application/cassandra/cassandra-service.yaml
service "cassandra" created
service "cassandra" created
# 作成を確認
$kubectl get svc cassandra
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cassandra ClusterIP None <none> 9042/TCP 40s
使用StatefulSet创建Cassandra环/验证Cassandra的StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cassandra
labels:
app: cassandra
spec:
serviceName: cassandra
replicas: 3
selector:
matchLabels:
app: cassandra
template:
metadata:
labels:
app: cassandra
spec:
terminationGracePeriodSeconds: 1800
containers:
- name: cassandra
image: gcr.io/google-samples/cassandra:v13
imagePullPolicy: Always
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7001
name: tls-intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
resources:
limits:
cpu: "500m"
memory: 1Gi
requests:
cpu: "500m"
memory: 1Gi
securityContext:
capabilities:
add:
- IPC_LOCK
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- nodetool drain
env:
- name: MAX_HEAP_SIZE
value: 512M
- name: HEAP_NEWSIZE
value: 100M
- name: CASSANDRA_SEEDS
value: "cassandra-0.cassandra.default.svc.cluster.local"
- name: CASSANDRA_CLUSTER_NAME
value: "K8Demo"
- name: CASSANDRA_DC
value: "DC1-K8Demo"
- name: CASSANDRA_RACK
value: "Rack1-K8Demo"
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
initialDelaySeconds: 15
timeoutSeconds: 5
# These volume mounts are persistent. They are like inline claims,
# but not exactly because the names need to match exactly one of
# the stateful pod volumes.
volumeMounts:
- name: cassandra-data
mountPath: /cassandra_data
# These are converted to volume claims by the controller
# and mounted at the paths mentioned above.
# do not use these in production until ssd GCEPersistentDisk or other ssd pd
volumeClaimTemplates:
- metadata:
name: cassandra-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: fast
resources:
requests:
storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
type: pd-ssd
储物柜 这可能不起作用。
虽说这样，还是试试看吧。
# マニフェストファイル適用.statefull set を storageclass を作成
$kubectl create -f https://k8s.io/examples/application/cassandra/cassandra-statefulset.yaml
statefulset.apps "cassandra" created
storageclass.storage.k8s.io "fast" created
# 一つのみ起動
$kubectl get statefulset cassandra
NAME DESIRED CURRENT AGE
cassandra 3 1 3m
# Pod は pending になっている
$ kubectl get pods -l="app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 0/1 Pending 0 4m
# やはり Volume の箇所でエラー
$ kubectl describe pod cassandra-0
(一部略)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 1s (x94 over 5m) default-scheduler pod has unbound PersistentVolumeClaims (repeated 3 times)
# pvc(PersistentVolumeClaim) を確認.エラー
$kubectl describe pvc cassandra-data-cassandra-0
Name: cassandra-data-cassandra-0
Namespace: default
StorageClass: fast
Status: Pending
Volume:
Labels: app=cassandra
Annotations: volume.beta.kubernetes.io/storage-provisioner=k8s.io/minikube-hostpath
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 57s (x26 over 6m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
还是失败了，无法创建卷。
先删除，然后重新创建。
# 一旦削除
$ kubectl delete -f https://k8s.io/examples/application/cassandra/cassandra-statefulset.yaml
statefulset.apps "cassandra" deleted
storageclass.storage.k8s.io "fast" deleted
在 EKS 中默认创建了使用 EBS 的 StorageClass，这是在 Manifest 文件中定义的。
$kubectl get storageclass gp2
NAME PROVISIONER AGE
gp2 (default) kubernetes.io/aws-ebs 6d
因此，需要修改清单文件以利用此功能。
删除存储类的定义，将使用的存储类更改为已存在的 gp2 类型。
$diff cassandra-statefulset.yaml cassandra-statefulset.yaml.1
89c89
< storageClassName: gp2
---
> storageClassName: fast
92a93,100
> ---
> kind: StorageClass
> apiVersion: storage.k8s.io/v1
> metadata:
> name: fast
> provisioner: k8s.io/minikube-hostpath
> parameters:
> type: pd-ssd
重新执行。
$kubectl apply -f cassandra-statefulset.yaml
嗯…同样的错误…
看起来之前创建失败的 pvc 仍然存在，并且尝试使用它会导致相同的错误。
根据以下进行删除。
$ kubectl delete pvc cassandra-data-cassandra-0
经过那之后，重新申请一次就成功了。
# 要求した3つの Pod が起動している
$kubectl get statefulset
NAME DESIRED CURRENT AGE
cassandra 3 3 4m
# 3つの Pod が作成されている
$ kubectl get pods -l="app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 5m
cassandra-1 1/1 Running 0 4m
cassandra-2 1/1 Running 0 3m
# pvc も3つ起動
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cassandra-data-cassandra-0 Bound pvc-9c0a9803-0964-11e9-b090-0ae6cc179478 1Gi RWO gp2 6m
cassandra-data-cassandra-1 Bound pvc-bd510f3a-0964-11e9-b090-0ae6cc179478 1Gi RWO gp2 5m
cassandra-data-cassandra-2 Bound pvc-ee1121f4-0964-11e9-b090-0ae6cc179478 1Gi RWO gp2 4m
修改Cassandra StatefulSet
修改已创建的StatefulSet。
将replicas字段从”3″修改为”4″。
# edit コマンドで編集.通常であればマニュフェストファイルを変更する
$kubectl edit statefulset cassandra
statefulset.apps "cassandra" edited
# 変更によって Pod 数が 3 から 4 になった
$ kubectl get statefulset
NAME DESIRED CURRENT AGE
cassandra 4 4 12m
# 新しい Pod cassandra-3 が出来た
$ kubectl get pods -l="app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 12m
cassandra-1 1/1 Running 0 11m
cassandra-2 1/1 Running 0 10m
cassandra-3 0/1 Running 0 43s
如前所述，StatefulSet 中的 Pod 名被命名为“(StatefulSet 名)-N”的形式，当添加一个 Pod 后，将创建 N+1 个 Pod 名。
不妨试试缩小规模看看。
# replica 数を 4 から 3 にする
$$kubectl edit statefulset cassandra
statefulset.apps "cassandra" edited
# 一番新しい Pod が削除される
$ kubectl get pods -l="app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 16m
cassandra-1 1/1 Running 0 15m
cassandra-2 1/1 Running 0 14m
此外，即使在上述情况下，pvc 也没有被删除，因此如果不需要的话就要进行删除。
$kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cassandra-data-cassandra-0 Bound pvc-9c0a9803-0964-11e9-b090-0ae6cc179478 1Gi RWO gp2 22m
cassandra-data-cassandra-1 Bound pvc-bd510f3a-0964-11e9-b090-0ae6cc179478 1Gi RWO gp2 21m
cassandra-data-cassandra-2 Bound pvc-ee1121f4-0964-11e9-b090-0ae6cc179478 1Gi RWO gp2 19m
cassandra-data-cassandra-3 Bound pvc-49d749e9-0966-11e9-b090-0ae6cc179478 1Gi RWO gp2 10m
$kubectl delete pvc cassandra-data-cassandra-3
persistentvolumeclaim "cassandra-data-cassandra-3" deleted
确认当Pod死机时的行为
在工作节点突然宕机或其他意外情况下，验证当 Pod 发生意外终止时的行为。
为了进行验证，预先将测试文件创建到挂载在即将被删除的 Pod 上的卷中，然后尝试删除该 Pod。
# volume 配下にテストファイルを作成
$kubectl exec -it cassandra-0 -- touch /cassandra_data/test.txt
# 存在確認
$ kubectl exec -it cassandra-0 -- ls /cassandra_data/
commitlog data hints lost+found saved_caches test.txt
# Pod を明示的に削除
$kubectl delete pod cassandra-0
pod "cassandra-0" deleted
# すぐに同じ名前で Pod が作れる
$kubectl get pods -l="app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 0/1 ContainerCreating 0 4s
cassandra-1 1/1 Running 0 29m
cassandra-2 1/1 Running 0 27m
# 同じ Volume を使っているので先程作成した test.txt が存在する
$kubectl exec -it cassandra-0 -- ls /cassandra_data/
commitlog data hints lost+found saved_caches test.txt
# pvc は同じ
$ kubectl describe pod cassandra-0 |grep Claim
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: cassandra-data-cassandra-0
# pvc は再作成などされておらず、同じ Volume がマウントされている
$kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cassandra-data-cassandra-0 Bound pvc-9c0a9803-0964-11e9-b090-0ae6cc179478 1Gi RWO gp2 33m
cassandra-data-cassandra-1 Bound pvc-bd510f3a-0964-11e9-b090-0ae6cc179478 1Gi RWO gp2 32m
cassandra-data-cassandra-2 Bound pvc-ee1121f4-0964-11e9-b090-0ae6cc179478 1Gi RWO gp2 31m