我尝试使用Kubernetes构建了一个Redis集群
简而言之
我尝试使用Kubernetes构建Redis集群的备忘录。
基本上我們直接使用了這個：
https://github.com/kelseyhightower/kubernetes-redis-cluster
-
- Redis server version、3.2.0
-
- データの永続化はしない
- redisクラスタ基本master×3にレプリカ用 slave×3
redis (loadbalancer)
|
|- [ redis-1(master) ] - [redis-4(slave)]
|- [ redis-2(master) ] - [redis-5(slave)]
|- [ redis-3(master) ] - [redis-6(slave)]
maxmemory-policy は、デフォルトvolatile-lru
`volatile-lru` remove the key with an expire set using an LRU algorithm
在保存会话时，预设设置有效期限。
因为在redis-1到redis6中使用了replicaset来创建pod，所以可以通过更改配置来启动多个pod。
可以逐个pod进行以下结构和确认。
儿童
建立
- kubernetesでreplicasetとserviceでpodとloadbalancerを起動する
$ kubectl create configmap redis-conf --from-file=redis.conf
$ kubectl create -f replicasets
$ kubectl create -f services
- podをredis clusterに繋ぐ
启动用于执行redis集群创建命令的pod
$ kubectl run -i --tty ubuntu --image=ubuntu --restart=Never /bin/bash
安装所需的库文件
$ apt-get update
$ apt-get install ruby vim wget redis-tools
$ gem install redis
$ wget http://download.redis.io/redis-stable/src/redis-trib.rb
$ chmod 755 redis-trib.rb
创建一个Redis集群
$ ./redis-trib.rb create --replicas 1 \
10.107.255.1:6379 \
10.107.255.2:6379 \
10.107.255.3:6379 \
10.107.255.4:6379 \
10.107.255.5:6379 \
10.107.255.6:6379
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.107.255.1:6379
10.107.255.2:6379
10.107.255.3:6379
Adding replica 10.107.255.4:6379 to 10.107.255.1:6379
Adding replica 10.107.255.5:6379 to 10.107.255.2:6379
Adding replica 10.107.255.6:6379 to 10.107.255.3:6379
M: d4dfb6366289f581e395428811af4bb6e1f07e2d 10.107.255.1:6379 10.104.1.34
slots:0-5460 (5461 slots) master
M: 4b23c907ef408e5efd6225fffd11a6cdff5de337 10.107.255.2:6379 10.104.1.35
slots:5461-10922 (5462 slots) master
M: 189ee8e87202326675adef4f507a80b5b5873369 10.107.255.3:6379 10.104.1.36
slots:10923-16383 (5461 slots) master
S: 37b2b6b625d8bfba7c8997abfe894df099761937 10.107.255.4:6379 10.104.1.37
replicates d4dfb6366289f581e395428811af4bb6e1f07e2d
S: f2229c73056a819f3cee6521d231e65ce1f4c180 10.107.255.5:6379 10.104.0.14
replicates 4b23c907ef408e5efd6225fffd11a6cdff5de337
S: ad1aa4c73849622068c96976998f778673a2ce4c 10.107.255.6:6379 10.104.1.38
replicates 189ee8e87202326675adef4f507a80b5b5873369
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 10.107.255.1:6379)
M: d4dfb6366289f581e395428811af4bb6e1f07e2d 10.107.255.1:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 189ee8e87202326675adef4f507a80b5b5873369 10.104.1.36:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 4b23c907ef408e5efd6225fffd11a6cdff5de337 10.104.1.35:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: ad1aa4c73849622068c96976998f778673a2ce4c 10.104.1.38:6379
slots: (0 slots) slave
replicates 189ee8e87202326675adef4f507a80b5b5873369
S: f2229c73056a819f3cee6521d231e65ce1f4c180 10.104.0.14:6379
slots: (0 slots) slave
replicates 4b23c907ef408e5efd6225fffd11a6cdff5de337
S: 37b2b6b625d8bfba7c8997abfe894df099761937 10.104.1.37:6379
slots: (0 slots) slave
replicates d4dfb6366289f581e395428811af4bb6e1f07e2d
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
核實
确认在关闭docker后，仍能从其他redis节点返回值。
- masterを1pod止めて、自動起動するか確認する
POD列表
$ kubectl get pods --show-all
NAME READY STATUS RESTARTS AGE
redis-1-2pg4v 1/1 Running 0 56m
redis-2-fbzks 1/1 Running 0 56m
redis-3-4fs93 1/1 Running 0 56m
redis-4-pjxt1 1/1 Running 0 56m
redis-5-bt1wm 1/1 Running 0 56m
redis-6-qpvvl 1/1 Running 0 56m
ubuntu 1/1 Running 0 20m
停止redis-1(master)的Pod。
$ kubectl stop pod redis-1-2pg4v
Command "stop" is deprecated, use "delete" instead.
pod "redis-1-2pg4v" deleted
全部Pod的列表
$ kubectl get pods --show-all
NAME READY STATUS RESTARTS AGE
redis-1-b7blk 1/1 Running 0 3m ← redis-1(master)のpodが新たに起動された
redis-2-fbzks 1/1 Running 0 1h
redis-3-4fs93 1/1 Running 0 1h
redis-4-pjxt1 1/1 Running 0 1h
redis-5-bt1wm 1/1 Running 0 1h
redis-6-qpvvl 1/1 Running 0 1h
ubuntu 1/1 Running 0 24m
- dockerを止めても正常にデータ取得できるか
a12948@gke-satsuma-redis-default-pool-f99bcf1a-cft1 ~ $ docker ps -a | grep redis-server
4b1cac0124e9 redis:3.2.0-alpine "redis-server /etc/re" 8 minutes ago Up 8 minutes k8s_redis.638d2d73_redis-1-b7blk_default_2df04edf-e91c-11e6-999b-42010af00135_72d2a8a6
6fef91ca8ca5 redis:3.2.0-alpine "redis-server /etc/re" 5 hours ago Exited (0) 8 minutes ago k8s_redis.638d2d73_redis-1-b7blk_default_2df04edf-e91c-11e6-999b-42010af00135_74b5a100
$ kubectl exec -it ubuntu sh
# redis-cli -h 10.107.244.37 -c
10.107.244.37:6379> get hoge
"fuga"
10.107.244.37:6379> get hogehoge
"fugafuga"
确认即使关闭节点也能从Redis节点返回值。
如果容器集群的节点只有一个，那么会导致无法自动启动的Pod出现，因此至少需要两个节点。
考虑到节点可能会宕机的情况，将集群设置如下：
-
- node最小サイズ3
- container cluster 自動スケーリング ON
$ kubectl get nodes
NAME STATUS AGE
gke-satsuma-redis-default-pool-f99bcf1a-cft1 Ready 2d
gke-satsuma-redis-default-pool-f99bcf1a-lj8d Ready 9m
$ kubectl delete node gke-satsuma-redis-default-pool-f99bcf1a-cft1
node "gke-satsuma-redis-default-pool-f99bcf1a-cft1" deleted
$ kubectl get nodes
NAME STATUS AGE
gke-satsuma-redis-default-pool-f99bcf1a-lj8d Ready 10m
$ kubectl exec -it ubuntu sh
# redis-cli -h 10.107.244.37 -c
10.107.244.37:6379> get hoge
-> Redirected to slot [1525] located at 10.104.0.20:6379
"fuga"
10.104.0.20:6379> get hogehoge
"fugafuga"
10.104.0.20:6379> get hoge
"fuga"
10.104.0.20:6379> get hogehoge
"fugafuga"
与服务关联，在所有 Redis 节点上进行随机访问，并确认能够成功获取值。
- service redisが redis-1〜redis-6endpointに向いているか確認
$ kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.107.240.1 <none> 443/TCP 2d
redis 10.107.244.37 <none> 6379/TCP,16379/TCP 52m
redis-1 10.107.255.1 <none> 6379/TCP,16379/TCP 52m
redis-2 10.107.255.2 <none> 6379/TCP,16379/TCP 52m
redis-3 10.107.255.3 <none> 6379/TCP,16379/TCP 52m
redis-4 10.107.255.4 <none> 6379/TCP,16379/TCP 52m
redis-5 10.107.255.5 <none> 6379/TCP,16379/TCP 52m
redis-6 10.107.255.6 <none> 6379/TCP,16379/TCP 52m
$ kubectl describe service redis
Name: redis
Namespace: default
Labels: <none>
Selector: app=redis
Type: ClusterIP
IP: 10.107.244.37
Port: redis 6379/TCP
Endpoints: 10.104.0.18:6379,10.104.0.19:6379,10.104.0.20:6379 + 3 more...
Port: cluster 16379/TCP
Endpoints: 10.104.0.18:16379,10.104.0.19:16379,10.104.0.20:16379 + 3 more...
Session Affinity: None
- masterでデータを挿入し、loadbalancerからデータが正しく取得できるか確認する
在redis-1（主节点）中插入数据。
$ kubectl exec -it redis-1-2pg4v -- sh
/data # redis-cli -c
127.0.0.1:6379> set hoge fuga
OK
在Redis-3（主节点）中插入数据。
$ kubectl exec -it redis-3-4fs93 -- sh
/data # redis-cli -c
127.0.0.1:6379> set hogehoge fugafuga
-> Redirected to slot [694] located at 10.104.1.44:6379
OK
尝试通过Redis（负载均衡器）获取数据。
$ kubectl exec -it ubuntu -- sh
# redis-cli -h 10.107.244.37 -c
10.107.244.37:6379> get hoge
-> Redirected to slot [1525] located at 10.104.1.44:6379
"fuga"
10.104.1.44:6379> get hogehoge
"fugafuga"
尝试在redis-2（主节点）中获取
$ kubectl exec -it redis-2-fbzks sh (feature/redis✱)
/data # redis-cli -c
127.0.0.1:6379> get hoge
-> Redirected to slot [1525] located at 10.104.1.46:6379
"fuga"
10.104.1.46:6379> get hogehoge
"fugafuga"
我试着在Redis-5（从服务器）中获取。
$ kubectl exec -it redis-5-bt1wm sh
/data # redis-cli -c
127.0.0.1:6379> get hoge
-> Redirected to slot [1525] located at 10.104.1.46:6379
"fuga"
10.104.1.46:6379> get hogehoge
"fugafuga"
验证
在GCP Redis上创建100万条数据。
$ redis-cli debug populate 1000000
10.104.0.6:6379> mget key:0
1) "value:0"
10.104.1.4:6379> mget key:99999
-> Redirected to slot [2036] located at 10.104.0.6:6379
1) "value:99999"
通过使用内存进行确认
$ redis-cli info
# Memory
used_memory:2278256
used_memory_human:2.17M
used_memory_rss:3805184
used_memory_peak:2278256
used_memory_peak_human:2.17M
used_memory_lua:36864
mem_fragmentation_ratio:1.67
mem_allocator:libc
根据所保存数据的字符串长度预估数据的数量，并需要另外进行内存可承受性验证。
