在Akamai中，通过VLAN和keepalived来实现Redis Sentinel的安全和冗余化

1 年 ago
清, 宇
6 minutes

Redis Sentinel是Redis的高可用性解决方案。为了确保应用程序的高可用性，可以使用冗余化的HAProxy模式。在Akamai的云端，提供了IP共享功能，可以使用私有IP来冗余化HAProxy。以下文章提供了有关步骤的参考。

 

为了将Redis Sentinel环境完全闭合在私有网络中进行构建，我们还介绍了支持VLAN的相关文章。

 

本文将说明将HAProxy的冗余和安全性两个要素结合起来的方法。主要的确认事项是在VLAN上实现HAProxy的冗余，使用keepalived。最终的设备结构如下图所示。

Redis-VLAN-HAProxy.jpg
不像公有IP和私有IP那样，VLAN是封闭在内部网络中的，因此不需要使用类似于lelastic的工具。
在VLAN中，可以自由使用IP，因此不需要IP共享。

准备 Redis Sentinel 在 VLAN 环境中。

参考这篇文章来设置VLAN环境。

 

redis-vlan-1.jpg

/等等/宿主机文件

请按照以下方式设置 VLAN 的 IP。

# VLAN
10.0.0.11 redis-1v # Redis Server 1
10.0.0.12 redis-2v # Redis Server 2
10.0.0.13 redis-3v # Redis Server 3
10.0.0.1 vlan-router # Act as a Router
10.0.0.21 vlan-haproxy1 # HAProxy 1
10.0.0.22 vlan-redisclient $ Application
10.0.0.23 vlan-haproxy2 # HAProxy 2
10.0.0.31 vlan-haproxy  # HAProxy (VIP)

新增HAProxy

接下来，根据以下文章的参考，我们准备haproxy2。

 

redis-vlan-haproxy2.jpg

使用Linode的克隆功能，简化工作流程。

克隆

clone.jpg
haproxy2-poweron.jpg

启动后，请重新确认安全设置，并在其中按以下方式更改主机名。

hostnamectl set-hostname hproxy2

在haproxy2上的haproxy.cfg文件

将绑定设置如下。

frontend ft_redis
    bind *:6379 name redis
    default_backend bk_redis

我将在haproxy1上进行相同的配置。如果进行了配置更改，将重新启动haproxy。

systemctl restart haproxy

在Redis1/2/3上启用防火墙。

在每个Redis服务器上，将firewalld的设置更改，以便可以从新的haproxy2和VIP访问。10.0.0.31是用于设置虚拟IP的设置。

  <source address="10.0.0.23"/>

  <source address="10.0.0.31"/>

在更改设置后，将重新启动firewalld。

firewall-cmd --reload

haproxy2 发送的确认请求

确认能够连接到所有的 Redis 服务器。

haproxy2:~# redis-cli -h redis-1v --tls --cacert ca.crt info replication | grep role
role:slave
haproxy2:~# redis-cli -h redis-2v --tls --cacert ca.crt info replication | grep role
role:master
haproxy2:~# redis-cli -h redis-3v --tls --cacert ca.crt info replication | grep role
role:slave

haproxy1 发送的确认信息

确认与haproxy1的连接。

haproxy1:~# redis-cli -h vlan-haproxy2 --tls --cacert ca.crt info replication | grep role
role:master
haproxy1:~# redis-cli -h vlan-haproxy1 --tls --cacert ca.crt info replication | grep role
role:master
haproxy2-setup-complete.jpg

保持keepalived设置

安装keepalived到HAProxy 1/2，并将10.0.0.31设置为虚拟IP（VIP）以使其正常运行。

haproxy-keepalived.jpg

安装 keepalived

将keepalived安装在haproxy1和haproxy2中。

apt install keepalived

保持活动.conf的配置

haproxy1 と happroxy2 で keepalived.conf を設定します。設定内容はそれぞれ異なります。

vi /etc/keepalived/keepalived.conf

haproxy1 的配置

vrrp_instance Instance1 {
    state MASTER
    interface eth0
    virtual_router_id 10
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass passWord
    }
    unicast_src_ip 10.0.0.21
    unicast_peer {
       10.0.0.23
    }
    virtual_ipaddress {
        10.0.0.31
    }
}

haproxy2的配置

haproxy1 との違いです。
1.state が MASTER ではなく BACKUP とする
2. priority を 100 ではなく 99 とする
3. Unicast_src_ip を自分自身の IP とする
4. unicast_peer を相手の IP とする

vrrp_instance Instance1 {
    state BACKUP
    interface eth0
    virtual_router_id 10
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass passWord
    }
    unicast_src_ip 10.0.0.23
    unicast_peer {
       10.0.0.21
    }
    virtual_ipaddress {
        10.0.0.31
    }
}

keepalived の起動

我们将在haproxy1/2上运行keepalived。

sudo systemctl enable keepalived
sudo systemctl start keepalived

确认 haproxy1 的 IP

eth0设备被分配了10.0.0.31的IP地址。

haproxy1:/etc/keepalived# ip -4 a show dev eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.31/32 scope global eth0
       valid_lft forever preferred_lft forever

请确认 HAProxy2 的 IP 地址。

eth0设备未分配10.0.0.31的地址。

haproxy2:/etc/keepalived# ip -4 a show dev eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.0.0.23/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever

保持活跃的操作验证

在Redis客户端（10.0.0.22）上启动三个终端，并执行以下三个命令。

ping 10.0.0.31

ping 10.0.0.21

redis-cli -h vlan-haproxy --tls --cacert redis-nj_ca.crt

我将对haproxy1进行重新启动。

reboot
redis-keepalived-haproxy.jpg

向 haproxy1 发出 ping 请求的结果

当haproxy1进行重新启动后，来自Redis客户端（10.0.0.22）的ping请求将在一半之后停止响应。

64 bytes from 10.0.0.21: icmp_seq=42 ttl=64 time=0.767 ms
64 bytes from 10.0.0.21: icmp_seq=43 ttl=64 time=2.02 ms
64 bytes from 10.0.0.21: icmp_seq=44 ttl=64 time=3.16 ms
64 bytes from 10.0.0.21: icmp_seq=45 ttl=64 time=1.37 ms
64 bytes from 10.0.0.21: icmp_seq=46 ttl=64 time=0.628 ms

對於 VIP 的 ping 測試結果

我们将确认 haproxy1 上的 10.0.0.31 是否已被 haproxy2 接管。在先前无法访问到 10.0.0.21 的时间点，我们已经成功访问到了 10.0.0.31。

64 bytes from 10.0.0.31: icmp_seq=42 ttl=64 time=0.467 ms
64 bytes from 10.0.0.31: icmp_seq=43 ttl=64 time=0.446 ms
64 bytes from 10.0.0.31: icmp_seq=44 ttl=64 time=0.355 ms
64 bytes from 10.0.0.31: icmp_seq=45 ttl=64 time=0.447 ms
64 bytes from 10.0.0.31: icmp_seq=46 ttl=64 time=0.365 ms
64 bytes from 10.0.0.31: icmp_seq=47 ttl=64 time=0.435 ms
64 bytes from 10.0.0.31: icmp_seq=48 ttl=64 time=0.372 ms
64 bytes from 10.0.0.31: icmp_seq=49 ttl=64 time=0.369 ms
64 bytes from 10.0.0.31: icmp_seq=50 ttl=64 time=0.357 ms

在这个时候，haproxy2的IP已经被改变了。keepalived正在运行，并且haproxy2拥有10.0.0.31的IP地址。

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.0.0.23/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.31/32 scope global eth0
       valid_lft forever preferred_lft forever

通过 HAProxy 从 Redis Client 连接到 Redis Server

即使关闭了haproxy1，仍然可以继续连接到Redis服务器。

redisclient:~# redis-cli -h vlan-haproxy --tls --cacert redis-nj_ca.crt
vlan-haproxy:6379> role
1) "master"
2) (integer) 407029129
3) 1) 1) "10.0.0.13"
      2) "6379"
      3) "407029129"
   2) 1) "10.0.0.11"
      2) "6379"
      3) "407029129"
vlan-haproxy:6379> <- このタイミングで haproxy1 を shutdown
vlan-haproxy:6379> get scott　
"tiger"
vlan-haproxy:6379> role
1) "master"
2) (integer) 407050682
3) 1) 1) "10.0.0.13"
      2) "6379"
      3) "407050416"
   2) 1) "10.0.0.11"
      2) "6379"
      3) "407050416"

重新啟動之後

由于haproxy1的启动，导致无法从haproxy2的IP看到10.0.0.31。

@haproxy2:/etc/keepalived# ip -4 a show dev eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.0.0.23/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever

通过haproxy2的keepalived.conf中的以下行配置判断，haproxy1的配置值优先级较低，并将IP权限授予haproxy1。

    priority 99

登录到haproxy1并确认IP地址。

@haproxy1:~# ip -4 a show dev eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.31/32 scope global eth0
       valid_lft forever preferred_lft forever

/var/log/syslog 可以被表达为 “系统日志” 或者 “系统日志文件”。

keepalived 的日志会被记录在 /var/log/syslog 中。当查看 haproxy1/2 的日志时，分别可以看到 Master 和 BACKUP 的状态切换。

HAProxy2就好了。

Jul 19 16:21:00 haproxy2 Keepalived_vrrp[44007]: (Instance1) Entering MASTER STATE
Jul 19 16:23:55 haproxy2 Keepalived_vrrp[44007]: (Instance1) Master received advert from 10.0.0.21 with higher priority 100, ours 99
Jul 19 16:23:55 haproxy2 Keepalived_vrrp[44007]: (Instance1) Entering BACKUP STATE
Jul 19 16:39:16 haproxy2 Keepalived_vrrp[44007]: (Instance1) Entering MASTER STATE
Jul 19 16:41:59 haproxy2 Keepalived_vrrp[44007]: (Instance1) Master received advert from 10.0.0.21 with higher priority 100, ours 99
Jul 19 16:41:59 haproxy2 Keepalived_vrrp[44007]: (Instance1) Entering BACKUP STATE

haproxy1 台服务器

Jul 19 16:20:59 haproxy1 Keepalived[528]: Stopping
Jul 19 16:20:59 haproxy1 systemd[1]: Stopping Keepalive Daemon (LVS and VRRP)...
Jul 19 16:23:50 haproxy1 systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
Jul 19 16:23:51 haproxy1 Keepalived[541]: Starting Keepalived v2.2.4 (08/21,2021)
Jul 19 16:23:51 haproxy1 Keepalived[541]: Running on Linux 6.2.9-x86_64-linode160 #1 SMP PREEMPT_DYNAMIC Wed Apr  5 15:30:32 EDT 2023 (built for Linux 5.15.27)
Jul 19 16:23:51 haproxy1 Keepalived[541]: Command line: '/usr/sbin/keepalived' '--dont-fork'
Jul 19 16:23:51 haproxy1 Keepalived[541]: Configuration file /etc/keepalived/keepalived.conf
Jul 19 16:23:51 haproxy1 Keepalived[541]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Jul 19 16:23:51 haproxy1 Keepalived[541]: Starting VRRP child process, pid=550
Jul 19 16:23:51 haproxy1 Keepalived[541]: Startup complete
Jul 19 16:23:51 haproxy1 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Jul 19 16:23:51 haproxy1 Keepalived_vrrp[550]: (Instance1) Entering BACKUP STATE (init)
Jul 19 16:23:52 haproxy1 Keepalived_vrrp[550]: (Instance1) received lower priority (99) advert from 10.0.0.23 - discarding
Jul 19 16:23:53 haproxy1 Keepalived_vrrp[550]: (Instance1) received lower priority (99) advert from 10.0.0.23 - discarding
Jul 19 16:23:54 haproxy1 Keepalived_vrrp[550]: (Instance1) received lower priority (99) advert from 10.0.0.23 - discarding
Jul 19 16:23:55 haproxy1 Keepalived_vrrp[550]: (Instance1) Entering MASTER STATE
Jul 19 16:39:16 haproxy1 Keepalived[541]: Stopping
Jul 19 16:39:16 haproxy1 systemd[1]: Stopping Keepalive Daemon (LVS and VRRP)...
Jul 19 16:41:55 haproxy1 systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
Jul 19 16:41:55 haproxy1 Keepalived[537]: Starting Keepalived v2.2.4 (08/21,2021)
Jul 19 16:41:55 haproxy1 Keepalived[537]: Running on Linux 6.2.9-x86_64-linode160 #1 SMP PREEMPT_DYNAMIC Wed Apr  5 15:30:32 EDT 2023 (built for Linux 5.15.27)
Jul 19 16:41:55 haproxy1 Keepalived[537]: Command line: '/usr/sbin/keepalived' '--dont-fork'
Jul 19 16:41:55 haproxy1 Keepalived[537]: Configuration file /etc/keepalived/keepalived.conf
Jul 19 16:41:55 haproxy1 Keepalived[537]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Jul 19 16:41:55 haproxy1 Keepalived[537]: Starting VRRP child process, pid=547
Jul 19 16:41:56 haproxy1 Keepalived[537]: Startup complete
Jul 19 16:41:56 haproxy1 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Jul 19 16:41:56 haproxy1 Keepalived_vrrp[547]: (Instance1) Entering BACKUP STATE (init)
Jul 19 16:41:56 haproxy1 Keepalived_vrrp[547]: (Instance1) received lower priority (99) advert from 10.0.0.23 - discarding
Jul 19 16:41:57 haproxy1 Keepalived_vrrp[547]: (Instance1) received lower priority (99) advert from 10.0.0.23 - discarding
Jul 19 16:41:58 haproxy1 Keepalived_vrrp[547]: (Instance1) received lower priority (99) advert from 10.0.0.23 - discarding
Jul 19 16:41:59 haproxy1 Keepalived_vrrp[547]: (Instance1) Entering MASTER STATE

最后最终会回到原本的状态。

redis-after-haproxy1-reboot.jpg

总结

Redis Sentinel 是 Redis 的高可用性解决方案。通过使用 VLAN，可以构建安全的网络。然后，通过使用 Keepalived，在提高整体便利性的同时，可以构建 HAProxy 的高可用性环境。

✖
广告
将在 10 秒后关闭
bannerAds