REDIS HA using VRRP
Before start doing anything read the URL’s bellow:
https://redis.io/topics/sentinel
https://redis.io/topics/cluster-spec
https://redis.io/commands/
AS per our requirement’s we need to provide REDIS for some applications installed in our DC. But as a data center we should also provide HA (High availability)
In our setup we will install 2 REDIS servers and one sentinel server, sentinel works as a cluster witness.
- REDIS Server Master ( Read / Write )
- REDIS Server Slave ( Read only )
- Monitoring server ( Sentinel Server )
| Server Name | Server Type | Port | Role | IPv4 |
| redis-1 | redis-server | 6379 | master + sentinel | 10.1.1.100 |
| redis-2 | redis-server | 6379 | Slave + sentinel | 10.1.1.101 |
| mon-01 | witness | sentinel only | 10.1.1.103 | |
| Visual ipv4 | Used by VRRP | 10.1.1.102 |
Prerequisites
Install 3 Centos server with static IPv4 Addresses.
Direct internet access is required to access epel YUM repo.
On each server do the following:
###
yum install –y epel-release
groupadd -g 1983 redis
useradd -u 1983 -s “/sbin/nologin” -c “redis user” -g 1983 redis
yum search redis && yum install –y redis
systemctl start redis.service && systemctl status redis.service
systemctl start redis-sentinel.service && systemctl status redis-sentinel.service
systemctl enable redis.service && systemctl enable redis-sentinel.service
###
Explanation:
#Install epel yum repo
yum install –y epel-release
# Add group redis and user redis user
groupadd -g 1983 redis
useradd -u 1983 -s “/sbin/nologin” -c “redis user” -g 1983 redis
#Install redis and redis sentinel
yum search redis && yum install –y redis
# start redis and check status
systemctl start redis.service && systemctl status redis.service
#start redis-sentinel and check status
systemctl start redis-sentinel.service && systemctl status redis-sentinel.service
# add redis and redis-sentinel to startup systemd
systemctl enable redis.service && systemctl enable redis-sentinel.service
MASTER REDIS SERVER
On the master server do the following:
Nano /etc/redis.conf
Change the following lines:
bind 10.1.1.100 127.0.0.1 10.1.1.103
requirepass naderbdotorg
# after editing the conf file restart redis service and redis sentinel service
systemctl restart redis.service && systemctl restart redis-sentinel.service
nano redis-sentinel.conf
protected-mode no
port 26379
dir “/tmp”
sentinel myid 2a196164939444702d10c118755a41fc72d67ee9
logfile “/var/log/redis/sentinel.log”
sentinel monitor redis-1 10.1.1.100 6379 2
sentinel down-after-milliseconds redis-1 5000
sentinel failover-timeout -redis-1 10000
sentinel auth-pass redis-1 Naderbdotorg
sentinel config-epoch redis-1 4
sentinel leader-epoch redis-1 4
Info:myid value is auto generated by installation of Redis, if left empty it will be auto generated
SLAVE REDIS SERVER
On the slave server do the following:
Nano /etc/redis.conf
bind 10.1.1.101 127.0.0.1requirepass Naderbdotorgmasterauth Naderbdotorgslaveof 10.1.1.100 6379
# after editing the conf file restart redis service and redis sentinel service
systemctl restart redis.service && systemctl restart redis-sentinel.service
nano redis-sentinel.conf
protected-mode no
port 26379
dir “/tmp”
sentinel myid 2a196164939444702d10c118755a41fc72d67111
sentinel monitor -redis-1 10.1.1.100 6379 2
logfile “/var/log/redis/sentinel.log”
sentinel down-after-milliseconds -redis-1 5000
sentinel failover-timeout -redis-1 10000
sentinel auth-pass -redis-1 Naderbdotorg
sentinel config-epoch -redis-1 0
sentinel leader-epoch -redis-1 0
sentinel known-slave -redis-1 10.1.1.101 6379
Witness Server
yum install –y epel-release
yum search redis && yum install –y redis
nano /etc/redis-sentinel.conf
past the following
sentinel myid 2a196164939444702d10c118755a41fc72d67ee9
sentinel monitor -redis-1 10.1.1.100 6379 2
sentinel down-after-milliseconds -redis-1 5000
sentinel failover-timeout -redis-1 10000
sentinel auth-pass -redis-1 Naderbdotorg
sentinel config-epoch -redis-1 0
sentinel leader-epoch -redis-1 0
KeepAlived
VRRP provider !
Yum install keepalived –y
On the master server:
nano /etc/keepalived/keepalived.conf
Copy what follows inside the keepalive.conf file
###
#Generated by Nader Barakat – 2018
# redis cluster using VRRP
vrrp_script redis_service_check {
script “/etc/keepalived/redis-ha-check”
interval 3
weight -5
fall 2
rise 1
}
vrrp_script redis_master_check {
script “/etc/keepalived/redis-master-check”
interval 3
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
interface ens192
state MASTER
priority 101
virtual_router_id 51
advert_int 1
accept
unicast_src_ip 10.1.1.100
unicast_peer {
10.1.1.101
}
virtual_ipaddress {
10.1.1.103
}
authentication {
auth_type PASS
auth_pass MOARedis2018
}
track_script {
redis_service_check
redis_master_check
}
}
###
Systemctl enable keepalived && systemctl start keepalived
nano /etc/keepalived/redis-ha-check
Copy what follows inside the redis-ha-check file
chmod +x /etc/keepalived/redis-ha-check
###
#!/bin/bash
counter=$(ps -C redis-server –no-heading|wc -l)
if [ “${counter}” = “0” ]; then
systemctl start redis.service && systemctl start redis-sentinel.service
sleep 2
counter=$(ps -C redis-server –no-heading|wc -l)
if [ “${counter}” = “0” ]; then
systemctl stop keepalived
fi
fi
###
The above script will check if the service of redis-server is up to keep visual ip to the master server if not able to keep the redis service up switch to slave server.
nano /etc/keepalived/redis-master-check
Copy what follows inside the redis-master-check file
chmod +x /etc/keepalived/redis-master-check
###
#!/bin/bash
#
#
counter=$(ps -C redis-server –no-heading|wc -l)
echo start
if [ “${counter}” = “1” ]; then
redis_master=$(redis-cli -h 10.1.1.100 -p 6379 -a Naderbdotorg info
replication | grep ‘role:slave’ | tr -d ‘\r’)
echo sex1 = $redis_master
if [ $redis_master = ‘role:slave’ ]; then
echo “Switch-redis master back to master”
redis-cli -h 10.1.1.100 -p 6379 -a Naderbdotorg SLAVEOF no one
redis-cli -h 10.1.1.101 -p 6379 -a Naderbdotorg SLAVEOF 10.1.1.100 6379
fifi
#################################################################
# This file is to restore master when its up and running
# also make the slave as slave of the sync <> master
# Nader Barakat – 2018
#################################################################
###
The script above to switch back the master server as master redis server when it’s restored.
Checking replication
redis-cli -h 10.1.1.100 -p 6379 -a Naderbdotorg INFO replication
redis-cli -h 10.1.1.101 -p 6379 -a Naderbdotorg INFO replication
redis-cli -h 10.1.1.103 -p 6379 -a Naderbdotorg INFO replication
I tested this method on nextcloud and some small application which is working without any issue.