Redis high availability using VRRP

REDIS HA using VRRP

Before start doing anything read the URL’s bellow:

https://redis.io/topics/sentinel
https://redis.io/topics/cluster-spec
https://redis.io/commands/

 

AS per our requirement’s we need to provide REDIS for some applications installed in our DC. But as a data center we should also provide HA (High availability)

In our setup we will install 2 REDIS servers and one sentinel server, sentinel works as a cluster witness.

  1. REDIS Server Master ( Read / Write )
  2. REDIS Server Slave ( Read only )
  3. Monitoring server ( Sentinel Server )

 

Server Name Server Type Port Role IPv4
redis-1 redis-server 6379 master + sentinel 10.1.1.100
redis-2 redis-server 6379 Slave + sentinel 10.1.1.101
mon-01 witness sentinel only 10.1.1.103
Visual ipv4   Used by VRRP 10.1.1.102

 

Prerequisites

Install 3 Centos server with static IPv4 Addresses.

Direct internet access is required to access epel YUM repo.

 

On each server do the following:

###

yum install –y epel-release
groupadd -g 1983 redis
useradd -u 1983 -s “/sbin/nologin” -c “redis user” -g 1983 redis
yum search redis && yum install –y  redis
systemctl start redis.service && systemctl status redis.service
systemctl start redis-sentinel.service && systemctl status redis-sentinel.service
systemctl enable redis.service && systemctl enable redis-sentinel.service

###

 

Explanation:

#Install epel yum repo
yum install –y epel-release

# Add group redis and user redis user
groupadd -g 1983 redis
useradd -u 1983 -s “/sbin/nologin” -c “redis user” -g 1983 redis

#Install redis and redis sentinel
yum search redis && yum install –y  redis
# start redis and check status
systemctl start redis.service && systemctl status redis.service

#start redis-sentinel and check status
systemctl start redis-sentinel.service && systemctl status redis-sentinel.service

# add redis and redis-sentinel  to startup systemd
systemctl enable redis.service && systemctl enable redis-sentinel.service

 

 

MASTER REDIS SERVER

On the master server do the following:

 

Nano /etc/redis.conf

 

 

Change the following lines:

bind 10.1.1.100 127.0.0.1 10.1.1.103
requirepass naderbdotorg

 

# after editing the conf file restart redis service and redis sentinel service

systemctl restart redis.service && systemctl restart redis-sentinel.service

 

nano redis-sentinel.conf

protected-mode no
port 26379
dir “/tmp”
sentinel myid 2a196164939444702d10c118755a41fc72d67ee9
logfile “/var/log/redis/sentinel.log”
sentinel monitor redis-1 10.1.1.100 6379 2
sentinel down-after-milliseconds redis-1 5000
sentinel failover-timeout -redis-1 10000
sentinel auth-pass redis-1 Naderbdotorg
sentinel config-epoch redis-1 4
sentinel leader-epoch redis-1 4

 

Info:myid value is auto generated by installation of Redis, if left empty it will be auto generated

 

SLAVE REDIS SERVER

On the slave server do the following:

 

Nano /etc/redis.conf

 

bind 10.1.1.101 127.0.0.1requirepass Naderbdotorgmasterauth Naderbdotorgslaveof 10.1.1.100 6379 

# after editing the conf file restart redis service and redis sentinel service

systemctl restart redis.service && systemctl restart redis-sentinel.service

 

nano redis-sentinel.conf

 

protected-mode no
port 26379
dir “/tmp”
sentinel myid 2a196164939444702d10c118755a41fc72d67111
sentinel monitor -redis-1 10.1.1.100 6379 2
logfile “/var/log/redis/sentinel.log”
sentinel down-after-milliseconds -redis-1 5000
sentinel failover-timeout -redis-1 10000
sentinel auth-pass -redis-1 Naderbdotorg
sentinel config-epoch -redis-1 0
sentinel leader-epoch -redis-1 0
sentinel known-slave -redis-1 10.1.1.101 6379

 

Witness Server

yum install –y epel-release

yum search redis && yum install –y  redis

nano /etc/redis-sentinel.conf

past the following

sentinel myid 2a196164939444702d10c118755a41fc72d67ee9
sentinel monitor -redis-1 10.1.1.100 6379 2
sentinel down-after-milliseconds -redis-1 5000
sentinel failover-timeout -redis-1 10000
sentinel auth-pass -redis-1 Naderbdotorg
sentinel config-epoch -redis-1 0
sentinel leader-epoch -redis-1 0

 

KeepAlived

VRRP provider !

Yum install keepalived –y

On the master server:

nano /etc/keepalived/keepalived.conf

Copy what follows inside the keepalive.conf file

###

#Generated by  Nader Barakat – 2018
# redis cluster using VRRP

vrrp_script redis_service_check {
script   “/etc/keepalived/redis-ha-check”
interval 3
weight   -5
fall     2
rise     1
}
vrrp_script redis_master_check {
script   “/etc/keepalived/redis-master-check”
interval 3
weight   -5
fall     2
rise     1
}
vrrp_instance VI_1 {
interface         ens192
state             MASTER
priority          101
virtual_router_id 51
advert_int        1
accept
unicast_src_ip    10.1.1.100

unicast_peer {
10.1.1.101
}
virtual_ipaddress {
10.1.1.103
}
authentication {
auth_type PASS
auth_pass MOARedis2018
}

track_script {
redis_service_check
redis_master_check
}
}

###

 

Systemctl enable keepalived && systemctl start keepalived

nano /etc/keepalived/redis-ha-check

Copy what follows inside the redis-ha-check file

chmod +x /etc/keepalived/redis-ha-check

###

#!/bin/bash
counter=$(ps -C redis-server –no-heading|wc -l)
if [ “${counter}” = “0” ]; then
systemctl start redis.service && systemctl start redis-sentinel.service
sleep 2
counter=$(ps -C redis-server –no-heading|wc -l)
if [ “${counter}” = “0” ]; then
systemctl stop keepalived
fi
fi

###

 

The above script will check if the service of redis-server is up to keep visual ip to the master server if not able to keep the redis service up switch to slave server.

nano /etc/keepalived/redis-master-check

Copy what follows inside the redis-master-check file

chmod +x /etc/keepalived/redis-master-check

###

#!/bin/bash
#
#
counter=$(ps -C redis-server –no-heading|wc -l)
echo start
if [ “${counter}” = “1” ]; then
redis_master=$(redis-cli -h 10.1.1.100 -p 6379 -a Naderbdotorg info
replication | grep ‘role:slave’ | tr -d ‘\r’)
echo sex1 = $redis_master
if [ $redis_master = ‘role:slave’ ]; then
echo “Switch-redis master back to master”
redis-cli -h 10.1.1.100 -p 6379 -a Naderbdotorg SLAVEOF no one
redis-cli -h 10.1.1.101 -p 6379 -a Naderbdotorg SLAVEOF 10.1.1.100 6379

fifi

#################################################################
# This file is to restore master when its up and running
# also make the slave as slave of the sync <> master
# Nader Barakat – 2018
#################################################################

###

The script above to switch back the master server as master redis server when it’s restored.

 

Checking replication

redis-cli -h 10.1.1.100 -p 6379 -a Naderbdotorg INFO replication

redis-cli -h 10.1.1.101 -p 6379 -a Naderbdotorg INFO replication

redis-cli -h 10.1.1.103 -p 6379 -a Naderbdotorg INFO replication

I tested this method on nextcloud and some small application which is working without any issue.

Skip to toolbar