nginx+keepalived实现负载均衡高可用

keepalived官网

keepalived github地址

keepalived官方文档

keepalived源码包官方下载地址

一、keepalived简介

Keepalived是用C语言编写的路由软件。该项目的主要目标是为Linux系统和基于Linux的基础架构提供负载均衡和高可用性的简单而强大的功能。 负载平衡框架依赖于提供第4层负载平衡的著名且广泛使用的Linux虚拟服务器(IPVS)内核模块。Keepalived实现了一组检查器,以根据其运行状况动态,自适应地维护和管理负载平衡的服务器池。另一方面,VRRP实现了高可用性协议。VRRP是路由器故障转移的基础。此外,Keepalived还实现了一组VRRP有限状态机的钩,从而提供了低级和高速协议交互。为了提供最快的网络故障检测,Keepalived实施BFD协议。VRRP状态转换可以考虑BFD提示来驱动快速状态转换。Keepalived框架可以独立使用,也可以一起使用以提供弹性基础架构。

二、keepalived负载均衡高可用配置过程

实验环境

角色 IP 主机名
keepalived-master 10.0.0.10 keepalived01
keepalived-slave 10.0.0.11 keepalived02
web01 10.0.0.51 nginx01
web02 10.0.0.52 nginx02

2.1 负载均衡端配置

2.1.1 安装keepalived

yum -y install keepalived

centos7.7中默认的keepalive版本是1.3.5,如果需要高版本请到keepalived github地址或者keepalived官网下载,这里仅做实验直接yum安装

$ keepalived -version
Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

2.1.2 lb01(keepalived master)编辑配置文件

lb01编辑keepalived配置文件/etc/keepalived/keepalived.conf

#备份原有文件
mv /etc/keepalived/keepalived.conf{,.bak}

#重新编辑文件
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {     
    router_id lb01
}

vrrp_instance VI_1 {
    state MASTER                    
    interface eth0                
    virtual_router_id 50           
    priority 150                
    advert_int 1                 
    authentication {            
        auth_type PASS
        auth_pass 123456
}
    virtual_ipaddress {
        10.0.0.100        
    }
}
EOF

配置文件参数含义

#配置含义
global_defs {     
    router_id lb01               #表示id身份,名称随意
}

vrrp_instance VI_1 {
    state MASTER                #状态,角色为master
    interface eth0                #VIP绑定的网卡
    virtual_router_id 50        #主和备的虚拟id号要相同,表明在一个组当中,名称随意,不要超过255
    priority 150                #优先级,越大优先级越高
    advert_int 1                #心跳检测,单位秒,备机每隔1秒检测一次主的存活状态 
    authentication {            #认证信息
        auth_type PASS
        auth_pass 123456
}
    virtual_ipaddress {
        10.0.0.100                #虚拟IP
    }
}

2.1.3 lb02(keepalived backup)编辑配置文件

#备份原有文件
mv /etc/keepalived/keepalived.conf{,.bak}

#重新编辑文件
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {     
    router_id lb02
}

vrrp_instance VI_1 {
    state BACKUP            
    interface eth0            
    virtual_router_id 50                                                         
    priority 100                
    advert_int 1                
    authentication {            
        auth_type PASS
        auth_pass 123456
}
    virtual_ipaddress {
        10.0.0.100
    }
}
EOF

配置文件参数含义

global_defs {     
    router_id lb02              #表示id身份,名称随意
}

vrrp_instance VI_1 {
    state BACKUP                #状态,因为是slave,所以是BACKUP
    interface eth0                #VIP绑定的网卡
    virtual_router_id 50        #主和备的虚拟id号要相同,表明在一个组当中,名称随意,不要                                                     超过255
    priority 100                #优先级,越大优先级越高
    advert_int 1                #心跳检测,单位秒,备机每隔1秒检测一次主的存活状态 
    authentication {            #认证信息
        auth_type PASS
        auth_pass 123456
}
    virtual_ipaddress {
        10.0.0.100                #虚拟IP
    }
}

对比keepalived的master与backup配置的区别

Keepalived配置区别 Master配置 Backup节配置
route_id(唯一标识) route_id lb01 route_id lb02
state(角色状态) state Master state Backup
priority(竞选优先级) priority 150 priority 100

2.1.4 启动keepalived

lb01和lb02相同操作

systemctl start keepalived && systemctl enable keepalived

2.1.5 检测lb01是否有vip

可以看到,启动keepalived后,lb01 eth0网卡就有了VIP 10.0.0.100

$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:f3:3c:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::21c:42ff:fef3:3c78/64 scope link
       valid_lft forever preferred_lft forever

2.1.6 验证VIP是否能够正常漂移

VIP首先在master上,当master宕机后,VIP会漂移至backup,当master恢复时VIP会自动漂移回来!!!

先停止lb01上的keepalived

systemctl stop keepalived

查看lb01上是否有VIP

可以看到,VIP此时已经没有了

$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:f3:3c:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::21c:42ff:fef3:3c78/64 scope link
       valid_lft forever preferred_lft forever

查看lb02上是否有VIP

当keepalived master宕机后,VIP就会漂移到lb02

$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:c1:0b:16 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.11/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::21c:42ff:fec1:b16/64 scope link
       valid_lft forever preferred_lft forever

重新启动lb01上的keepalived

systemctl start keepalived

#查看VIP
$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:f3:3c:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::21c:42ff:fef3:3c78/64 scope link
       valid_lft forever preferred_lft forever

2.1.7 编辑nginx配置文件

lb01和lb02编辑nginx配置文件

cat > /etc/nginx/conf.d/kd.test.conf <<'EOF'
upstream kd {
  server 10.0.0.51;
  server 10.0.0.52;
}

server {
  listen 80;
  server_name _;

  location / {
    proxy_pass http://kd;
  }
}
EOF

检测nginx语法并重载nginx

$ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ nginx -s reload

2.2 后端真实服务web配置

web01操作

2.2.1 编辑nginx配置文件

cat >> /etc/nginx/conf.d/kd.test.conf <<EOF
server {
 listen 80;
 server_name _;
 root /code;
 index index.html;
}
EOF

2.2.2 创建网站根目录

mkdir /code && echo "web01 web01 web01" > /code/index.html

2.2.3 检测nginx语法并重载nginx

$ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ nginx -s reload

web02操作

2.2.4 编辑nginx配置文件

cat >> /etc/nginx/conf.d/kd.test.conf <<EOF 
server {
    listen 80;
    server_name _;
    root /code;
    index index.html;
}
EOF

2.2.5 创建网站根目录

mkdir /code && echo "web02 web02 web02" > /code/index.html

2.2.6 检测nginx语法并重载nginx

$ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ nginx -s reload

2.3 浏览器验证负载均衡是否正常工作

浏览器访问VIP 10.0.0.100

kd

三、nginx宕机问题

如果Nginx宕机,会导致用户请求失败, 但Keepalived并不会进行切换,所以编写nginx检测脚本,检测nginx存活状态,不存活则kill掉nginx和keepalived,然后VIP进行漂移

3.1 编辑脚本

lb01操作

#创建存放脚本的目录
mkdir -p /server/scripts

#编辑脚本内容
cat >> /server/scripts/check_web.sh <<\EOF
#!/bin/sh
#使用while死循环
while true;do
nginxpid=$(ps -C nginx --no-header|wc -l)
#判断Nginx是否存活,如果不存活则尝试启动Nginx
if [ $nginxpid -eq 0 ];then
    systemctl start nginx
    sleep 5
    #5秒后再次获取一次Nginx状态
    nginxpid=$(ps -C nginx --no-header|wc -l) 
    #再次进行判断, 如Nginx还不存活则停止Keepalived,让地址进行漂移,并退出脚本  
    if [ $nginxpid -eq 0 ];then
        systemctl stop keepalived
        exit 1
   fi
fi
     sleep 5
done
EOF

#给脚本赋予执行权限
chmod +x /server/scripts/check_web.sh

#后台运行脚本
nohup sh /server/scripts/check_web.sh &

3.2 验证脚本是否生效

故意把nginx配置文件修改错误(比如删除一个括号或者分号),然后停止lb01上的nginx,因为上边的脚本中会判断nginx如果不存活就启动nginx,把nginx配置文件改错了nginx就无法正常启动过了

systemctl stop nginx

后台脚本检测到nginx不存活会kill掉keepalived,此时浏览器访问会报错,隔几秒就恢复了,因为VIP已经漂移至lb02上

kdnginx

3.3 在keepalived中调用检测脚本

⚠️nohup sh /server/scripts/check_web.sh &执行脚本属于手动执行,还可以在keepalived配置文件中配置调用监控脚本

lb01、lb02都需要操作

编辑keepalived配置文件/etc/keepalived/keepalived.conf

#在global_defs标签下增加如下内容
vrrp_script check_web {
    script "/server/scripts/check_web.sh"
    interval 10
    weight 50
}

#track_script标签要写在vrrp_instance VI_1{}中
track_script {
    check_web
}

⚠️⚠️⚠️vrrp_script{}中的interval时间需大于脚本中的sleep时间

否则会报错Keepalived_vrrp[2612]: /server/scripts/check_web.sh exited due to signal 15

这样就能够使keepalived自动调用监控脚本了,keepalived会根据配置文件中vrrp_script {}中的interval参数来决定每隔几秒执行脚本,interval后面的数字就表明执行脚本的间隔时间

在脚本中我们定义了,keepalived master上检测nginx是否存活,不存活尝试启动nginx,当无法成功启动nginx的时候停止master上的keepalived,然后让VIP漂移,这样的话就实现了当nginx服务挂掉时keepalived VIP能够漂移从而不影响客户端访问

四、keepalived高可用脑裂

概念

脑裂就是由于某些原因,导致两台keepalived高可用服务器在指定时间内,无法检测到对方的心跳消息,各自取得资源及服务的所有权,而此时的两台高可用服务器又都还活着,产生裂脑的情况下,master和backup上都会有VIP

产生脑裂的原因

1.服务器网络故障

2.服务器硬件故障发生损坏现象而崩溃

3.主备都开启firewalld防火墙

4.1 模拟脑裂

lb01和lb02开启firewalld防火墙

systemctl start firewalld

开启防火墙后产生裂脑原因

lb01和lb02都开启防火墙后,因为没有允许vrrp,当backup去检测master的时候无法检测到,此时backup认为master已经宕机,所以将VIP抢占;而master实际上并没有宕机,所以VIP不会漂移,这样就造成了master和backup都在抢占VIP。此时就出现了裂脑的情况

4.2 测试脑裂问题

在keepalived备上编写检测脚本, 测试如果能ping通主keepalived并且备节点还有VIP的话则认为产生了脑裂

lb02操作

#创建存放脚本的目录
mkdir -p /server/scripts

#编辑脚本内容
cat >/server/scripts/check_split_brain.sh <<'EOF'
#!/bin/sh
lb01_vip=10.0.0.10
lb01_ip=10.0.0.11
while true;do
    ping -c 2 -W 3 $lb01_ip &>/dev/null
    if [ $? -eq 0 -a `ip add|grep "$lb01_vip"|wc -l` -eq 1 ];then
        echo "keepalived出现了脑裂!!!"
    else
        echo "keepalived完全objk"
    fi
sleep 5
done
EOF

运行脚本,因为此时是脑裂的,keepalived主备上都有VIP,因为前期会报出现脑裂,当把主备上的firewalld关掉时就没有问题了

#脚本会一直运行,ctrl+c停止
$ sh /server/scripts/check_split_brain.sh
keepalived出现了脑裂!!!
keepalived出现了脑裂!!!
keepalived出现了脑裂!!!
keepalived出现了脑裂!!!
keepalived出现了脑裂!!!
keepalived出现了脑裂!!!
keepalived出现了脑裂!!!
keepalived完全objk
keepalived完全objk

五、源码安装keepalived

keepalived源码包官方下载地址

5.1 下载包

wget https://www.keepalived.org/software/keepalived-2.1.2.tar.gz

5.2 编译安装

安装依赖包

yum -y install openssl openssl-devel libnl3-devel pcre-devel

解压缩包

tar xf keepalived-2.1.2.tar.gz && cd keepalived-2.1.2

编译安装

./configure --prefix=/usr/local/keepalived
make && make install

5.3 配置keepalived

5.3.1 软连接keepalived命令

ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin

验证

$ keepalived -v
Keepalived v2.1.2 (06/14,2020)

Copyright(C) 2001-2020 Alexandre Cassen, <acassen@gmail.com>

Built with kernel headers for Linux 3.10.0
Running on Linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020

configure options: --prefix=/etc/keepalived --sysconfdir=/etc/keepalived

Config options:  LVS VRRP VRRP_AUTH OLD_CHKSUM_COMPAT FIB_ROUTING

System options:  PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV6_ADVANCED_API RTA_ENCAP RTA_EXPIRES RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTA_VIA FRA_OIFNAME IFA_FLAGS IP_MULTICAST_ALL NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION VRRP_VMAC IFLA_LINK_NETNSID CN_PROC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE SO_MARK SCHED_RESET_ON_FORK

5.3.2 keepalived.service文件

⚠️keepalived编译安装完成后会自动生成文件/usr/lib/systemd/system/keepalived.service

5.3.3 拷贝安装目录下keepalived配置文件

否则会报错Unable to find configuration file /etc/keepalived/keepalived.conf (glob returned 3)

mkdir /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

5.3.4 拷贝启动文件(可选)

cp /root/keepalived-2.1.2/keepalived/etc/init.d/keepalived /etc/init.d

5.3.5 启动keepalived并设置开机自启

systemctl daemon-reload && systemctl enable keepalived && systemctl start keepalived
泡泡吐肥皂o © gitbook.pptfz.top 2021 all right reserved,powered by Gitbook文件修订时间: 秃笔南波湾!!!

results matching ""

    No results matching ""