------------------- HAProxy ------------------- Yaohong_Lin 2019/03/05 ============================================================ 1.TCP proxy SSH layer 4 ============================================================ +----+----+ +----+----+ | clients | | RS | +----+----+ +----+----+ | GW=10.10.10.1 | IP=10.10.10.20 | | SSH PORT=22 ------+---+------------+--+-----> pass : 10.10.10.17:9001 --TO-- 10.10.10.20:22 | +--+--+ | LB1 | +-----+ | IP=10.10.10.17 | | PORT=9001 | haproxy Config on haproxy (LB1): ------------------------ global chroot /application/haproxy/var/chroot daemon user haproxy group haproxy log 127.0.0.1:514 local0 warning pidfile /application/haproxy/var/run/haproxy.pid maxconn 4000 nbproc 8 defaults log global retries 3 option redispatch listen www bind 10.10.10.17:80 mode tcp balance roundrobin timeout connect 15s timeout server 15s server web01 10.10.10.20:20 check port 20 inter 5000 fall 5 =================================================== 2. Simple HTTP load-balancing with cookie insertion =================================================== A web application often saturates the front-end server with high CPU loads, due to the scripting language involved. It also relies on a back-end database which is not much loaded. User contexts are stored on the server itself, and not in the database, so that simply adding another server with simple IP/TCP load-balancing would not work. +-------+ |clients| clients and/or reverse-proxy +---+---+ | -+-----+--------+---- | _|_db +--+--+ (___) | web |---(___) +-----+ (___) 192.168.1.1 192.168.1.2 Replacing the web server with a bigger SMP system would cost much more than adding low-cost pizza boxes. The solution is to buy N cheap boxes and install the application on them. Install haproxy on the old one which will spread the load across the new boxes. 192.168.1.1 192.168.1.11-192.168.1.14 192.168.1.2 -------+-----------+-----+-----+-----+--------+--------- | | | | | _|_db +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___) | LB1 |------| A | | B | | C | | D |----(___) +-----+ +---+ +---+ +---+ +---+ (___) haproxy 4 cheap web servers Or shared VIP=192.168.1.1 192.168.1.3 192.168.1.4 192.168.1.11-192.168.1.14 192.168.1.2 -------+------------+-----------+-----+-----+-----+--------+---- | | | | | | _|_db +--+--+ +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___) | LB1 | | LB2 | | A | | B | | C | | D | (___) +-----+ +-----+ +---+ +---+ +---+ +---+ (___) haproxy haproxy 4 cheap web servers keepalived keepalived ================================== 3.Config on haproxy (LB1) : Listen ================================== global chroot /application/haproxy/var/chroot daemon user haproxy group haproxy log 127.0.0.1:514 local0 info pidfile /application/haproxy/var/run/haproxy.pid maxconn 4000 nbproc 8 defaults log global mode http retries 3 option redispatch timeout connect 5000ms timeout client 50000ms timeout server 50000ms #ulimit -n 65535 option abortonclose option dontlognull option httpclose listen linyaohong bind *:80 mode http stats enable stats hide-version stats uri /admin?status stats realm Private lands stats auth admin:111111 balance roundrobin option httpclose option forwardfor cookie SERVERID insert indirect timeout connect 15s timeout client 15s timeout server 15s #option httpchk #option httpchk GET /test/get.html #option httpchk HEAD /test/index.html HTTP/1.1\r\nHost:www.linyaohong.com option httpchk HEAD /test/index.html HTTP/1.0 server www01 10.10.10.20:80 cookie A check port 80 inter 5000 rise 3 fall 5 #server www01 10.10.10.20:80 cookie A check port 80 inter 5000 rise 3 fall 5 server www02 10.10.10.21:80 cookie A check port 80 inter 5000 rise 3 fall 5 #server www03 10.10.10.21:80 maxconn 2048 weight 8 check port 80 inter 5000 rise 3 fall 5 3.1 Config on haproxy (LB1) : frontend backend -------------------------------------------- global chroot /application/haproxy/var/chroot daemon user haproxy group haproxy log 127.0.0.1:514 local0 info pidfile /application/haproxy/var/run/haproxy.pid maxconn 4000 nbproc 8 defaults log global mode http retries 3 option redispatch timeout connect 5000ms timeout client 5000ms timeout server 5000ms option abortonclose option dontlognull option httpclose stats enable stats hide-version stats uri /admin?status stats realm Private lands stats auth admin:111111 frontend web bind 10.10.10.25:80 acl www_deamon hdr(host) -i www.linyaohong.com acl bbs_deamon hdr(host) -i bbs.linyaohong.com acl blog_deamon hdr(host) -i blog.linyaohong.com acl 123_deamon hdr(host) -i test.linyaohong.com redirect prefix https://www.baidu.com code 301 if 123_deamon use_backend blog if blog_deamon use_backend bbs if bbs_deamon default_backend www backend blog balance roundrobin option httpclose option forwardfor server www01 10.10.10.20:80 cookie A check port 80 inter 5000 rise 3 fall 5 #server www02 10.10.10.21:80 cookie A check port 80 inter 5000 rise 3 fall 5 backend bbs balance roundrobin option httpclose option forwardfor #server www01 10.10.10.20:80 cookie A check port 80 inter 5000 rise 3 fall 5 server www02 10.10.10.21:80 cookie A check port 80 inter 5000 rise 3 fall 5 backend www balance roundrobin option httpclose option forwardfor server www01 10.10.10.20:80 cookie A check port 80 inter 5000 rise 3 fall 5 server www02 10.10.10.21:80 cookie A check port 80 inter 5000 rise 3 fall 5 3.2 Config on haproxy (LB1) : frontend -uri ------------------------------------------- frontend web bind 10.10.10.25:80 acl www_static path_beg /nginx/ use_backend nginxtools if www_static acl www_php path_beg /php/ use_backend phptools if www_php acl www_java path_beg /resin/ use_backend javatools if www_java default_backend nginxtools 3.3 Config on haproxy (LB1) : frontend -extension ------------------------------------------------- acl www_static path_end .gif .png .jpg .css .js acl www_images url_reg \.(css|js|html|png|css?.*|js?.*)$ use_backend nginxtools if www_static or www_images default_backend nginxtools 3.4 Config on haproxy (LB1) : frontend -user-agent ------------------------------------------------- frontend web bind 10.10.10.25:80 acl iphone_users hdr_sub(user-agent) -i iphone redirect prefix http://iphone.linyaohong.com if iphone_users #use_backend iphone if iphone_users acl android_users hdr_sub(user-agent) -i android redirect prefix http://android.linyaohong.com if android_users #use_backend android if android acl valid_ip src 10.10.10.0/24 block if !valid_ip default_backend nginxtools from configuration: To select a different backend for requests to static contents on the "www" site and to every request on the "img", "video", "download" and "ftp" hosts : acl url_static path_beg /static /images /img /css acl url_static path_end .gif .png .jpg .css .js acl host_www hdr_beg(host) -i www acl host_static hdr_beg(host) -i img. video. download. ftp. # now use backend "static" for all static-only hosts, and for static urls # of host "www". Use backend "www" for the rest. use_backend static if host_static or host_www url_static use_backend www if host_www ================================== 4 Config on haproxy (LB1) : error ================================== errorfile Return a file contents instead of errors generated by HAProxy May be used in sections : defaults | frontend | listen | backend yes | yes | yes | yes Example : errorfile 400 /etc/haproxy/errorfiles/400badreq.http errorfile 403 /etc/haproxy/errorfiles/403forbid.http errorfile 503 /etc/haproxy/errorfiles/503sorry.http errorfile 503 /tmp/503.html errorloc errorloc 503 http://www.xxx.com/503.html errorloc 504 http://www.xxx.com errorloc 502 http://www.xxx.com but not uri 403 ============== Install proxy： ============== [ ! -f haproxy-1.6.14.tar.gz ] && wget https://www.haproxy.org/download/1.6/src/haproxy-1.6.14.tar.gz tar zxvf haproxy-1.6.14.tar.gz cd haproxy-1.6.14 yum -y install gcc gcc-c++ make TARGET=linux2628 ARCH=X86_64 make PREFIX=/server/application/haproxy-1.6.14 install ln -s /server/application/haproxy-1.6.14/ /server/application/haproxy [ -z "`grep "net.ipv4.ip_forward" /etc/sysctl.conf`" ] && echo net.ipv4.ip_forward = 1 >>/etc/sysctl.conf sysctl -p cd /server/application/haproxy mkdir -p bin conf logs var/run var/chroot echo local0.* /server/application/haproxy/logs/haproxy.log>>/etc/rsyslog.conf tail -1 /etc/rsyslog.conf sed -i s#'SYSLOGD_OPTIONS=""'#'SYSLOGD_OPTIONS="-c 2 -m 0 -r -x"'#g /etc/sysconfig/rsyslog tail -1 /etc/sysconfig/rsyslog sed -i s/'#$ModLoad imudp'/'$ModLoad imudp'/g /etc/rsyslog.conf sed -i s/'#$UDPServerRun 514'/'$UDPServerRun 514'/g /etc/rsyslog.conf systemctl restart rsyslog echo "/server/application/haproxy-1.6.14/bin/haproxy start" >>/etc/rc.local chmod +x /etc/rc.local ==================== Profile description： ==================== global -------------------------------------------------------------------------------------------------------------------------------- 修改haproxy的工作目录至指定目录并在放弃权限前执行chroot()操作，可以提升haproxy安全级别 chroot /application/haproxy/var/chroot 以守护进程的方式运行 daemon 指定运行haproxy的用户和组 user haproxy group haproxy 全局日志配置，log关键字，指定只用127.0.0.1上的rsyslog服务中的local0日志设备， 记录日志等级为emerg|alert|crit|err|warning|notice|info|debug log 127.0.0.1:514 local0 warning 定义haproxy的pid文件文件，方便管理和查看 pidfile /application/haproxy/var/run/haproxy.pid 定义haproxy的最大连接数 maxconn 4000 设置haproxy启动的进程数，该值的设置应该和服务器的CPU核心数一致，即常见的2颗8核心CPU的服务器，即共有16核心， 则可以将其值设置为：<=16 ，创建多个进程数，可以减少每个进程的任务队列，但是过多的进程数也可能会导致进程的崩溃。这里我设置为16 nbproc 1 设置最大打开的文件描述符数，在1.4的官方文档中提示，该值会自动计算，所以不建议进行设置 #ulimit -n 65536 defaults --------------------------------------------------------------------------------------------------------------------------------- 继承global中log的定义 log global mode语法：mode {http|tcp|health} 。http是七层模式，tcp是四层模式，health是健康检测，返回OK mode http 定义连接后端服务器的失败重连次数，连接失败此处超过 此值 后，将后端服务器标记为不可用 retries 3 当使用了cookie时，haproxy将会将其请求的后端服务器的serverID插入到cookie中，以保证会话的SESSION持久性； 而此时，如果后端的服务器宕掉了，但是客户端的cookie是不会刷新的， 如果设置此参数，将会将客户的请求强制定向到另外一个后端server上，以保证服务的正常。 option redispatch 设置成功连接到一台服务器的最长等待时间，默认单位是毫秒，老版本的haproxy使用 contimeout 5000 timeout connect 5000ms 35-120m 设置连接客户端发送数据时的成功连接最长等待时间，默认单位是毫秒，老版本haproxy使用 clitimeout 3000 timeout client 50000ms 35-120m 设置服务器端回应客户度数据发送的最长等待时间，默认单位是毫秒，老版本haproxy使用 srvtimeout 3000 timeout server 50000ms 35-120m 当服务器负载很高的时候，自动结束掉当前队列处理比较久的链接 option abortonclose 启用该项，日志中将不会记录空连接。所谓空连接就是在上游的负载均衡器或者监控系统为了探测该服务是否存活可用时，需要定期的连接或者获取某一固定的组件或页面，或者探测扫描端口是否在监听或开放等动作被称为空连接 option dontlognull 使用该参数，每处理完一个request时，haproxy都会去检查http头中的Connection的值，如果该值不是close，haproxy将会将其删除，如果该值为空将会添加为：Connection: close。 使每个客户端和服务器端在完成一次传输后都会主动关闭TCP连接。与该参数类似的另外一个参数是“option forceclose”，该参数的作用是强制关闭对外的服务通道，因为有的服务器端收到Connection: close时，也不会自动关闭TCP连接， 如果客户端也不关闭，连接就会一直处于打开，直到超时 option httpclose