Skip to Content
Skip Breadcrumb

How does the SNI Routing works in HAProxy

Posted on Modified on by Aleksandar Lazic

I use the excellent service from Beamy-Lake to have my own Nextcloud and ejabberd instance. As I travel a lot I faced the problem that in some WIFI’s some ports are forbidden for outgoing communication 😱. The solution is to use a Software which can handle TCP and HTTP via port 443 so that I can use my nextcloud and my jabber client on the same port.

Introduction

TLS/SSL

Today’s communication should be done via Transport Layer Security (TLS) Protocol Version 1.3 or The Transport Layer Security (TLS) Protocol Version 1.2.

The encrypted communication is good for the people as the Information’s which are transported are not easy readable on the wire. This means that the network packages which are in the Internet are more or less secure. This implies that the network package which reaches the target Server have only some small parts of unencrypted data.

HTTP

At HTTP level was a common decision criteria for a Webserver which hosts several websites on the same Server is the Host Header (A.1.1. Multihomed Web Servers)

The HTTP Server (HAProxy, NGINX, Caddyserver ,…) parse the HTTP protocol and decides which route or content should be used or deliverd.

This techniques is used for a long time and it’s rock solid.

HTTP is here a short cut for all HTTP versions which supports Host Headers (HTTP/1.1,HTTP2).

The Problem

Now let me explain the problem which I solve with the HAProxy SNI Routing.

To be able to read the Host Header the Server must read the HTTP protocol and therefore it must be decrypt the TLS package. But that means that the listener (frontend) must have all certificates which is sometimes not possible.

Luckily there is a Transport Layer Security (TLS) Extensions: Extension Definitions which have the Server Name Indication defined and is available since January 2011.

The generic Solution

This SNI (Server Name Indication) is part of the (extended) client hello which is plain text. Now as the client can tell the server which Host the client want’s to reach the server can decide which route or content should be deliverd.

In Kubernetes are several Ingress Controllers based on HAProxy.

NGINX use for the same function the Module ngx_stream_ssl_preread_module.

Envoy use for the same function the TLS Inspector see How do I setup SNI

The OpenShift router based on The HAProxy Template Router works exactly as described in the HAProxy Solution below.

HAProxy Solution

Picture

HAProxy SNI Routing

Prosa

I describe here the picture above with the configuration below.

In the picture is the TCP frontend public_ssl the main entry point for the port 443.

As you can see this frontend section is pretty small because of the power of HAProxies map feature. Based on the req.ssl_sni is it possible to decide different backends for different SNI which must be able to handle the TLS Handshake.

In this case I have only one backend backend be_sni_xmpp before HAProxy forward the request to the default_backend be_sni.

The backend be_sni forwards the request to the frontend https-in on the same server, but this could be any destination which HAProxy supports. The request will now be decrypted in the http mode as the listener (frontend https-in) have the required certificates and key to decrypt the request. It’s the same flow for the listner (listen xmppc2s-backend).

HAProxy Config

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log stdout format raw daemon debug
    log-send-hostname cloud.DOMAIN

    maxconn     5000

    # ssl-default-bind-options ssl-min-ver TLSv1.0 no-tls-tickets
    tune.ssl.default-dh-param 3072

    # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy-1.8.0&openssl=1.1.0i&hsts=yes&profile=modern
    # set default parameters to the intermediate configuration
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.1 no-tls-tickets

    ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-server-options ssl-min-ver TLSv1.1 no-tls-tickets
    
    # https://www.haproxy.com/blog/dynamic-configuration-haproxy-runtime-api/
    stats socket ipv4@127.0.0.1:9999 level admin
    stats socket /var/run/haproxy.sock mode 666 level admin
    stats timeout 2m

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    tcp
    log                     global
    option                  dontlognull
    #option                  logasap
    option                  srvtcpka
    option                  log-separate-errors
    retries                 3
    timeout http-request    10s
    timeout queue           2m
    timeout connect         10s
    timeout client          5m
    timeout server          5m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 750

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------

##
## Frontend for HTTP
##
frontend http-in
    bind :::80 v4v6
    mode http
    option httplog

    tcp-request inspect-delay 5s
    tcp-request content accept if HTTP

    # redirect http to https .
    http-request redirect scheme https unless { ssl_fc }

##
## Frontend for HTTPS
##
frontend public_ssl

    bind :::443 v4v6 

    option tcplog
    log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq ssl_fc_has_sni '%[ssl_fc_has_sni]' sni:'%[capture.req.hdr(0)]' ssl_fc_sni '%[ssl_fc_sni]' ssl_fc_protocol '%[ssl_fc_protocol]' ssl_bc '%[ssl_bc]' ssl_bc_alpn '%[ssl_bc_alpn]' ssl_bc_protocol '%[ssl_bc_protocol]' ssl_c_i_dn '%[ssl_c_i_dn()]' ssl_c_s_dn '%[ssl_c_s_dn()]' ssl_f_i_dn '%[ssl_f_i_dn()]' ssl_f_s_dn '%[ssl_f_s_dn]' ssl_fc_cipher '%[ssl_fc_cipher]' "

    tcp-request inspect-delay 5s
    tcp-request content capture req.ssl_sni len 25
    tcp-request content accept if { req.ssl_hello_type 1 }
	
    # https://www.haproxy.com/blog/introduction-to-haproxy-maps/
    use_backend %[req.ssl_sni,lower,map(/usr/local/etc/haproxy/tcp-domain2backend-map.txt)]

    default_backend be_sni

##########################################################################
# TLS SNI
#
# When using SNI we can terminate encryption with dedicated certificates.
##########################################################################
backend be_sni
  server fe_sni 127.0.0.1:10444 weight 10 send-proxy-v2-ssl-cn

backend be_sni_xmpp
  server fe_sn_xmpp 127.0.0.1:10442 weight 10 send-proxy-v2-ssl-cn

# handle https incomming
frontend https-in

    # terminate ssl 
    bind 127.0.0.1:10444 accept-proxy ssl strict-sni alpn h2,http/1.1 crt /usr/local/etc/haproxy-certs

    mode http
    option forwardfor
    option httplog
    option http-use-htx
    option http-ignore-probes

    # Strip off Proxy headers to prevent HTTpoxy (https://httpoxy.org/)
    http-request del-header Proxy

    http-request set-header Host %[req.hdr(host),lower]
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Host %[req.hdr(host),lower]
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request set-header X-Forwarded-Proto-Version h2 if { ssl_fc_alpn -i h2 }
    http-request add-header Forwarded for=\"[%[src]]\";host=%[req.hdr(host),lower];proto=%[req.hdr(X-Forwarded-Proto)];proto-version=%[req.hdr(X-Forwarded-Proto-Version)]

    # Add hsts https://www.haproxy.com/blog/haproxy-and-http-strict-transport-security-hsts-header-in-http-redirects/
    # http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"

    # https://www.haproxy.com/blog/introduction-to-haproxy-maps/
    use_backend %[req.hdr(host),lower,map(/usr/local/etc/haproxy/http-domain2backend-map.txt)]

#---------------------------------------------------------------------
#  backends
#---------------------------------------------------------------------
## backend for cloud.DOMAIN
backend nextcloud-backend
    mode http
    option http-use-htx
    option httpchk GET / HTTP/1.1\r\nHost:\ BACKEND_VHOST
    server short-cloud 127.0.0.1:81 check 


## backend for dashboard.DOMAIN
backend dashboard-backend
    mode http
    option http-use-htx
    server short-cloud 127.0.0.1:82 check

## backend for upload.DOMAIN
backend httpupload-backend
    log global
    mode http
    option http-use-htx
    server short-cloud 127.0.0.1:8443 check

## backend for DOMAIN (XMPP C2S) direct TLS/SSL
listen xmppc2s-backend
    
    bind 127.0.0.1:10442 accept-proxy ssl strict-sni crt /usr/local/etc/haproxy-certs

    log global
    log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq ssl_fc_has_sni '%[ssl_fc_has_sni]' sni:'%[capture.req.hdr(0)]' ssl_fc_sni '%[ssl_fc_sni]' ssl_fc_protocol '%[ssl_fc_protocol]' ssl_bc '%[ssl_bc]' ssl_bc_alpn '%[ssl_bc_alpn]' ssl_bc_protocol '%[ssl_bc_protocol]' ssl_c_i_dn '%[ssl_c_i_dn()]' ssl_c_s_dn '%[ssl_c_s_dn()]' ssl_f_i_dn '%[ssl_f_i_dn()]' ssl_f_s_dn '%[ssl_f_s_dn]' ssl_fc_cipher '%[ssl_fc_cipher]' "

    # if you want to have the client IP in ejabberd
    # add send-proxy-v2-ssl-cn and in ejabberd use_proxy_protocol: true
    server me2d-cloud 127.0.0.1:5223 check ssl check-ssl verify none check-sni str('DOMAIN') sni str('DOMAIN') ssl-min-ver TLSv1.2

#---------------------------------------------------------------------
#  stats page is hosted at different port
#---------------------------------------------------------------------
listen stats
  bind *:10000
  mode http
  stats enable
  stats hide-version
  stats realm Haproxy\ Statistics
  stats uri /stats
  stats auth "${STAT_USER}:${STAT_PASS}"

TCP Map

In the file tcp-domain2backend-map.txt is defined which domain maps to which backend on the TCP SNI level. I strongly suggest to read Introduction to HAProxy Maps

jabber.mydomain.im be_sni_xmpp

HTTP Map

In the file http-domain2backend-map.txt is defined which domain maps to which backend on the TLS decrypted level. I strongly suggest to read Introduction to HAProxy Maps

# http backends
nextcloud.MyDomain.com nextcloud-backend
dashboard.MyDomain.com dashboard-backend 
jabupload.MyDomain.com httpupload-backend

Run

The haproxy is running via podman from my haproxy image as it supports TLS 1.3. This images is based on the following source haproxy19-centos.
The flags are documented in podman-run(1)

The configuration files are all in /haproxy/etc and the Certificates are all in /haproxy/certs. In env_stats_auth.txt is the username and password for stats auth "${STAT_USER}:${STAT_PASS}".

 podman run -dt --name=my_haproxy \
   --expose 80-80 --expose 443-443 --expose 9999-9999 --expose 10000-10000 \
   -p 80-80 -p 443-443 -p 9999-9999 -p 10000-10000 \
   --network host \
   -v /haproxy/etc:/usr/local/etc/haproxy:ro \
   -v /haproxy/certs:/usr/local/etc/haproxy-certs:ro \
   --env-file /haproxy/etc/env_stats_auth.txt \
   me2digital/haproxy19 haproxy -f /usr/local/etc/haproxy/haproxy.cfg

SSL labs output

After all this work and tuning I was able to get a A+ on the SSL Labs Report with TLS 1.3 available.

SSL Labs Report

SSL Labs Report Config

You can contact me for any further questions and orders