HAProxy, Lua & Redis: Connection pool

In the previous post, we see how making simple connection to a Redis database. The performances are a little bit low. Now we will see how making a connection pool with Lua.

The advantages are:

• Redis initialisation only one time
• TCP connection  established one time.

This solution is more fast than the previous.

The first step is creating a redis wrapper with initialise and manage the Redis connection pool. This wrapper use the FIFO described here.

One of the big trap of this code is the increment of the number of connection established which is done before the effective establishment. The reason is a little bit tricky. This variable (r.nb_cur) is global (because the object is global) and it is shared by all process trying to establishing connection. When the command tcp:connect() is performed, the Lua give back the hand to HAProxy, and it may execute another Lua process which try to establish another connection.

If the connection accounting is not set before connection, other processes try to establish connection because the accounting is not yet incremented. The result is that the total amount of connections is upper than the maximum value of pool.

There is the code of the redis-wrapper file:

-- redis-wrapper : Add connection pool to the Lua Redis library
-- 
-- Copyright 2018 Thierry Fournier 
--
-- This lib is compliant with HAProxy cosockets
--
package.path  = package.path  .. ";redis-lua/src/?.lua"
require("fifo")

redis_wrapper = {}
redis_wrapper.redis = require("redis")

redis_wrapper.new = function(host, port, pool_sz)
        local r = {}
        r.host = host
        r.port = port
        r.pool = Fifo.new()
        r.nb_max = pool_sz
        r.nb_cur = 0
        r.nb_avail = 0
        r.nb_err = 0
        setmetatable(r, redis_wrapper.meta);
        return r
end

redis_wrapper.meta = {}
redis_wrapper.meta.__index = {}
redis_wrapper.meta.__index.new_conn = function(r)

        -- Limit the max number of connections
        if r.nb_cur >= r.nb_max then
                return nil
        end

        -- Increment the number of connexions before the real
        -- connexion, because the connection does a yield and
        -- another connexion may be started. If the creation
        -- fails decrement the counter.
        r.nb_cur = r.nb_cur + 1

        -- Redis session
        local sess = {}

        -- create and connect new tcp socket
        sess.tcp = core.tcp();
        sess.tcp:settimeout(1);
        if sess.tcp:connect(r.host, r.port) == nil then
                r.nb_cur = r.nb_cur - 1
                return nil
        end

        -- use the lib_redis library with this new socket
        sess.client = redis_wrapper.redis.connect({socket=sess.tcp});

        -- One more session created
        r.nb_avail = r.nb_avail + 1
        return sess

end

redis_wrapper.meta.__index.get = function(r, wait)
        local tspent = 0
        local conn
        while true do

                -- Get entry from pool
                conn = r.pool:pop()
                if conn ~= nil then
                        r.nb_avail = r.nb_avail - 1
                        return conn
                end

                -- Slot available: create new connection
                if r.nb_cur < r.nb_max then
                        conn = r:new_conn()
                        if conn ~= nil then
                                r.nb_avail = r.nb_avail - 1
                                return conn
                        end
                end

                -- no slot available wait a while
                if tspent >= wait then
                        return nil
                end
                core.msleep(50)
                tspent = tspent + 50
        end
end

redis_wrapper.meta.__index.release = function(r, conn)
        r.nb_avail = r.nb_avail + 1
        r.pool:push(conn)
end

redis_wrapper.meta.__index.renew = function(r, conn)
        if conn ~= nil then
                conn.tcp:close()
        end
        r.nb_cur = r.nb_cur - 1
        conn = r:new_conn()
        if conn ~= nil then
                r:release(conn)
        end
end

Now the usage of this code in  the main HAroxy Lua file. Note the function pcall. Is is explain int the previous post.

r_wrap = redis_wrapper.new("127.0.0.1", 6379, 20)

core.register_action("redis-accounting-v2", { "http-req", "http-res", "tcp-req", "tcp-res" }, function(txn)
        local conn
        local ip

        -- Get client information
        ip = txn.sf:src()

        -- Get lib_redis connection. If no connection avalaible, wait a bit.
        conn = r_wrap:get(1000)
        if conn ~= nil then
                local ret = pcall(conn.client.incrby, conn.client, ip, 1)
                if ret == false then
                        r_wrap:renew()
                else
                        r_wrap:release(conn)
                end
        end
end)

The HAProxy configuration file sample

global
        lua-load samples.lua
        stats socket /tmp/haproxy.sock mode 644 level admin
        tune.ssl.default-dh-param 2048

defaults
        timeout client 1m
        timeout server 1m

listen sample6
        mode http
        bind *:10060
        http-request lua.redis-accounting-v2
        http-request redirect location /ok

Benchmark

I bench this solution on my laptop. It have a i7-4600U CPU @ 2.10GHz. 2 core, 4 threads. I reserve one core for haproxy, one thread for the injector, and one thread fr redis. The setup is:

• CPU1: Redis-4.0.8
• CPU2: Injector: http://1wt.eu/tools/inject/
• CPU3: HAProxy-1.7.8 with Lua support

A reference test: With the same HAProxy configuration without the Lua process (# http-request lua.redis-accounting-v2), we reach about 70 000 HTTP request per second with an approximate ratio of CPU consummation 25% user and 75% system. Note that the test is limited by the injector who reach 100% CPU.

The result are quite better than the tests without the connection pool. We reach 33 500 HTTP requests per seconds (8x better).

We show a huge consummation of the CPU "user". The ratio is 66% user for 33% system. The reference is 25% user. I deduct that the gap from 25% to 66% is done by the Lua execution.

I try a last test without the lib Redis. I write himself the Redis protocol without the library. I reuse the connection pool. I alway initialize the Redis library, but I don't use it. Here the new code:

core.register_action("redis-accounting-v2-1", { "http-req", "http-res", "tcp-req", "tcp-res" }, function(txn)
        local conn
        local ip

        -- Get client information
        ip = txn.sf:src()

        -- Get lib_redis connection. If no connection avalaible, wait a bit.
        conn = r_wrap:get(1000)
        if conn ~= nil then
                if conn.tcp:send("*3\r\n$6\r\nINCRBY\r\n$9\r\n127.0.0.1\r\n$1\r\n1\r\n") == nil then
                        conn.tcp:close()
                        r_wrap:renew()
                elseif conn.tcp:receive() == nil then
                        conn.tcp:close()
                        r_wrap:renew()
                end
                r_wrap:release(conn)
        end
end)

The result is 15% better (38k vs 33k). 15% is a good gap, but writing Redis protocol from scratch is complicated. I conclude that the best way for "production" is using Redis library and pools.

Benchmark summary

Method	Result	Note
Calibration test: without Lua	70 000 req/s	25% user / 75% system, limited by injector
Basic usage of Redis	4 300 req/s	98% user / 2% system
Redis with pools	33 000 req/s	66% user / 33% system
Redis with pool and without Redis library	38 000 req/s	50% user / 50% system