HAProxy, Lua & Redis: Basic usage

This article shows a basic usage of Redis with HAProxy and Lua. This way is absolutely expensive because a network connection is open for each request.

I suppose that a local Redis server is installed.

First step is getting the Redis project. Checkout this project in the directory containing HAProxy Lua scripts. The lin to the libaray and the doc are here: https://github.com/nrk/redis-lua.

$ git clone https://github.com/nrk/redis-lua.git
Cloning into 'redis-lua'...
remote: Counting objects: 1439, done.
remote: Total 1439 (delta 0), reused 0 (delta 0), pack-reused 1439
Receiving objects: 100% (1439/1439), 337.30 KiB | 0 bytes/s, done.
Resolving deltas: 100% (647/647), done.
Checking connectivity... done.

For using this library in Lua script, we must adapt the search Lua library path, and obviously load the package.

package.path  = package.path  .. ";redis-lua/src/?.lua"

redis = require("redis")

Now we want to use redis for doing something. Two attention points:

• First, we must use HAProxy cosocket in place of LuaSocket package. This is done in the configuration of the connexion. Look for {socket=tcp}
• Second, the Lua call must be protected. This Redis library doesn't return error: it directly fail :-(. We must catch the fail for a clean termination of the process. Look for pcall(client.incrby, client, ip, 1)
• Third, never use the quit() functions. This function call the LuaSocket function shutdown() which is not implemented in HAProxy.

We register an action which perform accounting on the IP source.

core.register_action("redis-accounting", { "http-req", "http-res", "tcp-req", "tcp-res" }, function(txn)

        -- create and connect new tcp socket
        local tcp = core.tcp();
        if tcp == nil then
                return
        end
        tcp:settimeout(1);
        if tcp:connect("127.0.0.1", 6379) == nil then
                return
        end

        -- use the redis library with this new socket
        local client = redis.connect({socket=tcp});

        -- Send redis accouting command
        local ip = txn.sf:src()
        pcall(client.incrby, client, ip, 1);

        -- Close connection
        tcp:close()
end)

And the HAProxy configuration

global
        lua-load samples.lua
        stats socket /tmp/haproxy.sock mode 644 level admin
        tune.ssl.default-dh-param 2048

defaults
        timeout client 1m
        timeout server 1m

listen sample5
        mode http
        bind *:10050
        http-request lua.redis-accounting
        http-request redirect location /ok

Now you can test:

$ redis-cli get '127.0.0.1'
(nil)
$ curl -s http://127.0.0.1:10050/
$ redis-cli get '127.0.0.1'
"1"
$ curl -s http://127.0.0.1:10050/
$ redis-cli get '127.0.0.1'
"2"
$ curl -s http://127.0.0.1:10050/
$ redis-cli get '127.0.0.1'
"3"

benchmark

I bench this solution on my laptop. It have a i7-4600U CPU @ 2.10GHz. 2 core, 4 threads. I reserve one core for haproxy, one thread for the injector, and one thread fr redis. The setp is:

• CPU1: Redis-4.0.8
• CPU2: Injector: http://1wt.eu/tools/inject/
• CPU3: HAProxy-1.7.8 with Lua support

A reference test: With the same HAProxy configuration without the Lua process (# http-request lua.redis-accounting), we reach about 70 000 HTTP request per second with an approximate ratio of CPU consummation 25% user and 75% system. Note that the test is limited by the injector who reach 100% CPU.
The results are not surprising:

We are limited by the HAproxy CPU. The consomation is about 98% user for the HAProxy process. Redis and the injector does nothing: about 15% cpu fr redis and 10% for the injector.

HAProxy process 4300 requests / second. HAProxy is very slow because the lib Redis is initialized to each request, the initialization takes a lot of CPU. In other way, the TCP connection is also initialized for each connection.