HAProxy & Lua: Fifo and asynchronous actions

In some cases, it will be great to process asynchronously some task. Maybe our email server is very slow and you dont want to add this server processing time to the response time of the HTTP request.

The solution is provided in two parts:

• A Lua FIFO
• An asychronous action action using tasks and the FIFO.

The FIFO

The fifo is a simple script inspired from the "PIL" manual. There is the library:

-- This library provides fifo functions
--
-- Usage:
--
--   fifo = Fifo.new()
--   fifo:push(data)
--   daa = fifo:pop()

Fifo = {}
Fifo.meta = {}
Fifo.meta.__index = {}
Fifo.new = function()
        local fifo = {}
        fifo.first = 1 -- Always the first data
        fifo.last = 1 -- Alway the last available + 1
        fifo.data = {}
        setmetatable(fifo, Fifo.meta)
        return fifo
end
Fifo.meta.__index.push = function(fifo, data)
        fifo.data[fifo.last] = data
        fifo.last = fifo.last + 1
end
Fifo.meta.__index.pop = function(fifo, data)
        if fifo.first == fifo.last then
                return nil
        end
        local data = fifo.data[fifo.first]
        fifo.data[fifo.first] = nil
        fifo.first = fifo.first + 1
        if fifo.first == fifo.last then
                fifo.first = 1
                fifo.last = 1
        end
        return data
end

The usage of this library is easy. Create new Fifo object. Push and pop elements. Like this

require("fifo")
fifo = Fifo.new()
fifo:push("a")
fifo:push("b")
fifo:push("c")
print(fifo:pop())
print(fifo:pop())
print(fifo:pop())
print(fifo:pop())

This Little sample displays:

$ lua ./test.lua
a
b
c
nil

Asynchronous tasks with Lua & HAProxy

Now, we will use this library to stack task executed asynchronously. We use:

• A FIFO for storing tasks
• An HAProxy Lua task for executing it
• The email library to send email

Fisrt step is initializing the FIFO. The FIFO is initialized in the main part of the Lua file. It it just:

require("fifo")

fifo_email = Fifo.new()

The second step is creation an action which send emails. This action put data where sent to the user. In this example, data is a copy of received request.

core.register_action("async_send_email", { "tcp-req", "http-req", "tcp-res", "http-res" }, function(txn)
   fifo_email:push("Request received:\n" .. txn.req:dup())
end)

And now, the task function which effectively send the data. This function pop the FIFO each seconds looking for jobs.

core.register_task(function()
   local ret
   local reason
   local server = "127.0.0.1"
   local port = 25
   local domain = "arpalert.org"
   local from = "haproxy@arpalert.org"
   local to = "admin@arpalert.org"

   while true do
      -- Process queue
      local data = fifo_email:pop()
      if data == nil then
         core.sleep(1)
      else
         -- Execute action
         local msg = "From: " .. from .. "\r\n" ..
                "To: " .. to .. "\r\n" ..
                "Subject: test - " .. os.date() .. "\r\n" ..
                "\r\n" ..
                data .. "\r\n"
         ret, reason = smtp_send_email(server, port, domain, from, to, msg);
         if ret == false then
            txn:Warning("Can't send email: " .. reason)
         end
      end
   end
end)

Finaly a little bit of haproxy configuration:

global
   lua-load samples.lua
   stats socket /tmp/haproxy.sock mode 644 level admin

defaults
   timeout client 1m
   timeout server 1m

listen sample4
   mode http
   bind *:10040
   http-request lua.async_send_email
   http-request redirect location /ok

That's all