Date Fri 06 May 2016 Tags Collectd / FreeBSD / Ports / Net-Mgmt / Patch / Collectd / OpenTSDB / NginX / LUA

If you, like me, are using net-mgmt/collectd5 and databases/opentsdb to collect your metrics, you probably are using write_tsdb to push your metrics to OpenTSDB.

Sometimes you, like me, find frustrating how collectd set its metrics' names and you would like to move part of the metric's name (i.e. network interface name) to a tag (so that you can aggregate using it).

Keeping this as simple as possible we can let NginX (using its Lua engine) to do this job for us, let's see how.

First of all, we need to enable WITH_CURL in net-mgmt/collectd5 because of write_http and we have to enable write_http output plugin in /usr/local/etc/collectd.conf:

LoadPlugin write_http <Plugin "write_http" > <Node "tsdb" > URL "http://127.0.0.1:9090/tsdb" Format "JSON" </Node> </Plugin>

As you can see we are instructing collectd to publish metrics via HTTP using a JSON format to 127.0.0.1:9090/tsdb, where there is a nginx listening with this config file:

server { listen *:9090 ; access_log /var/log/nginx/tsdb_access.log main ; error_log /var/log/nginx/tsdb_error.log ; location /tsdb { rewrite_by_lua_file /usr/local/etc/nginx/sites/tsdb_proxy.lua ; } location / { return 405 ; } }

where /usr/local/etc/nginx/sites/tsdb_proxy.lua is translating our metrics:

local cjson = require( "cjson" ) if ngx.req.get_method() == "POST" then ngx.req.read_body() local params = ngx.req.get_post_args() -- Define a prefix local metric_prefix = "my.own.metric" -- Connect to OpenTSDB local sock = ngx.socket.connect("127.0.0.1", 4242) local tsdb_payload = "" for k,v in pairs(params) do local k_tbl = cjson.decode(k) local count_tbl = 1 for _ in pairs(k_tbl) do local metric_host = k_tbl[count_tbl]['host'] local metric_name_tmp = k_tbl[count_tbl]['plugin'] local metric_tstamp = math.floor(k_tbl[count_tbl]['time']) local metric_name = "" local metric_value = 0 local metric_tags = "host=" .. metric_host if metric_name_tmp == "cpu" or metric_name_tmp == "memory" then metric_name = metric_name_tmp .. "." .. k_tbl[count_tbl]['type_instance'] .. "." .. k_tbl[count_tbl]['type'] metric_value = k_tbl[count_tbl]['values'][1] tsdb_payload = tsdb_payload .. "put " .. metric_prefix .. "." .. metric_name .. " " .. metric_tstamp .. " " .. metric_value .. " " .. metric_tags .. "\r

" elseif metric_name_tmp == "interface" then local interface_name = k_tbl[count_tbl]['plugin_instance'] metric_tags = metric_tags .. " interface=" .. interface_name for i, v in ipairs(k_tbl[count_tbl]['dsnames']) do metric_name = metric_name_tmp .. "." .. k_tbl[count_tbl]['type'] .. "." .. v metric_value = k_tbl[count_tbl]['values'][i] tsdb_payload = tsdb_payload .. "put " .. metric_prefix .. "." .. metric_name .. " " .. metric_tstamp .. " " .. metric_value .. " " .. metric_tags .. "\r

" end elseif metric_name_tmp == "load" then for i, v in ipairs(k_tbl[count_tbl]['dsnames']) do metric_name = metric_name_tmp .. "." .. k_tbl[count_tbl]['type_instance'] .. "." .. v metric_value = k_tbl[count_tbl]['values'][i] tsdb_payload = tsdb_payload .. "put " .. metric_prefix .. "." .. metric_name .. " " .. metric_tstamp .. " " .. metric_value .. " " .. metric_tags .. "\r

" end elseif metric_name_tmp == "statsd" then --[[ Here we will include logic for metrics from statsd --]] else ngx.log(ngx.STDERR, "[WARNING]: Unknown metric_name " .. metric_name) end count_tbl = count_tbl + 1 end end if tsdb_payload ~= "" then sock:send(tsdb_payload) sock:setkeepalive() end end -- All safe, let's get out of here ngx.exit(200)

This will translate this JSON:

[{"values":[6584721,4633500],"dstypes":["derive","derive"],"dsnames":["rx","tx"],"time":1460360931.658,"interval":10.000,"host":"my.own.host","plugin":"interface","plugin_instance":"vtnet0","type":"if_packets","type_instance":""},[...],{"values":[0,0],"dstypes":["derive","derive"],"dsnames":["rx","tx"],"time":1460360931.658,"interval":10.000,"host":"my.own.host","plugin":"interface","plugin_instance":"vtnet0","type":"if_errors","type_instance":""}]

into this:

put my.own.metric.interface.if_packets.rx 1460360931 6584721 host=my.own.host interface=vtnet0 put my.own.metric.interface.if_packets.tx 1460360931 4633500 host=my.own.host interface=vtnet0 put my.own.metric.interface.if_errors.rx 1460360931 0 host=my.own.host interface=vtnet0 put my.own.metric.interface.if_errors.tx 1460360931 0 host=my.own.host interface=vtnet0

That's better than:

put my.own.metric.interface.vtnet0.if_packets.rx 1460360931 6584721 host=my.own.host put my.own.metric.interface.vtnet0.if_packets.tx 1460360931 4633500 host=my.own.host put my.own.metric.interface.vtnet0.if_errors.rx 1460360931 0 host=my.own.host put my.own.metric.interface.vtnet0.if_errors.tx 1460360931 0 host=my.own.host

As you may notice, we used the cjson library in tsdb_proxy.lua, so we need to install devel/lua-cjson, too.