At BackerKit, we occasionally see high volumes of traffic from malicious clients. (Kickstarter has faced a similar problem.) These DDoS attacks result in degraded performance and frustrate our customers. Not cool!

We implemented Kickstarter’s Rack::Attack and configured constraints on the number of requests allowed in a time period based on IP address on our troublesome endpoints. Yay, problem solved!

Like most tools, Rack::Attack requires tuning; our initial stab at configuration led to customers being blocked. We needed a way to clear out blocked IPs that were friendly.

Unblocking Friendlies

We started with tooling developers could use to manually clear a key. Keys of blocked IPs look like rack::attack:allow2ban:ban:104.62.144.205 . Given a target IP address we could go find that key in our cache (we use Redis) and delete it.

Our first manual step was to filter the Rails’ cache’s keys and find ones used by Rack::Attack that includes our target IP to unblock.

Rails.cache.data.keys .find_all{ |k| k.include?(':ban:') } .find_all{ |x| x.include?('104.62.144.205') }

We could then delete the found Redis key. Note this approach will work on only cache backends supporting the data method. Notably, ActiveSupport::Cache::FileStore (frequently used in development) does not support it. In addition, this operation is O(N) so beware of performance issues on large caches.

We wanted to type less and make fewer mistakes, so we wrapped some of that logic into the following helper:

# config/initializers/rack_attack.rb module BannedIpGetters def banned_ips with do |conn| return conn.keys.find_all { |x| x.include?(':ban:') } end end end ActiveSupport::Cache::RedisStore.include(BannedIpGetters)

the above helper lets us write:

ip_key_to_unblock = Rails.cache.banned_ips.find_all{|x| x.include?('104.62.144.205')} Rails.cache.delete(ip_key_to_unblock)

Empowering our Coworkers

Fixing these issues became time-consuming for our dev team, so we created a simple dashboard for our coworkers to use. The dashboard lets them see which IPs have been blacklisted and provides a button to unblock a given IP.

The controller presents an index page with a list of banned IPs and an endpoint that allows for deletion of a blocked key:

class Staff::RackAttacksController < Staff::BaseController def index ips = Rails.cache.data.keys.find_all { |x| x.include?('rack::attack') }.sort @banned_ips, @other_ips = ips.partition { |x| x.include?(':ban:') } end def destroy if params[:ip].starts_with?('rack::attack') Rails.cache.delete(params[:ip]) end redirect_to :back, notice: "#{params[:ip]} unblocked" end end

Our view looks like:

<% @banned_ips.each do |ip| %> <%= ip %> <%= link_to 'Lookup', "http://ip-lookup.net/?ip=#{ip.split(':').last}", target:'_blank' %> <%= link_to 'Unblock', staff_rack_attack_path(id: 1, ip: ip), method: :delete %> <% end %>

Now our non-technical teammates can go to this admin page and quickly find out what Rack::Attack is doing.

Next Steps