Using Unicorn Worker Killer to help reduce queue backlog on Heroku with Unicorn

In light of the update from Heroku CTO Adam Wiggins and having been part of the efforts to test the larger Dynos with Unicorn on Heroku I felt it necessary to share Unicorn Worker Killer; a tool we have found indispensable.

Based on configurable thresholds for memory and number of requests received it will kill off a Unicorn Worker. For those of you on Heroku this is valuable in that it then returns all the requests back to the Heroku random routing. While not an exact correlation between too many requests and memory size it does help keep individual workers from becoming too overloaded.

Add this to your Gemfile:

gem 'unicorn-worker-killer'

The gem suggests that you configure your `config.ru` file for the thresholds. This can be cumbersome on Heroku if you need to test out different settings.

Thankfully you can also control the thresholds via environment variables. In config.ru do:

max_request_min =  ENV['MAX_REQUEST_MIN'].to_i || 3072
max_request_max =  ENV['MAX_REQUEST_MAX'].to_i || 4096

# Max requests per worker
use Unicorn::WorkerKiller::MaxRequests, max_request_min, max_request_max

oom_min = ((ENV['OOM_MIN'].to_i || 192) * (1024**2))
oom_max = ((ENV['OOM_MAX'].to_i || 256) * (1024**2))

# Max memory size (RSS) per worker
use Unicorn::WorkerKiller::Oom, oom_min, oom_max

The run this on the heroku command line:

heroku config:add OOM_MAX=256 memory_limit_min =192 MAX_REQUEST_MIN=3072 MAX_REQUEST_MAX=4096 -a unicon-ttm-sandbox

That will add in the variables needed to control the thresholds. The example shows the defaults though we found dropping the OOM_MAX to 216 worked best

Heroku: Scaling Dynos at night and in the morning

I needed a rake task to auto scale Dynos on Heroku; wind them down at night and wind them up in the morning. I found this helpful post on Stackoverflow. I made a few modifications and would up with this:

namespace :scale_dynos do
  require 'heroku-api'
  desc "scales up dynos"
  task :up do
    dyno_max = [6,0].include?(Time.now.wday) ? ENV['WEEKEND_DYNO_MAX'] : ENV['DYNO_MAX']
    heroku = Heroku::API.new(:api_key => ENV['HEROKU_API_KEY'])
    heroku.post_ps_scale(ENV['APP_NAME'], 'web', dyno_max.to_i)
  end

  desc "scales down dynos"
  task :down do
    dyno_min = ENV['DYNO_MIN']
    heroku = Heroku::API.new(:api_key =>  ENV['HEROKU_API_KEY'])
    heroku.post_ps_scale(ENV['APP_NAME'], 'web', dyno_min.to_i)
  end
end

It adjusts for the weekend to scale to a different level if you set the ENV varaible. Additionally here is how I set the Heroku config vars. The add command does them all at once but the remove goes through one by one, removes the VAR and restarts the app.

heroku config:add DYNO_MIN=1 DYNO_MAX=25 WEEKEND_DYNO_MAX=1 APP_NAME=<your app name> HEROKU_API_KEY=<your api key> -a <your app name>
heroku config:remove DYNO_MIN DYNO_MAX WEEKEND_DYNO_MAX APP_NAME HEROKU_API_KEY -a <your app name>

You will need the heroku-api gem. I also set the Gemfile entry to this:

gem 'heroku-api', :require => false

Since the rake task is called twice a day there was no need to load the gem into memory for the rest of the time.