Using Puma on Heroku instead of Unicorn

A few years back I wrote about using Unicorn on Heroku along with Unicorn Worker Killer. Oyr need was to increase the number of web requests an app could handle on Heroku. Puma tested faster at the time but did not offer worker processes. You have to use threads and for an app that was never intended to be threaded running on Ruby MRI it was too much work to adjust. Puma has since added the concept of Worker Processes and, given the performance advantages found with Puma it made sense to install it. Still, I was used to some variability with Unicorn and wanted to replicate some of the same things. This article from Heroku offered most of the help I needed though I did reduce the default thread count from 5 to 1. Additionally the section on the use of the Rack Timeout gem was helpful though I wound up doing this so that I could adjust the timeout via environment variable:

 Rack::Timeout.timeout = 20 # seconds  

What I did not find was helpful instructions on queue length. Puma defaults to 1024 and the previously mentioned article from Heroku cautions strongly against altering that queue length. Still, from previous performance testing on Heroku sometimes a shorter queue length is warranted (DO YOUR OWN TESTING!). This blog post originally suggested altering the queue length but again he cautions against it on Heroku. If you find that a shorter queue length helps then it is a setting in the config/puma.rb file:

 backlog = Integer(ENV['PUMA_BACKLOG'] || 20)  
 bind "tcp://0.0.0.0:#{port}?backlog=#{backlog}"   

Good luck and Heroku on

A rake task to automate deployment and database migration to Heroku

One of my major annoyances with Heroku is that the deploy does not automatically run any database migrations. Seriously?! So I ran into this tonight when I am trying to explain to a seasoned SA that while Heroku automatically will recompile your assets for you it will not do the migrations. A Google turned up this Stackoverflow question which in turn led to this gist. And I am excited to be trying it out.

Using Unicorn Worker Killer to help reduce queue backlog on Heroku with Unicorn

In light of the update from Heroku CTO Adam Wiggins and having been part of the efforts to test the larger Dynos with Unicorn on Heroku I felt it necessary to share Unicorn Worker Killer; a tool we have found indispensable.

Based on configurable thresholds for memory and number of requests received it will kill off a Unicorn Worker. For those of you on Heroku this is valuable in that it then returns all the requests back to the Heroku random routing. While not an exact correlation between too many requests and memory size it does help keep individual workers from becoming too overloaded.

Add this to your Gemfile:

gem 'unicorn-worker-killer'

The gem suggests that you configure your `config.ru` file for the thresholds. This can be cumbersome on Heroku if you need to test out different settings.

Thankfully you can also control the thresholds via environment variables. In config.ru do:

max_request_min =  ENV['MAX_REQUEST_MIN'].to_i || 3072
max_request_max =  ENV['MAX_REQUEST_MAX'].to_i || 4096

# Max requests per worker
use Unicorn::WorkerKiller::MaxRequests, max_request_min, max_request_max

oom_min = ((ENV['OOM_MIN'].to_i || 192) * (1024**2))
oom_max = ((ENV['OOM_MAX'].to_i || 256) * (1024**2))

# Max memory size (RSS) per worker
use Unicorn::WorkerKiller::Oom, oom_min, oom_max

The run this on the heroku command line:

heroku config:add OOM_MAX=256 memory_limit_min =192 MAX_REQUEST_MIN=3072 MAX_REQUEST_MAX=4096 -a unicon-ttm-sandbox

That will add in the variables needed to control the thresholds. The example shows the defaults though we found dropping the OOM_MAX to 216 worked best

Heroku plugin providing insight into the Postgres database for your application

Heroku provides a lovely plugin called heroku-pg-extras which can provide some insight into the inner workings of the postgres databases connected to your application. By default it will hit the database listed in the DATABASE_URL variable for Heroku. However you can specify the database using the database color url; e.g. HEROKU_POSTGRES_PURPLE_URL.

The plugin does the following:

• cache_hit - calculates your cache hit rate (effective databases are at 99% and up)
• index_usage - calculates your index hit rate (effective databases are at 99% and up)
• ps - view active queries with execution time
• locks - display queries with active locks
• blocking - display queries holding locks other queries are waiting to be released
• kill - -f,--force; terminates the connection in addition to cancelling the query
• total_index_size - show the total size of the indexes in MB
• index_size - show the size of the indexes in MB descending by size
• seq_scans - show the count of seq_scans by table descending by order
• long_running_queries - show queries taking longer than 5 minutes 
• bloat - show table and index bloat in your database ordered by most wasteful
• mandlebrot - show the mandelbrot set

These are based on the stats collections process used by postgres which has its own body of literature in collection and analysis starting with their main documentation. It may be worth opening a ticket with Heroku to clear the stats first as they accumulate over time. Changes to the application will not be readily apparent if you still have the stats from the previous app state. Therefore you may want to open a ticket with Heroku to clean the stats (you need to be a superuser to do this yourself and Heroku does not give you superuser access).

Disclaimer: I am a contributer to the project

Heroku with Unicorn backlog settings and performance

In light of this post from Rap Genius and subsequent blow up on Hacker News I decided to share what we did to play with the request queue on Unicorn & Heroku.

In the app/config/unicorn.rb we changed the backlog line to this:

:backlog => Integer(ENV['UNICORN_BACKLOG'] || 200)

And then we can alter the backlog as necessary via:

heroku config:set UNICORN_BACKLOG=25 -a <app_name>

We found 25 to be a sweet spot for two Unicorn Workers but it may be different with four (which we are experimenting with now). Again this is also relative to the app so your results may (and probably will) be different.

Update:

I neglected to mention that you have to move the port declaration from config.ru and put it in the unicorn.rb too. The full line should be something like this:

listen ENV['PORT'], :backlog => Integer(ENV['UNICORN_BACKLOG'] || 200)

Heroku: Scaling Dynos at night and in the morning

I needed a rake task to auto scale Dynos on Heroku; wind them down at night and wind them up in the morning. I found this helpful post on Stackoverflow. I made a few modifications and would up with this:

namespace :scale_dynos do
  require 'heroku-api'
  desc "scales up dynos"
  task :up do
    dyno_max = [6,0].include?(Time.now.wday) ? ENV['WEEKEND_DYNO_MAX'] : ENV['DYNO_MAX']
    heroku = Heroku::API.new(:api_key => ENV['HEROKU_API_KEY'])
    heroku.post_ps_scale(ENV['APP_NAME'], 'web', dyno_max.to_i)
  end

  desc "scales down dynos"
  task :down do
    dyno_min = ENV['DYNO_MIN']
    heroku = Heroku::API.new(:api_key =>  ENV['HEROKU_API_KEY'])
    heroku.post_ps_scale(ENV['APP_NAME'], 'web', dyno_min.to_i)
  end
end

It adjusts for the weekend to scale to a different level if you set the ENV varaible. Additionally here is how I set the Heroku config vars. The add command does them all at once but the remove goes through one by one, removes the VAR and restarts the app.

heroku config:add DYNO_MIN=1 DYNO_MAX=25 WEEKEND_DYNO_MAX=1 APP_NAME=<your app name> HEROKU_API_KEY=<your api key> -a <your app name>
heroku config:remove DYNO_MIN DYNO_MAX WEEKEND_DYNO_MAX APP_NAME HEROKU_API_KEY -a <your app name>

You will need the heroku-api gem. I also set the Gemfile entry to this:

gem 'heroku-api', :require => false

Since the rake task is called twice a day there was no need to load the gem into memory for the rest of the time.