代码之家 › 专栏 › 技术社区 › Nick ONeill

是否可以避免Heroku上的ActiveRecord::ConnectionTimeoutError?

sidekiq unicorn heroku activerecord ruby-on-rails

Nick ONeill · 技术社区 · 11 年前

在Heroku上,我有一个rails应用程序在运行,它既有两个网络dyno,也有一个工人dyno。我在Sidekiq上全天运行数千个工作任务,但偶尔会引发ActiveRecord::ConnectionTimeoutError(每天大约50次)。我已经按如下方式设置了我的独角兽服务器

worker_processes 4
timeout 30
preload_app true

before_fork do |server, worker|
    # As suggested here: https://devcenter.heroku.com/articles/rails-unicorn
    Signal.trap 'TERM' do
        puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
        Process.kill 'QUIT', Process.pid
    end

    if defined?(ActiveRecord::Base)
        ActiveRecord::Base.connection.disconnect!
    end
end

after_fork do |server,worker|
    if defined?(ActiveRecord::Base)
        config = Rails.application.config.database_configuration[Rails.env]
        config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
        config['pool']            = ENV['DB_POOL'] || 10
        ActiveRecord::Base.establish_connection(config)
    end

    Sidekiq.configure_client do |config|
        config.redis = { :size => 1 }
    end

    Sidekiq.configure_server do |config|
        config = Rails.application.config.database_configuration[Rails.env]
        config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
        config['pool']            = ENV['DB_POOL'] || 10
        ActiveRecord::Base.establish_connection(config)
    end
end

在heroku上,我将DB_POOL配置变量设置为2作为 recommended by Heroku 。这些错误应该发生吗?避免这样的错误似乎很奇怪,不是吗?你有什么建议?

2 回复 | 直到 11 年前

stereoscott 8 年前

sidekiq服务器(实际执行延迟任务的服务器上运行的进程)默认情况下最多会拨出25个线程来处理队列中的工作。如果任务需要,这些线程中的每一个都可以通过ActiveRecord请求与主数据库的连接。

如果你只有一个由5个连接组成的连接池,但你有25个线程试图连接,那么5秒钟后,如果线程无法从池中获得可用的连接,它们就会放弃,并且你会得到一个连接超时错误。

将Sidekiq服务器的池大小设置为更接近您的并发级别(使用 -c 启动进程时的标志)将有助于缓解这个问题,但代价是打开更多到数据库的连接。例如,如果你在Heroku上使用Postgres,他们的一些计划限制为20个,而其他计划的连接限制为500个( source ).

如果您运行的是像Unicorn这样的多进程服务器环境,您还需要监控每个分叉进程建立的连接数量。如果您有4个独角兽进程,默认连接池大小为5,那么您的独角兽环境在任何给定时间都可能有20个实时连接。你可以在上阅读更多信息 Heroku's docs 还要注意的是,DB池的大小并不意味着每个dyno现在都会有那么多打开的连接,只是如果需要一个新的连接,它会被创建,直到创建了最大数量的连接。

话虽如此,我的做法如下。

# config/initializers/unicorn.rb

if ENV['RACK_ENV'] == 'development'
  worker_processes 1
  listen "#{ENV['BOXEN_SOCKET_DIR']}/rails_app"
  timeout 120
else
  worker_processes Integer(ENV["WEB_CONCURRENCY"] || 2)
  timeout 29
end

# The timeout mechanism in Unicorn is an extreme solution that should be avoided whenever possible. 
# It will help catch bugs in your application where and when your application forgets to use timeouts,
# but it is expensive as it kills and respawns a worker process.
# see http://unicorn.bogomips.org/Application_Timeouts.html

# Heroku recommends a timeout of 15 seconds. With a 15 second timeout, the master process will send a 
# SIGKILL to the worker process if processing a request takes longer than 15 seconds. This will 
# generate a H13 error code and youâll see it in your logs. Note, this will not generate any stacktraces 
# to assist in debugging. Using Rack::Timeout, we can get a stacktrace in the logs that can be used for
# future debugging, so we set that value to something less than this one

preload_app true # for new relic

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end

end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to sent QUIT'
  end

  Rails.logger.info("Done forking unicorn processes")

  #https://devcenter.heroku.com/articles/concurrency-and-database-connections
  if defined?(ActiveRecord::Base)

    db_pool_size = if ENV["DB_POOL"]
      ENV["DB_POOL"]
    else
      ENV["WEB_CONCURRENCY"] || 2
    end

    config = Rails.application.config.database_configuration[Rails.env]
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    config['pool']              = ENV['DB_POOL'] || 2
    ActiveRecord::Base.establish_connection(config)

    # Turning synchronous_commit off can be a useful alternative when performance is more important than exact certainty about the durability of a transaction
    ActiveRecord::Base.connection.execute "update pg_settings set setting='off' where name = 'synchronous_commit';"    

    Rails.logger.info("Connection pool size for unicorn is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
  end

end

对于sidekiq:

# config/initializers/sidekiq.rb

Sidekiq.configure_server do |config|

  sidekiq_pool = ENV['SIDEKIQ_DB_POOL'] || 20

  if defined?(ActiveRecord::Base)
    Rails.logger.debug("Setting custom connection pool size of #{sidekiq_pool} for Sidekiq Server")
    db_config = Rails.application.config.database_configuration[Rails.env]
    db_config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    cb_config['pool']              = sidekiq_pool
    ActiveRecord::Base.establish_connection(db_config)

    Rails.logger.info("Connection pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
  end
end

如果一切顺利,当您启动流程时,您会在日志中看到类似的内容:

Setting custom connection pool size of 10 for Sidekiq Server
Connection pool size for Sidekiq Server is now: 20
Done forking unicorn processes
   (1.4ms)  update pg_settings set setting='off' where name = 'synchronous_commit';
Connection pool size for unicorn is now: 2

来源:

nmott 11 年前

对于Sidekiq服务器配置,建议使用 db_pool 数字与您的并发性相同,我假设您已将其设置为大于2。

假设设置 数据库工具 正在工作 unicorn.rb (我没有这样做的经验)一个潜在的解决方案是设置另一个环境变量来控制Sidekiq 数据库工具 直接地

如果您的sidekiq并发性为20,那么如下所示:

配置变量- SIDEKIQ_DB_POOL = 20

Sidekiq.configure_server do |config|
  config = Rails.application.config.database_configuration[Rails.env]
  config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
  config['pool']            = ENV['SIDEKIQ_DB_POOL'] || 10
  ActiveRecord::Base.establish_connection(config)
end

这样可以确保您有两个独立的池,分别针对您的网络工作者进行优化 DB_POOL 和你的背景工作者 SIDEKIQ_DB_POOL