BigBinary is building a suite of products under neeto. We currently have around 22 products under development and all of the products are using Sidekiq. After the launch of Solid Queue, we decided to migrate neetoForm from Sidekiq to Solid Queue.
Please note that Solid Queue currently doesn't support cron-style or recurring jobs. There is a PR open regarding this issue. We have only partially migrated to Solid Queue. For recurring jobs, we are still using Sidekiq. Once the PR is merged, we will migrate completely to Solid Queue.
Migrating to Solid Queue from Sidekiq
Here is a step-by-step migration guide you can use to migrate your Rails application from Sidekiq to Solid Queue.
1. Installation
- Add gem "solid_queue" to your Rails application's Gemfile and run bundle install.
- Run bin/rails generate solid_queue:install which copies the config file and the required migrations.
- Run the migrations using bin/rails db:migrate.
2. Configuration
The installation step should have created a config/solid_queue.yml file. Uncomment the file and modify it as per your needs. Here is how the file looks for our application.
1default: &default 2 dispatchers: 3 - polling_interval: 1 4 batch_size: 500 5 workers: 6 - queues: "auth" 7 threads: 3 8 processes: 1 9 polling_interval: 0.1 10 - queues: "urgent" 11 threads: 3 12 processes: 1 13 polling_interval: 0.1 14 - queues: "low" 15 threads: 3 16 processes: 1 17 polling_interval: 2 18 - queues: "*" 19 threads: 3 20 processes: 1 21 polling_interval: 1 22 23development: 24 <<: *default 25 26staging: 27 <<: *default 28 29heroku: 30 <<: *default 31 32test: 33 <<: *default 34 35production: 36 <<: *default
3. Starting Solid Queue
On your development machine, you can start Solid Queue by running the following command.
1bundle exec rake solid_queue:start
This will start Solid Queue's supervisor process and will start processing any enqueued jobs. The supervisor process forks workers and dispatchers according to the configuration provided in the config/solid_queue.yml file. The supervisor process also controls the heartbeats of workers and dispatchers, and sends signals to stop and start them when needed.
Since we use foreman, we added the above command to our Procfile.
1# Procfile 2web: bundle exec puma -C config/puma.rb 3worker: bundle exec sidekiq -C config/sidekiq.yml 4solidqueueworker: bundle exec rake solid_queue:start 5release: bundle exec rake db:migrate
4. Setting the Active Job queue adapter
You can set the Active Job queue adapter to :solid_queue by adding the following line in your application.rb file.
1# application.rb 2config.active_job.queue_adapter = :solid_queue
The above change sets the queue adapter at the application level for all the jobs. However, since we wanted to use Solid Queue for our regular jobs and continue using Sidekiq for cron jobs, we didn't make the above change in application.rb.
Instead, we created a new base class which inherited from ApplicationJob and set the queue adapter to :solid_queue inside that.
1# sq_base_job.rb 2class SqBaseJob < ApplicationJob 3 self.queue_adapter = :solid_queue 4end
Then we made all the classes implementing regular jobs inherit from this new class SqBaseJob instead of ApplicationJob.
1# send_email_job.rb 2- class SendEmailJob < ApplicationJob 3+ class SendEmailJob < SqBaseJob 4 # ... 5end
By making the above change, all our regular jobs got enqueued via Solid Queue instead of Sidekiq.
But, we realized later that emails were still being sent via Sidekiq. On debugging and looking into Rails internals, we found that ActionMailer uses ActionMailer::MailDeliveryJob for enqueuing or sending emails.
ActionMailer::MailDeliveryJob inherits from ActiveJob::Base rather than the application's ApplicationJob. So even if we set the queue_adapter in application_job.rb, it won't work. ActionMailer::MailDeliveryJob fallbacks to using the adapter defined in application.rb or environment-specific (production.rb / staging.rb / development.rb) config files. But we can't do that because we still want to use Sidekiq for cron jobs.
To use Solid Queue for mailers, we needed to override the queue_adapter for mailers. We can do that in application_mailer.rb.
1# application_mailer.rb 2class ApplicationMailer < ActionMailer::Base 3 # ... 4 ActionMailer::MailDeliveryJob.queue_adapter = :solid_queue 5end
This change is only until we use both Sidekiq and Solid Queue. Once cron style jobs feature lands in Solid Queue, we can remove this override and set the queue_adapter directly in application.rb which will enforce the setting globally.
5. Code changes
For migrating from Sidekiq to Solid Queue, we had to make the following changes to the syntax for enqueuing a job.
- Replaced .perform_async with .perform_later.
- Replaced .perform_at with .set(...).perform_later(...).
1- SendMailJob.perform_async 2+ SendMailJob.perform_later 3 4- SendMailJob.perform_at(1.minute.from_now) 5+ SendMailJob.set(wait: 1.minute).perform_later
At some places we were storing the Job ID on a record, for querying the job's status or for cancelling the job. For such cases, we made the following change.
1def disable_form_at_deadline 2- job_id = DisableFormJob.perform_at(deadline, self.id) 3- self.disable_job_id = job_id 4+ job_id = DisableFormJob.set(wait_until: deadline).perform_later(self.id) 5+ self.disable_job_id = job.job_id 6end 7 8def cancel_form_deadline 9- Sidekiq::Status.cancel(self.disable_job_id) 10+ SolidQueue::Job.find_by(active_job_id: self.disable_job_id).destroy! 11 self.disable_job_id = nil 12end
6. Error handling and retries
Initially, we thought the on_thread_error configuration provided by Solid Queue can be used for error handling. However, during the development phase, we noticed that it wasn't capturing errors. We raised an issue with Solid Queue as we thought it was a bug.
Rosa Gutiérrez responded on the issue and clarified the following.
on_thread_error wasn't intended for errors on the job itself, but rather errors in the thread that's executing the job, but around the job itself. For example, if you had an Active Record's thread pool too small for your number of threads and you got an error when trying to check out a new connection, on_thread_error would be called with that.
For errors in the job itself, you could try to hook into Active Job's itself.
Based on the above information, we modified our SqBaseJob base class to handle the exceptions and report it to Honeybadger.
1# sq_base_job.rb 2class SqBaseJob < ApplicationJob 3 self.queue_adapter = :solid_queue 4 5 rescue_from(Exception) do |exception| 6 context = { 7 error_class: self.class.name, 8 args: self.arguments, 9 scheduled_at: self.scheduled_at, 10 job_id: self.job_id 11 } 12 Honeybadger.notify(exception, context:) 13 raise exception 14 end 15end
Remember we mentioned that ActionMailer doesn't inherit from ApplicationJob. So similarly, we would have to handle exceptions for Mailers separately.
1# application_mailer.rb 2class ApplicationMailer < ActionMailer::Base 3 # ... 4 ActionMailer::MailDeliveryJob.rescue_from(Exception) do |exception| 5 context = { 6 error_class: self.class.name, 7 args: self.arguments, 8 scheduled_at: self.scheduled_at, 9 job_id: self.job_id 10 } 11 Honeybadger.notify(exception, context:) 12 raise exception 13 end 14end
For retries, unlike Sidekiq, Solid Queue doesn't include any automatic retry mechanism, it relies on Active Job for this. We wanted our application to retry sending emails in case of any errors. So we added the retry logic in the ApplicationMailer.
1# application_mailer.rb 2class ApplicationMailer < ActionMailer::Base 3 # ... 4 ActionMailer::MailDeliveryJob.retry_on StandardError, attempts: 3 5end
Note that, although the queue adapter configuration can be removed from application_mailer.rb once the entire application migrates to Solid Queue, error handling and retry override cannot be removed because of the way ActionMailer::MailDeliveryJob inherits from ActiveJob::Base rather than application's ApplicationJob.
7. Testing
Once all the above changes were done, it was obvious that a lot of tests were failing. Apart from fixing the usual failures related to the syntax changes, some of the tests were failing inconsistently. On debugging, we found that the affected tests were all related to controllers, specifically tests inheriting from ActionDispatch::IntegrationTest.
We tried debugging and searched for solutions when we stumbled upon Ben Sheldon's comment on one of Good Job's issues. Ben points out that this is actually an issue in Rails where Rails sometimes inconsistently overrides ActiveJob's queue_adapter setting with TestAdapter. A PR is already open for the fix. Thankfully, Ben, in the same comment, also mentioned a workaround for it until the fix has been added to Rails.
We added the workaround in our test helper_methods.rb and called the method in each of our controller tests which were failing.
1# test/support/helper_methods.rb 2def ensure_consistent_test_adapter_is_used 3 # This is a hack mentioned here: https://github.com/bensheldon/good_job/issues/846#issuecomment-1432375562 4 # The actual issue is in Rails for which a PR is pending merge 5 # https://github.com/rails/rails/pull/48585 6 (ActiveJob::Base.descendants + [ActiveJob::Base]).each(&:disable_test_adapter) 7end
1# test/controllers/exports_controller_test.rb 2class ExportsControllerTest < ActionDispatch::IntegrationTest 3 def setup 4 ensure_consistent_test_adapter_is_used 5 # ... 6 end 7 8 # ... 9end
8. Monitoring
Basecamp has released mission_control-jobs which can be used to monitor background jobs. It is generic, so it can be used with any compatible ActiveJob adapter.
Add gem "mission_control-jobs" to your Gemfile and run bundle install.
Mount the mission control route in your routes.rb file.
1# routes.rb 2Rails.application.routes.draw do 3 # ... 4 mount MissionControl::Jobs::Engine, at: "/jobs"
By default, mission control would try to load the adapter specified in your application.rb or individual environment-specific files. Currently, Sidekiq isn't compatible with mission control, so you will face an error while loading the dashboard at /jobs. The fix is to explicitly specify solid_queue to the list of mission control adapters.
1# application.rb 2# ... 3config.mission_control.jobs.adapters = [:solid_queue]
Now, visiting /jobs on your site should load a dashboard where you can monitor your Solid Queue jobs.
But that isn't enough. There is no authentication. For development environments it is fine, but the /jobs route would be exposed on production too. By default, Mission Control's controllers will extend the host app's ApplicationController. If no authentication is enforced, /jobs will be available to everyone.
To implement some kind of authentication, we can specify a different controller as the base class for Mission Control's controllers and add the authentication there.
1# application.rb 2# ... 3MissionControl::Jobs.base_controller_class = "MissionControlController"
1# app/controllers/mission_control_controller.rb 2class MissionControlController < ApplicationController 3 before_action :authenticate!, if: :restricted_env? 4 5 private 6 7 def authenticate! 8 authenticate_or_request_with_http_basic do |username, password| 9 username == "solidqueue" && password == Rails.application.secrets.mission_control_password 10 end 11 end 12 13 def restricted_env? 14 Rails.env.staging? || Rails.env.production? 15 end 16end
Here, we have specified that MissionControlController would be our base controller for mission control related controllers. Then in MissionControlController we implemented basic authentication for staging and production environments.
Observations
We haven't had any complaints so far. Solid Queue offers simplicity, requires no additional infrastructure and provides visibility for managing jobs since they are stored in the database.
In the coming days, we will migrate all of our 22 neeto products to Solid Queue. And once cron style job support lands in Solid Queue, we will completely migrate from Sidekiq.