San Francisco, USA

5214F Diamond Heights Blvd #553
San Francisco, CA 94131

Pune, India

203, Jewel Towers, 2nd Floor
Lane Number 5, Koregaon Park
Pune 411001, India

301 - 275 - 3997
hello@BigBinary.com

Rails 6 adds ActiveRecord::Relation#annotate

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails 6 has added ActiveRecord::Relation#annotate to allow adding comments to the SQL queries generated by the ActiveRecord::Relation instance.

Here is how it can be used.

>> User.annotate("User whose name starts with 'A'").where("name LIKE ?", "A%")

SELECT "users".* FROM "users"
WHERE (name LIKE 'A%')
/* User whose name starts with 'A' */
LIMIT ?  [["LIMIT", 11]]

ActiveRecord::Relation#annotate allows to add multiple annotations on a query

>> bigbinary = Organization.find_by!(name: "BigBinary")
>> User.annotate("User whose name starts with 'A'")
       .annotate("AND belongs to BigBinary organization")
       .where("name LIKE ?", "A%")
       .where(organization: bigbinary)

SELECT "users".* FROM "users"
WHERE (name LIKE 'A%') AND "users"."organization_id" = ?
/* Users whose name starts with 'A' */
/* AND belongs to BigBinary organization */
LIMIT ?  [["organization_id", 1], ["LIMIT", 11]]

Also, ActiveRecord::Relation#annotate allows annotating scopes and model associations.

class User < ActiveRecord::Base
  scope :active, -> { where(status: 'active').annotate("Active users") }
end

>> User.active
SELECT "users".* FROM "users"
/* Active users */
LIMIT ?  [["LIMIT", 11]]

Check out the pull request for more details on this.


Rails 6 adds hooks to Active Job around retries and discards

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Before Rails 6

Before Rails 6, we have to provide a custom block to perform custom logging and monitoring around retries and discards of the jobs defined using Active Job framework.

class Container::DeleteJob < ActiveJob::Base
  retry_on Timeout::Error, wait: 2.seconds, attempts: 3 do |job, error|
    message = "Stopped retrying #{job.class} (JID #{job.job_id})
               with #{job.arguments.join(', ')} due to
               '#{error.class} - #{error.message}'.
               This job was retried for #{job.executions} times.".squish

    BackgroundJob::ErrorLogger.log(message)
  end

  discard_on Container::NotFoundError do |job, error|
    message = "Discarded #{job.class} (JID #{job.job_id})
               with #{job.arguments.join(', ')} due to
               '#{error.class} - #{error.message}' error.".squish

    BackgroundJob::ErrorLogger.log(message)
  end

  def perform(container_id)
    Container::DeleteService(container_id).process

    # Will raise Container::NotFoundError
    # if no container is found with 'container_id'.

    # Might raise Timeout::Error when the remote API is not responding.
  end
end

Notice the custom blocks provided to retry_on and discard_on methods to an individual job in the above example.

Extracting such custom logic to a base class or to a 3rd-party gem is possible but it will be non-standard and will be a bit difficult task.

An alternative approach is to subscribe to the hooks instrumented using Active Support Instrumentation API which is a standard and recommended way. Prior versions of Rails 6 already instruments some hooks such as enqueue_at.active_job, enqueue.active_job, perform_start.active_job, and perform.active_job. Unfortunately no hook is instrumented around retries and discards of an Active Job prior to Rails 6.

Rails 6

Rails 6 has introduced hooks to Active Job around retries and discards to which one can easily subscribe using Active Support Instrumentation API to perform custom logging and monitoring or to collect any custom information.

The newly introduced hooks are enqueue_retry.active_job, retry_stopped.active_job and discard.active_job.

Let’s discuss each of these hooks in detail.

Note that whenever we say a job, it means a job of type ActiveJob.

enqueue_retry.active_job

The enqueue_retry.active_job hook is instrumented when a job is enqueued to retry again due to occurrence of an exception which is configured using the retry_on method in the job’s definition. This hook is triggered only when above condition is satisfied and the number of executions of the job is less than the number of attempts defined using the retry_on method. The number of attempts is by default set to 5 if not defined explicitly.

This is how we would subscribe to this hook and perform custom logging in our Rails application.

ActiveSupport::Notifications.subscribe "enqueue_retry.active_job" do |*args|
  event = ActiveSupport::Notifications::Event.new *args
  payload = event.payload
  job = payload[:job]
  error = payload[:error]
  message = "#{job.class} (JID #{job.job_id})
             with arguments #{job.arguments.join(', ')}
             will be retried again in #{payload[:wait]}s
             due to error '#{error.class} - #{error.message}'.
             It is executed #{job.executions} times so far.".squish

  BackgroundJob::Logger.log(message)
end

Note that the BackgroundJob::Logger above is our custom logger. If we want, we can add any other logic instead.

We will change the definition of Container::DeleteJob job as below.

class Container::DeleteJob < ActiveJob::Base
  retry_on Timeout::Error, wait: 2.seconds, attempts: 3

  def perform(container_id)
    Container::DeleteService(container_id).process

    # Will raise Timeout::Error when the remote API is not responding.
  end
end

Let’s enqueue this job.

Container::DeleteJob.perform_now("container-1234")

Assume that this job keeps throwing Timeout::Error exception due to a network issue. The job will be retried twice since it is configured to retry when a Timeout::Error exception occurs up to maximum 3 attempts. While retrying this job, Active Job will instrument enqueue_retry.active_job hook along with the necessary job payload.

Since we have already subscribed to this hook, our subscriber would log something like this with the help of BackgroundJob::Logger.log.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 1 times so far.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 2 times so far.
retry_stopped.active_job

The retry_stopped.active_job hook is triggered when a job is retried up to the available number of attempts.

Let’s see how this hook is triggered.

Along with the subscription for the enqueue_retry.active_job hook, let’s subscribe to the retry_stopped.active_job hook, too.

ActiveSupport::Notifications.subscribe "retry_stopped.active_job" do |*args|
  event = ActiveSupport::Notifications::Event.new *args
  payload = event.payload
  job = payload[:job]
  error = payload[:error]
  message = "Stopped processing #{job.class} (JID #{job.job_id})
             further with arguments #{job.arguments.join(', ')}
             since it failed due to '#{error.class} - #{error.message}' error
             which reoccurred #{job.executions} times.".squish

  BackgroundJob::Logger.log(message)
end

Let’s keep the Container::DeleteJob job’s definition unchanged and enqueue it again.

Container::DeleteJob.perform_now("container-1234")

We will assume that the job will keep throwing Timeout::Error exception due to a network issue.

In the logs recorded using BackgroundJob::Logger.log, we should see something like this.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 1 times so far.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 2 times so far.

Stopped processing Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) further with arguments container-1234 since it failed due to 'Timeout::Error - Couldn't establish connection to cluster within 10s' error which reoccurred 3 times.

Notice the last entry in the logs above and its order.

discard.active_job

The discard.active_job hook is triggered when a job’s further execution is discarded due to occurrence of an exception which is configured using discard_on method.

To see how this hook is triggered, we will subscribe to the discard.active_job hook.

ActiveSupport::Notifications.subscribe "discard.active_job" do |*args|
  event = ActiveSupport::Notifications::Event.new *args
  payload = event.payload
  job = payload[:job]
  error = payload[:error]
  message = "Discarded #{job.class} (JID #{job.job_id})
             with arguments #{job.arguments.join(', ')}
             due to '#{error.class} - #{error.message}' error.".squish

  BackgroundJob::Logger.log(message)
end

We will configure our Container::DeleteJob job to discard when Container::NotFoundError exception occurs while executing the job.

class Container::DeleteJob < ActiveJob::Base
  discard_on Container::NotFoundError
  retry_on Timeout::Error, wait: 2.seconds, attempts: 3

  def perform(container_id)
    Container::DeleteService(container_id).process

    # Will raise Container::NotFoundError
    # if no container is found with 'container_id'.

    # Will raise Timeout::Error when the remote API is not responding.
  end
end

Let’s enqueue this job and assume that it would throw Container::NotFoundError exception.

Container::DeleteJob.perform_now("unknown-container-9876")

We should see following in the logs recorded by BackgroundJob::Logger.log.

Discarded Container::DeleteJob (JID e9b1cb5c-6d2d-49ae-b1d7-fef44f09ab8d) with arguments unknown-container-9876 due to 'Container::NotFoundError - Container 'unknown-container-9876' was not found' error.

Notes

  1. These new hooks are also instrumented for the jobs which are enqueued using perform_later method since both perform_now and perform_later calls perform method under the hood.

  2. Active Job already subscribes to the these hooks and writes them using the Rails’ default logger.

  3. If a block is provided to retry_on or discard_on methods then an applicable hook is instrumented first and then the given block is yielded.


Rails 6 adds support for Multi Environment credentials

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

In Rails 5.2, encrypted credentails are stored in the file config/credentials.yml.enc. This is a single flat file which is encrypted by the key located in config/master.key.

Rails 5.2 does not support storing credentials of different environments with different encryption keys. If we want environment specific encrypted credentials, we’ll have to follow this workaround.

Rails 6 has added support for Multi Environment credentials. With this change, credentials that belong to different environments can be stored in separate files with their own encryption key.

Let’s see how this works in Rails 6.0.0.beta3

Rails 6.0.0.beta3

If we want to add credentials to be used in staging environment, we can run

rails edit:credentials --environment staging

This will create the credentials file config/credentials/staging.yml.enc and a staging specific encryption key config/credentials/staging.key and open the credentials file in your text editor.

Let’s add our AWS access key id here.

aws:
  access_key_id: "STAGING_KEY"

We can then access the access_key_id in staging environment.

>> RAILS_ENV=staging rails c

pry(main)> Rails.application.credentials.aws[:access_key_id]

=> "STAGING_KEY"

Which takes precedence: Global or Environment Specific credentials?

Credentials added to global file config/credentials.yml.enc will not be loaded in environments which have their own environment specific credentials file (config/credentials/$environment.yml.enc).

So if we decide to add the following to the global credentials file, these credentials will not be available in staging. Since we already have a environment specific credentials file for staging.

aws:
  access_key_id: "DEFAULT_KEY"
stripe:
  secret_key: "DEFAULT_SECRET_KEY"
>> RAILS_ENV=staging rails c

pry(main)> Rails.application.credentials.aws[:access_key_id]

=> "STAGING_KEY"

pry(main)> Rails.application.credentials.stripe[:secret_key]

Traceback (most recent call last):
        1: from (irb):6
NoMethodError (undefined method `[]' for nil:NilClass)

Here is the relevant pull request


Rails 6 adds before? and after? to Date and Time

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails 6 adds before? and after? to Date , DateTime , Time and ActiveSupport::TimeWithZone classes.

before? and after? are aliases to < (less than) and > (greater than) methods respectively.

Let’s checkout how it works.

Rails 5.2

Let’s try calling before? on a date object in Rails 5.2.

>> Date.new(2019, 3, 31).before?(Date.new(2019, 4, 1))

=> NoMethodError: undefined method 'before?' for Sun, 31 Mar 2019:Date
	from (irb):1

>> Date.new(2019, 3, 31) < Date.new(2019, 4, 1)

=> true

Rails 6.0.0.beta2

Now, let’s compare Date , DateTime , Time and ActiveSupport::TimeWithZone objects using before? and after? in Rails 6.

>> Date.new(2019, 3, 31).before?(Date.new(2019, 4, 1))

=> true

>> Date.new(2019, 3, 31).after?(Date.new(2019, 4, 1))

=> false

>> DateTime.parse('2019-03-31').before?(DateTime.parse('2019-04-01'))

=> true

>> DateTime.parse('2019-03-31').after?(DateTime.parse('2019-04-01'))

=> false

>> Time.parse('2019-03-31').before?(Time.parse('2019-04-01'))

=> true

>> Time.parse('2019-03-31').after?(Time.parse('2019-04-01'))

=> false

>> ActiveSupport::TimeWithZone.new(Time.utc(2019, 3, 31, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]).before?(ActiveSupport::TimeWithZone.new(Time.utc(2019, 4, 1, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]))

=> true

>> ActiveSupport::TimeWithZone.new(Time.utc(2019, 3, 31, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]).after?(ActiveSupport::TimeWithZone.new(Time.utc(2019, 4, 1, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]))

=> false

Here is the relevant pull request for adding before? and after? methods and the pull request for moving before? and after? to DateAndTime::Calculations.


Rails 6 adds Array#extract!

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails 6 added extract! on Array class. extract! removes and returns the elements for which the given block returns true.

extract! is different from reject! in the way that reject! returns the array after removing the elements whereas extract! returns removed elements from the array.

Let’s checkout how it works.

Rails 6.0.0.beta2

Let’s pluck all the user emails and then extract emails which include gmail.com.

>> emails = User.pluck(:email)
SELECT "users"."email" FROM "users"

=> ["amit.choudhary@bigbinary.com", "amit@gmail.com", "mark@gmail.com", "sam@gmail.com"]

>> emails.extract! { |email| email.include?('gmail.com') }

=> ["amit@gmail.com", "mark@gmail.com", "sam@gmail.com"]

>> emails

=> ["amit.choudhary@bigbinary.com"]

>> emails = User.pluck(:email)
SELECT "users"."email" FROM "users"

=> ["amit.choudhary@bigbinary.com", "amit@gmail.com", "mark@gmail.com", "sam@gmail.com"]

>> emails.reject! { |email| email.include?('gmail.com') }

=> ["amit.choudhary@bigbinary.com"]

>> emails

=> ["amit.choudhary@bigbinary.com"]

Here is the relevant pull request.


Rails 6 adds Enumerable#index_with

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails 6 added index_with on Enumerable module. This will help in creating a hash from an enumerator with default or fetched values.

Before Rails 6, we can achieve this by calling map along with to_h.

index_with takes both value or a block as a parameter.

Let’s checkout how it works.

Rails 5.2

Let’s create a hash from an array in Rails 5.2 using map and to_h.

>> address = Address.first
SELECT "addresses".* FROM "addresses"
ORDER BY "addresses"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Address id: 1, first_name: "Amit", last_name: "Choudhary", state: "California", created_at: "2019-03-21 10:03:57", updated_at: "2019-03-21 10:03:57">

>> NAME_ATTRIBUTES = [:first_name, :last_name]

=> [:first_name, :last_name]

>> NAME_ATTRIBUTES.map { |attr| [attr, address.public_send(attr)] }.to_h

=> {:first_name=>"Amit", :last_name=>"Choudhary"}

Rails 6.0.0.beta2

Now let’s create the same hash from the array using index_with in Rails 6.

>> address = Address.first
SELECT "addresses".* FROM "addresses"
ORDER BY "addresses"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Address id: 1, first_name: "Amit", last_name: "Choudhary", state: "California", created_at: "2019-03-21 10:02:47", updated_at: "2019-03-21 10:02:47">

>> NAME_ATTRIBUTES = [:first_name, :last_name]

=> [:first_name, :last_name]

>> NAME_ATTRIBUTES.index_with { |attr| address.public_send(attr) }

=> {:first_name=>"Amit", :last_name=>"Choudhary"}

>> NAME_ATTRIBUTES.index_with('Default')

=> {:first_name=>"Default", :last_name=>"Default"}

Here is the relevant pull request.


Rails 6 adds private option to delegate method

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails 6 adds :private option to delegate method. After this addition, we can delegate methods in the private scope.

Let’s checkout how it works.

Rails 6.0.0.beta2

Let’s create two models named as Address and Order. Let’s also delegate validate_state method in Order to Address.

class Address < ApplicationRecord
  validates :first_name, :last_name, :state, presence: true

  DELIVERABLE_STATES = ['New York']

  def validate_state
    unless DELIVERABLE_STATES.include?(state)
      errors.add(:state, :invalid)
    end
  end
end

class Order < ApplicationRecord
  belongs_to :address

  delegate :validate_state, to: :address
end

>> Order.first
SELECT "orders".* FROM "orders"
ORDER BY "orders"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Order id: 1, amount: 0.1e2, address_id: 1, created_at: "2019-03-21 10:02:58", updated_at: "2019-03-21 10:17:44">

>> Address.first
SELECT "addresses".* FROM "addresses"
ORDER BY "addresses"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Address id: 1, first_name: "Amit", last_name: "Choudhary", state: "California", created_at: "2019-03-21 10:02:47", updated_at: "2019-03-21 10:02:47">

>> Order.first.validate_state
SELECT "orders".* FROM "orders"
ORDER BY "orders"."id" ASC LIMIT $1  [["LIMIT", 1]]

SELECT "addresses".* FROM "addresses"
WHERE "addresses"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]

=> ["is invalid"]

Now, let’s add private: true to the delegation.

class Order < ApplicationRecord
  belongs_to :address

  delegate :validate_state, to: :address, private: true
end

>> Order.first.validate_state
SELECT "orders".* FROM "orders"
ORDER BY "orders"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> Traceback (most recent call last):
        1: from (irb):7
NoMethodError (private method 'validate_state' called for #<Order:0x00007fb9d72fc1f8>
Did you mean?  validate)

As we can see, Rails now raises an exception of private method called if private option is set with delegate method.

Here is the relevant pull request.


Rails 6 allows spaces in postgres table names

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails 6 allows spaces in tables names in PostgreSQL. Before Rails 6, if we try to create a table named as user reviews, Rails tries to create a table named as reviews in schema named as user.

Let’s checkout how it works.

Rails 5.2

Let’s create a table user reviews in Rails 5.2.

>> class CreateUserReviews < ActiveRecord::Migration[5.2]
>>   def change
>>     create_table 'user reviews' do |t|
>>       t.string :value
>>
>>       t.timestamps
>>     end
>>   end
>> end

=> :change

>> CreateUserReviews.new.change
-- create_table("user reviews")
CREATE TABLE "user"."reviews" ("id" bigserial primary key, "value" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

=> Traceback (most recent call last):
        2: from (irb):10
        1: from (irb):3:in 'change'
ActiveRecord::StatementInvalid (PG::InvalidSchemaName: ERROR:  schema "user" does not exist)
LINE 1: CREATE TABLE "user"."reviews" ("id" bigserial primary key, "...
                     ^
: CREATE TABLE "user"."reviews" ("id" bigserial primary key, "value" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

We can see that Rails 5.2 raised an exception and tried to create table named as reviews in user schema.

Rails 6.0.0.beta2

Now, let’s create a table user reviews in Rails 6.

>> class CreateUserReviews < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table 'user reviews' do |t|
>>       t.string :value
>>
>>       t.timestamps
>>     end
>>   end
>> end

=> :change

>> CreateUserReviews.new.change
-- create_table("user reviews")
CREATE TABLE "user reviews" ("id" bigserial primary key, "value" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> #<PG::Result:0x00007f9d633c5458 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

Now, we can see that the SQL generated is correct and Rails successfully created a table named as user reviews.

Here is the relevant pull request.


Rails 6 adds if_not_exists option to create_table

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails 6 added if_not_exists to create_table option to create a table if it doesn’t exist.

Before Rails 6, we could use ActiveRecord::Base.connection.table_exists?.

Default value of if_not_exists option is false.

Rails 5.2

Let’s create users table in Rails 5.2.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users)
CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

=> #<PG::Result:0x00007fd73e711cf0 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

Now let’s try creating users table again with if_not_exists option.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users, if_not_exists: true do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>true})
CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

=> Traceback (most recent call last):
        2: from (irb):121
        1: from (irb):114:in 'change'
ActiveRecord::StatementInvalid (PG::DuplicateTable: ERROR:  relation "users" already exists)
: CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

We can see that Rails 5.2 ignored if_not_exists option and tried creating the table again.

Now let’s try ActiveRecord::Base.connection.table_exists? with Rails 5.2.

>> class CreateUsers < ActiveRecord::Migration[5.2]
>>   def change
>>     unless ActiveRecord::Base.connection.table_exists?('users')
>>       create_table :users do |t|
>>         t.string :name
>>
>>         t.timestamps
>>       end
>>     end
>>   end
>> end

>> CreateUsers.new.change

=> nil

We can see that create_table :users never executed because ActiveRecord::Base.connection.table_exists?('users') returned true.

Rails 6.0.0.beta2

Let’s create users table in Rails 6 with if_not_exists option set as true.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users, if_not_exists: true do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>true})
CREATE TABLE IF NOT EXISTS "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> #<PG::Result:0x00007fc4614fef48 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>true})
CREATE TABLE IF NOT EXISTS "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> #<PG::Result:0x00007fc46513fde0 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

We can see that no exception was raised when we tried creating users table the second time.

Now let’s see what happens if we set if_not_exists to false.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users, if_not_exists: false do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>false})
CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> Traceback (most recent call last):
        2: from (irb):23
        1: from (irb):15:in `change'
ActiveRecord::StatementInvalid (PG::DuplicateTable: ERROR:  relation "users" already exists
)

As we can see, Rails raised an exception here because if_not_exists was set to false.

Here is the relevant pull request.


Rails 6 has added a way to change the database of the app

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails allows us to use different databases using the database.yml config file. It uses sqlite3 as the default database when a new Rails app is created. But it is also possible to use different databases such as MySQL or PostgreSQL. The contents of database.yml change as per the database. Also each database has a different adapter. We need to include the gems pg or mysql2 accordingly.

Before Rails 6, it was not possible to change the contents of database.yml automatically. But now a command has been added to do this automatically.

Let’s say our app has started with sqlite and now we have to switch to MySQL.

$ rails db:system:change --to=mysql
    conflict  config/database.yml
Overwrite /Users/prathamesh/Projects/reproductions/squish_app/config/database.yml? (enter "h" for help) [Ynaqdhm] Y
       force  config/database.yml
        gsub  Gemfile
        gsub  Gemfile

Our database.yml is now changed to contain the configuration for MySQL database and the Gemfile also gets updated automatically with addition of mysql2 gem in place of sqlite3.

This command also takes care of using proper gem versions in the Gemfile when the database backend is changed.


Rails 6 adds parallel testing

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

We frequently think about how good it would be if we could run tests in parallel on local so there would be less wait time for tests to be completed. Wait times increase considerably when the count of tests are on the higher side, which is a common case for a lot of applications.

Though CI tools like CircleCi and Travis CI provide a feature to run tests in parallel, there still wasn’t a straightforward way to parallelize tests on local before Rails 6.

Before Rails 6, if we wanted to parallelize tests, we would use Parallel Tests.

Rails 6 adds the parallelization of tests by default. Rails 6 added parallelize as a class method on ActiveSupport::TestCase which takes a hash as a parameter with the keys workers and with. The worker key is responsible for setting the number of parallel workers. The default value of the worker key is :number_of_processors, which finds the number of processors on the machine and sets it as the number of parallel workers. with takes two values - :processes, which is the default one, and :threads as a value.

Rails 6 also added two hooks - parallelize_setup, which is called before the processes are forked, and parallelize_teardown, which is called after the processes are killed. Rails 6 also handles creation of multiple databases and namespacing of those databases for parallel tests out of the box.

If we want to disable parallel testing, we can set the value of workers as 1 or less.

Rails 6.0.0.beta2

class ActiveSupport::TestCase
  parallelize_setup do |worker|
    # setup databases
  end
   parallelize_teardown do |worker|
    # cleanup database
  end

  # Run tests in parallel with specified workers
  parallelize(workers: :number_of_processors)

  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
  fixtures :all

  # Add more helper methods to be used by all tests here...
end

Rails 6 also provides an environment variable PARALLEL_WORKERS to set the number of parallel workers on runtime.

$ PARALLEL_WORKERS=10 bin/rails test

Here is the relevant pull request for adding parallelize and pull request for setting number of processors as default workers count.


Rails 6 adds CPU time, idle time and allocations to ActiveSupport::Notifications::Event

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails provides an easy way to instrument events and ability to subscribe to those events using Active Support Instrumentation API.

Before Rails 6

Before Rails 6, the subscriber of an event can access the event’s start time, end time and the duration along with the other event information.

To demonstrate how to access this information from an event, we will instrument a custom event custom_sleep_event and attach a subscriber to that event.

ActiveSupport::Notifications.subscribe('custom_sleep_event') do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)

  puts "Event: #{event.inspect}"
  puts "Started: #{event.time}"
  puts "Finished: #{event.end}"
  puts "Duration (ms): #{event.duration}"
end

ActiveSupport::Notifications.instrument('custom_sleep_event') do
  sleep 2
end

The event subscriber should print something similar.

Event: #<ActiveSupport::Notifications::Event:0x00007f952fc6a0b8 @name="custom_sleep_event", @payload={}, @time=2019-04-11 16:58:52 +0530, @transaction_id="e82231ab65b7af3c85ec", @end=2019-04-11 16:58:54 +0530, @children=[], @duration=nil>

Started: 2019-04-11 16:58:52 +0530

Finished: 2019-04-11 16:58:54 +0530

Duration (ms): 2001.287

Improvements and additions made to ActiveSupport::Notifications::Event in Rails 6

Rails 6 has improved the way an event’s duration is computed and also added useful information accessible on an event object such as CPU time, idle time and allocations.

Let’s discuss it in more detail.

1. CLOCK_MONOTONIC instead of CLOCK_REALTIME

Before Rails 6, Time.now is used for recording the event’s start time and end time. To avoid issues with the machine changing time, Rails 6 now uses Concurrent.monotonic_time instead of Time.now to record the event’s both start time and end time accurately.

Initially Process.clock_gettime(Process::CLOCK_MONOTONIC) was used which later modified to use Concurrent.monotonic_time. Note that Concurrent.monotonic_time is same but returns more precise time than Process.clock_gettime(Process::CLOCK_MONOTONIC).

Time.now or Process.clock_gettime(Process::CLOCK_REALTIME) can jump forwards and backwards as the system time-of-day clock is changed. Whereas, clock time using CLOCK_MONOTONIC returns the absolute wall-clock time since an unspecified time in the past (for example, system start-up time, or the Epoch). The CLOCK_MONOTONIC does not change with the system time-of-day clock, it just keeps advances forwards at one tick per tick and resets if the system is rebooted. In general, CLOCK_MONOTONIC is recommended to compute the elapsed time between two events. To read more about the differences between CLOCK_REALTIME and CLOCK_MONOTONIC, please check the discussion on this Stackoverflow thread. Another article written by Luca Guidi on the same topic is a recommended read.

2. No need to create hand made event objects on our own

Since it is a common practice to initialize an event using ActiveSupport::Notifications::Event.new(*args) in the event subscriber block, Rails 6 now makes this a bit easy. If the block passed to the subscriber only takes one argument then the Active Support Notification framework now yields an event object to the block.

Therefore, the subscriber definition below

ActiveSupport::Notifications.subscribe('an_event') do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)

  puts "Event #{event.name} received."
end

now can be simplified in Rails 6 as follows.

ActiveSupport::Notifications.subscribe('an_event') do |event|
  puts "Event #{event.name} received."
end
3. CPU time and idle time

Rails 6 now computes elapsed CPU time of an event with the help of Process.clock_gettime(Process::CLOCK_PROCESS_CPUTIME_ID).

System (kernel) keeps track of CPU time per process. The clock time returned using CLOCK_PROCESS_CPUTIME_ID represents the CPU time that has passed since the process started. Since a process may not always get all CPU cycles between start and finish of the process, the process often has to (sleep and) share CPU time among other processes. If the system puts a process to sleep, then the time spend waiting is not counted in the process’ CPU time.

The CPU time of an event can be fetched using the #cpu_time method.

Also, Rails 6 now computes the idle time of an event, too. The idle time of an event represents the difference between the event’s #duration and #cpu_time. Note that the #duration is computed using the difference between the event’s monotonic time at the start (#time) and the monotonic time at the end (#end).

Let’s see how to get these time values.

ActiveSupport::Notifications.subscribe('custom_sleep_event') do |event|
  puts "Event: #{event.inspect}"
  puts "Started: #{event.time}"
  puts "Finished: #{event.end}"
  puts "Duration (ms): #{event.duration}"
  puts "CPU time (ms): #{event.cpu_time}"
  puts "Idle time (ms): #{event.idle_time}"
end

ActiveSupport::Notifications.instrument('custom_sleep_event') do
  sleep 2
end

It prints this.

Event: #<ActiveSupport::Notifications::Event:0x00007fb02ac72400 @name="custom_sleep_event", @payload={}, @time=29514.525707, @transaction_id="43ca8e1c378b6b00d861", @end=29516.528971, @children=[], @duration=nil, @cpu_time_start=2.238801, @cpu_time_finish=2.238874, @allocation_count_start=835821, @allocation_count_finish=835821>

Started: 29514.525707

Finished: 29516.528971

Duration (ms): 2003.2639999990351

CPU time (ms): 0.07299999999998974

Idle time (ms): 2003.190999999035

Notice the @cpu_time_start and @cpu_time_finish counters in the inspected event object representation which are used to calculate the CPU time.

4. Allocations

We will now know how many objects were allocated between the start and end of an event using event’s #allocations method.

ActiveSupport::Notifications.subscribe('custom_sleep_event') do |event|
  puts "Event: #{event.inspect}"
  puts "Started: #{event.time}"
  puts "Finished: #{event.end}"
  puts "Duration (ms): #{event.duration}"
  puts "CPU time (ms): #{event.cpu_time}"
  puts "Idle time (ms): #{event.idle_time}"
  puts "# of objects allocated: #{event.allocations}"
end

ActiveSupport::Notifications.instrument('custom_sleep_event') do
  sleep 2
end

The above example should print something like this.

Event: #<ActiveSupport::Notifications::Event:0x00007fed8c4e33c0 @name="custom_sleep_event", @payload={}, @time=30503.508897, @transaction_id="5330165dae2b49fbe143", @end=30505.513547, @children=[], @duration=nil, @cpu_time_start=2.813231, @cpu_time_finish=2.813404, @allocation_count_start=834227, @allocation_count_finish=834228>

Started: 30503.508897

Finished: 30505.513547

Duration (ms): 2004.6499999989464

CPU time (ms): 0.17299999999975668

Idle time (ms): 2004.4769999989467

# of objects allocated: 1

Notice the @allocation_count_finish and @allocation_count_start counters in the inspected event object representation which are used to calculate the number of objects allocated during an event whose difference is (834228 - 834227 = 1).

In case of JRuby, the allocations would be zero.

5. #start! and #finish!

Two public methods #start! and #finish! have been introduced to ActiveSupport::Notifications::Event which can be used to record more information.

The #start! method resets the @time, @cpu_time_start and @allocation_count_start counters. Similarly, the #finish! method also resets the @end, @cpu_time_finish and @allocation_count_finish counters.


To learn more about this feature, please check rails/rails#33449.


Rails 6 allows configurable attribute name on has_secure_password

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

has_secure_password is used to encrypt and authenticate passwords using BCrypt . It assumes the model has a column named password_digest.

Before Rails 6, has_secure_password did not accept any attribute as a parameter. So, if we needed BCrypt encryption on a different column other than password_digest, we would have to manually encrypt the value before storing it.

Rails 6 makes it easy and allows custom attributes as a parameter to has_secure_password. has_secure_password still defaults to password so it works with previous versions of Rails. has_secure_password still needs the column named column_name_digest defined on the model.

has_secure_password also adds the authenticate_column_name method to authenticate the custom column.

Let’s check out how it works.

Rails 5.2

>> class User < ApplicationRecord
>>   has_secure_password
>> end

=> [ActiveModel::Validations::ConfirmationValidator]

>> user = User.create(email: 'amit.choudhary@bigbinary.com', password: 'amit.choudhary')
BEGIN
User Create (0.8ms)  INSERT INTO "users" ("email", "password_digest", "created_at", "updated_at") VALUES ($1, $2, $3, $4) RETURNING "id"  [["email", "amit.choudhary@bigbinary.com"], ["password_digest", "$2a$10$g6ZJNgakn4I1w/qjAx3vM.I76QSNjFCHtTtT9ovko/9Th50SEmIBO"], ["created_at", "2019-03-17 23:30:13.754379"], ["updated_at", "2019-03-17 23:30:13.754379"]]
COMMIT

=> #<User id: 1, email: "amit.choudhary@bigbinary.com", password_digest: "$2a$10$g6ZJNgakn4I1w/qjAx3vM.I76QSNjFCHtTtT9ovko/9...", created_at: "2019-03-17 23:30:13", updated_at: "2019-03-17 23:30:13">

>> user.authenticate('amit.choudhary')

=> #<User id: 1, email: "amit.choudhary@bigbinary.com", password_digest: "$2a$10$g6ZJNgakn4I1w/qjAx3vM.I76QSNjFCHtTtT9ovko/9...", created_at: "2019-03-17 23:30:13", updated_at: "2019-03-17 23:30:13">

>> class User < ApplicationRecord
>>   has_secure_password :transaction_password
>> end

=> NoMethodError: undefined method 'fetch' for :transaction_password:Symbol
	from (irb):9:in '<class:User>'
	from (irb):8

Rails 6.0.0.beta2

>> class User < ApplicationRecord
>>   has_secure_password
>>   has_secure_password :transaction_password
>> end

=> [ActiveModel::Validations::ConfirmationValidator]

>> user = User.create(email: 'amit.choudhary@bigbinary.com', password: 'amit.choudhary', transaction_password: 'amit.choudhary')
BEGIN
User Create (0.5ms)  INSERT INTO "users" ("email", "password_digest", "transaction_password_digest", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["email", "amit.choudhary@bigbinary.com"], ["password_digest", "$2a$10$nUiO7E2XrIJx/sSdpG0JAOL00uFvPRH7kXHLk5f/6qA1zLPHIrpPy"], ["transaction_password_digest", "$2a$10$l6cTpHwV9xOEn2.OumI29OnualGpvr1CgrNrbuMuHyGTltko8eBG2"], ["created_at", "2019-03-17 23:42:28.723431"], ["updated_at", "2019-03-17 23:42:28.723431"]]
COMMIT

=> #<User id: 5, email: "amit.choudhary@bigbinary.com", password_digest: [FILTERED], transaction_password_digest: [FILTERED], created_at: "2019-03-17 23:42:28", updated_at: "2019-03-17 23:42:28">

>> user.authenticate('amit.choudhary')

=> #<User id: 5, email: "amit.choudhary@bigbinary.com", password_digest: [FILTERED], transaction_password_digest: [FILTERED], created_at: "2019-03-17 23:42:28", updated_at: "2019-03-17 23:42:28">

>> user.authenticate_transaction_password('amit.choudhary')

=> #<User id: 5, email: "amit.choudhary@bigbinary.com", password_digest: [FILTERED], transaction_password_digest: [FILTERED], created_at: "2019-03-17 23:42:28", updated_at: "2019-03-17 23:42:28">

Here is the relevant pull request.


Rails 6 allows to override the ActiveModel::Errors#full_message format at the model level and at the attribute level

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Before Rails 6

Before Rails 6, the default format %{attribute} %{message} is used to display validation error message for a model’s attribute.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be blank")
=> "Title cannot be blank"

The default format can be overridden globally using a language-specific locale file.

# config/locales/en.yml

en:
  errors:
    format:
      "'%{attribute}' %{message}"

With this change, the full error message is changed for all the attributes of all models.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be blank")
=> "'Title' cannot be blank"

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "'First name' cannot be blank"

This trick works in some cases but it doesn’t work if we have to customize the error messages on the basis of specific models or attributes.

Before Rails 6, there is no easy way to generate error messages like shown below.

The article's title cannot be empty

or

First name of a person cannot be blank

If we change the errors.format to The article's %{attribute} %{message} in config/locales/en.yml then that format will be unexpectedaly used for other models, too.

# config/locales/en.yml

en:
  errors:
    format:
      "The article's %{attribute} %{message}"

This is what will happen if we make such a change.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be empty")
=> "The article's Title cannot be empty"

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "The article's First name cannot be blank"

Notice the error message generated for the :first_name attribute of User model. This does not look the way we want, right?

Let’s see what is changed in Rails 6 to overcome this problem.

Enhancements made to ActiveModel::Errors#full_message in Rails 6

Overriding the format of error message globally using errors.format is still supported in Rails 6.

In addition to that, Rails 6 now also supports overriding the error message’s format at the model level and at the attribute level.

In order to enable this support, we need to explicitly set config.active_model.i18n_customize_full_message to true in the Rails configuration file, preferably in config/application.rb which is implicitly set to false by default.

Overriding model level format

We can customize the full error message format for each model separately.

# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article:
          format: "`%{attribute}`: %{message}"
        user:
          format: "%{attribute} of the user %{message}"

The full error messages will look like this.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be empty")
=> "`Title`: cannot be empty"
>> article.valid?
=> false
>> article.errors.full_messages
=> ["`Title`: can't be blank"]

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "First name of the user cannot be blank"

>> comment = Comment.new
=> #<Comment id: nil, message: nil, author_id: nil, created_at: nil, updated_at: nil>
>> comment.errors.full_message(:message, "is required")
=> "Message is required"

Notice how the default format %{attribute} %{message} is used for generating the full error messages for the Comment model since its format is not being overridden.

Since the other methods such as ActiveModel::Errors#full_messages, ActiveModel::Errors#full_messages_for, ActiveModel::Errors#to_hash etc. use the ActiveModel::Errors#full_message method under the hood, we get the full error messages according to the custom format in the returned values of these methods respectively as expected.

Overriding attribute level format

Similar to customizing format at the model level, we can customize the error format for specific attributes of individual models.

# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article:
          attributes:
            title:
              format: "The article's title %{message}"
        user:
          attributes:
            first_name:
              format: "%{attribute} of a person %{message}"

With such a configuration, we get the customized error message for the title attribute of the Article model.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be empty")
=> "The article's title cannot be empty"
>> article.errors.full_message(:description, "cannot be empty")
=> "Description cannot be empty"

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "First name of a person cannot be blank"
>> user.errors.full_message(:last_name, "cannot be blank")
=> "Last name cannot be blank"

Note that the error messages for the rest of the attributes were generated using the default %{attribute} %{message} format for which we didn’t add custom formats in the config/locales/en.yml manifest.

Overriding model level format of deeply nested models
# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article/comments/attachments:
          format: "%{message}"
>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:'comments/attachments.file_name', "is required")
=> "is required"
>> article.errors.full_message(:'comments/attachments.path', "cannot be blank")
=> "cannot be blank"
>> article.errors.full_message(:'comments.message', "cannot be blank")
=> "Comments message cannot be blank"
Overriding attribute level format of deeply nested models
# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article/comments/attachments:
          attributes:
            file_name:
              format: "File name of an attachment %{message}"
>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:'comments/attachments.file_name', "is required")
=> "File name of an attachment is required"
>> article.errors.full_message(:'comments/attachments.path', "cannot be blank")
=> "Comments/attachments path cannot be blank"

Precedence

The custom formats specified in the locale file has the following precedence in the high to low order.

  • activerecord.errors.models.article/comments/attachments.attributes.file_name.format
  • activerecord.errors.models.article/comments/attachments.format
  • activerecord.errors.models.article.attributes.title.format
  • activerecord.errors.models.article.format
  • errors.format

To learn more, please checkout rails/rails#32956 and rails/rails#35789.


Rails 6 adds ActiveRecord::Relation#extract_associated

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Before Rails 6, if we want to extract associated records from an ActiveRecord::Relation, we would use preload and collect.

For example, we want to fetch subscriptions of some users. The query would look as shown below.

Rails 5.2

User.where(blocked: false).preload(:subscriptions).collect(&:subscriptions)

=> # returns collection of subscription records

ActiveRecord::Relation#extract_associated provides a shorthand to acheive same result and is more readable than former.

Rails 6.0.0.beta3

User.where(blocked: false).extract_associated(:subscriptions)

=> # returns the same collection of subscription records

Here’s the relevant pull request for this change.


Rails 6 adds implicit_order_column

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails 6 added implicit_order_column on ActiveRecord::ModelSchema which allows us to define a custom column for implicit ordering on the model level. If there is no implicit_order_column defined, Rails takes a primary key as the implicit order column. Also, before Rails 6, the primary key was used to order records implicitly by default.

This has impact on methods like first , last and many more where implicit ordering is used.

Let’s checkout how it works.

Rails 5.2

>> class User < ApplicationRecord
>>   validates :name, presence: true
>> end

=> {:presence=>true}

>> User.first
SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 1, name: "Amit", created_at: "2019-03-11 00:18:41", updated_at: "2019-03-11 00:18:41">

>> User.last
SELECT "users".* FROM "users" ORDER BY "users"."id" DESC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 2, name: "Mark", created_at: "2019-03-11 00:20:42", updated_at: "2019-03-11 00:20:42">

>> class User < ApplicationRecord
>>   validates :name, presence: true
>>   self.implicit_order_column = "updated_at"
>> end

=> Traceback (most recent call last):
        2: from (irb):10
        1: from (irb):12:in '<class:User>'
NoMethodError (undefined method 'implicit_order_column=' for #<Class:0x00007faf4d6cb408>)

Rails 6.0.0.beta2

>> class User < ApplicationRecord
>>   validates :name, presence: true
>> end

=> {:presence=>true}

>> User.first
SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 1, name: "Amit", created_at: "2019-03-11 00:18:41", updated_at: "2019-03-11 00:18:41">

>> User.last
SELECT "users".* FROM "users" ORDER BY "users"."id" DESC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 2, name: "Mark", created_at: "2019-03-11 00:20:42", updated_at: "2019-03-11 00:20:42">

>> class User < ApplicationRecord
>>   validates :name, presence: true
>>   self.implicit_order_column = "updated_at"
>> end

=> "updated_at"

>> User.find(1).touch
SELECT "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]
UPDATE "users" SET "updated_at" = $1 WHERE "users"."id" = $2  [["updated_at", "2019-03-11 00:23:33.369021"], ["id", 1]]

=> true

>> User.first
SELECT "users".* FROM "users" ORDER BY "users"."updated_at" ASC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 2, name: "Mark", created_at: "2019-03-11 00:20:42", updated_at: "2019-03-11 00:23:09">

>> User.last
SELECT "users".* FROM "users" ORDER BY "users"."updated_at" DESC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 1, name: "Amit", created_at: "2019-03-11 00:18:41", updated_at: "2019-03-11 00:23:33">

Here is the relevant pull request.


Bulk insert support in Rails 6

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Rails 6 has added support for bulk inserts similar to how bulk update is supported using update_all and bulk delete is supported using delete_all.

Bulk inserts can be performed using newly added methods: insert_all, insert_all! and upsert_all.

All of these new methods allow the insertion of multiple records of the same model into the database. A single INSERT SQL query is prepared by these methods and a single sql statement is sent to the database, without instantiating the model or invoking Active Record callbacks or validations.

During bulk insertion, violation of primary key, violation of unique indexes, and violation of unique constraints is possible. Rails leverages database-specific features to either skip, or upsert the duplicates, depending on the case.

Let’s discuss insert_all, insert_all! and upsert_all methods in detail, which are all used to perform bulk insert.

We will create an articles table with two unique indexes.

create_table :articles do |t|
  t.string :title, null: false
  t.string :slug, null: false
  t.string :author, null: false
  t.text :description

  t.index :slug, unique: true
  t.index [:title, :author], unique: true
end

Note that we do not allow duplicate slug columns. We also prevent records from having duplicate title and author columns together.

To try out the examples provided in this blog post, please ensure to always clean up the articles table before running any example.

1. Performing bulk inserts by skipping duplicates

Let’s say we want to insert multiple articles at once into the database. It is possible that certain records may violate the unique constraint(s) of the table. Such records are considered duplicates.

In other words, rows or records are determined to be unique by every unique index on the table by default.

To skip the duplicate rows or records, and insert the rest of the records at once, we can use ActiveRecord::Persistence#insert_all method.

1.1 Behavior with PostgreSQL

Let’s run the following example on a PostgreSQL database.

result = Article.insert_all(
  [
    { id: 1,
      title: 'Handling 1M Requests Per Second',
      author: 'John',
      slug: '1m-req-per-second' },

    { id: 1, # duplicate 'id' here
      title: 'Type Safety in Elm',
      author: 'George',
      slug: 'elm-type-safety' },

    { id: 2,
      title: 'Authentication with Devise - Part 1',
      author: 'Laura',
      slug: 'devise-auth-1' },

    { id: 3,
      title: 'Authentication with Devise - Part 1',
      author: 'Laura', # duplicate 'title' & 'author' here
      slug: 'devise-auth-2' },

    { id: 4,
      title: 'Dockerizing and Deploying Rails App to Kubernetes',
      author: 'Paul',
      slug: 'rails-on-k8s' },

    { id: 5,
      title: 'Elm on Rails',
      author: 'Amanda',
      slug: '1m-req-per-second' }, # duplicate 'slug' here

    { id: 6,
      title: 'Working Remotely',
      author: 'Greg',
      slug: 'working-remotely' }
  ]
)
# Bulk Insert (2.3ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests  [...snip...] 'working-remotely') ON CONFLICT  DO NOTHING RETURNING "id"

puts result.inspect
#<ActiveRecord::Result:0x00007fb6612a1ad8 @columns=["id"], @rows=[[1], [2], [4], [6]], @hash_rows=nil, @column_types={"id"=>#<ActiveModel::Type::Integer:0x00007fb65f420078 @precision=nil, @scale=nil, @limit=8, @range=-9223372036854775808...9223372036854775808>}>

puts Article.count
# 4

The insert_all method accepts a mandatory argument which should be an array of hashes with the attributes of the same model. The keys in all hashes should be same.

Notice the ON CONFLICT DO NOTHING clause in the INSERT query. This clause is supported by PostgreSQL and SQLite databases. This instructs the database that when there is a conflict or a unique key constraint violation during bulk insert operation, to skip the conflicting record silently and proceed with the insertion of the next record.

In the above example, we have exactly 3 records which violate various unique constraints defined on the articles table.

One of the records being inserted has a duplicate id: 1 attribute, which violates unique primary key constraint. Another record that has duplicate title: 'Authentication with Devise - Part 1', author: 'Laura' attributes violates the multi-column unique index defined on title and author columns. Another record has duplicate slug: '1m-req-per-second' attributes violates the unique index defined on the slug column.

All of these records that violate any unique constraint or unique index are skipped and are not inserted into the database.

If successful, ActiveRecord::Persistence#insert_all returns an instance of ActiveRecord::Result. The contents of the result vary per database. In case of PostgreSQL database, this result instance holds information about the successfully inserted records such as the chosen column names, values of the those columns in each successfully inserted row, etc.

For PostgreSQL, by default, insert_all method appends RETURNING "id" clause to the SQL query where id is the primary key(s). This clause instructs the database to return the id of every successfully inserted record. By inspecting the result, especially the @columns=["id"], @rows=[[1], [2], [4], [6]] attributes of the result instance, we can see that the records having id attribute with values 1, 2, 4 and 6 were successfully inserted.

What if we want to see more attributes and not just the id attribute of the successfully inserted records in the result?

We should use the optional returning option, which accepts an array of attribute names, which should be returned for all successfully inserted records!

result = Article.insert_all(
  [
    { id: 1,
      title: 'Handling 1M Requests Per Second',
      author: 'John',
      slug: '1m-req-per-second' },
    #...snip...
  ],
  returning: %w[ id title ]
)
# Bulk Insert (2.3ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests  [...snip...] 'working-remotely') ON CONFLICT  DO NOTHING RETURNING "id","title"

puts result.inspect
#<ActiveRecord::Result:0x00007f902a1196f0 @columns=["id", "title"], @rows=[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]], @hash_rows=nil, @column_types={"id"=>#<ActiveModel::Type::Integer:0x00007f90290ca8d0 @precision=nil, @scale=nil, @limit=8, @range=-9223372036854775808...9223372036854775808>, "title"=>#<ActiveModel::Type::String:0x00007f9029978298 @precision=nil, @scale=nil, @limit=nil>}>

puts result.pluck("id", "title").inspect
#[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]]

Notice how the INSERT query appends RETURNING "id","title" clause and the result now holds the id and title attributes of the successfully inserted records.

1.2 Behavior with SQLite

Similar to PostgreSQL, the violating records are skipped during the bulk insert operation performed using insert_all when we run our example on a SQLite database.

result = Article.insert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    #...snip...
  ]
)
# Bulk Insert (1.6ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests [...snip...] 'working-remotely') ON CONFLICT  DO NOTHING

puts result.inspect
#<ActiveRecord::Result:0x00007fa9df448ff0 @columns=[], @rows=[], @hash_rows=nil, @column_types={}>

puts Article.pluck(:id, :title)
#[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]]

puts Article.count
# 4

Note that since SQLite does not support RETURING clause, it is not being added to the SQL query. Therefore, the returned ActiveRecord::Result instance does not contain any useful information.

If we try to explicitly use the returning option when the database being used is SQLite, then the insert_all method throws an error.

Article.insert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    #...snip...
  ],
  returning: %w[ id title ]
)
# ActiveRecord::ConnectionAdapters::SQLite3Adapter does not support :returning (ArgumentError)
1.3 Behavior with MySQL

The records that violate primary key, unique key constraints, or unique indexes are skipped during bulk insert operation performed using insert_all on a MySQL database.

result = Article.insert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    #...snip...
  ]
)
# Bulk Insert (20.3ms)  INSERT INTO `articles`(`id`,`title`,`author`,`slug`) VALUES (1, 'Handling 1M Requests [...snip...] 'working-remotely') ON DUPLICATE KEY UPDATE `id`=`id`

puts result.inspect
#<ActiveRecord::Result:0x000055d6cfea7580 @columns=[], @rows=[], @hash_rows=nil, @column_types={}>

puts Article.pluck(:id, :title)
#[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]]

puts Article.count
# 4

Here, the ON DUPLICATE KEY UPDATE 'id'='id' clause in the INSERT query is essentially doing the same thing as the ON CONFLICT DO NOTHING clause supported by PostgreSQL and SQLite.

Since MySQL does not support RETURING clause, it is not being included in the SQL query and therefore, the result doesn’t contain any useful information.

Explicitly trying to use returning option with insert_all method on a MySQL database throws ActiveRecord::ConnectionAdapters::Mysql2Adapter does not support :returning error.

2. Performing bulk inserts by skipping duplicates on a specified unique constraint but raising exception if records violate other unique constraints

In the previous case, we were skipping the records that were violating any unique constraints. In some case, we may want to skip duplicates caused by only a specific unique index but abort transaction if the other records violate any other unique constraints.

The optional unique_by option of the insert_all method allows to define such a unique constraint.

2.1 Behavior with PostgreSQL and SQLite

Let’s see an example to skip duplicate records that violate only the specified unique index :index_articles_on_title_and_author using unique_by option. The duplicate records that do not violate index_articles_on_title_and_author index are not skipped, and therefore throw an error.

result = Article.insert_all(
  [
    { .... },
    { .... }, # duplicate 'id' here
    { .... },
    { .... }, # duplicate 'title' and 'author' here
    { .... },
    { .... }, # duplicate 'slug' here
    { .... }
  ],
  unique_by: :index_articles_on_title_and_author
)
# PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "articles_pkey" (ActiveRecord::RecordNotUnique)
# DETAIL:  Key (id)=(1) already exists.

In case of SQLite, the error appears as shown below.

# SQLite3::ConstraintException: UNIQUE constraint failed: articles.id (ActiveRecord::RecordNotUnique)

In this case we get ActiveRecord::RecordNotUnique error which is caused by the violation of primary key constraint on the id column.

It didn’t skip the second record in the example above which violated the unique index on primary key id since the unique_by option was specified with a different unique index.

When an exception occurs, no record persists to the database since insert_all executes just a single SQL query.

The unique_by option can be identified by columns or a unique index name.

``

``
unique_by: :index_articles_on_title_and_author
# is same as
unique_by: %i[ title author ]

# Also,
unique_by: :slug
# is same as
unique_by: %i[ :slug ]
# and also same as
unique_by: :index_articles_on_slug

Let’s remove (or fix) the record that has duplicate primary key and re-run the above example.

result = Article.insert_all(
  [
    { .... },
    { .... },
    { .... },
    { .... }, # duplicate 'title' and 'author' here
    { .... },
    { .... }, # duplicate 'slug' here
    { .... }
  ],
  unique_by: :index_articles_on_title_and_author
)
# PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_articles_on_slug" (ActiveRecord::RecordNotUnique)
# DETAIL:  Key (slug)=(1m-req-per-second) already exists.

In case of SQLite, the error looks appears as shown below.

# SQLite3::ConstraintException: UNIQUE constraint failed: articles.slug (ActiveRecord::RecordNotUnique)

The ActiveRecord::RecordNotUnique error in the example above now says the index_articles_on_slug unique constraint is violated. Note how it intentionally didn’t raise an error for the unique constraint violated on the title and author columns by the the fourth record in the examplea above.

Now we will remove (or fix) the record that has same slug.

result = Article.insert_all(
  [
    { .... },
    { .... },
    { .... },
    { .... }, # duplicate 'title' and 'author' here
    { .... },
    { .... },
    { .... }
  ],
  unique_by: :index_articles_on_title_and_author
)
# Bulk Insert (2.5ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests Per Second', [...snip...] 'working-remotely') ON CONFLICT ("title","author") DO NOTHING RETURNING "id"

puts result.inspect
#<ActiveRecord::Result:0x00007fada2069828 @columns=["id"], @rows=[[1], [7], [2], [4], [5], [6]], @hash_rows=nil, @column_types={"id"=>#<ActiveModel::Type::Integer:0x00007fad9fdb9df0 @precision=nil, @scale=nil, @limit=8, @range=-9223372036854775808...9223372036854775808>}>

Here, the fourth record was skipped since that record violates the unique index index_articles_on_title_and_author specified by the unique_by option.

Similarly, we can specify a different unique index using the unique_by option. For example, if we specify unique_by: :slug option then the records containing duplicate slug columns will be skipped, but would raise ActiveRecord::RecordNotUnique exception if any record violates other unique constraints.

2.2 Behavior with MySQL

The unique_by option is not supported when the database is MySQL.

3. Raising exception if any of the records being bulk inserted violate any unique constraints

The insert_all! method (with bang version) never skips a duplicate record. If a record violates any unique constraints, then insert_all! method would simply throw an ActiveRecord::RecordNotUnique error.

When database is PostgreSQL, insert_all! method can accept optional returning option, which we discussed in depth in 1.1 section above.

The unique_by option is not supported by the insert_all! method.

4. Performing bulk upserts (updates or inserts)

So far, in the sections 1, 2 and 3 above, we discussed either skipping the duplicates or raising an exception if a duplicate is encountered during bulk inserts. Sometimes, we want to update the existing record when a duplicate occurs otherwise insert a new record. This operation is called upsert because either it tries to update the record, or if there is no record to update, then it tries to insert.

The upsert_all method in Rails 6 allows performing bulk upserts.

Let’s see it’s usage and behavior with different database systems.

4.1 upsert_all in MySQL

Let’s try to bulk upsert multiple articles containing some duplicates.

result = Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, .... }, # duplicate 'id' here
    { id: 2, .... },
    { id: 3, .... }, # duplicate 'title' and 'author' here
    { id: 4, .... },
    { id: 5, .... }, # duplicate 'slug' here
    { id: 6, .... }
  ]
)
# Bulk Insert (26.3ms)  INSERT INTO `articles`(`id`,`title`,`author`,`slug`) VALUES (1, 'Handling 1M Requests Per Second', 'John', [...snip...] 'working-remotely') ON DUPLICATE KEY UPDATE `title`=VALUES(`title`),`author`=VALUES(`author`),`slug`=VALUES(`slug`)

puts result.inspect
#<ActiveRecord::Result:0x000055a43c1fae10 @columns=[], @rows=[], @hash_rows=nil, @column_types={}>

puts Article.count
# 5

puts Article.all
#<ActiveRecord::Relation [#<Article id: 1, title: "Type Safety in Elm", slug: "elm-type-safety", author: "George", description: nil>, #<Article id: 2, title: "Authentication with Devise - Part 1", slug: "devise-auth-2", author: "Laura", description: nil>, #<Article id: 4, title: "Dockerizing and Deploying Rails App to Kubernetes", slug: "rails-on-k8s", author: "Paul", description: nil>, #<Article id: 5, title: "Elm on Rails", slug: "1m-req-per-second", author: "Amanda", description: nil>, #<Article id: 6, title: "Working Remotely", slug: "working-remotely", author: "Greg", description: nil>]>

The persisted records in the database look exactly as intended. Let’s discuss it in detail.

The second row in the input array that has the id: 1 attribute replaced the first row, which also had the duplicate id: 1 attribute.

The fourth row that has id: 3 replaced the attributes of the third row since both had duplicate “title” and “author” attributes.

The rest of the rows were not duplicates or no longer became duplicates, and therefore were inserted without any issues.

Note that the returning and unique_by options are not supported in the upsert_all method when the database is MySQL.

4.2 upsert_all in SQLite

Let’s try to execute the same example from the above section 4.1 when database is SQLite.

result = Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
    { id: 4, title: 'Dockerizing and Deploying Rails App to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ]
)
# Bulk Insert (4.0ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests Per Second', [...snip...] 'working-remotely') ON CONFLICT ("id") DO UPDATE SET "title"=excluded."title","author"=excluded."author","slug"=excluded."slug"

# SQLite3::ConstraintException: UNIQUE constraint failed: articles.title, articles.author (ActiveRecord::RecordNotUnique)

The bulk upsert operation failed in the above example due to ActiveRecord::RecordNotUnique exception.

Why it didn’t work similar to MySQL?

As per the documentation of MySQL, an upsert operation takes place if a new record violates any unique constraint.

Whereas, in case of SQLite, by default, new record replaces existing record when both the existing and new record have the same primary key. If a record violates any other unique constraints other than the primary key, it then raises ActiveRecord::RecordNotUnique exception.

The ON CONFLICT ("id") DO UPDATE clause in the SQL query above conveys the same intent.

Therefore, upsert_all in SQLite doesn’t behave exactly same as in MySQL.

As a workaround, we will need to upsert records with the help of multiple upsert_all calls with the usage of unique_by option.

If a duplicate record is encountered during the upsert operation, which violates the unique index specified using unique_by option then it will replace the attributes of the existing matching record.

Let’s try to understand this workaround with another example.

Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
  ],
  unique_by: :id
)

Article.upsert_all(
  [
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
    { id: 4, title: 'Dockerizing and Deploying Rails App to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ],
  unique_by: %i[ title author ]
)

puts Article.count
# 5

puts Article.all
#<ActiveRecord::Relation [#<Article id: 1, title: "Type Safety in Elm", slug: "elm-type-safety", author: "George", description: nil>, #<Article id: 2, title: "Authentication with Devise - Part 1", slug: "devise-auth-2", author: "Laura", description: nil>, #<Article id: 4, title: "Dockerizing and Deploying Rails App to Kubernetes", slug: "rails-on-k8s", author: "Paul", description: nil>, #<Article id: 5, title: "Elm on Rails", slug: "1m-req-per-second", author: "Amanda", description: nil>, #<Article id: 6, title: "Working Remotely", slug: "working-remotely", author: "Greg", description: nil>]>

Here, we first tried to upsert all the records which violated the unique primary key index on id column. Later, we upsert successfully all the remaining records, which violated the unique index on the title and author columns.

Note that since the first record’s slug attribute was already replaced with the second record’s slug attribute; the last second record having id: 5 didn’t raise an exception because of duplicate slug column.

4.3 upsert_all in PostgreSQL

We will run the same example in the 4.1 section above with PostgreSQL database.

result = Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
    { id: 4, title: 'Dockerizing and Deploying Rails App to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ]
)
# PG::CardinalityViolation: ERROR:  ON CONFLICT DO UPDATE command cannot affect row a second time (ActiveRecord::StatementInvalid)
# HINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.

The bulk upsert operation failed in the above example due to ActiveRecord::StatementInvalid exception which was caused by another PG::CardinalityViolation exception.

The PG::CardinalityViolation exception originates from here.

The PG::CardinalityViolation exception occurs when a row cannot be updated a second time in the same ON CONFLICT DO UPDATE SQL query. PostgreSQL assumes this behavior would lead the same row to updated a second time in the same SQL query, in unspecified order, non-deterministically.

PostgreSQL recommends it is the developer’s responsibility to prevent this situation from occurring.

Here’s more discussion about this issue - rails/rails#35519.

Therefore, the upsert_all method doesn’t work as intended due to the above limitation in PostgreSQL.

As a workaround, we can divide the single upsert_all query into multiple upsert_all queries with the use of unique_by option similar to how we did in case of SQLite in the 4.2 section above.

Article.insert_all(
  [
    { id: 1, title: 'Handling 1M requests per second', author: 'John', slug: '1m-req-per-second' },
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 4, title: 'Dockerizing and deploy Rails app to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ]
)

Article.upsert_all(
  [
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
  ]
)

Article.upsert_all(
  [
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
  ],
  unique_by: :index_articles_on_title_and_author
)

Article.upsert_all(
  [
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
  ]
)

puts Article.count
# 5

puts Article.all
#<ActiveRecord::Relation [#<Article id: 1, title: "Type Safety in Elm", slug: "elm-type-safety", author: "George", description: nil>, #<Article id: 2, title: "Authentication with Devise - Part 1", slug: "devise-auth-2", author: "Laura", description: nil>, #<Article id: 4, title: "Dockerizing and deploy Rails app to Kubernetes", slug: "rails-on-k8s", author: "Paul", description: nil>, #<Article id: 5, title: "Elm on Rails", slug: "1m-req-per-second", author: "Amanda", description: nil>, #<Article id: 6, title: "Working Remotely", slug: "working-remotely", author: "Greg", description: nil>]>

For reference, note that the upsert_all method also accepts returning option for PostgreSQL which we have already discussed in the 1.1 section above.

5. insert, insert! and upsert

Rails 6 has also introduced three more additional methods namely insert, insert! and upsert for convenience.

The insert method inserts a single record into the database. If that record violates a uniqueness constrain, then the insert method will skip inserting record into the database without raising an exception.

Similarly, the insert! method also inserts a single record into the database, but will raise ActiveRecord::RecordNotUnique exception if that record violates a uniqueness constraint.

The upsert method inserts or updates a single record into the database similar to how upsert_all does.

The methods insert, insert! and upsert are wrappers around insert_all, insert_all! and upsert_all respectively.

Let’s see some examples to understand the usage of these methods.

Article.insert({ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, unique_by: :slug)

# is same as

Article.insert_all([{ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }], unique_by: :slug)
Article.insert!({ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, returning: %w[ id title ])

# is same as

Article.insert_all!([{ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }], returning: %w[ id title ])
Article.upsert({ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, unique_by: :slug, returning: %w[ id title ])

# is same as

Article.upsert_all([{ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }], unique_by: :slug, returning: %w[ id title ])

To learn more about the bulk insert feature and its implementation, please check rails/rails#35077, rails/rails#35546 and rails/rails#35854.


Rails 6 drops support for PostgreSQL version less than 9.3

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

Before Rails 6, Rails was supporting PostgreSQL from version 9.1 and above. But in Rails 6, support for versions less than 9.3 is dropped. If your PostgreSQL version is less than 9.3 then an error is shown as follows.

Your version of PostgreSQL (90224) is too old. Active Record supports PostgreSQL >= 9.3.

Travis CI uses PostgreSQL 9.2 by default in their images. So this error can occur while testing the app on Travis CI with Rails 6. It can be resolved by using an addon for PostgreSQL.


Rails 6 requires Ruby 2.5 or newer

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

As per rails/rails#34754, a Rails 6 app requires Ruby version 2.5 or newer.

Let’s discuss what we need to know if we are dealing with Rails 6.

Ensuring a valid Ruby version is set while creating a new Rails 6 app

While creating a new Rails 6 app, we need to ensure that the current Ruby version in the shell is set to 2.5 or newer.

If it is set to an older version then the same version will be used by the rails new command to set the Ruby version in .ruby-version and in Gemfile respectively in the created Rails app.

$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

$ rails new meme-wizard
      create
      create  README.md
      create  Rakefile
      create  .ruby-version
      create  config.ru
      create  .gitignore
      create  Gemfile
      [...] omitted the rest of the output

$ cd meme-wizard && grep -C 2 -Rn -a "2.3.1" .
./.ruby-version:1:2.3.1
--
--
./Gemfile-2-git_source(:github) { |repo| "https://github.com/#{repo}.git" }
./Gemfile-3-
./Gemfile:4:ruby '2.3.1'
./Gemfile-5-
./Gemfile-6-# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'

An easy fix for this is to install a Ruby version 2.5 or newer and use that version prior to running the rails new command.

$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

$ rbenv local 2.6.0

$ ruby -v
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-darwin18]

$ rails new meme-wizard

$ cd meme-wizard && grep -C 2 -Rn -a "2.6.0" .
./.ruby-version:1:2.6.0
--
--
./Gemfile-2-git_source(:github) { |repo| "https://github.com/#{repo}.git" }
./Gemfile-3-
./Gemfile:4:ruby '2.6.0'
./Gemfile-5-
./Gemfile-6-# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'

Upgrading an older Rails app to Rails 6

While upgrading an older Rails app to Rails 6, we need to update the Ruby version to 2.5 or newer in .ruby-version and Gemfile files respectively.

What else do we need to know?

Since Ruby 2.5 has added Hash#slice method, the extension method with the same name defined by activesupport/lib/active_support/core_ext/hash/slice.rb has been removed from Rails 6.

Similarly, Rails 6 has also removed the extension methods Hash#transform_values and Hash#transform_values! from Active Support in favor of the native methods with the same names which exist in Ruby. These methods were introduced in Ruby 2.4 natively.

If we try to explicitly require active_support/core_ext/hash/transform_values then it would print a deprecation warning.

>> require "active_support/core_ext/hash/transform_values"
# DEPRECATION WARNING: Ruby 2.5+ (required by Rails 6) provides Hash#transform_values natively, so requiring active_support/core_ext/hash/transform_values is no longer necessary. Requiring it will raise LoadError in Rails 6.1. (called from irb_binding at (irb):1)
=> true

Database seeding task uses inline Active Job adapter in Rails 6

This blog is part of our Rails 6 series. Rails 6.0.0.rc1 was recently released.

We use the db:seed task to seed the database in Rails apps. Recently an issue was reported on Rails issue tracker where the db:seed task was not finishing.

In development environment, Rails uses async adapter as the default Active Job adapter. The Async adapter runs jobs with an in-process thread pool.

This specific issue was happening because the seed task was trying to attach a file using Active Storage. Active Storage adds a job in the background during the attachment process. This task was not getting executed properly using the async adapter and it was causing the seed task to hang without exiting.

It was found out that by using the inline adapter in development environment, this issue goes away. But making a wholesale change of making the default adapter in development environment as inline adapter defeats the purpose of having the async adapter as default in the first place.

Instead a change is made to execute all the code related to seeding using inline adapter. The inline adapter makes sure that all the code will be executed immediately.

As the inline adapter does not allow queuing up the jobs in future, this can result into an error if the seeding code somehow triggers such jobs. This issue is already reported on Github.

Update

Active Job is optional framework and we can skip it completely. Now that seeding depends on presence of Active Job, it was throwing an error when Active Job was not part of the application. Also, executing the jobs inline automatically, when users has set the Active Job queue adapter to something of their choice was surprising for the users. So a change has been made to load the seeds inline only when Active Job is included in the application and the queue adapter is async. This makes it backward compatible as well it does not change user’s choice of queue adapter automatically.