San Francisco, USA

5214F Diamond Heights Blvd #553
San Francisco, CA 94131

Pune, India

203, Jewel Towers, 2nd Floor
Lane Number 5, Koregaon Park
Pune 411001, India

301 - 275 - 3997
hello@BigBinary.com

Rails 6 allows to override the ActiveModel::Errors#full_message format at the model level and at the attribute level

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Before Rails 6

Before Rails 6, the default format %{attribute} %{message} is used to display validation error message for a model’s attribute.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be blank")
=> "Title cannot be blank"

The default format can be overridden globally using a language-specific locale file.

# config/locales/en.yml

en:
  errors:
    format:
      "'%{attribute}' %{message}"

With this change, the full error message is changed for all the attributes of all models.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be blank")
=> "'Title' cannot be blank"

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "'First name' cannot be blank"

This trick works in some cases but it doesn’t work if we have to customize the error messages on the basis of specific models or attributes.

Before Rails 6, there is no easy way to generate error messages like shown below.

The article's title cannot be empty

or

First name of a person cannot be blank

If we change the errors.format to The article's %{attribute} %{message} in config/locales/en.yml then that format will be unexpectedaly used for other models, too.

# config/locales/en.yml

en:
  errors:
    format:
      "The article's %{attribute} %{message}"

This is what will happen if we make such a change.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be empty")
=> "The article's Title cannot be empty"

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "The article's First name cannot be blank"

Notice the error message generated for the :first_name attribute of User model. This does not look the way we want, right?

Let’s see what is changed in Rails 6 to overcome this problem.

Enhancements made to ActiveModel::Errors#full_message in Rails 6

Overriding the format of error message globally using errors.format is still supported in Rails 6.

In addition to that, Rails 6 now also supports overriding the error message’s format at the model level and at the attribute level.

In order to enable this support, we need to explicitly set config.active_model.i18n_customize_full_message to true in the Rails configuration file, preferably in config/application.rb which is implicitly set to false by default.

Overriding model level format

We can customize the full error message format for each model separately.

# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article:
          format: "`%{attribute}`: %{message}"
        user:
          format: "%{attribute} of the user %{message}"

The full error messages will look like this.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be empty")
=> "`Title`: cannot be empty"
>> article.valid?
=> false
>> article.errors.full_messages
=> ["`Title`: can't be blank"]

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "First name of the user cannot be blank"

>> comment = Comment.new
=> #<Comment id: nil, message: nil, author_id: nil, created_at: nil, updated_at: nil>
>> comment.errors.full_message(:message, "is required")
=> "Message is required"

Notice how the default format %{attribute} %{message} is used for generating the full error messages for the Comment model since its format is not being overridden.

Since the other methods such as ActiveModel::Errors#full_messages, ActiveModel::Errors#full_messages_for, ActiveModel::Errors#to_hash etc. use the ActiveModel::Errors#full_message method under the hood, we get the full error messages according to the custom format in the returned values of these methods respectively as expected.

Overriding attribute level format

Similar to customizing format at the model level, we can customize the error format for specific attributes of individual models.

# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article:
          attributes:
            title:
              format: "The article's title %{message}"
        user:
          attributes:
            first_name:
              format: "%{attribute} of a person %{message}"

With such a configuration, we get the customized error message for the title attribute of the Article model.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be empty")
=> "The article's title cannot be empty"
>> article.errors.full_message(:description, "cannot be empty")
=> "Description cannot be empty"

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "First name of a person cannot be blank"
>> user.errors.full_message(:last_name, "cannot be blank")
=> "Last name cannot be blank"

Note that the error messages for the rest of the attributes were generated using the default %{attribute} %{message} format for which we didn’t add custom formats in the config/locales/en.yml manifest.

Overriding model level format of deeply nested models
# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article/comments/attachments:
          format: "%{message}"
>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:'comments/attachments.file_name', "is required")
=> "is required"
>> article.errors.full_message(:'comments/attachments.path', "cannot be blank")
=> "cannot be blank"
>> article.errors.full_message(:'comments.message', "cannot be blank")
=> "Comments message cannot be blank"
Overriding attribute level format of deeply nested models
# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article/comments/attachments:
          attributes:
            file_name:
              format: "File name of an attachment %{message}"
>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:'comments/attachments.file_name', "is required")
=> "File name of an attachment is required"
>> article.errors.full_message(:'comments/attachments.path', "cannot be blank")
=> "Comments/attachments path cannot be blank"

Precedence

The custom formats specified in the locale file has the following precedence in the high to low order.

  • activerecord.errors.models.article/comments/attachments.attributes.file_name.format
  • activerecord.errors.models.article/comments/attachments.format
  • activerecord.errors.models.article.attributes.title.format
  • activerecord.errors.models.article.format
  • errors.format

To learn more, please checkout rails/rails#32956 and rails/rails#35789.


Rails 6 adds ActiveRecord::Relation#extract_associated

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Before Rails 6, if we want to extract associated records from an ActiveRecord::Relation, we would use preload and collect.

For example, we want to fetch subscriptions of some users. The query would look as shown below.

Rails 5.2

User.where(blocked: false).preload(:subscriptions).collect(&:subscriptions)

=> # returns collection of subscription records

ActiveRecord::Relation#extract_associated provides a shorthand to acheive same result and is more readable than former.

Rails 6.0.0.beta3

User.where(blocked: false).extract_associated(:subscriptions)

=> # returns the same collection of subscription records

Here’s the relevant pull request for this change.


Rails 6 adds implicit_order_column

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Rails 6 added implicit_order_column on ActiveRecord::ModelSchema which allows us to define a custom column for implicit ordering on model level. If there is no implicit_order_column defined, Rails takes primary key as implicit order column. Before Rails 6 too, primary key is used to order records implicitly by default.

This has impact on methods like first , last and many more where implicit ordering is used.

Let’s checkout how it works.

Rails 5.2

>> class User < ApplicationRecord
>>   validates :name, presence: true
>> end

=> {:presence=>true}

>> User.first
SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 1, name: "Amit", created_at: "2019-03-11 00:18:41", updated_at: "2019-03-11 00:18:41">

>> User.last
SELECT "users".* FROM "users" ORDER BY "users"."id" DESC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 2, name: "Mark", created_at: "2019-03-11 00:20:42", updated_at: "2019-03-11 00:20:42">

>> class User < ApplicationRecord
>>   validates :name, presence: true
>>   self.implicit_order_column = "updated_at"
>> end

=> Traceback (most recent call last):
        2: from (irb):10
        1: from (irb):12:in '<class:User>'
NoMethodError (undefined method 'implicit_order_column=' for #<Class:0x00007faf4d6cb408>)

Rails 6.0.0.beta2

>> class User < ApplicationRecord
>>   validates :name, presence: true
>> end

=> {:presence=>true}

>> User.first
SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 1, name: "Amit", created_at: "2019-03-11 00:18:41", updated_at: "2019-03-11 00:18:41">

>> User.last
SELECT "users".* FROM "users" ORDER BY "users"."id" DESC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 2, name: "Mark", created_at: "2019-03-11 00:20:42", updated_at: "2019-03-11 00:20:42">

>> class User < ApplicationRecord
>>   validates :name, presence: true
>>   self.implicit_order_column = "updated_at"
>> end

=> "updated_at"

>> User.find(1).touch
SELECT "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]
UPDATE "users" SET "updated_at" = $1 WHERE "users"."id" = $2  [["updated_at", "2019-03-11 00:23:33.369021"], ["id", 1]]

=> true

>> User.first
SELECT "users".* FROM "users" ORDER BY "users"."updated_at" ASC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 2, name: "Mark", created_at: "2019-03-11 00:20:42", updated_at: "2019-03-11 00:23:09">

>> User.last
SELECT "users".* FROM "users" ORDER BY "users"."updated_at" DESC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 1, name: "Amit", created_at: "2019-03-11 00:18:41", updated_at: "2019-03-11 00:23:33">

Here is the relevant pull request.

Also note that in Rails 6 if UUID is used as primary key, created_at column is used by default for implicit ordering. Check out this pull request for more.


Bulk insert support in Rails 6

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Rails 6 has added support for bulk inserts similar to how bulk update is supported using update_all and bulk delete is supported using delete_all.

Bulk inserts can be performed using newly added methods: insert_all, insert_all! and upsert_all.

All of these new methods allow the insertion of multiple records of the same model into the database. A single INSERT SQL query is prepared by these methods and a single sql statement is sent to the database, without instantiating the model or invoking Active Record callbacks or validations.

During bulk insertion, violation of primary key, violation of unique indexes, and violation of unique constraints is possible. Rails leverages database-specific features to either skip, or upsert the duplicates, depending on the case.

Let’s discuss insert_all, insert_all! and upsert_all methods in detail, which are all used to perform bulk insert.

We will create an articles table with two unique indexes.

create_table :articles do |t|
  t.string :title, null: false
  t.string :slug, null: false
  t.string :author, null: false
  t.text :description

  t.index :slug, unique: true
  t.index [:title, :author], unique: true
end

Note that we do not allow duplicate slug columns. We also prevent records from having duplicate title and author columns together.

To try out the examples provided in this blog post, please ensure to always clean up the articles table before running any example.

1. Performing bulk inserts by skipping duplicates

Let’s say we want to insert multiple articles at once into the database. It is possible that certain records may violate the unique constraint(s) of the table. Such records are considered duplicates.

In other words, rows or records are determined to be unique by every unique index on the table by default.

To skip the duplicate rows or records, and insert the rest of the records at once, we can use ActiveRecord::Persistence#insert_all method.

1.1 Behavior with PostgreSQL

Let’s run the following example on a PostgreSQL database.

result = Article.insert_all(
  [
    { id: 1,
      title: 'Handling 1M Requests Per Second',
      author: 'John',
      slug: '1m-req-per-second' },

    { id: 1, # duplicate 'id' here
      title: 'Type Safety in Elm',
      author: 'George',
      slug: 'elm-type-safety' },

    { id: 2,
      title: 'Authentication with Devise - Part 1',
      author: 'Laura',
      slug: 'devise-auth-1' },

    { id: 3,
      title: 'Authentication with Devise - Part 1',
      author: 'Laura', # duplicate 'title' & 'author' here
      slug: 'devise-auth-2' },

    { id: 4,
      title: 'Dockerizing and Deploying Rails App to Kubernetes',
      author: 'Paul',
      slug: 'rails-on-k8s' },

    { id: 5,
      title: 'Elm on Rails',
      author: 'Amanda',
      slug: '1m-req-per-second' }, # duplicate 'slug' here

    { id: 6,
      title: 'Working Remotely',
      author: 'Greg',
      slug: 'working-remotely' }
  ]
)
# Bulk Insert (2.3ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests  [...snip...] 'working-remotely') ON CONFLICT  DO NOTHING RETURNING "id"

puts result.inspect
#<ActiveRecord::Result:0x00007fb6612a1ad8 @columns=["id"], @rows=[[1], [2], [4], [6]], @hash_rows=nil, @column_types={"id"=>#<ActiveModel::Type::Integer:0x00007fb65f420078 @precision=nil, @scale=nil, @limit=8, @range=-9223372036854775808...9223372036854775808>}>

puts Article.count
# 4

The insert_all method accepts a mandatory argument which should be an array of hashes with the attributes of the same model. The keys in all hashes should be same.

Notice the ON CONFLICT DO NOTHING clause in the INSERT query. This clause is supported by PostgreSQL and SQLite databases. This instructs the database that when there is a conflict or a unique key constraint violation during bulk insert operation, to skip the conflicting record silently and proceed with the insertion of the next record.

In the above example, we have exactly 3 records which violate various unique constraints defined on the articles table.

One of the records being inserted has a duplicate id: 1 attribute, which violates unique primary key constraint. Another record that has duplicate title: 'Authentication with Devise - Part 1', author: 'Laura' attributes violates the multi-column unique index defined on title and author columns. Another record has duplicate slug: '1m-req-per-second' attributes violates the unique index defined on the slug column.

All of these records that violate any unique constraint or unique index are skipped and are not inserted into the database.

If successful, ActiveRecord::Persistence#insert_all returns an instance of ActiveRecord::Result. The contents of the result vary per database. In case of PostgreSQL database, this result instance holds information about the successfully inserted records such as the chosen column names, values of the those columns in each successfully inserted row, etc.

For PostgreSQL, by default, insert_all method appends RETURNING "id" clause to the SQL query where id is the primary key(s). This clause instructs the database to return the id of every successfully inserted record. By inspecting the result, especially the @columns=["id"], @rows=[[1], [2], [4], [6]] attributes of the result instance, we can see that the records having id attribute with values 1, 2, 4 and 6 were successfully inserted.

What if we want to see more attributes and not just the id attribute of the successfully inserted records in the result?

We should use the optional returning option, which accepts an array of attribute names, which should be returned for all successfully inserted records!

result = Article.insert_all(
  [
    { id: 1,
      title: 'Handling 1M Requests Per Second',
      author: 'John',
      slug: '1m-req-per-second' },
    #...snip...
  ],
  returning: %w[ id title ]
)
# Bulk Insert (2.3ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests  [...snip...] 'working-remotely') ON CONFLICT  DO NOTHING RETURNING "id","title"

puts result.inspect
#<ActiveRecord::Result:0x00007f902a1196f0 @columns=["id", "title"], @rows=[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]], @hash_rows=nil, @column_types={"id"=>#<ActiveModel::Type::Integer:0x00007f90290ca8d0 @precision=nil, @scale=nil, @limit=8, @range=-9223372036854775808...9223372036854775808>, "title"=>#<ActiveModel::Type::String:0x00007f9029978298 @precision=nil, @scale=nil, @limit=nil>}>

puts result.pluck("id", "title").inspect
#[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]]

Notice how the INSERT query appends RETURNING "id","title" clause and the result now holds the id and title attributes of the successfully inserted records.

1.2 Behavior with SQLite

Similar to PostgreSQL, the violating records are skipped during the bulk insert operation performed using insert_all when we run our example on a SQLite database.

result = Article.insert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    #...snip...
  ]
)
# Bulk Insert (1.6ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests [...snip...] 'working-remotely') ON CONFLICT  DO NOTHING

puts result.inspect
#<ActiveRecord::Result:0x00007fa9df448ff0 @columns=[], @rows=[], @hash_rows=nil, @column_types={}>

puts Article.pluck(:id, :title)
#[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]]

puts Article.count
# 4

Note that since SQLite does not support RETURING clause, it is not being added to the SQL query. Therefore, the returned ActiveRecord::Result instance does not contain any useful information.

If we try to explicitly use the returning option when the database being used is SQLite, then the insert_all method throws an error.

Article.insert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    #...snip...
  ],
  returning: %w[ id title ]
)
# ActiveRecord::ConnectionAdapters::SQLite3Adapter does not support :returning (ArgumentError)
1.3 Behavior with MySQL

The records that violate primary key, unique key constraints, or unique indexes are skipped during bulk insert operation performed using insert_all on a MySQL database.

result = Article.insert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    #...snip...
  ]
)
# Bulk Insert (20.3ms)  INSERT INTO `articles`(`id`,`title`,`author`,`slug`) VALUES (1, 'Handling 1M Requests [...snip...] 'working-remotely') ON DUPLICATE KEY UPDATE `id`=`id`

puts result.inspect
#<ActiveRecord::Result:0x000055d6cfea7580 @columns=[], @rows=[], @hash_rows=nil, @column_types={}>

puts Article.pluck(:id, :title)
#[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]]

puts Article.count
# 4

Here, the ON DUPLICATE KEY UPDATE 'id'='id' clause in the INSERT query is essentially doing the same thing as the ON CONFLICT DO NOTHING clause supported by PostgreSQL and SQLite.

Since MySQL does not support RETURING clause, it is not being included in the SQL query and therefore, the result doesn’t contain any useful information.

Explicitly trying to use returning option with insert_all method on a MySQL database throws ActiveRecord::ConnectionAdapters::Mysql2Adapter does not support :returning error.

2. Performing bulk inserts by skipping duplicates on a specified unique constraint but raising exception if records violate other unique constraints

In the previous case, we were skipping the records that were violating any unique constraints. In some case, we may want to skip duplicates caused by only a specific unique index but abort transaction if the other records violate any other unique constraints.

The optional unique_by option of the insert_all method allows to define such a unique constraint.

2.1 Behavior with PostgreSQL and SQLite

Let’s see an example to skip duplicate records that violate only the specified unique index :index_articles_on_title_and_author using unique_by option. The duplicate records that do not violate index_articles_on_title_and_author index are not skipped, and therefore throw an error.

result = Article.insert_all(
  [
    { .... },
    { .... }, # duplicate 'id' here
    { .... },
    { .... }, # duplicate 'title' and 'author' here
    { .... },
    { .... }, # duplicate 'slug' here
    { .... }
  ],
  unique_by: :index_articles_on_title_and_author
)
# PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "articles_pkey" (ActiveRecord::RecordNotUnique)
# DETAIL:  Key (id)=(1) already exists.

In case of SQLite, the error appears as shown below.

# SQLite3::ConstraintException: UNIQUE constraint failed: articles.id (ActiveRecord::RecordNotUnique)

In this case we get ActiveRecord::RecordNotUnique error which is caused by the violation of primary key constraint on the id column.

It didn’t skip the second record in the example above which violated the unique index on primary key id since the unique_by option was specified with a different unique index.

When an exception occurs, no record persists to the database since insert_all executes just a single SQL query.

The unique_by option can be identified by columns or a unique index name.

``

``
unique_by: :index_articles_on_title_and_author
# is same as
unique_by: %i[ title author ]

# Also,
unique_by: :slug
# is same as
unique_by: %i[ :slug ]
# and also same as
unique_by: :index_articles_on_slug

Let’s remove (or fix) the record that has duplicate primary key and re-run the above example.

result = Article.insert_all(
  [
    { .... },
    { .... },
    { .... },
    { .... }, # duplicate 'title' and 'author' here
    { .... },
    { .... }, # duplicate 'slug' here
    { .... }
  ],
  unique_by: :index_articles_on_title_and_author
)
# PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_articles_on_slug" (ActiveRecord::RecordNotUnique)
# DETAIL:  Key (slug)=(1m-req-per-second) already exists.

In case of SQLite, the error looks appears as shown below.

# SQLite3::ConstraintException: UNIQUE constraint failed: articles.slug (ActiveRecord::RecordNotUnique)

The ActiveRecord::RecordNotUnique error in the example above now says the index_articles_on_slug unique constraint is violated. Note how it intentionally didn’t raise an error for the unique constraint violated on the title and author columns by the the fourth record in the examplea above.

Now we will remove (or fix) the record that has same slug.

result = Article.insert_all(
  [
    { .... },
    { .... },
    { .... },
    { .... }, # duplicate 'title' and 'author' here
    { .... },
    { .... },
    { .... }
  ],
  unique_by: :index_articles_on_title_and_author
)
# Bulk Insert (2.5ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests Per Second', [...snip...] 'working-remotely') ON CONFLICT ("title","author") DO NOTHING RETURNING "id"

puts result.inspect
#<ActiveRecord::Result:0x00007fada2069828 @columns=["id"], @rows=[[1], [7], [2], [4], [5], [6]], @hash_rows=nil, @column_types={"id"=>#<ActiveModel::Type::Integer:0x00007fad9fdb9df0 @precision=nil, @scale=nil, @limit=8, @range=-9223372036854775808...9223372036854775808>}>

Here, the fourth record was skipped since that record violates the unique index index_articles_on_title_and_author specified by the unique_by option.

Similarly, we can specify a different unique index using the unique_by option. For example, if we specify unique_by: :slug option then the records containing duplicate slug columns will be skipped, but would raise ActiveRecord::RecordNotUnique exception if any record violates other unique constraints.

2.2 Behavior with MySQL

The unique_by option is not supported when the database is MySQL.

3. Raising exception if any of the records being bulk inserted violate any unique constraints

The insert_all! method (with bang version) never skips a duplicate record. If a record violates any unique constraints, then insert_all! method would simply throw an ActiveRecord::RecordNotUnique error.

When database is PostgreSQL, insert_all! method can accept optional returning option, which we discussed in depth in 1.1 section above.

The unique_by option is not supported by the insert_all! method.

4. Performing bulk upserts (updates or inserts)

So far, in the sections 1, 2 and 3 above, we discussed either skipping the duplicates or raising an exception if a duplicate is encountered during bulk inserts. Sometimes, we want to update the existing record when a duplicate occurs otherwise insert a new record. This operation is called upsert because either it tries to update the record, or if there is no record to update, then it tries to insert.

The upsert_all method in Rails 6 allows performing bulk upserts.

Let’s see it’s usage and behavior with different database systems.

4.1 upsert_all in MySQL

Let’s try to bulk upsert multiple articles containing some duplicates.

result = Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, .... }, # duplicate 'id' here
    { id: 2, .... },
    { id: 3, .... }, # duplicate 'title' and 'author' here
    { id: 4, .... },
    { id: 5, .... }, # duplicate 'slug' here
    { id: 6, .... }
  ]
)
# Bulk Insert (26.3ms)  INSERT INTO `articles`(`id`,`title`,`author`,`slug`) VALUES (1, 'Handling 1M Requests Per Second', 'John', [...snip...] 'working-remotely') ON DUPLICATE KEY UPDATE `title`=VALUES(`title`),`author`=VALUES(`author`),`slug`=VALUES(`slug`)

puts result.inspect
#<ActiveRecord::Result:0x000055a43c1fae10 @columns=[], @rows=[], @hash_rows=nil, @column_types={}>

puts Article.count
# 5

puts Article.all
#<ActiveRecord::Relation [#<Article id: 1, title: "Type Safety in Elm", slug: "elm-type-safety", author: "George", description: nil>, #<Article id: 2, title: "Authentication with Devise - Part 1", slug: "devise-auth-2", author: "Laura", description: nil>, #<Article id: 4, title: "Dockerizing and Deploying Rails App to Kubernetes", slug: "rails-on-k8s", author: "Paul", description: nil>, #<Article id: 5, title: "Elm on Rails", slug: "1m-req-per-second", author: "Amanda", description: nil>, #<Article id: 6, title: "Working Remotely", slug: "working-remotely", author: "Greg", description: nil>]>

The persisted records in the database look exactly as intended. Let’s discuss it in detail.

The second row in the input array that has the id: 1 attribute replaced the first row, which also had the duplicate id: 1 attribute.

The fourth row that has id: 3 replaced the attributes of the third row since both had duplicate “title” and “author” attributes.

The rest of the rows were not duplicates or no longer became duplicates, and therefore were inserted without any issues.

Note that the returning and unique_by options are not supported in the upsert_all method when the database is MySQL.

4.2 upsert_all in SQLite

Let’s try to execute the same example from the above section 4.1 when database is SQLite.

result = Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
    { id: 4, title: 'Dockerizing and Deploying Rails App to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ]
)
# Bulk Insert (4.0ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests Per Second', [...snip...] 'working-remotely') ON CONFLICT ("id") DO UPDATE SET "title"=excluded."title","author"=excluded."author","slug"=excluded."slug"

# SQLite3::ConstraintException: UNIQUE constraint failed: articles.title, articles.author (ActiveRecord::RecordNotUnique)

The bulk upsert operation failed in the above example due to ActiveRecord::RecordNotUnique exception.

Why it didn’t work similar to MySQL?

As per the documentation of MySQL, an upsert operation takes place if a new record violates any unique constraint.

Whereas, in case of SQLite, by default, new record replaces existing record when both the existing and new record have the same primary key. If a record violates any other unique constraints other than the primary key, it then raises ActiveRecord::RecordNotUnique exception.

The ON CONFLICT ("id") DO UPDATE clause in the SQL query above conveys the same intent.

Therefore, upsert_all in SQLite doesn’t behave exactly same as in MySQL.

As a workaround, we will need to upsert records with the help of multiple upsert_all calls with the usage of unique_by option.

If a duplicate record is encountered during the upsert operation, which violates the unique index specified using unique_by option then it will replace the attributes of the existing matching record.

Let’s try to understand this workaround with another example.

Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
  ],
  unique_by: :id
)

Article.upsert_all(
  [
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
    { id: 4, title: 'Dockerizing and Deploying Rails App to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ],
  unique_by: %i[ title author ]
)

puts Article.count
# 5

puts Article.all
#<ActiveRecord::Relation [#<Article id: 1, title: "Type Safety in Elm", slug: "elm-type-safety", author: "George", description: nil>, #<Article id: 2, title: "Authentication with Devise - Part 1", slug: "devise-auth-2", author: "Laura", description: nil>, #<Article id: 4, title: "Dockerizing and Deploying Rails App to Kubernetes", slug: "rails-on-k8s", author: "Paul", description: nil>, #<Article id: 5, title: "Elm on Rails", slug: "1m-req-per-second", author: "Amanda", description: nil>, #<Article id: 6, title: "Working Remotely", slug: "working-remotely", author: "Greg", description: nil>]>

Here, we first tried to upsert all the records which violated the unique primary key index on id column. Later, we upsert successfully all the remaining records, which violated the unique index on the title and author columns.

Note that since the first record’s slug attribute was already replaced with the second record’s slug attribute; the last second record having id: 5 didn’t raise an exception because of duplicate slug column.

4.3 upsert_all in PostgreSQL

We will run the same example in the 4.1 section above with PostgreSQL database.

result = Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
    { id: 4, title: 'Dockerizing and Deploying Rails App to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ]
)
# PG::CardinalityViolation: ERROR:  ON CONFLICT DO UPDATE command cannot affect row a second time (ActiveRecord::StatementInvalid)
# HINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.

The bulk upsert operation failed in the above example due to ActiveRecord::StatementInvalid exception which was caused by another PG::CardinalityViolation exception.

The PG::CardinalityViolation exception originates from here.

The PG::CardinalityViolation exception occurs when a row cannot be updated a second time in the same ON CONFLICT DO UPDATE SQL query. PostgreSQL assumes this behavior would lead the same row to updated a second time in the same SQL query, in unspecified order, non-deterministically.

PostgreSQL recommends it is the developer’s responsibility to prevent this situation from occurring.

Here’s more discussion about this issue - rails/rails#35519.

Therefore, the upsert_all method doesn’t work as intended due to the above limitation in PostgreSQL.

As a workaround, we can divide the single upsert_all query into multiple upsert_all queries with the use of unique_by option similar to how we did in case of SQLite in the 4.2 section above.

Article.insert_all(
  [
    { id: 1, title: 'Handling 1M requests per second', author: 'John', slug: '1m-req-per-second' },
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 4, title: 'Dockerizing and deploy Rails app to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ]
)

Article.upsert_all(
  [
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
  ]
)

Article.upsert_all(
  [
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
  ],
  unique_by: :index_articles_on_title_and_author
)

Article.upsert_all(
  [
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
  ]
)

puts Article.count
# 5

puts Article.all
#<ActiveRecord::Relation [#<Article id: 1, title: "Type Safety in Elm", slug: "elm-type-safety", author: "George", description: nil>, #<Article id: 2, title: "Authentication with Devise - Part 1", slug: "devise-auth-2", author: "Laura", description: nil>, #<Article id: 4, title: "Dockerizing and deploy Rails app to Kubernetes", slug: "rails-on-k8s", author: "Paul", description: nil>, #<Article id: 5, title: "Elm on Rails", slug: "1m-req-per-second", author: "Amanda", description: nil>, #<Article id: 6, title: "Working Remotely", slug: "working-remotely", author: "Greg", description: nil>]>

For reference, note that the upsert_all method also accepts returning option for PostgreSQL which we have already discussed in the 1.1 section above.

5. insert, insert! and upsert

Rails 6 has also introduced three more additional methods namely insert, insert! and upsert for convenience.

The insert method inserts a single record into the database. If that record violates a uniqueness constrain, then the insert method will skip inserting record into the database without raising an exception.

Similarly, the insert! method also inserts a single record into the database, but will raise ActiveRecord::RecordNotUnique exception if that record violates a uniqueness constraint.

The upsert method inserts or updates a single record into the database similar to how upsert_all does.

The methods insert, insert! and upsert are wrappers around insert_all, insert_all! and upsert_all respectively.

Let’s see some examples to understand the usage of these methods.

Article.insert({ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, unique_by: :slug)

# is same as

Article.insert_all([{ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }], unique_by: :slug)
Article.insert!({ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, returning: %w[ id title ])

# is same as

Article.insert_all!([{ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }], returning: %w[ id title ])
Article.upsert({ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, unique_by: :slug, returning: %w[ id title ])

# is same as

Article.upsert_all([{ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }], unique_by: :slug, returning: %w[ id title ])

To learn more about the bulk insert feature and its implementation, please check rails/rails#35077, rails/rails#35546 and rails/rails#35854.


Rails 6 drops support for PostgreSQL version less than 9.3

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Before Rails 6, Rails was supporting PostgreSQL from version 9.1 and above. But in Rails 6, support for versions less than 9.3 is dropped. If your PostgreSQL version is less than 9.3 then an error is shown as follows.

Your version of PostgreSQL (90224) is too old. Active Record supports PostgreSQL >= 9.3.

Travis CI uses PostgreSQL 9.2 by default in their images. So this error can occur while testing the app on Travis CI with Rails 6. It can be resolved by using an addon for PostgreSQL.


Rails 6 requires Ruby 2.5 or newer

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

As per rails/rails#34754, a Rails 6 app requires Ruby version 2.5 or newer.

Let’s discuss what we need to know if we are dealing with Rails 6.

Ensuring a valid Ruby version is set while creating a new Rails 6 app

While creating a new Rails 6 app, we need to ensure that the current Ruby version in the shell is set to 2.5 or newer.

If it is set to an older version then the same version will be used by the rails new command to set the Ruby version in .ruby-version and in Gemfile respectively in the created Rails app.

$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

$ rails new meme-wizard
      create
      create  README.md
      create  Rakefile
      create  .ruby-version
      create  config.ru
      create  .gitignore
      create  Gemfile
      [...] omitted the rest of the output

$ cd meme-wizard && grep -C 2 -Rn -a "2.3.1" .
./.ruby-version:1:2.3.1
--
--
./Gemfile-2-git_source(:github) { |repo| "https://github.com/#{repo}.git" }
./Gemfile-3-
./Gemfile:4:ruby '2.3.1'
./Gemfile-5-
./Gemfile-6-# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'

An easy fix for this is to install a Ruby version 2.5 or newer and use that version prior to running the rails new command.

$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

$ rbenv local 2.6.0

$ ruby -v
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-darwin18]

$ rails new meme-wizard

$ cd meme-wizard && grep -C 2 -Rn -a "2.6.0" .
./.ruby-version:1:2.6.0
--
--
./Gemfile-2-git_source(:github) { |repo| "https://github.com/#{repo}.git" }
./Gemfile-3-
./Gemfile:4:ruby '2.6.0'
./Gemfile-5-
./Gemfile-6-# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'

Upgrading an older Rails app to Rails 6

While upgrading an older Rails app to Rails 6, we need to update the Ruby version to 2.5 or newer in .ruby-version and Gemfile files respectively.

What else do we need to know?

Since Ruby 2.5 has added Hash#slice method, the extension method with the same name defined by activesupport/lib/active_support/core_ext/hash/slice.rb has been removed from Rails 6.

Similarly, Rails 6 has also removed the extension methods Hash#transform_values and Hash#transform_values! from Active Support in favor of the native methods with the same names which exist in Ruby. These methods were introduced in Ruby 2.4 natively.

If we try to explicitly require active_support/core_ext/hash/transform_values then it would print a deprecation warning.

>> require "active_support/core_ext/hash/transform_values"
# DEPRECATION WARNING: Ruby 2.5+ (required by Rails 6) provides Hash#transform_values natively, so requiring active_support/core_ext/hash/transform_values is no longer necessary. Requiring it will raise LoadError in Rails 6.1. (called from irb_binding at (irb):1)
=> true

Database seeding task uses inline Active Job adapter in Rails 6

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

We use the db:seed task to seed the database in Rails apps. Recently an issue was reported on Rails issue tracker where the db:seed task was not finishing.

In development environment, Rails uses async adapter as the default Active Job adapter. The Async adapter runs jobs with an in-process thread pool.

This specific issue was happening because the seed task was trying to attach a file using Active Storage. Active Storage adds a job in the background during the attachment process. This task was not getting executed properly using the async adapter and it was causing the seed task to hang without exiting.

It was found out that by using the inline adapter in development environment, this issue goes away. But making a wholesale change of making the default adapter in development environment as inline adapter defeats the purpose of having the async adapter as default in the first place.

Instead a change is made to execute all the code related to seeding using inline adapter. The inline adapter makes sure that all the code will be executed immediately.

As the inline adapter does not allow queuing up the jobs in future, this can result into an error if the seeding code somehow triggers such jobs. This issue is already reported on Github.


Rails 6 adds ActiveRecord::Relation#reselect

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Rails have rewhere and reorder methods to change the previously set conditions attributes to new attributes which are given as an argument to method.

Before Rails 6, if you want to change the previously set select statement attributes to new attributes, it was done as follows.

>> Post.select(:title, :body).unscope(:select).select(:views)

   SELECT "posts"."views" FROM "posts" LIMIT ? ["LIMIT", 1]]

In Rails 6, ActiveRecord::Relation#reselect method is added.

The reselect method is similar to rewhere and reorder. reselect is a short-hand for unscope(:select).select(fields).

Here is how reselect method can be used.

>> Post.select(:title, :body).reselect(:views)

   SELECT "posts"."views" FROM "posts" LIMIT ? ["LIMIT", 1]]

Check out the pull request for more details on this.


Rails 6 adds ActiveModel::Errors#of_kind?

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Rails 6 added of_kind? on ActiveModel::Errors. It returns true if ActiveModel::Errors object has provided key and message associated with it. The default message is :invalid.

of_kind? is same as ActiveModel::Errors#added? but it doesn’t take extra options as parameter.

Let’s checkout how it works.

Rails 6.0.0.beta2

>> class User < ApplicationRecord
>>   validates :name, presence: true
>> end

>> user = User.new

=> => #<User id: nil, name: nil, password: nil, created_at: nil, updated_at: nil>

>> user.valid?

=> false

>> user.errors

=> #<ActiveModel::Errors:0x00007fc462a1d140 @base=#<User id: nil, name: nil, password: nil, created_at: nil, updated_at: nil>, @messages={:name=>["can't be blank"]}, @details={:name=>[{:error=>:blank}]}>

>> user.errors.of_kind?(:name)

=> false

>> user.errors.of_kind?(:name, :blank)

=> true

>> user.errors.of_kind?(:name, "can't be blank")

=> true

>> user.errors.of_kind?(:name, "is blank")

=> false

Here is the relevant pull request.


Rails 6 shows routes in expanded format

The output of rails routes is in the table format.

$ rails routes
   Prefix Verb   URI Pattern               Controller#Action
    users GET    /users(.:format)          users#index
          POST   /users(.:format)          users#create
 new_user GET    /users/new(.:format)      users#new
edit_user GET    /users/:id/edit(.:format) users#edit
     user GET    /users/:id(.:format)      users#show
          PATCH  /users/:id(.:format)      users#update
          PUT    /users/:id(.:format)      users#update
          DELETE /users/:id(.:format)      users#destroy

If we have long route names, they don’t fit on the terminal window as the output lines wrap with each other.

Example of overlapping routes

Rails 6 has added a way to display the routes in an expanded format.

We can pass --expanded switch to the rails routes command to see this in action.

$ rails routes --expanded

--[ Route 1 ]--------------------------------------------------------------
Prefix            | users
Verb              | GET
URI               | /users(.:format)
Controller#Action | users#index
--[ Route 2 ]--------------------------------------------------------------
Prefix            |
Verb              | POST
URI               | /users(.:format)
Controller#Action | users#create
--[ Route 3 ]--------------------------------------------------------------
Prefix            | new_user
Verb              | GET
URI               | /users/new(.:format)
Controller#Action | users#new
--[ Route 4 ]--------------------------------------------------------------
Prefix            | edit_user
Verb              | GET
URI               | /users/:id/edit(.:format)
Controller#Action | users#edit

This shows the output of the routes command in much more user friendly manner.

The --expanded switch can be used in conjunction with other switches for searching specific routes.


Rails 6 adds ActiveModel::Errors#slice!

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Rails 6 added slice! on ActiveModel::Errors. With this addition, it becomes quite easy to select just few keys from errors and show or return them. Before Rails 6, we needed to convert ActiveModel::Errors object to a hash before slicing the keys.

Let’s checkout how it works.

Rails 5.2

>> user = User.new

=> #<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>

>> user.valid?

=> false

>> user.errors

=> #<ActiveModel::Errors:0x00007fc46700df10 @base=#<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>, @messages={:email=>["can't be blank"], :password=>["can't be blank"]}, @details={:email=>[{:error=>:blank}], :password=>[{:error=>:blank}]}>

>> user.errors.slice!

=> Traceback (most recent call last):
        1: from (irb):16
NoMethodError (undefined method 'slice!' for #<ActiveModel::Errors:0x00007fa1f0e46eb8>)
Did you mean?  slice_when

>> errors = user.errors.to_h
>> errors.slice!(:email)

=> {:password=>["can't be blank"]}

>> errors

=> {:email=>["can't be blank"]}

Rails 6.0.0.beta2

>> user = User.new

=> #<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>

>> user.valid?

=> false

>> user.errors

=> #<ActiveModel::Errors:0x00007fc46700df10 @base=#<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>, @messages={:email=>["can't be blank"], :password=>["can't be blank"]}, @details={:email=>[{:error=>:blank}], :password=>[{:error=>:blank}]}>

>> user.errors.slice!(:email)

=> {:password=>["can't be blank"]}

>> user.errors

=> #<ActiveModel::Errors:0x00007fc46700df10 @base=#<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>, @messages={:email=>["can't be blank"]}, @details={:email=>[{:error=>:blank}]}>

Here is the relevant pull request.


Rails 6 adds create_or_find_by and create_or_find_by!

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Rails 6 added create_or_find_by and create_or_find_by!. Both methods rely on unique constraints on database level. If creation fails because of unique constraints on one or all of the given columns, it tries to find the record using find_by!.

create_or_find_by is an improvement over find_or_create_by because find_or_create_by first queries for the record and then inserts it if none is found. This might lead to a race condition.

As mentioned by DHH in the pull request, create_or_find_by has few cons too:

  • The table must have unique constraints on relevant columns.
  • This method relies on exception handling which is generally slower.

create_or_find_by! raises exception when creation fails because of validations.

Let’s see how both methods work in Rails 6.0.0.beta2.

Rails 6.0.0.beta2

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> class User < ApplicationRecord
>>   validates :name, presence: true
>> end


>> User.create_or_find_by(name: 'Amit')
BEGIN
INSERT INTO "users" ("name", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id"  [["name", "Amit"], ["created_at", "2019-03-07 09:33:23.391719"], ["updated_at", "2019-03-07 09:33:23.391719"]]
COMMIT

=> #<User id: 1, name: "Amit", created_at: "2019-03-07 09:33:23", updated_at: "2019-03-07 09:33:23">

>> User.create_or_find_by(name: 'Amit')
BEGIN
INSERT INTO "users" ("name", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id"  [["name", "Amit"], ["created_at", "2019-03-07 09:46:37.189068"], ["updated_at", "2019-03-07 09:46:37.189068"]]
ROLLBACK

=> #<User id: 1, name: "Amit", created_at: "2019-03-07 09:33:23", updated_at: "2019-03-07 09:33:23">

>> User.create_or_find_by(name: nil)
BEGIN
COMMIT

=> #<User id: nil, name: nil, created_at: nil, updated_at: nil>

>> User.create_or_find_by!(name: nil)

=> Traceback (most recent call last):
        1: from (irb):2
ActiveRecord::RecordInvalid (Validation failed: Name can't be blank)

Here is the relevant pull request.

Also note that create_or_find_by can lead to primary keys running out if the type of primary key is int. This happens because each time create_or_find_by hits ActiveRecord::RecordNotUnique, it does not rollback auto-increment of primary key. The problem is discussed in this pull request.


Rails 6 raises ActiveModel::MissingAttributeError when update_columns is used with non-existing attribute

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Rails 6 raises ActiveModel::MissingAttributeError when update_columns is used with a non-existing attribute. Before Rails 6, update_columns raises ActiveRecord::StatementInvalid error.

Rails 5.2

>> User.first.update_columns(email: 'amit@bigbinary.com')
SELECT  "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]
UPDATE "users" SET "email" = $1 WHERE "users"."id" = $2  [["email", "amit@bigbinary.com"], ["id", 1]]

=> Traceback (most recent call last):
        1: from (irb):8
ActiveRecord::StatementInvalid (PG::UndefinedColumn: ERROR:  column "email" of relation "users" does not exist)
LINE 1: UPDATE "users" SET "email" = $1 WHERE "users"."id" = $2
                           ^
: UPDATE "users" SET "email" = $1 WHERE "users"."id" = $2

Rails 6.0.0.beta2

>> User.first.update_columns(email: 'amit@bigbinary.com')
SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT ?  [["LIMIT", 1]]

Traceback (most recent call last):
        1: from (irb):1
ActiveModel::MissingAttributeError (can't write unknown attribute `email`)

Here is the relevant commit.


Rails 6 changed ActiveRecord::Base.configurations result to an object

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Rails 6 changed return value of ActiveRecord::Base.configurations to an object of ActiveRecord::DatabaseConfigurations. Before Rails 6, ActiveRecord::Base.configurations returned a hash with all the database configurations. We can call to_h on object of ActiveRecord::DatabaseConfigurations to get a hash.

A method named as configs_for has also been added on to fetch configurations for a particular environment.

Rails 5.2

>> ActiveRecord::Base.configurations

=> {"development"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}, "test"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/test.sqlite3"}, "production"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/production.sqlite3"}}

Rails 6.0.0.beta2

>> ActiveRecord::Base.configurations

=> #<ActiveRecord::DatabaseConfigurations:0x00007fc18274f9f0 @configurations=[#<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fc18274f680 @env_name="development", @spec_name="primary", @config={"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fc18274f608 @env_name="test", @spec_name="primary", @config={"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/test.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fc18274f590 @env_name="production", @spec_name="primary", @config={"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/production.sqlite3"}>]>

>> ActiveRecord::Base.configurations.to_h

=> {"development"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}, "test"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/test.sqlite3"}, "production"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/production.sqlite3"}}

>> ActiveRecord::Base.configurations['development']

=> {"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}

>> ActiveRecord::Base.configurations.configs_for(env_name: "development")

=> [#<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fc18274f680 @env_name="development", @spec_name="primary", @config={"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}>]

Here is the relevant pull request.


Rails 6 shows unpermitted params in logs in color

Strong parameters allow us to control the user input in our Rails app. In development environment the unpermitted parameters are shown in the log as follows.

Unpermitted params before Rails 6

It is easy to miss this message in the flurry of other messages.

Rails 6 has added a change to show these params in red color for better visibility.

Unpermitted params after Rails 6


Marketing strategy at BigBinary

BigBinary started in 2011. Here are our revenue numbers for the last 7 years.

BigBinary revenue

We achieved this to date without having any outbound marketing and sales strategy.

  • We have never sent a cold email.
  • We have never sent a cold LinkedIn message.
  • The only time we advertised was a period of two months when we tried Google advertisements, with no outcomes.
  • We do not sponsor any podcast.
  • We have not had a sales person.
  • We have not had a marketing person.

We have kept our head down and have focused on what we do best, such as designing, developing, debugging, devops, and blogging.

This is what has worked out for us so far:

  • We contribute to the community through blog posts and open source.
  • We sponsor community events like Rails Girls and Ruby Conf India.
  • We sponsor many React and Ruby meetups.
  • We focus on keeping our existing clients happy.

Over the years I have come across many people who aspire to be freelancers. While it is not for everyone, I encourage them to give freelancing a try.

The greatest hindrance I have seen is that they stress over sales and marketing, and as it should be. Being a freelancer means constant need to find your next client.

I’m not here to say what others ought to do. I’m here to say what has worked out for BigBinary over the last 7 years.

While we plan to experiment with new forms of marketing, networking, and sales channel as we grow, it is not the end-all-be-all for freelancers. While marketing, networking, and sales may be effective for some, it was not how we started BigBinary and may not be how you want to start as well.

For us at BigBinary, it has been writing blogs. When we come across a potentially intriguing blog topic, we save the topic by creating a Github issue. When we have downtime, we pick up a topic from our issues list. It’s as simple as that and has been our primary driver of growth thus far.

While you should experiment to find out what works best for you, you need to find out what suits your personality. If you are good at teaching through videos, consider creating your own YouTube channel. If you contribute to open source, try creating a blog about your efforts and learnings. If you are good at concentrating on a niche technology, build your marketing and business around that.

I can confidently say that majority of people I met and who want to be freelancer would do fine if they simply share what they are learning. Most of these people do technical work. Some of them already blog and others can blog. A blog is a decent start nearly everybody will say. I’m saying that it is a good end too.

If you do not want to do any other form of marketing then that’s fine too. Just blogging will work out fine for you just like it has worked out fine for us at BigBinary.

Just because you are going to be a freelancer you don’t have to change who you are. If you don’t like sending cold emails then don’t. If you do not like networking then that’s alright as well. Write personal emails, dump corporate talk, show compassion and be genuine.

So go on and do some freelancing. It would teach you a lot about software development, business, life, managing money, creating value and capturing value. It will be rough at times. And it would be hard at times. But it would also be a ton of fun.


Rails 6 adds delete_by and destroy_by as ActiveRecord::Relation methods

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

As described by DHH in the issue, Rails has find_or_create_by, find_by and similar methods to create and find the records matching the specified conditions. Rails was missing similar feature for deleting/destroying the record(s).

Before Rails 6, deleting/destroying the record(s) which are matching the given condition was done as shown below.

  # Example to destroy all authors matching the given condition
  Author.find_by(email: "abhay@example.com").destroy
  Author.where(email: "abhay@example.com", rating: 4).destroy_all

  # Example to delete all authors matching the given condition
  Author.find_by(email: "abhay@example.com").delete
  Author.where(email: "abhay@example.com", rating: 4).delete_all

The above examples were missing the symmetry like find_or_create_by and find_by methods.

In Rails 6, the new delete_by and destroy_by methods have been added as ActiveRecord::Relation methods. ActiveRecord::Relation#delete_by is short-hand for relation.where(conditions).delete_all. Similarly, ActiveRecord::Relation#destroy_by is short-hand for relation.where(conditions).destroy_all.

Here is how it can be used.

  # Example to destroy all authors matching the given condition using destroy_by
  Author.destroy_by(email: "abhay@example.com")
  Author.destroy_by(email: "abhay@example.com", rating: 4)

  # Example to destroy all authors matching the given condition using delete_by
  Author.delete_by(email: "abhay@example.com")
  Author.delete_by(email: "abhay@example.com", rating: 4)

Check out the pull request for more details on this.


Rails 6 adds ActiveRecord::Relation#touch_all

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

Before moving forward, we need to understand what touch method is. touch is used to update updated_at timestamp by default to current time. It also takes custom time or different columns as parameters.

Rails 6 has added touch_all on ActiveRecord::Relation to touch multiple records in a go. Before Rails 6, we needed to iterate on all records using an iterator to achieve this.

Let’s take an example in which we call touch_all on all User records.

Rails 5.2

>> User.count
SELECT COUNT(\*) FROM "users"

=> 3

>> User.all.touch_all

=> Traceback (most recent call last):1: from (irb):2
NoMethodError (undefined method 'touch_all' for #<User::ActiveRecord_Relation:0x00007fe6261f9c58>)

>> User.all.each(&:touch)
SELECT "users".* FROM "users"
begin transaction
  UPDATE "users" SET "updated_at" = ? WHERE "users"."id" = ?  [["updated_at", "2019-03-05 17:45:51.495203"], ["id", 1]]
commit transaction
begin transaction
  UPDATE "users" SET "updated_at" = ? WHERE "users"."id" = ?  [["updated_at", "2019-03-05 17:45:51.503415"], ["id", 2]]
commit transaction
begin transaction
  UPDATE "users" SET "updated_at" = ? WHERE "users"."id" = ?  [["updated_at", "2019-03-05 17:45:51.509058"], ["id", 3]]
commit transaction

=> [#<User id: 1, name: "Sam", created_at: "2019-03-05 16:09:29", updated_at: "2019-03-05 17:45:51">, #<User id: 2, name: "John", created_at: "2019-03-05 16:09:43", updated_at: "2019-03-05 17:45:51">, #<User id: 3, name: "Mark", created_at: "2019-03-05 16:09:45", updated_at: "2019-03-05 17:45:51">]

Rails 6.0.0.beta2

>> User.count
SELECT COUNT(*) FROM "users"

=> 3

>> User.all.touch_all
UPDATE "users" SET "updated_at" = ?  [["updated_at", "2019-03-05 16:08:47.490507"]]

=> 3

touch_all returns count of the records on which it is called.

touch_all also takes a custom time or different columns as parameters.

Rails 6.0.0.beta2

>> User.count
SELECT COUNT(*) FROM "users"

=> 3

>> User.all.touch_all(time: Time.new(2019, 3, 2, 1, 0, 0))
UPDATE "users" SET "updated_at" = ?  [["updated_at", "2019-03-02 00:00:00"]]

=> 3

>> User.all.touch_all(:created_at)
UPDATE "users" SET "updated_at" = ?, "created_at" = ?  [["updated_at", "2019-03-05 17:55:41.828347"], ["created_at", "2019-03-05 17:55:41.828347"]]

=> 3

Here is the relevant pull request.


Rails 6 adds negative scopes on enum

This blog is part of our Rails 6 series. Rails 6.0.0.beta3 was recently released.

When an enum attribute is defined on a model, Rails adds some default scopes to filter records based on values of enum on enum field.

Here is how enum scope can be used.

class Post < ActiveRecord::Base
  enum status: %i[drafted active trashed]
end

Post.drafted # => where(status: :drafted)
Post.active  # => where(status: :active)

In Rails 6, negative scopes are added on the enum values.

As mentioned by DHH in the pull request,

these negative scopes are convenient when you want to disallow access in controllers

Here is how they can be used.

class Post < ActiveRecord::Base
  enum status: %i[drafted active trashed]
end

Post.not_drafted # => where.not(status: :drafted)
Post.not_active  # => where.not(status: :active)

Check out the pull request for more details on this.


MJIT Support in Ruby 2.6

This blog is part of our Ruby 2.6 series. Ruby 2.6.0 was released on Dec 25, 2018.

What is JIT?

JIT stands for Just-In-Time compiler. JIT converts repetitive code into bytecode which can then be sent to the processor directly, hence, saving time by not compiling the same piece of code over and over.

Ruby 2.6

MJIT is introduced in Ruby 2.6. It is most commonly known as MRI JIT or Method Based JIT.

It is a part of the Ruby 3x3 project started by Matz. The name “Ruby 3x3” signifies Ruby 3.0 will be 3 times faster than Ruby 2.0 and it will focus mainly on performance. In addition to performance, it also aims for the following things:

  1. Portability
  2. Stability
  3. Security

MJIT is still in development, therefore, MJIT is optional in Ruby 2.6. If you are running Ruby 2.6, then you can execute the following commnad.

ruby --help

You will see following options.

--Jit-wait # Wait program execution until code compiles.
--jit-verbose=num # Level information MJIT compiler prints for Ruby program.
--jit-min-calls=num # Minimum count in loops for which MJIT should work.
--jit-max-cache
--jit-save-temps # Save compiled library to the file.

Vladimir Makarov proposed improving performance by replacing VM instructions with RTL(Register Transfer Language) and introducing the Method based JIT compiler.

Vladimir explained MJIT architecture in his RubyKaigi 2017 conference keynote.

Ruby’s compiler converts the code to YARV(Yet Another Ruby VM) instructions and then these instructions are run by the Ruby Virtual Machine. Code that is executed too often is converted to RTL instructions, which runs faster.

Let’s take a look at how MJIT works.

# mjit.rb

require 'benchmark'

puts Benchmark.measure {
  def test_while
    start_time = Time.now
    i = 0

    while i < 4
      i += 1
    end

    i
    puts Time.now - start_time
  end

  4.times { test_while }
}

Let’s run this code with MJIT options and check what we got.

ruby --jit --jit-verbose=1 --jit-wait --disable-gems mjit.rb
Time taken is 4.0e-06
Time taken is 0.0
Time taken is 0.0
Time taken is 0.0
  0.000082   0.000032   0.000114 (  0.000105)
Successful MJIT finish

Nothing interesting right? And why is that? because we are iterating the loop for 4 times and default value for MJIT to work is 5. We can always decide after how many calls MJIT should work by providing --jit-min-calls=#number option.

Let’s tweak the program a bit so MJIT gets to work.

require 'benchmark'

puts Benchmark.measure {
  def test_while
    start_time = Time.now
    i = 0

    while i < 4_00_00_000
      i += 1
    end

    puts "Time taken is #{Time.now - start_time}"
  end

  10.times { test_while }
}

After running the above code we can see some work done by MJIT.

Time taken is 0.457916
Time taken is 0.455921
Time taken is 0.454672
Time taken is 0.452823
JIT success (72.5ms): block (2 levels) in <main>@mjit.rb:15 -> /var/folders/v6/_6sh53vn5gl3lct18w533gr80000gn/T//_ruby_mjit_p66220u0.c
JIT success (140.9ms): test_while@mjit.rb:4 -> /var/folders/v6/_6sh53vn5gl3lct18w533gr80000gn/T//_ruby_mjit_p66220u1.c
JIT compaction (23.0ms): Compacted 2 methods -> /var/folders/v6/_6sh53vn5gl3lct18w533gr80000gn/T//_ruby_mjit_p66220u2.bundle
Time taken is 0.463703
Time taken is 0.102852
Time taken is 0.103335
Time taken is 0.103299
Time taken is 0.103252
Time taken is 0.103261
  2.797843   0.005357   3.141944 (  2.801391)
Successful MJIT finish

Here’s what’s happening. Method ran 4 times and on the 5th call it found it is running same code again. So MJIT started a separate thread to convert the code into RTL instructions, which created a shared object library. Next, threads took that shared code and executed directly. As we passed option --jit-verbose=1 we can see what MJIT did.

What we are seeing in output is the following:

  1. Time taken to compile.
  2. What block of code is compiled by JIT.
  3. Location of compiled code.

We can open the file and see how MJIT converted the piece of code to binary instructions but for that we need to pass another option which is --jit-save-temps and then just inspect those files.

After compiling the code to RTL instructions, take a look at the execution time. It dropped down to 0.10 ms from 0.46 ms. That’s a neat speed bump.

Here is a comparation across some of the Ruby versions for some basic operations.

Ruby time comparison in different versions

Rails comparison on Ruby 2.5, Ruby 2.6 and Ruby 2.6 with JIT

Create a rails application with different Ruby versions and start a server. We can start the rails server with the JIT option, as shown below.

RUBYOPT="--jit" bundle exec rails s

Now, we can start testing the performance on servers. We found that Ruby 2.6 is faster than Ruby 2.5, but enabling JIT in Ruby 2.6 does not add more value to the Rails application.

MJIT status and future directions

  • It is in an early development stage.
  • Does not work on windows.
  • Needs more time to mature.
  • Needs more optimisations.
  • MJIT can use GCC or LLVM in the future C Compilers.

Further reading

  1. Ruby 3x3 Performance Goal
  2. The method JIT compiler for Ruby2.6
  3. Vladimir Makarov’s Ruby Edition