San Francisco, USA

5214F Diamond Heights Blvd #553
San Francisco, CA 94131

Pune, India

203, Jewel Towers, 2nd Floor
Lane Number 5, Koregaon Park
Pune 411001, India

301 - 275 - 3997
hello@BigBinary.com

Rails 6 adds support of symbol keys with ActiveSupport::HashWithIndifferentAccess#assoc

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails 6 added support of symbol keys with ActiveSupport::HashWithIndifferentAccess#assoc.

Please note that documentation of ActiveSupport::HashWithIndifferentAccess#assoc in Rails 5.2 shows that ActiveSupport::HashWithIndifferentAccess#assoc works with symbol keys but it doesn’t.

In Rails 6, ActiveSupport::HashWithIndifferentAccess implements a hash where string and symbol keys are considered to be the same.

Before Rails 6, HashWithIndifferentAccess#assoc used to work with just string keys.

Let’s checkout how it works.

Rails 5.2

Let’s create an object of ActiveSupport::HashWithIndifferentAccess and call assoc on that object.

>> info = { name: 'Mark', email: 'mark@bigbinary.com' }.with_indifferent_access

=> {"name"=>"Mark", "email"=>"mark@bigbinary.com"}

>> info.assoc(:name)

=> nil

>> info.assoc('name')

=> ["name", "Mark"]

We can see that assoc does not work with symbol keys with ActiveSupport::HashWithIndifferentAccess in Rails 5.2.

Rails 6.0.0.beta2

Now, let’s call assoc on the same hash in Rails 6 with both string and symbol keys.

>> info = { name: 'Mark', email: 'mark@bigbinary.com' }.with_indifferent_access

=> {"name"=>"Mark", "email"=>"mark@bigbinary.com"}

>> info.assoc(:name)

=> ["name", "Mark"]

>> info.assoc('name')

=> ["name", "Mark"]

As we can see, assoc works perfectly fine with both string and symbol keys with ActiveSupport::HashWithIndifferentAccess in Rails 6.

Here is the relevant pull request.


Rails 6 preserves status of #html_safe? on sliced and multiplied HTML safe strings

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Before Rails 6

Before Rails 6, calling #html_safe? on a slice of an HTML safe string returns nil.

>> html_content = "<div>Hello, world!</div>".html_safe
# => "<div>Hello, world!</div>"
>> html_content.html_safe?
# => true
>> html_content[0..-1].html_safe?
# => nil

Also, before Rails 6, the ActiveSupport::SafeBuffer#* method does not preserve the HTML safe status as well.

>> line_break = "<br />".html_safe
# => "<br />"
>> line_break.html_safe?
# => true
>> two_line_breaks = (line_break * 2)
# => "<br /><br />"
>> two_line_breaks.html_safe?
# => nil

Rails 6 returns expected status of #html_safe?

In Rails 6, both of the above cases have been fixed properly.

Therefore, we will now get the status of #html_safe? as expected.

>> html_content = "<div>Hello, world!</div>".html_safe
# => "<div>Hello, world!</div>"
>> html_content.html_safe?
# => true
>> html_content[0..-1].html_safe?
# => true

>> line_break = "<br />".html_safe
# => "<br />"
>> line_break.html_safe?
# => true
>> two_line_breaks = (line_break * 2)
# => "<br /><br />"
>> two_line_breaks.html_safe?
# => true

Please check rails/rails#33808 and rails/rails#36012 for the relevant changes.


Recyclable cache keys in Rails

Recyclable cache keys or cache versioning was introduced in Rails 5.2. Large applications frequently need to invalidate their cache because cache store has limited memory. We can optimize cache storage and minimize cache miss using recyclable cache keys.

Recyclable cache keys is supported by all cache stores that ship with Rails.

Before Rails 5.2, cache_key’s format was {model_name}/{id}-{update_at}. Here model_name and id are always constant for an object and updated_at changes on every update.

Rails 5.1

>> post = Post.last

>> post.cache_key
=> "posts/1-20190522104553296111"

# Update post
>> post.touch

>> post.cache_key
=> "posts/1-20190525102103422069" # cache_key changed

In Rails 5.2, #cache_key returns {model_name}/{id} and new method #cache_version returns {updated_at}.

Rails 5.2

>> ActiveRecord::Base.cache_versioning = true

>> post = Post.last

>> post.cache_key
=> "posts/1"

>> post.cache_version
=> "20190522070715422750"

>> post.cache_key_with_version
=> "posts/1-20190522070715422750"

Let’s update post instance and check cache_key and cache_version’s behaviour.

>> post.touch

>> post.cache_key
=> "posts/1" # cache_key remains same

>> post.cache_version
=> "20190527062249879829" # cache_version changed

To use cache versioning feature, we have to enable ActiveRecord::Base.cache_versioning configuration. By default cache_versioning config is set to false for backward compatibility.

We can enable cache versioning configuration globally as shown below.

ActiveRecord::Base.cache_versioning = true
# or
config.active_record.cache_versioning = true

Cache versioning config can be applied at model level.

class Post < ActiveRecord::Base
  self.cache_versioning = true
end

# Or, when setting `#cache_versioning` outside the model -

Post.cache_versioning = true

Let’s understand the problem step by step with cache keys before Rails 5.2.

Rails 5.1 (without cache versioning)

1. Write post instance to cache using fetch api.

>> before_update_cache_key = post.cache_key
=> "posts/1-20190527062249879829"

>> Rails.cache.fetch(before_update_cache_key) { post }
=> #<Post id: 1, title: "First Post", created_at: "2019-05-22 17:23:22", updated_at: "2019-05-27 06:22:49">

2. Update post instance using touch.

>> post.touch
   (0.1ms)  begin transaction
  Post Update (1.6ms)  UPDATE "posts" SET "updated_at" = ? WHERE "posts"."id" = ?  [["updated_at", "2019-05-27 08:01:52.975653"], ["id", 1]]
   (1.2ms)  commit transaction
=> true

3. Verify stale cache_key in cache store.

>> Rails.cache.fetch(before_update_cache_key)
=> #<Post id: 1, title: "First Post", created_at: "2019-05-22 17:23:22", updated_at: "2019-05-27 06:22:49">

4. Write updated post instance to cache using new cache_key.

>> after_update_cache_key = post.cache_key
=> "posts/1-20190527080152975653"

>> Rails.cache.fetch(after_update_cache_key) { post }
=> #<Post id: 1, title: "First Post", created_at: "2019-05-22 17:23:22", updated_at: "2019-05-27 08:01:52">

5. Cache store now has two copies of post instance.

>> Rails.cache.fetch(before_update_cache_key)
=> #<Post id: 1, title: "First Post", created_at: "2019-05-22 17:23:22", updated_at: "2019-05-27 06:22:49">

>> Rails.cache.fetch(after_update_cache_key)
=> #<Post id: 1, title: "First Post", created_at: "2019-05-22 17:23:22", updated_at: "2019-05-27 08:01:52">

cache_key and its associated instance becomes irrelevant as soon as an instance is updated. But it stays in cache store until it is manually invalidated.

This sometimes result in overflowing cache store with stale keys and data. In applications that extensively use cache store, a huge chunk of cache store gets filled with stale data frequently.

Now let’s take a look at the same example. This time with cache versioning to understand how recyclable cache keys help optimize cache storage.

Rails 5.2 (cache versioning)

1. Write post instance to cache store with version option.

>> ActiveRecord::Base.cache_versioning = true

>> post = Post.last

>> cache_key = post.cache_key
=> "posts/1"

>> before_update_cache_version = post.cache_version
=> "20190527080152975653"

>> Rails.cache.fetch(cache_key, version: before_update_cache_version) { post }
=> #<Post id: 1, title: "First Post", created_at: "2019-05-22 17:23:22", updated_at: "2019-05-27 08:01:52">

2. Update post instance.

>> post.touch
   (0.1ms)  begin transaction
  Post Update (0.4ms)  UPDATE "posts" SET "updated_at" = ? WHERE "posts"."id" = ?  [["updated_at", "2019-05-27 09:09:15.651029"], ["id", 1]]
   (0.7ms)  commit transaction
=> true

3. Verify stale cache_version in cache store.

>> Rails.cache.fetch(cache_key, version: before_update_cache_version)
=> #<Post id: 1, title: "First Post", created_at: "2019-05-22 17:23:22", updated_at: "2019-05-27 08:01:52">

4. Write updated post instance to cache.

>> after_update_cache_version = post.cache_version
=> "20190527090915651029"

>> Rails.cache.fetch(cache_key, version: after_update_cache_version) { post }
=> #<Post id: 1, title: "First Post", created_at: "2019-05-22 17:23:22", updated_at: "2019-05-27 09:09:15">

5. Cache store has replaced old copy of post with new version automatically.

>> Rails.cache.fetch(cache_key, version: before_update_cache_version)
=> nil

>> Rails.cache.fetch(cache_key, version: after_update_cache_version)
=> #<Post id: 1, title: "First Post", created_at: "2019-05-22 17:23:22", updated_at: "2019-05-27 09:09:15">

Above example shows how recyclable cache keys maintains single, latest copy of an instance. Stale versions are removed automatically when new version is added to cache store.

Rails 6 added #cache_versioning for ActiveRecord::Relation.

ActiveRecord::Base.collection_cache_versioning configuration should be enabled to use cache versioning feature on collections. It is set to false by default.

We can enable this configuration as shown below.

ActiveRecord::Base.collection_cache_versioning = true
# or
config.active_record.collection_cache_versioning = true

Before Rails 6, ActiveRecord::Relation had cache_key in format {table_name}/query-{query-hash}-{count}-{max(updated_at)}.

In Rails 6, cache_key is split in stable part cache_key - {table_name}/query-{query-hash} and volatile part cache_version - {count}-{max(updated_at)}.

For more information, check out blog on ActiveRecord::Relation#cache_key in Rails 5.

Rails 5.2

>> posts = Post.all

>> posts.cache_key
=> "posts/query-00644b6a00f2ed4b925407d06501c8fb-3-20190522172326885804"

Rails 6

>> ActiveRecord::Base.collection_cache_versioning = true

>> posts = Post.all

>> posts.cache_key
=> "posts/query-00644b6a00f2ed4b925407d06501c8fb"

>> posts.cache_version
=> "3-20190522172326885804"

Cache versioning works similarly for ActiveRecord::Relation as ActiveRecord::Base.

In case of ActiveRecord::Relation, if number of records change and/or record(s) are updated, then same cache_key is written to cache store with new cache_version and updated records.

Conclusion

Previously, cache invalidation had to be done manually either by deleting cache or setting cache expire duration. Cache versioning invalidates stale data automatically and keeps latest copy of data, saving on storage and performance drastically.

Check out the pull request and commit for more details.


Rails 6 deprecates where.not working as NOR and will change to NAND in Rails 6.1

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

A notable deprecation warning has been added in Rails 6 when using where.not with multiple attributes.

Before Rails 6, if we use where.not with multiple attributes, it applies logical NOR (NOT(A) AND NOT(B)) in WHERE clause of the query. This does not always work as expected.

Let’s look at an example to understand this better.

We have Post model with a polymorphic association.

Rails 5.2
>> Post.all
=> #<ActiveRecord::Relation [
#<Post id: 1, title: "First Post", source_type: "Feed", source_id: 100>,
#<Post id: 2, title: "Second Post", source_type: "Feed", source_id: 101>]>

>> Post.where(source_type: "Feed", source_id: 100)
=> #<ActiveRecord::Relation [#<Post id: 1, title: "First Post", source_type: "Feed", source_id: 100>]>

>> Post.where.not(source_type: "Feed", source_id: 100)
=> #<ActiveRecord::Relation []>

In the last query, we expect ActiveRecord to fetch one record.

Let’s check SQL generated for the above case.

>> Post.where.not(source_type: "Feed", source_id: 100).to_sql

=> SELECT "posts".* FROM "posts" WHERE "posts"."source_type" != 'Feed' AND "posts"."source_id" != 100

where.not applies AND to the negation of source_type and source_id, and fails to fetch expected records.

In such cases, correct implementation of where.not would be logical NAND (NOT(A) OR NOT(B)).

Let us query posts table using NAND this time.

>> Post.where("source_type != 'Feed' OR source_id != 100")

   SELECT "posts".* FROM "posts" WHERE (source_type != 'Feed' OR source_id != 100)

=> #<ActiveRecord::Relation [#<Post id: 2, title: "Second Post", source_type: "Feed", source_id: 101>]>

Above query works as expected and returns one record. Rails 6.1 will change where.not working to NAND similar to the above query.

Rails 6.0.0.rc1
>> Post.where.not(source_type: "Feed", source_id: 100)

DEPRECATION WARNING: NOT conditions will no longer behave as NOR in Rails 6.1. To continue using NOR conditions, NOT each conditions manually (`.where.not(:source_type => ...).where.not(:source_id => ...)`). (called from irb_binding at (irb):1)

=> #<ActiveRecord::Relation []>

It is well mentioned in deprecation warning that if we wish to use NOR condition with multiple attributes, we can chain multiple where.not using a single predicate.

>> Post.where.not(source_type: "Feed").where.not(source_id: 100)

Here’s the relevant discussion and pull request for this change.


Rails 6 adds support for Optimizer Hints

Rails 6 has added support to provide optimizer hints.

What is Optimizer Hints?

Many relational database management systems (RDBMS) have a query optimizer. The job of the query optimizer is to determine the most efficient and fast plan to execute a given SQL query. Query optimizer has to consider all possible query execution plans before it can determine which plan is the optimal plan for executing the given SQL query and then compile and execute that query.

An optimal plan is chosen by the query optimizer by calculating the cost of each possible plans. Typically, when the number of tables referenced in a join query increases, then the time spent in query optimization grows exponentially which often affects the system’s performance. The fewer the execution plans the query optimizer needs to evaluate, the lesser time is spent in compiling and executing the query.

As an application designer, we might have more context about the data stored in our database. With the contextual knowledge about our database, we might be able to choose a more efficient execution plan than the query optimizer.

This is where the optimizer hints or optimizer guidelines come into picture.

Optimizer hints allow us to control the query optimizer to choose a certain query execution plan based on the specific criteria. In other words, we can hint the optimizer to use or ignore certain optimization plans using optimizer hints.

Usually, optimizer hints should be provided only when executing a complex query involving multiple table joins.

Note that the optimizer hints only affect an individual SQL statement. To alter the optimization strategies at the global level, there are different mechanisms supported by different databases. Optimizer hints provide finer control over other mechanisms which allow altering optimization plans by other means.

Optimizer hints are supported by many databases such as MySQL, PostgreSQL with the help of pg_hint_plan extension, Oracle, MS SQL, IBM DB2, etc. with varying syntax and options.

Optimizer Hints in Rails 6

Before Rails 6, we have to execute a raw SQL query to use the optimizer hints.

query = "SELECT
            /*+ JOIN_ORDER(articles, users) MAX_EXECUTION_TIME(60000) */
            articles.*
         FROM articles
         INNER JOIN users
         ON users.id = articles.user_id
         WHERE (published_at > '2019-02-17 13:15:44')
        ".squish

ActiveRecord::Base.connection.execute(query)

In the above query, we provided two optimizer hints to MySQL .

/*+ HINT_HERE ANOTHER_HINT_HERE ... */

Another approach to use optimizer hints prior to Rails 6 is to use a monkey patch like this.

In Rails 6, using optimizer hints is easier.

The same example looks like this in Rails 6.

Article
  .joins(:user)
  .where("published_at > ?", 2.months.ago)
  .optimizer_hints(
    "JOIN_ORDER(articles, users)",
    "MAX_EXECUTION_TIME(60000)"
  )

This produces the same SQL query as above but the result is of type ActiveRecord::Relation.

In PostgreSQL (using the pg_hint_plan extension), the optimizer hints have a different syntax.

Article
  .joins(:user)
  .where("published_at > ?", 2.months.ago)
  .optimizer_hints("Leading(articles users)", "SeqScan(articles)")

Please checkout the documentation of each database separately to learn the support and syntax of optimizer hints.

To learn more, please checkout this PR which introduced the #optimization_hints method to Rails 6.

Bonus example: Using optimizer hints to speedup a slow SQL statement in MySQL

Consider that we have articles table with some indexes.

class CreateArticles < ActiveRecord::Migration[6.0]
  def change
    create_table :articles do |t|
      t.string :title, null: false
      t.string :slug, null: false
      t.references :user
      t.datetime :published_at
      t.text :description

      t.timestamps

      t.index :slug, unique: true
      t.index [:published_at]
      t.index [:slug, :user_id]
      t.index [:published_at, :user_id]
      t.index [:title, :slug]
    end
  end
end

Let’s try to fetch all the articles which have been published in the last 2 months.

>> Article.joins(:user).where("published_at > ?", 2.months.ago)
# Article Load (10.5ms)  SELECT `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:38:18.647296')
=> #<ActiveRecord::Relation [#<Article id: 20, title: "Article 20", slug: "article-20", user_id: 1, ...]>

Let’s use EXPLAIN to investigate why it is taking 10.5ms to execute this query.

>> Article.joins(:user).where("published_at > ?", 2.months.ago).explain
# Article Load (13.9ms)  SELECT `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:39:05.380577')
=> # EXPLAIN for: SELECT `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:39:05.380577')
# +--------+----------+----------------+-----------+------+----------+-------+
# | select |   table  | possible_keys  | key       | rows | filtered | Extra |
# | _type  |          |                |           |      |          |       |
# +--------+----------+----------------+-----------+------+----------+-------+
# | SIMPLE |   users  | PRIMARY        | PRIMARY   | 2    | 100.0    | Using |
# |        |          |                |           |      |          | index |
# +--------+----------+----------------+-----------+------+----------+-------+
# | SIMPLE | articles | index          | index     | 9866 | 10.0     | Using |
# |        |          | _articles      | _articles |      |          | where |
# |        |          | _on_user_id,   | _on       |      |          |       |
# |        |          | index          | _user_id  |      |          |       |
# |        |          | _articles      |           |      |          |       |
# |        |          | _on            |           |      |          |       |
# |        |          | _published_at, |           |      |          |       |
# |        |          | index          |           |      |          |       |
# |        |          | _articles      |           |      |          |       |
# |        |          | _on            |           |      |          |       |
# |        |          | _published_at  |           |      |          |       |
# |        |          | _and_user_id   |           |      |          |       |
# +--------+----------+----------------+-----------+------+----------+-------+

According to the above table, it appears that the query optimizer is considering users table first and then the articles table.

The rows column indicates the estimated number of rows the query optimizer must examine to execute the query.

The filtered column indicates an estimated percentage of table rows that will be filtered by the table condition.

The formula rows x filtered gives the number of rows that will be joined with the following table.

Also,

  • For users table, the number of rows to be joined with the following table is 2 x 100% = 2,
  • For articles table, the number of rows to be joined with the following table is 500 * 7.79 = 38.95.

Since the articles tables contain more records which references very few records from the users table, it would be better to consider the articles table first and then the users table.

We can hint MySQL to consider the articles table first as follows.

>> Article.joins(:user).where("published_at > ?", 2.months.ago).optimizer_hints("JOIN_ORDER(articles, users)")
# Article Load (2.2ms)  SELECT `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:54:06.230651')
=> #<ActiveRecord::Relation [#<Article id: 20, title: "Article 20", slug: "article-20", user_id: 1, ...]>

Note that it took 2.2ms now to fetch the same records by providing JOIN_ORDER(articles, users) optimization hint.

Let’s try to EXPLAIN what changed by using this JOIN_ORDER(articles, users) optimization hint.

>> Article.joins(:user).where("published_at > ?", 2.months.ago).optimizer_hints("JOIN_ORDER(articles, users)").explain
# Article Load (4.1ms)  SELECT /*+ JOIN_ORDER(articles, users) */ `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:55:24.335152')
=> # EXPLAIN for: SELECT /*+ JOIN_ORDER(articles, users) */ `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:55:24.335152')
# +--------+----------+----------------+-----------+------+----------+--------+
# | select |   table  | possible_keys  | key       | rows | filtered | Extra  |
# | _type  |          |                |           |      |          |        |
# +--------+----------+----------------+-----------+------+----------+--------+
# | SIMPLE | articles | index          | index     | 769  | 100.0    | Using  |
# |        |          | _articles      | _articles |      |          | index  |
# |        |          | _on_user_id,   | _on       |      |          | condi  |
# |        |          | index          | _publish  |      |          | tion;  |
# |        |          | _articles      | ed_at,    |      |          | Using  |
# |        |          | _on            |           |      |          | where  |
# |        |          | _published_at, |           |      |          |        |
# |        |          | index          |           |      |          |        |
# |        |          | _articles      |           |      |          |        |
# |        |          | _on            |           |      |          |        |
# |        |          | _published_at  |           |      |          |        |
# |        |          | _and_user_id   |           |      |          |        |
# +--------+----------+----------------+-----------+------+----------+--------+
# | SIMPLE | users    | PRIMARY        | PRIMARY   | 2    | 100.0    | Using  |
# |        |          |                |           |      |          | index  |
# +--------+----------+----------------+-----------+------+----------+--------+

The result of the EXPLAIN query shows that the articles table was considered first and then the users table as expected. We can also see that the index_articles_on_published_at index key was considered from the possible keys to execute the given query. The filtered column for both tables shows that the number of filtered rows was 100% which means no filtering of rows occurred.

We hope this example helps in understanding how to use #explain and #optimization_hints methods in order to investigate and debug the performance issues and then fixing it.


Rails 6 reports object allocations made while rendering view templates

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Recently, Rails 6 added allocations feature to ActiveSupport::Notifications::Event. Using this feature, an event subscriber can see how many number of objects were allocated during the event’s start time and end time. We have written in detail about this feature here.

By taking the benefit of this feature, Rails 6 now reports the allocations made while rendering a view template, a partial and a collection.

Started GET "/articles" for ::1 at 2019-04-15 17:24:09 +0530
Processing by ArticlesController#index as HTML
  Rendering articles/index.html.erb within layouts/application
  Rendered shared/_ad_banner.html.erb (Duration: 0.1ms | Allocations: 6)
  Article Load (1.3ms)  SELECT "articles".* FROM "articles"
  ↳ app/views/articles/index.html.erb:5
  Rendered collection of articles/_article.html.erb [100 times] (Duration: 6.1ms | Allocations: 805)
  Rendered articles/index.html.erb within layouts/application (Duration: 17.6ms | Allocations: 3901)
Completed 200 OK in 86ms (Views: 83.6ms | ActiveRecord: 1.3ms | Allocations: 29347)

Notice the Allocations: information in the above logs.

We can see that

  • 6 objects were allocated while rendering shared/_ad_banner.html.erb view partial,
  • 805 objects were allocated while rendering a collection of 100 articles/_article.html.erb view partials,
  • and 3901 objects were allocated while rendering articles/index.html.erb view template.

We can use this information to understand how much time was spent while rendering a view template and how many objects were allocated in the process’ memory between the time when that view template had started rendering and the time when that view template had finished rendering.

To learn more about this feature, please check rails/rails#34136.

Note that we can also collect this information by subscribing to Action View hooks.

ActiveSupport::Notifications.subscribe /^render_.+.action_view$/ do |event|
  views_path = Rails.root.join("app/views/").to_s
  template_identifier = event.payload[:identifier]
  template_name = template_identifier.sub(views_path, "")
  message = "[#{event.name}] #{template_name} (Allocations: #{event.allocations})"

  ViewAllocationsLogger.log(message)
end

This should log something like this.

[render_partial.action_view] shared/_ad_banner.html.erb (Allocations: 43)

[render_collection.action_view] articles/_article.html.erb (Allocations: 842)

[render_template.action_view] articles/index.html.erb (Allocations: 4108)

Rails 6 adds ActiveRecord::Relation#annotate

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails 6 has added ActiveRecord::Relation#annotate to allow adding comments to the SQL queries generated by the ActiveRecord::Relation instance.

Here is how it can be used.

>> User.annotate("User whose name starts with 'A'").where("name LIKE ?", "A%")

SELECT "users".* FROM "users"
WHERE (name LIKE 'A%')
/* User whose name starts with 'A' */
LIMIT ?  [["LIMIT", 11]]

ActiveRecord::Relation#annotate allows to add multiple annotations on a query

>> bigbinary = Organization.find_by!(name: "BigBinary")
>> User.annotate("User whose name starts with 'A'")
       .annotate("AND belongs to BigBinary organization")
       .where("name LIKE ?", "A%")
       .where(organization: bigbinary)

SELECT "users".* FROM "users"
WHERE (name LIKE 'A%') AND "users"."organization_id" = ?
/* Users whose name starts with 'A' */
/* AND belongs to BigBinary organization */
LIMIT ?  [["organization_id", 1], ["LIMIT", 11]]

Also, ActiveRecord::Relation#annotate allows annotating scopes and model associations.

class User < ActiveRecord::Base
  scope :active, -> { where(status: 'active').annotate("Active users") }
end

>> User.active
SELECT "users".* FROM "users"
/* Active users */
LIMIT ?  [["LIMIT", 11]]

Check out the pull request for more details on this.


Rails 6 adds hooks to Active Job around retries and discards

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Before Rails 6

Before Rails 6, we have to provide a custom block to perform custom logging and monitoring around retries and discards of the jobs defined using Active Job framework.

class Container::DeleteJob < ActiveJob::Base
  retry_on Timeout::Error, wait: 2.seconds, attempts: 3 do |job, error|
    message = "Stopped retrying #{job.class} (JID #{job.job_id})
               with #{job.arguments.join(', ')} due to
               '#{error.class} - #{error.message}'.
               This job was retried for #{job.executions} times.".squish

    BackgroundJob::ErrorLogger.log(message)
  end

  discard_on Container::NotFoundError do |job, error|
    message = "Discarded #{job.class} (JID #{job.job_id})
               with #{job.arguments.join(', ')} due to
               '#{error.class} - #{error.message}' error.".squish

    BackgroundJob::ErrorLogger.log(message)
  end

  def perform(container_id)
    Container::DeleteService(container_id).process

    # Will raise Container::NotFoundError
    # if no container is found with 'container_id'.

    # Might raise Timeout::Error when the remote API is not responding.
  end
end

Notice the custom blocks provided to retry_on and discard_on methods to an individual job in the above example.

Extracting such custom logic to a base class or to a 3rd-party gem is possible but it will be non-standard and will be a bit difficult task.

An alternative approach is to subscribe to the hooks instrumented using Active Support Instrumentation API which is a standard and recommended way. Prior versions of Rails 6 already instruments some hooks such as enqueue_at.active_job, enqueue.active_job, perform_start.active_job, and perform.active_job. Unfortunately no hook is instrumented around retries and discards of an Active Job prior to Rails 6.

Rails 6

Rails 6 has introduced hooks to Active Job around retries and discards to which one can easily subscribe using Active Support Instrumentation API to perform custom logging and monitoring or to collect any custom information.

The newly introduced hooks are enqueue_retry.active_job, retry_stopped.active_job and discard.active_job.

Let’s discuss each of these hooks in detail.

Note that whenever we say a job, it means a job of type ActiveJob.

enqueue_retry.active_job

The enqueue_retry.active_job hook is instrumented when a job is enqueued to retry again due to occurrence of an exception which is configured using the retry_on method in the job’s definition. This hook is triggered only when above condition is satisfied and the number of executions of the job is less than the number of attempts defined using the retry_on method. The number of attempts is by default set to 5 if not defined explicitly.

This is how we would subscribe to this hook and perform custom logging in our Rails application.

ActiveSupport::Notifications.subscribe "enqueue_retry.active_job" do |*args|
  event = ActiveSupport::Notifications::Event.new *args
  payload = event.payload
  job = payload[:job]
  error = payload[:error]
  message = "#{job.class} (JID #{job.job_id})
             with arguments #{job.arguments.join(', ')}
             will be retried again in #{payload[:wait]}s
             due to error '#{error.class} - #{error.message}'.
             It is executed #{job.executions} times so far.".squish

  BackgroundJob::Logger.log(message)
end

Note that the BackgroundJob::Logger above is our custom logger. If we want, we can add any other logic instead.

We will change the definition of Container::DeleteJob job as below.

class Container::DeleteJob < ActiveJob::Base
  retry_on Timeout::Error, wait: 2.seconds, attempts: 3

  def perform(container_id)
    Container::DeleteService(container_id).process

    # Will raise Timeout::Error when the remote API is not responding.
  end
end

Let’s enqueue this job.

Container::DeleteJob.perform_now("container-1234")

Assume that this job keeps throwing Timeout::Error exception due to a network issue. The job will be retried twice since it is configured to retry when a Timeout::Error exception occurs up to maximum 3 attempts. While retrying this job, Active Job will instrument enqueue_retry.active_job hook along with the necessary job payload.

Since we have already subscribed to this hook, our subscriber would log something like this with the help of BackgroundJob::Logger.log.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 1 times so far.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 2 times so far.
retry_stopped.active_job

The retry_stopped.active_job hook is triggered when a job is retried up to the available number of attempts.

Let’s see how this hook is triggered.

Along with the subscription for the enqueue_retry.active_job hook, let’s subscribe to the retry_stopped.active_job hook, too.

ActiveSupport::Notifications.subscribe "retry_stopped.active_job" do |*args|
  event = ActiveSupport::Notifications::Event.new *args
  payload = event.payload
  job = payload[:job]
  error = payload[:error]
  message = "Stopped processing #{job.class} (JID #{job.job_id})
             further with arguments #{job.arguments.join(', ')}
             since it failed due to '#{error.class} - #{error.message}' error
             which reoccurred #{job.executions} times.".squish

  BackgroundJob::Logger.log(message)
end

Let’s keep the Container::DeleteJob job’s definition unchanged and enqueue it again.

Container::DeleteJob.perform_now("container-1234")

We will assume that the job will keep throwing Timeout::Error exception due to a network issue.

In the logs recorded using BackgroundJob::Logger.log, we should see something like this.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 1 times so far.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 2 times so far.

Stopped processing Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) further with arguments container-1234 since it failed due to 'Timeout::Error - Couldn't establish connection to cluster within 10s' error which reoccurred 3 times.

Notice the last entry in the logs above and its order.

discard.active_job

The discard.active_job hook is triggered when a job’s further execution is discarded due to occurrence of an exception which is configured using discard_on method.

To see how this hook is triggered, we will subscribe to the discard.active_job hook.

ActiveSupport::Notifications.subscribe "discard.active_job" do |*args|
  event = ActiveSupport::Notifications::Event.new *args
  payload = event.payload
  job = payload[:job]
  error = payload[:error]
  message = "Discarded #{job.class} (JID #{job.job_id})
             with arguments #{job.arguments.join(', ')}
             due to '#{error.class} - #{error.message}' error.".squish

  BackgroundJob::Logger.log(message)
end

We will configure our Container::DeleteJob job to discard when Container::NotFoundError exception occurs while executing the job.

class Container::DeleteJob < ActiveJob::Base
  discard_on Container::NotFoundError
  retry_on Timeout::Error, wait: 2.seconds, attempts: 3

  def perform(container_id)
    Container::DeleteService(container_id).process

    # Will raise Container::NotFoundError
    # if no container is found with 'container_id'.

    # Will raise Timeout::Error when the remote API is not responding.
  end
end

Let’s enqueue this job and assume that it would throw Container::NotFoundError exception.

Container::DeleteJob.perform_now("unknown-container-9876")

We should see following in the logs recorded by BackgroundJob::Logger.log.

Discarded Container::DeleteJob (JID e9b1cb5c-6d2d-49ae-b1d7-fef44f09ab8d) with arguments unknown-container-9876 due to 'Container::NotFoundError - Container 'unknown-container-9876' was not found' error.

Notes

  1. These new hooks are also instrumented for the jobs which are enqueued using perform_later method since both perform_now and perform_later calls perform method under the hood.

  2. Active Job already subscribes to the these hooks and writes them using the Rails’ default logger.

  3. If a block is provided to retry_on or discard_on methods then an applicable hook is instrumented first and then the given block is yielded.


Rails 6 adds support for Multi Environment credentials

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

In Rails 5.2, encrypted credentails are stored in the file config/credentials.yml.enc. This is a single flat file which is encrypted by the key located in config/master.key.

Rails 5.2 does not support storing credentials of different environments with different encryption keys. If we want environment specific encrypted credentials, we’ll have to follow this workaround.

Rails 6 has added support for Multi Environment credentials. With this change, credentials that belong to different environments can be stored in separate files with their own encryption key.

Let’s see how this works in Rails 6.0.0.beta3

Rails 6.0.0.beta3

If we want to add credentials to be used in staging environment, we can run

rails edit:credentials --environment staging

This will create the credentials file config/credentials/staging.yml.enc and a staging specific encryption key config/credentials/staging.key and open the credentials file in your text editor.

Let’s add our AWS access key id here.

aws:
  access_key_id: "STAGING_KEY"

We can then access the access_key_id in staging environment.

>> RAILS_ENV=staging rails c

pry(main)> Rails.application.credentials.aws[:access_key_id]

=> "STAGING_KEY"

Which takes precedence: Global or Environment Specific credentials?

Credentials added to global file config/credentials.yml.enc will not be loaded in environments which have their own environment specific credentials file (config/credentials/$environment.yml.enc).

So if we decide to add the following to the global credentials file, these credentials will not be available in staging. Since we already have a environment specific credentials file for staging.

aws:
  access_key_id: "DEFAULT_KEY"
stripe:
  secret_key: "DEFAULT_SECRET_KEY"
>> RAILS_ENV=staging rails c

pry(main)> Rails.application.credentials.aws[:access_key_id]

=> "STAGING_KEY"

pry(main)> Rails.application.credentials.stripe[:secret_key]

Traceback (most recent call last):
        1: from (irb):6
NoMethodError (undefined method `[]' for nil:NilClass)

Here is the relevant pull request


Rails 6 adds before? and after? to Date and Time

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails 6 adds before? and after? to Date , DateTime , Time and ActiveSupport::TimeWithZone classes.

before? and after? are aliases to < (less than) and > (greater than) methods respectively.

Let’s checkout how it works.

Rails 5.2

Let’s try calling before? on a date object in Rails 5.2.

>> Date.new(2019, 3, 31).before?(Date.new(2019, 4, 1))

=> NoMethodError: undefined method 'before?' for Sun, 31 Mar 2019:Date
	from (irb):1

>> Date.new(2019, 3, 31) < Date.new(2019, 4, 1)

=> true

Rails 6.0.0.beta2

Now, let’s compare Date , DateTime , Time and ActiveSupport::TimeWithZone objects using before? and after? in Rails 6.

>> Date.new(2019, 3, 31).before?(Date.new(2019, 4, 1))

=> true

>> Date.new(2019, 3, 31).after?(Date.new(2019, 4, 1))

=> false

>> DateTime.parse('2019-03-31').before?(DateTime.parse('2019-04-01'))

=> true

>> DateTime.parse('2019-03-31').after?(DateTime.parse('2019-04-01'))

=> false

>> Time.parse('2019-03-31').before?(Time.parse('2019-04-01'))

=> true

>> Time.parse('2019-03-31').after?(Time.parse('2019-04-01'))

=> false

>> ActiveSupport::TimeWithZone.new(Time.utc(2019, 3, 31, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]).before?(ActiveSupport::TimeWithZone.new(Time.utc(2019, 4, 1, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]))

=> true

>> ActiveSupport::TimeWithZone.new(Time.utc(2019, 3, 31, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]).after?(ActiveSupport::TimeWithZone.new(Time.utc(2019, 4, 1, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]))

=> false

Here is the relevant pull request for adding before? and after? methods and the pull request for moving before? and after? to DateAndTime::Calculations.


Rails 6 adds Array#extract!

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails 6 added extract! on Array class. extract! removes and returns the elements for which the given block returns true.

extract! is different from reject! in the way that reject! returns the array after removing the elements whereas extract! returns removed elements from the array.

Let’s checkout how it works.

Rails 6.0.0.beta2

Let’s pluck all the user emails and then extract emails which include gmail.com.

>> emails = User.pluck(:email)
SELECT "users"."email" FROM "users"

=> ["amit.choudhary@bigbinary.com", "amit@gmail.com", "mark@gmail.com", "sam@gmail.com"]

>> emails.extract! { |email| email.include?('gmail.com') }

=> ["amit@gmail.com", "mark@gmail.com", "sam@gmail.com"]

>> emails

=> ["amit.choudhary@bigbinary.com"]

>> emails = User.pluck(:email)
SELECT "users"."email" FROM "users"

=> ["amit.choudhary@bigbinary.com", "amit@gmail.com", "mark@gmail.com", "sam@gmail.com"]

>> emails.reject! { |email| email.include?('gmail.com') }

=> ["amit.choudhary@bigbinary.com"]

>> emails

=> ["amit.choudhary@bigbinary.com"]

Here is the relevant pull request.


Rails 6 adds Enumerable#index_with

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails 6 added index_with on Enumerable module. This will help in creating a hash from an enumerator with default or fetched values.

Before Rails 6, we can achieve this by calling map along with to_h.

index_with takes both value or a block as a parameter.

Let’s checkout how it works.

Rails 5.2

Let’s create a hash from an array in Rails 5.2 using map and to_h.

>> address = Address.first
SELECT "addresses".* FROM "addresses"
ORDER BY "addresses"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Address id: 1, first_name: "Amit", last_name: "Choudhary", state: "California", created_at: "2019-03-21 10:03:57", updated_at: "2019-03-21 10:03:57">

>> NAME_ATTRIBUTES = [:first_name, :last_name]

=> [:first_name, :last_name]

>> NAME_ATTRIBUTES.map { |attr| [attr, address.public_send(attr)] }.to_h

=> {:first_name=>"Amit", :last_name=>"Choudhary"}

Rails 6.0.0.beta2

Now let’s create the same hash from the array using index_with in Rails 6.

>> address = Address.first
SELECT "addresses".* FROM "addresses"
ORDER BY "addresses"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Address id: 1, first_name: "Amit", last_name: "Choudhary", state: "California", created_at: "2019-03-21 10:02:47", updated_at: "2019-03-21 10:02:47">

>> NAME_ATTRIBUTES = [:first_name, :last_name]

=> [:first_name, :last_name]

>> NAME_ATTRIBUTES.index_with { |attr| address.public_send(attr) }

=> {:first_name=>"Amit", :last_name=>"Choudhary"}

>> NAME_ATTRIBUTES.index_with('Default')

=> {:first_name=>"Default", :last_name=>"Default"}

Here is the relevant pull request.


Rails 6 adds private option to delegate method

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails 6 adds :private option to delegate method. After this addition, we can delegate methods in the private scope.

Let’s checkout how it works.

Rails 6.0.0.beta2

Let’s create two models named as Address and Order. Let’s also delegate validate_state method in Order to Address.

class Address < ApplicationRecord
  validates :first_name, :last_name, :state, presence: true

  DELIVERABLE_STATES = ['New York']

  def validate_state
    unless DELIVERABLE_STATES.include?(state)
      errors.add(:state, :invalid)
    end
  end
end

class Order < ApplicationRecord
  belongs_to :address

  delegate :validate_state, to: :address
end

>> Order.first
SELECT "orders".* FROM "orders"
ORDER BY "orders"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Order id: 1, amount: 0.1e2, address_id: 1, created_at: "2019-03-21 10:02:58", updated_at: "2019-03-21 10:17:44">

>> Address.first
SELECT "addresses".* FROM "addresses"
ORDER BY "addresses"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Address id: 1, first_name: "Amit", last_name: "Choudhary", state: "California", created_at: "2019-03-21 10:02:47", updated_at: "2019-03-21 10:02:47">

>> Order.first.validate_state
SELECT "orders".* FROM "orders"
ORDER BY "orders"."id" ASC LIMIT $1  [["LIMIT", 1]]

SELECT "addresses".* FROM "addresses"
WHERE "addresses"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]

=> ["is invalid"]

Now, let’s add private: true to the delegation.

class Order < ApplicationRecord
  belongs_to :address

  delegate :validate_state, to: :address, private: true
end

>> Order.first.validate_state
SELECT "orders".* FROM "orders"
ORDER BY "orders"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> Traceback (most recent call last):
        1: from (irb):7
NoMethodError (private method 'validate_state' called for #<Order:0x00007fb9d72fc1f8>
Did you mean?  validate)

As we can see, Rails now raises an exception of private method called if private option is set with delegate method.

Here is the relevant pull request.


Rails 6 allows spaces in postgres table names

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails 6 allows spaces in tables names in PostgreSQL. Before Rails 6, if we try to create a table named as user reviews, Rails tries to create a table named as reviews in schema named as user.

Let’s checkout how it works.

Rails 5.2

Let’s create a table user reviews in Rails 5.2.

>> class CreateUserReviews < ActiveRecord::Migration[5.2]
>>   def change
>>     create_table 'user reviews' do |t|
>>       t.string :value
>>
>>       t.timestamps
>>     end
>>   end
>> end

=> :change

>> CreateUserReviews.new.change
-- create_table("user reviews")
CREATE TABLE "user"."reviews" ("id" bigserial primary key, "value" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

=> Traceback (most recent call last):
        2: from (irb):10
        1: from (irb):3:in 'change'
ActiveRecord::StatementInvalid (PG::InvalidSchemaName: ERROR:  schema "user" does not exist)
LINE 1: CREATE TABLE "user"."reviews" ("id" bigserial primary key, "...
                     ^
: CREATE TABLE "user"."reviews" ("id" bigserial primary key, "value" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

We can see that Rails 5.2 raised an exception and tried to create table named as reviews in user schema.

Rails 6.0.0.beta2

Now, let’s create a table user reviews in Rails 6.

>> class CreateUserReviews < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table 'user reviews' do |t|
>>       t.string :value
>>
>>       t.timestamps
>>     end
>>   end
>> end

=> :change

>> CreateUserReviews.new.change
-- create_table("user reviews")
CREATE TABLE "user reviews" ("id" bigserial primary key, "value" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> #<PG::Result:0x00007f9d633c5458 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

Now, we can see that the SQL generated is correct and Rails successfully created a table named as user reviews.

Here is the relevant pull request.


Rails 6 adds if_not_exists option to create_table

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails 6 added if_not_exists to create_table option to create a table if it doesn’t exist.

Before Rails 6, we could use ActiveRecord::Base.connection.table_exists?.

Default value of if_not_exists option is false.

Rails 5.2

Let’s create users table in Rails 5.2.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users)
CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

=> #<PG::Result:0x00007fd73e711cf0 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

Now let’s try creating users table again with if_not_exists option.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users, if_not_exists: true do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>true})
CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

=> Traceback (most recent call last):
        2: from (irb):121
        1: from (irb):114:in 'change'
ActiveRecord::StatementInvalid (PG::DuplicateTable: ERROR:  relation "users" already exists)
: CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

We can see that Rails 5.2 ignored if_not_exists option and tried creating the table again.

Now let’s try ActiveRecord::Base.connection.table_exists? with Rails 5.2.

>> class CreateUsers < ActiveRecord::Migration[5.2]
>>   def change
>>     unless ActiveRecord::Base.connection.table_exists?('users')
>>       create_table :users do |t|
>>         t.string :name
>>
>>         t.timestamps
>>       end
>>     end
>>   end
>> end

>> CreateUsers.new.change

=> nil

We can see that create_table :users never executed because ActiveRecord::Base.connection.table_exists?('users') returned true.

Rails 6.0.0.beta2

Let’s create users table in Rails 6 with if_not_exists option set as true.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users, if_not_exists: true do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>true})
CREATE TABLE IF NOT EXISTS "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> #<PG::Result:0x00007fc4614fef48 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>true})
CREATE TABLE IF NOT EXISTS "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> #<PG::Result:0x00007fc46513fde0 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

We can see that no exception was raised when we tried creating users table the second time.

Now let’s see what happens if we set if_not_exists to false.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users, if_not_exists: false do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>false})
CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> Traceback (most recent call last):
        2: from (irb):23
        1: from (irb):15:in `change'
ActiveRecord::StatementInvalid (PG::DuplicateTable: ERROR:  relation "users" already exists
)

As we can see, Rails raised an exception here because if_not_exists was set to false.

Here is the relevant pull request.


Rails 6 has added a way to change the database of the app

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails allows us to use different databases using the database.yml config file. It uses sqlite3 as the default database when a new Rails app is created. But it is also possible to use different databases such as MySQL or PostgreSQL. The contents of database.yml change as per the database. Also each database has a different adapter. We need to include the gems pg or mysql2 accordingly.

Before Rails 6, it was not possible to change the contents of database.yml automatically. But now a command has been added to do this automatically.

Let’s say our app has started with sqlite and now we have to switch to MySQL.

$ rails db:system:change --to=mysql
    conflict  config/database.yml
Overwrite /Users/prathamesh/Projects/reproductions/squish_app/config/database.yml? (enter "h" for help) [Ynaqdhm] Y
       force  config/database.yml
        gsub  Gemfile
        gsub  Gemfile

Our database.yml is now changed to contain the configuration for MySQL database and the Gemfile also gets updated automatically with addition of mysql2 gem in place of sqlite3.

This command also takes care of using proper gem versions in the Gemfile when the database backend is changed.


Rails 6 adds parallel testing

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

We frequently think about how good it would be if we could run tests in parallel on local so there would be less wait time for tests to be completed. Wait times increase considerably when the count of tests are on the higher side, which is a common case for a lot of applications.

Though CI tools like CircleCi and Travis CI provide a feature to run tests in parallel, there still wasn’t a straightforward way to parallelize tests on local before Rails 6.

Before Rails 6, if we wanted to parallelize tests, we would use Parallel Tests.

Rails 6 adds the parallelization of tests by default. Rails 6 added parallelize as a class method on ActiveSupport::TestCase which takes a hash as a parameter with the keys workers and with. The worker key is responsible for setting the number of parallel workers. The default value of the worker key is :number_of_processors, which finds the number of processors on the machine and sets it as the number of parallel workers. with takes two values - :processes, which is the default one, and :threads as a value.

Rails 6 also added two hooks - parallelize_setup, which is called before the processes are forked, and parallelize_teardown, which is called after the processes are killed. Rails 6 also handles creation of multiple databases and namespacing of those databases for parallel tests out of the box.

If we want to disable parallel testing, we can set the value of workers as 1 or less.

Rails 6.0.0.beta2

class ActiveSupport::TestCase
  parallelize_setup do |worker|
    # setup databases
  end
   parallelize_teardown do |worker|
    # cleanup database
  end

  # Run tests in parallel with specified workers
  parallelize(workers: :number_of_processors)

  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
  fixtures :all

  # Add more helper methods to be used by all tests here...
end

Rails 6 also provides an environment variable PARALLEL_WORKERS to set the number of parallel workers on runtime.

$ PARALLEL_WORKERS=10 bin/rails test

Here is the relevant pull request for adding parallelize and pull request for setting number of processors as default workers count.


Rails 6 adds CPU time, idle time and allocations to ActiveSupport::Notifications::Event

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Rails provides an easy way to instrument events and ability to subscribe to those events using Active Support Instrumentation API.

Before Rails 6

Before Rails 6, the subscriber of an event can access the event’s start time, end time and the duration along with the other event information.

To demonstrate how to access this information from an event, we will instrument a custom event custom_sleep_event and attach a subscriber to that event.

ActiveSupport::Notifications.subscribe('custom_sleep_event') do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)

  puts "Event: #{event.inspect}"
  puts "Started: #{event.time}"
  puts "Finished: #{event.end}"
  puts "Duration (ms): #{event.duration}"
end

ActiveSupport::Notifications.instrument('custom_sleep_event') do
  sleep 2
end

The event subscriber should print something similar.

Event: #<ActiveSupport::Notifications::Event:0x00007f952fc6a0b8 @name="custom_sleep_event", @payload={}, @time=2019-04-11 16:58:52 +0530, @transaction_id="e82231ab65b7af3c85ec", @end=2019-04-11 16:58:54 +0530, @children=[], @duration=nil>

Started: 2019-04-11 16:58:52 +0530

Finished: 2019-04-11 16:58:54 +0530

Duration (ms): 2001.287

Improvements and additions made to ActiveSupport::Notifications::Event in Rails 6

Rails 6 has improved the way an event’s duration is computed and also added useful information accessible on an event object such as CPU time, idle time and allocations.

Let’s discuss it in more detail.

1. CLOCK_MONOTONIC instead of CLOCK_REALTIME

Before Rails 6, Time.now is used for recording the event’s start time and end time. To avoid issues with the machine changing time, Rails 6 now uses Concurrent.monotonic_time instead of Time.now to record the event’s both start time and end time accurately.

Initially Process.clock_gettime(Process::CLOCK_MONOTONIC) was used which later modified to use Concurrent.monotonic_time. Note that Concurrent.monotonic_time is same but returns more precise time than Process.clock_gettime(Process::CLOCK_MONOTONIC).

Time.now or Process.clock_gettime(Process::CLOCK_REALTIME) can jump forwards and backwards as the system time-of-day clock is changed. Whereas, clock time using CLOCK_MONOTONIC returns the absolute wall-clock time since an unspecified time in the past (for example, system start-up time, or the Epoch). The CLOCK_MONOTONIC does not change with the system time-of-day clock, it just keeps advances forwards at one tick per tick and resets if the system is rebooted. In general, CLOCK_MONOTONIC is recommended to compute the elapsed time between two events. To read more about the differences between CLOCK_REALTIME and CLOCK_MONOTONIC, please check the discussion on this Stackoverflow thread. Another article written by Luca Guidi on the same topic is a recommended read.

2. No need to create hand made event objects on our own

Since it is a common practice to initialize an event using ActiveSupport::Notifications::Event.new(*args) in the event subscriber block, Rails 6 now makes this a bit easy. If the block passed to the subscriber only takes one argument then the Active Support Notification framework now yields an event object to the block.

Therefore, the subscriber definition below

ActiveSupport::Notifications.subscribe('an_event') do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)

  puts "Event #{event.name} received."
end

now can be simplified in Rails 6 as follows.

ActiveSupport::Notifications.subscribe('an_event') do |event|
  puts "Event #{event.name} received."
end
3. CPU time and idle time

Rails 6 now computes elapsed CPU time of an event with the help of Process.clock_gettime(Process::CLOCK_PROCESS_CPUTIME_ID).

System (kernel) keeps track of CPU time per process. The clock time returned using CLOCK_PROCESS_CPUTIME_ID represents the CPU time that has passed since the process started. Since a process may not always get all CPU cycles between start and finish of the process, the process often has to (sleep and) share CPU time among other processes. If the system puts a process to sleep, then the time spend waiting is not counted in the process’ CPU time.

The CPU time of an event can be fetched using the #cpu_time method.

Also, Rails 6 now computes the idle time of an event, too. The idle time of an event represents the difference between the event’s #duration and #cpu_time. Note that the #duration is computed using the difference between the event’s monotonic time at the start (#time) and the monotonic time at the end (#end).

Let’s see how to get these time values.

ActiveSupport::Notifications.subscribe('custom_sleep_event') do |event|
  puts "Event: #{event.inspect}"
  puts "Started: #{event.time}"
  puts "Finished: #{event.end}"
  puts "Duration (ms): #{event.duration}"
  puts "CPU time (ms): #{event.cpu_time}"
  puts "Idle time (ms): #{event.idle_time}"
end

ActiveSupport::Notifications.instrument('custom_sleep_event') do
  sleep 2
end

It prints this.

Event: #<ActiveSupport::Notifications::Event:0x00007fb02ac72400 @name="custom_sleep_event", @payload={}, @time=29514.525707, @transaction_id="43ca8e1c378b6b00d861", @end=29516.528971, @children=[], @duration=nil, @cpu_time_start=2.238801, @cpu_time_finish=2.238874, @allocation_count_start=835821, @allocation_count_finish=835821>

Started: 29514.525707

Finished: 29516.528971

Duration (ms): 2003.2639999990351

CPU time (ms): 0.07299999999998974

Idle time (ms): 2003.190999999035

Notice the @cpu_time_start and @cpu_time_finish counters in the inspected event object representation which are used to calculate the CPU time.

4. Allocations

We will now know how many objects were allocated between the start and end of an event using event’s #allocations method.

ActiveSupport::Notifications.subscribe('custom_sleep_event') do |event|
  puts "Event: #{event.inspect}"
  puts "Started: #{event.time}"
  puts "Finished: #{event.end}"
  puts "Duration (ms): #{event.duration}"
  puts "CPU time (ms): #{event.cpu_time}"
  puts "Idle time (ms): #{event.idle_time}"
  puts "# of objects allocated: #{event.allocations}"
end

ActiveSupport::Notifications.instrument('custom_sleep_event') do
  sleep 2
end

The above example should print something like this.

Event: #<ActiveSupport::Notifications::Event:0x00007fed8c4e33c0 @name="custom_sleep_event", @payload={}, @time=30503.508897, @transaction_id="5330165dae2b49fbe143", @end=30505.513547, @children=[], @duration=nil, @cpu_time_start=2.813231, @cpu_time_finish=2.813404, @allocation_count_start=834227, @allocation_count_finish=834228>

Started: 30503.508897

Finished: 30505.513547

Duration (ms): 2004.6499999989464

CPU time (ms): 0.17299999999975668

Idle time (ms): 2004.4769999989467

# of objects allocated: 1

Notice the @allocation_count_finish and @allocation_count_start counters in the inspected event object representation which are used to calculate the number of objects allocated during an event whose difference is (834228 - 834227 = 1).

In case of JRuby, the allocations would be zero.

5. #start! and #finish!

Two public methods #start! and #finish! have been introduced to ActiveSupport::Notifications::Event which can be used to record more information.

The #start! method resets the @time, @cpu_time_start and @allocation_count_start counters. Similarly, the #finish! method also resets the @end, @cpu_time_finish and @allocation_count_finish counters.


To learn more about this feature, please check rails/rails#33449.


Rails 6 allows configurable attribute name on has_secure_password

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

has_secure_password is used to encrypt and authenticate passwords using BCrypt . It assumes the model has a column named password_digest.

Before Rails 6, has_secure_password did not accept any attribute as a parameter. So, if we needed BCrypt encryption on a different column other than password_digest, we would have to manually encrypt the value before storing it.

Rails 6 makes it easy and allows custom attributes as a parameter to has_secure_password. has_secure_password still defaults to password so it works with previous versions of Rails. has_secure_password still needs the column named column_name_digest defined on the model.

has_secure_password also adds the authenticate_column_name method to authenticate the custom column.

Let’s check out how it works.

Rails 5.2

>> class User < ApplicationRecord
>>   has_secure_password
>> end

=> [ActiveModel::Validations::ConfirmationValidator]

>> user = User.create(email: 'amit.choudhary@bigbinary.com', password: 'amit.choudhary')
BEGIN
User Create (0.8ms)  INSERT INTO "users" ("email", "password_digest", "created_at", "updated_at") VALUES ($1, $2, $3, $4) RETURNING "id"  [["email", "amit.choudhary@bigbinary.com"], ["password_digest", "$2a$10$g6ZJNgakn4I1w/qjAx3vM.I76QSNjFCHtTtT9ovko/9Th50SEmIBO"], ["created_at", "2019-03-17 23:30:13.754379"], ["updated_at", "2019-03-17 23:30:13.754379"]]
COMMIT

=> #<User id: 1, email: "amit.choudhary@bigbinary.com", password_digest: "$2a$10$g6ZJNgakn4I1w/qjAx3vM.I76QSNjFCHtTtT9ovko/9...", created_at: "2019-03-17 23:30:13", updated_at: "2019-03-17 23:30:13">

>> user.authenticate('amit.choudhary')

=> #<User id: 1, email: "amit.choudhary@bigbinary.com", password_digest: "$2a$10$g6ZJNgakn4I1w/qjAx3vM.I76QSNjFCHtTtT9ovko/9...", created_at: "2019-03-17 23:30:13", updated_at: "2019-03-17 23:30:13">

>> class User < ApplicationRecord
>>   has_secure_password :transaction_password
>> end

=> NoMethodError: undefined method 'fetch' for :transaction_password:Symbol
	from (irb):9:in '<class:User>'
	from (irb):8

Rails 6.0.0.beta2

>> class User < ApplicationRecord
>>   has_secure_password
>>   has_secure_password :transaction_password
>> end

=> [ActiveModel::Validations::ConfirmationValidator]

>> user = User.create(email: 'amit.choudhary@bigbinary.com', password: 'amit.choudhary', transaction_password: 'amit.choudhary')
BEGIN
User Create (0.5ms)  INSERT INTO "users" ("email", "password_digest", "transaction_password_digest", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["email", "amit.choudhary@bigbinary.com"], ["password_digest", "$2a$10$nUiO7E2XrIJx/sSdpG0JAOL00uFvPRH7kXHLk5f/6qA1zLPHIrpPy"], ["transaction_password_digest", "$2a$10$l6cTpHwV9xOEn2.OumI29OnualGpvr1CgrNrbuMuHyGTltko8eBG2"], ["created_at", "2019-03-17 23:42:28.723431"], ["updated_at", "2019-03-17 23:42:28.723431"]]
COMMIT

=> #<User id: 5, email: "amit.choudhary@bigbinary.com", password_digest: [FILTERED], transaction_password_digest: [FILTERED], created_at: "2019-03-17 23:42:28", updated_at: "2019-03-17 23:42:28">

>> user.authenticate('amit.choudhary')

=> #<User id: 5, email: "amit.choudhary@bigbinary.com", password_digest: [FILTERED], transaction_password_digest: [FILTERED], created_at: "2019-03-17 23:42:28", updated_at: "2019-03-17 23:42:28">

>> user.authenticate_transaction_password('amit.choudhary')

=> #<User id: 5, email: "amit.choudhary@bigbinary.com", password_digest: [FILTERED], transaction_password_digest: [FILTERED], created_at: "2019-03-17 23:42:28", updated_at: "2019-03-17 23:42:28">

Here is the relevant pull request.


Rails 6 allows to override the ActiveModel::Errors#full_message format at the model level and at the attribute level

This blog is part of our Rails 6 series. Rails 6.0 was recently released.

Before Rails 6

Before Rails 6, the default format %{attribute} %{message} is used to display validation error message for a model’s attribute.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be blank")
=> "Title cannot be blank"

The default format can be overridden globally using a language-specific locale file.

# config/locales/en.yml

en:
  errors:
    format:
      "'%{attribute}' %{message}"

With this change, the full error message is changed for all the attributes of all models.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be blank")
=> "'Title' cannot be blank"

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "'First name' cannot be blank"

This trick works in some cases but it doesn’t work if we have to customize the error messages on the basis of specific models or attributes.

Before Rails 6, there is no easy way to generate error messages like shown below.

The article's title cannot be empty

or

First name of a person cannot be blank

If we change the errors.format to The article's %{attribute} %{message} in config/locales/en.yml then that format will be unexpectedaly used for other models, too.

# config/locales/en.yml

en:
  errors:
    format:
      "The article's %{attribute} %{message}"

This is what will happen if we make such a change.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be empty")
=> "The article's Title cannot be empty"

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "The article's First name cannot be blank"

Notice the error message generated for the :first_name attribute of User model. This does not look the way we want, right?

Let’s see what is changed in Rails 6 to overcome this problem.

Enhancements made to ActiveModel::Errors#full_message in Rails 6

Overriding the format of error message globally using errors.format is still supported in Rails 6.

In addition to that, Rails 6 now also supports overriding the error message’s format at the model level and at the attribute level.

In order to enable this support, we need to explicitly set config.active_model.i18n_customize_full_message to true in the Rails configuration file, preferably in config/application.rb which is implicitly set to false by default.

Overriding model level format

We can customize the full error message format for each model separately.

# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article:
          format: "`%{attribute}`: %{message}"
        user:
          format: "%{attribute} of the user %{message}"

The full error messages will look like this.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be empty")
=> "`Title`: cannot be empty"
>> article.valid?
=> false
>> article.errors.full_messages
=> ["`Title`: can't be blank"]

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "First name of the user cannot be blank"

>> comment = Comment.new
=> #<Comment id: nil, message: nil, author_id: nil, created_at: nil, updated_at: nil>
>> comment.errors.full_message(:message, "is required")
=> "Message is required"

Notice how the default format %{attribute} %{message} is used for generating the full error messages for the Comment model since its format is not being overridden.

Since the other methods such as ActiveModel::Errors#full_messages, ActiveModel::Errors#full_messages_for, ActiveModel::Errors#to_hash etc. use the ActiveModel::Errors#full_message method under the hood, we get the full error messages according to the custom format in the returned values of these methods respectively as expected.

Overriding attribute level format

Similar to customizing format at the model level, we can customize the error format for specific attributes of individual models.

# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article:
          attributes:
            title:
              format: "The article's title %{message}"
        user:
          attributes:
            first_name:
              format: "%{attribute} of a person %{message}"

With such a configuration, we get the customized error message for the title attribute of the Article model.

>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:title, "cannot be empty")
=> "The article's title cannot be empty"
>> article.errors.full_message(:description, "cannot be empty")
=> "Description cannot be empty"

>> user = User.new
=> #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
>> user.errors.full_message(:first_name, "cannot be blank")
=> "First name of a person cannot be blank"
>> user.errors.full_message(:last_name, "cannot be blank")
=> "Last name cannot be blank"

Note that the error messages for the rest of the attributes were generated using the default %{attribute} %{message} format for which we didn’t add custom formats in the config/locales/en.yml manifest.

Overriding model level format of deeply nested models
# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article/comments/attachments:
          format: "%{message}"
>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:'comments/attachments.file_name', "is required")
=> "is required"
>> article.errors.full_message(:'comments/attachments.path', "cannot be blank")
=> "cannot be blank"
>> article.errors.full_message(:'comments.message', "cannot be blank")
=> "Comments message cannot be blank"
Overriding attribute level format of deeply nested models
# config/locales/en.yml

en:
  activerecord:
    errors:
      models:
        article/comments/attachments:
          attributes:
            file_name:
              format: "File name of an attachment %{message}"
>> article = Article.new
=> #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
>> article.errors.full_message(:'comments/attachments.file_name', "is required")
=> "File name of an attachment is required"
>> article.errors.full_message(:'comments/attachments.path', "cannot be blank")
=> "Comments/attachments path cannot be blank"

Precedence

The custom formats specified in the locale file has the following precedence in the high to low order.

  • activerecord.errors.models.article/comments/attachments.attributes.file_name.format
  • activerecord.errors.models.article/comments/attachments.format
  • activerecord.errors.models.article.attributes.title.format
  • activerecord.errors.models.article.format
  • errors.format

To learn more, please checkout rails/rails#32956 and rails/rails#35789.