Introduction to ES6 generators

Generators in JavaScript are the functions that can be paused and resumed.

function* genFunc(){...}

Calling genFunc() doesn’t execute the body of the function.

const genObj = genFunc();

Instead, it returns generator object which can be used to control the execution of the generator.

genObj.next() will start executing the function and it will execute till it reaches yield keyword.

We can think of yield as return for now. It is similar to C# yield.

function* genFunc(){
  console.log('First');
  yield;
  console.log('Second');
}
const genObj = genFunc();
genObj.next();

/*
output

First
*/

Calling genObj.next() again will resume the further execution after yield.

function* genFunc(){
  console.log('First');
  yield;
  console.log('Second');
}
const genObj = genFunc();
genObj.next();
genObj.next();

/*
output

First
Second
*/

Here’s how the execution looks like:

Generator

yield is not allowed inside non-generator functions. That is, yielding in callbacks doesn’t work.

We can also pass the data to generator function via next.

function* genFunc(){
  console.log('First');
  const input = yield;
  console.log(input);
  console.log('Second');
}
const genObj = genFunc();
genObj.next();
genObj.next('Third');

/*
output

First
Third
Second
*/

We can retrieve the yielded values via the generator object genObj:

function* genFunc(){
  console.log('First');
  const input = yield;
  console.log(input);
  console.log('Second');
  yield 'Forth';
}
const genObj = genFunc();
genObj.next();
const result = genObj.next('Third');
console.log(result);

/*
output

First
Third
Second
{
  done: false,
  value: "Forth"
}
*/

We can have multiple yield in the function as shown in the above example.

Once, the execution of the generator function is completed, further calls to genObj.next() will have no effect.

Further reading

We highly recommend Exploring ES6 by Dr. Axel Rauschmayer to go deeper on this topic.

Configuring CircleCI for JRuby

Recently we worked with a client where we had to run a part of their multi-threaded code in JRuby for performance reasons. They have been using CircleCI with MRI for running tests. In this post I will explain how we configured CircleCI to run the same tests using both JRuby and MRI.

CircleCI uses circle.yml file for configuration. Before configuring JRuby, this is how it looked like:

  machine:
    ruby:
      version:
        2.1.5

  dependencies:
    pre:
      - ./bundle_install_circle_ci.sh
    cache_directories:
      - "~/vendor/bundle"

  test:
    override:
      - bundle exec rspec --format progress --format documentation --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/app.xml:
          parallel: true
          pwd: app
          files:
            - spec/**/*_spec.rb


Here are the steps to enable JRuby in CircleCI.

Specify the JDK version

We need to specify a JDK version before using JRuby.

  machine:
    java:
      version:
        openjdk7

Install proper dependencies

We needed to use JRuby 9.0.4.0 but the version of JRuby that came with Ubuntu 12.04 image of CircleCI was different. We added rvm install command as follows to install specific version that we wanted. Also we can configure any script (like bundle install) that needs to run before running tests.

  dependencies:
    pre:

      - rvm install jruby-9.0.4.0
      - ./bundle_install_jruby_circle_ci.sh

    cache_directories:
      - "~/vendor/bundle"

Configure JRuby

We used rvm-exec to set JRuby for running tests for this particular component in the test section. Otherwise by default it picks up MRI.

  test:
    override:

      - rvm-exec jruby-9.0.4.0 bash -c "bundle exec rspec --format progress --format documentation --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/app_jruby.xml":
          parallel: true
          pwd: app
          files:
            - spec/**/*_spec.rb

Improving test runs on JRuby

Once we started running tests with JRuby, we observed it was taking comparatively slower to finish all tests. Most of the time was spent in starting the JVM. We made it faster by setting --dev parameter in JRUBY_OPTS environment variable. This parameter improves JRuby boot time and it shaved more than a minute time for us.

  machine:
    environment:
      JRUBY_OPTS: '--dev'

Here is the final circle.yml file:

# circle.yml

machine:
  ruby:
    version:
      2.1.5

  java:
    version:
      openjdk7

  environment:
    JRUBY_OPTS: '--dev'

dependencies:
  pre:
    - rvm install jruby-9.0.4.0
    - ./bundle_install_jruby_circle_ci.sh

  cache_directories:
    - "~/vendor/bundle"

test:
  override:
    - rvm-exec jruby-9.0.4.0 bash -c "bundle exec rspec --format progress --format documentation --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/app_jruby.xml":
        parallel: true
        pwd: app
        files:
          - spec/**/*_spec.rb

    - bundle exec rspec --format progress --format documentation --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/app_mri.xml:
        parallel: true
        pwd: app
        files:
          - spec/**/*_spec.rb

Opening non HTTPS sites in WebView in React Native

Using WebView in a React Native application allows us to reuse already built web pages.

With iOS 9 or higher if our application attempts to connect to any HTTP server that doesn’t support the latest SSL technology (TLSv1.2), WebView will fail and will not be able load the web pages.

Let’s see this in action. Here we are trying to access a HTTP site via WebView in React Native.

<WebView
  style={styles.container}
  source={ {uri: "http://del.icio.us"} } />

Running that on an iPhone simulator with iOS version 9.0 or greater would show following error.

Error Loading Page
Domain: NSURLErrorDomain
Error Code: -1022
Description: The resource could not be loaded because
the App Transport Security policy requires the use of a
secure connection

Ideally, the site we are trying to connect to should have HTTPS enabled. However there might be cases where we need to connect to sites where HTTPS is not enabled.

For example, while developing the app, we might want to connect to local server which is running just HTTP.

Fix - Using Xcode

To access HTTP sites inside our WebView, we need to open the project in Xcode and open the Info.plist file.

In the list of keys, we will find App Transport Security Settings.

When we expand the section, we would find localhost inside the Exception Domains and the key NSTemporaryExceptionAllowsInsecureHTTPLoads with value true.

Because of this setting when we are connecting to localhost then app runs even if server is running on HTTP and not on HTTPS.

info plist before

So in order to make our non HTTPS site run, we need to add our website url to this whitelisting.

When we hover over the Exception Domains key, we would see a + sign at the right hand side.

Click on it and add our domain here. Set the type to dictionary.

Now click on the domain we just entered and add NSTemporaryExceptionAllowsInsecureHTTPLoads with type Boolean and value YES similar to the one present for localhost

info plist after

Re-run the app using react-native run-ios from the terminal and now the site would be loaded.

If it doesn’t work, then prior to running the app, do a clean build from xcode.

Fix using any IDE

By making the changes from XCode, if we look at the changes in info.plist file, we would find a few lines of code added.

So if we don’t want to open Xcode for the fix, we can add the following lines directly in our info.plist.

Edit the node for the key NSAppTransportSecurity so that the whole node now looks like this:

<key>NSAppTransportSecurity</key>
<dict>
    <key>NSExceptionDomains</key>
    <dict>
        <key>del.icio.us</key>
        <dict>
            <key>NSTemporaryExceptionAllowsInsecureHTTPLoads</key>
            <true/>
        </dict>
        <key>localhost</key>
        <dict>
            <key>NSTemporaryExceptionAllowsInsecureHTTPLoads</key>
            <true/>
        </dict>
    </dict>
</dict>

Be sure to re-run the app using react-native run-ios. Now let’s see how to allow all the HTTP sites instead of whitelisting each and everyone

Using Xcode

Using Xcode : To allow all the non HTTPS sites, just delete the Exception Domains from Xcode inside info.plist and add a new key Allow Arbitrary Loads with the value true.

Using any IDE

Our NSAppTransportSecurity should just contain the following.

<key>NSAppTransportSecurity</key>
<dict>
    <key>NSAllowsArbitraryLoads</key>
    <true/>
</dict>

Send HTTP request headers on each WebView request in React Native

Using WebView in a React Native application allows us to reuse already built web pages.

HTTP Headers are name/value pairs that appear in both request and response messages. The purpose of headers is to supply the web server with additional information and control how content is returned.

In React Native, while opening web pages via WebView Component, we can pass headers to the HTTP request. Refer to our previous blog for more on this.

<WebView
  source={
    {
      uri: "http://localhost:3000",
      headers: {"custom-app-header": "react-native-ios-app"}
    }
  }
/>

But there is a bug. On subsequent requests from inside the WebView, the headers are not passed in the request.

First, let’s try to understand by recreating the bug. We have created a simple node server which will act as the backend for the application and log the request along with the custom header.

Here is our server.js

var http = require('http');
var port = 9000;

function logRequest(request) {
  console.log("Processing request for: ", request.url)
  console.log("Custom Header: ", request.headers["custom-app-header"])
  console.log("Request Processed\n")
}

http.createServer(function(request, response) {
  response.writeHead(200, {"Content-Type": "text/html"});
  switch(request.url) {
    case "/":
      response.write("<html><body>Welcome<a href='/bye'>Bye</a></body></html>");
      logRequest(request)
      break;
    case "/bye":
      response.write("<html><body>Bye<a href='/'>Welcome</a></body></html>");
      logRequest(request)
      break;
    default:
      break;
  }
  response.end();
}).listen(port);

As we can see, the welcome page has a link to bye and vice versa. Let’s start the node server by running node server.js.

When we run the app on the simulator, the welcome page opens up, and in the server log, we can verify that the request header is being passed.

Processing request for:  /
Custom Header:  react-native-ios-app
Request Processed

But when we click on the Bye link from the Welcome page, the server doesn’t receive the request header, which can be verified from the log.

Processing request for:  /bye
Custom Header:  undefined
Request Processed

And it can be verified again that for any subsequent clicks the request header does not get passed. We can click on Welcome and check the log again.

Processing request for:  /
Custom Header:  undefined
Request Processed

We recently encountered this bug and created an issue here. Untill the issue is fixed, we have found a workaround.

Workaround

WebView provides a prop onLoadStart which accepts a function that is invoked when the WebView starts loading.

We can use this prop to know when a link is clicked and then re-render the WebView component with the new url. Re-rendering the WebView component will load the page as if it’s the first page and then the request headers would be passed.

We know that in React, a component re-renders itself when any of its state changes. The only thing which changes here is the url, so let’s move the url to a state and initialize it to the Welcome page which is the root of the application. And then use the onLoadStart prop to change the url state to the clicked url.

Here’s the new code.

class testApp extends Component {
  state = {
    url: "http://localhost:3000",
  };

  render() {
    return (
      <WebView
        onLoadStart={(navState) => this.setState({url: navState.nativeEvent.url})}
        source={
          {
            uri: this.state.url,
            headers: {"custom-app-header": "react-native-ios-app"}
          }
        }
      />
    );
  }
}

Now when we run the app, we can verify in the backend that the request headers are being sent even when we click on Bye link.

Processing request for:  /bye
Custom Header:  undefined
Request Processed

Processing request for:  /bye
Custom Header:  react-native-ios-app
Request Processed

One thing to note here is that, when we click on the Bye link, the request is not intercepted from reaching the server. We are just resending the request by means of a component re-render with the new url.

Hence in the log, we see two requests. First request took place when user clicked on the link, and the second request occurred when the component got re-rendered with the required request headers.

This workaround might help us to pass the request headers which we intend to send to the backend server until the issue gets fixed.

ActionController::Parameters no longer inherits from HashWithIndifferentAccess in Rails 5

This blog is part of our Rails 5 series.

We are all guilty of treating ActionController::Parameters as a plain hash at some point or the other. But with Rails 5, ActionController::Parameters will no longer inherit from HashWithIndifferentAccess.

Inheriting from HashWithIndifferentAccess allowed programmers to call enumerable methods over ActionController::Parameters, which caused ActionController::Parameters to lose its @permitted state there by rendering Strong Parameters as a barebone Hash. This change would discourage such operations.

However since this change would have meant a major impact on all of the upgrading applications as they would have crashed with a NoMethodErrorfor all of those undesired methods. Hence this feature would go through a deprecation cycle, showing deprecation warnings for all of those HashWithIndifferentAccess method usages.

class Parameters

...

def method_missing(method_sym, *args, &block)
  if @parameters.respond_to?(method_sym)
    message = <<-DEPRECATE.squish
      Method #{method_sym} is deprecated and will be removed in Rails 5.1,
      as `ActionController::Parameters` no longer inherits from
      hash. Using this deprecated behavior exposes potential security
      problems. If you continue to use this method you may be creating
      a security vulnerability in your app that can be exploited. Instead,
      consider using one of these documented methods which are not
      deprecated: http://api.rubyonrails.org/v#{ActionPack.version}/classes/ActionController/Parameters.html
    DEPRECATE
    ActiveSupport::Deprecation.warn(message)
    @parameters.public_send(method_sym, *args, &block)
  else
    super
  end
end

...

end

If you need to convert ActionController::Parameters in a true hash then it supports to_h method. Also ActionController::Parameters will continue to have methods like fetch, slice, slice!, except, except!, extract!, delete etc. You can take a detailed look at them here.

Rails 5 fixes ambiguous column issue for projected fields in group by query

This blog is part of our Rails 5 series.

users(:id, :name)
posts(:id, :title, :user_id)
comments(:id, :description, :user_id, :post_id)

>> Post.joins(:comments).group(:user_id).count
Mysql2::Error: Column 'user_id' in field list is ambiguous: SELECT COUNT(*) AS count_all, user_id AS user_id FROM `posts` INNER JOIN `comments` ON `comments`.`post_id` = `posts`.`id` GROUP BY user_id

As we can see user_id has conflict in both projection and GROUP BY as they are not prepended with the table name posts in the generated SQL and thus, raising SQL error Column 'user_id' in field list is ambiguous.

Fix in Rails 5

This issue has been addressed in Rails 5 with this pull request.

With this fix, we can now group by columns having same name in both the tables.

users(:id, :name)
posts(:id, :title, :user_id)
comments(:id, :description, :user_id, :post_id)

>> Post.joins(:comments).group(:user_id).count
SELECT COUNT(*) AS count_all, "posts"."user_id" AS posts_user_id FROM "posts" INNER JOIN "comments" ON "comments"."post_id" = "posts"."id" GROUP BY "posts"."user_id"

=> { 1 => 1 }

This shows that now both projection and Group By are prepended with the posts table name and hence fixing the conflict.

Rails 5 adds Expression Indexes and Operator Classes support for PostgreSQL

This blog is part of our Rails 5 series.

Let’s assume that in our health care application we have a page which shows all Patients. This page also has a filter and it allows us to filter patients by their name.

We could implement the filter as shown here.

Patient.where("lower(first_name) = ?", first_name.downcase)

There might be many users with the same name. In such cases, to speed up the search process, we can add an index. But, adding a regular index will not trigger an index scan since we are using an expression in the where clause i.e lower(name). In such cases, we can leverage expression indexes given by PostgreSQL.

Before Rails 5 adding an expression index is not straightforward since the migrate api does not support it. In order to add one we would need to ditch schema.rb and start using structure.sql. We would also need to add following migration.

def up
  execute <<-SQL
    CREATE INDEX patient_lower_name_idx ON patients (lower(name));
  SQL
end

def down
  execute <<-SQL
    DROP INDEX patient_lower_name_idx;
  SQL
end

Rails 5 adds support for expression indexes

Rails 5 provides ability to add an expression index using add_index method as follows:

def change
  add_index :patients,
            'lower(last_name)',
            name: "index_patients_on_name_unique",
            unique: true
end

And we also get to keep schema.rb.

Time goes on. everyone is happy with the search functionality until one day a new requirement comes along which is, in short, to have partial matches on patient names.

We modify our search as follows:

User.where("lower(name) like ?", "%#{name.downcase}%")

Since the query is different from before, PostgreSQL query planner will not take the already existing btree index into account and will revert to a sequential scan.

Quoting directly from Postgresql documents,

'The operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops support B-tree indexes on the types text, varchar, and char respectively. The difference from the default operator classes is that the values are compared strictly character by character rather than according to the locale-specific collation rules. This makes these operator classes suitable for use by queries involving pattern matching expressions (LIKE or POSIX regular expressions) when the database does not use the standard "C" locale.'

We need to add an operator class to the previous index for the query planner to utilize the index that we created earlier.

Rails 5 adds support for specifying operator classes on expression indexes

In order to add an index with an operator class we could write our migration as shown below.

def change
  remove_index :patients, name: :index_patients_on_name_unique
  add_index :patients,  'lower(last_name) varchar_pattern_ops',
                        name: "index_patients_on_name_unique",
                        unique: true
end

Attach arbitrary metadata to an Active Job in Rails 5

This blog is part of our Rails 5 series.

Rails 4.2 came with built-in support for executing jobs in the background using Active Job. Along with many enhancements to Active Job, Rails 5 provides the ability to attach arbitrary metadata to any job.

Consider the scenario where we would like to get notified when a job has failed for more than three times. With the new enhancement, we can make it work by overriding serialize and deserialize methods of the job class.

class DeliverWebhookJob < ActiveJob::Base
  def serialize
    super.merge('attempt_number' => (@attempt_number || 0) + 1)
  end

  def deserialize(job_data)
    super(job_data)
    @attempt_number = job_data['attempt_number']
  end

  rescue_from(TimeoutError) do |ex|
    notify_job_tried_x_times(@attempt_number) if @attempt_number == 3
    retry_job(wait: 10)
  end
end

Earlier, deserialization was performed by #deserialize class method and therefore was inaccessible from the job instance. With the new changes, deserialization is delegated to the deserialize method on job instance thereby allowing it to attach arbitrary metadata when it gets serialized and read it back when it gets performed.

Partial template name need not be a valid Ruby identifier in Rails 5

This blog is part of our Rails 5 series.

Before Rails 5, partials name should start with underscore and should be followed by any combination of letters, numbers and underscores.

This rule was required because before commit, rendering a partial without giving :object or :collection used to generate a local variable with the partial name by default and a variable name in ruby can’t have dash and other things like that.

In the following case we have a file named _order-details.html.erb. Now let’s try to use this partial.

<!DOCTYPE html>
<html>
<body>
  <%= render :partial => 'order-details' %>
</body>
</html>

We will get following error, if we try to render above view in Rails 4.x.

ActionView::Template::Error (The partial name (order-details) is not a valid Ruby identifier;
make sure your partial name starts with underscore,
and is followed by any combination of letters, numbers and underscores.):
    2: <html>
    3: <body>
    4: Following code is rendered through partial named _order-details.erb
    5: <%= render :partial => 'order-details' %>
    6: </body>
    7: </html>

In the above the code failed because the partial name has a dash which is not a valid ruby variable name.

In Rails 5, we can give our partials any name which starts with underscore.

Rails 5 allows passing record being validated to error message generator

This blog is part of our Rails 5 series.

Active Record validations by default provides an error messages, based on applied attributes. But some time we need to display a custom error message while validating a record.

We can give custom error message by passing String or Proc to :message.

class Book < ActiveRecord::Base
  # error message with a string
  validates_presence_of :title, message: 'You must provide the title of book.'

  # error message with a proc
  validates_presence_of :price,
      :message => Proc.new { |error, attributes|
      "#{attributes[:key]} cannot be blank."
      }
end

What’s new in Rails 5 ?

Rails 5 allows passing record to error message generator. Now we can pass current record object in a proc as an argument, so that we can write custom error message based on current object.

Revised example with current record object.

class Book < ActiveRecord::Base
  # error message with simple string
  validates_presence_of :title, message: 'You must provide the title of book.'

  # error message with proc using current record object
  validates_presence_of :price,
      :message => Proc.new { |book, data|
      "You must provide #{data[:attribute]} for #{book.title}"
      }
end

Apply platform specific styles in stylesheet in React Native

While writing cross platform applications we need to add platform specific styles. There are many ways of accomplishing this.

React Native introduced Platform.select helper in version 0.28 which allows us to write platform specific styles in a concise way. In this blog we will see how to use this newly introduced feature.

StyleSheet

A basic stylesheet file might look something like this.

import {StyleSheet} from 'react-native'

export default StyleSheet.create({
  container: {
    flex: 1,
  },

  containerIOS: {
    padding: 4,
    margin: 2,
  },

  containerAndroid: {
    padding: 6,
  },
});

Now let’s see how we can re-write this StyleSheet using Platform.select.

import {StyleSheet, Platform} from 'react-native';

export default StyleSheet.create({
  container: {
    flex: 1,
    ...Platform.select({
      ios: {
        padding: 4,
        margin: 2
      },
      android: {
        padding: 6
      }
    }),
  },
});

Passing user agent or custom header in React Native WebView

Using WebView in a React Native application allows us to reuse already built web pages.

We have seen that in most of the “web view” based applications the links in header are mostly turned in native navigational components. It means server should not be sending header if React Native component is asking for web pages. However those headers should be present if the request is not coming from the React Native app.

Passing custom request headers

We can configure React Native app to pass custom request headers when request is made to the server to fetch pages.

let customHeaders = {
  "X-DemoApp-Version": "1.1",
  "X-DemoApp-Type": "demo-app-react-native"
}

While invoking WebView component we can pass customHeaders as shown below.

renderWebView() {
  return (
    <WebView
      source={ {uri: this.props.url, headers: customHeaders} }
    />
  )
}

Passing user agent

React Native also allows us to pass “userAgent” as a prop. However it is only supported by android version of React Native.

renderWebView() {
  return (
    <WebView
      source={ {uri: this.props.url} }
      userAgent="demo-react-native-app"
    />
  )
}

For iOS, we would need to add the following lines to our AppDelegate.m to set the userAgent.

NSString *newAgent = @"demo-react-native-app";
NSDictionary *dictionary = [[NSDictionary alloc] initWithObjectsAndKeys:newAgent, @"UserAgent", nil];
[[NSUserDefaults standardUserDefaults] registerDefaults:dictionary];

Skip mailers while generating Rails 5 app

This blog is part of our Rails 5 series.

We can now skip requiring Action Mailer while generating Rails 5 app.

$ rails new my_app --skip-action-mailer

# OR

$ rails new my_app -M

This comments out requiring action_mailer/railtie in application.rb.

It also omits mailer specific configurations such as config.action_mailer.raise_delivery_errors and config.action_mailer.perform_caching in production/development and config.action_mailer.delivery_method by default in test environment.

# application.rb

require "rails"
require "active_model/railtie"
require "active_job/railtie"
require "active_record/railtie"
require "action_controller/railtie"
# require "action_mailer/railtie"
require "action_view/railtie"
require "action_cable/engine"
require "sprockets/railtie"
require "rails/test_unit/railtie"

As, we can see action_mailer/railtie is commented out.

Errors can be indexed with nested attributes in Rails 5

This blog is part of our Rails 5 series.

We use accepts_nested_attributes_for when we want a single form to cater to multiple models. By using this we can easily provide attributes for associated models.

In Rails 4.x, if a validation fails for one or more of the associated models, then it is not possible to figure out from error message, which of the associated model object is the error related to.

class Product < ApplicationRecord
  has_many :variants
  accepts_nested_attributes_for :variants
end

class Variant < ApplicationRecord
  validates :display_name, :price, presence: true
end

>> product = Product.new(name: 'Table')
>> variant1 = Variant.new(price: 10)
>> variant2 = Variant.new(display_name: 'Brown')
>> product.variants = [variant1, variant2]
>> product.save
=> false

>> product.error.messages
=> {:"variants.display_name"=>["can't be blank"], :"variants.price"=>["can't be blank"]}

In the example above we can see that if this error message is sent as JSON API, we cannot find out which variant save failed because of which attribute.

This works well when we render forms using Active Record models, as errors are available on individual instances. But, the issue arises with an API call, where we don’t have access to these instances.

Rails 5 allows indexing of errors on nested attributes

In Rails 5, we can add an index to errors on nested models.

We can add the option index_errors: true to has_many association to enable this behavior on individual association.

class Product < ApplicationRecord
  has_many :variants, index_errors: true
  accepts_nested_attributes_for :variants
end

class Variant < ApplicationRecord
  validates :display_name, :price, presence: true
end

>> product = Product.new(name: 'Table')
>> variant1 = Variant.new(price: 10)
>> variant2 = Variant.new(display_name: 'Brown')
>> product.variants = [variant1, variant2]
>> product.save
=> false

>> product.error.messages
=> {:"variants[0].display_name"=>["can't be blank"], :"variants[1].price"=>["can't be blank"]}

Using global configuration

In order to make this change global, we can set configuration config.active_record.index_nested_attribute_errors = true which is false by default.

config.active_record.index_nested_attribute_errors = true

class Product < ApplicationRecord
  has_many :variants
  accepts_nested_attributes_for :variants
end

class Variant < ApplicationRecord
  validates :display_name, :price, presence: true
end

This will work exactly same as an example with has_many :variants, index_errors: true in Product.

Rails 5 doesn't convert empty array to nil when deep munging params

This blog is part of our Rails 5 series.

In older Rails version (< 3.2), when an empty array was passed to a where clause or to a find_by query, it generated SQL with an IS NULL clause.

User.find_by_email([]).to_sql
#=> "SELECT * FROM users WHERE email IS NULL"

User.find_by_email([nil]).to_sql
#=> "SELECT * FROM users WHERE email IS NULL"

Also, when JSON data of the request was parsed and params got generated the deep munging converted empty arrays to nil.

For example, When the following JSON data is posted to a Rails controller

{"property_grouping":{"project_id":289,"name":"test group2","property_ids":[]}}

It gets converted into the following params in the controller.

{"property_grouping"=>{"project_id"=>289, "name"=>"test group2", "property_ids"=>nil, },
"action"=>"...", "controller"=>"...", "format"=>"json"}

This in combination with the fact that Active Record constructs IS NULL query when blank array is passed became one of the security threats and one of the most complained issues in Rails.

The security threat we had was that it was possible for an attacker to issue unexpected database queries with “IS NULL” where clauses. Though there was no threat of an insert being carried out, there could be scope for firing queries that would check for NULL even if it wasn’t intended.

In later version of Rails(> 3.2), we had a different way of handling blank arrays in Active Record find_by and where clauses.

User.find_by_email([]).to_sql
#=> "SELECT "users".* FROM "users" WHERE 1=0 LIMIT 1"

User.find_by_email([nil]).to_sql
#=> "SELECT * FROM users WHERE email IS NULL"

As you can see a conditional for empty array doesn’t trigger IS NULL query, which solved part of the problem.

We still had conversion of empty array to nil in the deep munging in place and hence there was still a threat of undesired behavior when request contained empty array.

One way to handle it was to add before_action hooks to the action that could modify the value to empty array if it were nil.

In Rails 5, empty array does not get converted to nil in deep munging. With this change, the empty array will persist as is from request to the params in the controller.

Use as option to encode a request for a specific mime type in controller tests in Rails 5

This blog is part of our Rails 5 series.

Before Rails 5, while sending requests with integration test setup, we needed to add format option to send request with different Mime type.

class ProductsControllerTest < ActionController::TestCase
  def test_create

    post :create, { product: { name: 'Rails 5 book' } }.to_json,
        format: :json,
        headers: { 'Content-Type' => 'application/json' }

    assert_equal 'application/json', request.content_type
    assert_equal({ id: 1, name: 'Rails 5 book' }, JSON.parse(response.body))
  end
end

This format for writing tests with JSON type is lengthy and needs too much information to be passed to request as well.

Improvement with Rails 5

In Rails 5, we can provide Mime type while sending request by passing it with as option and all the other information like headers and format will be passed automatically.

class ProductsControllerTest < ActionDispatch::IntegrationTest
  def test_create
    post products_url, params: { product: { name: 'Rails 5 book' } }, as: :json

    assert_equal 'application/json', request.content_type
    assert_equal({ 'id' => 1, 'name' => 'Rails 5 book' }, response.parsed_body)
  end
end

As we can notice, we don’t need to parse JSON anymore.

With changes in this PR, we can fetch parsed response without needing to call JSON.parse at all.

Custom Mime Type

We can also register our own encoders for any registered Mime Type.

class ProductsControllerTest < ActionDispatch::IntegrationTest

  def setup
    Mime::Type.register 'text/custom', :custom

    ActionDispatch::IntegrationTest.register_encoder :custom,
      param_encoder: -> params { params.to_custom },
      response_parser: -> body { body }

  end

  def test_index
    get products_url, params: { name: 'Rails 5 book' }, as: :custom

    assert_response :success
    assert_equal 'text/custom', request.content_type
  end
end

Controller actions respond with 204 No Content by default in Rails 5 if no response is specified

This blog is part of our Rails 5 series.

Before Rails 5, when we forget to add template for an action, we get ActionView::MissingTemplate exception.

class UsersController < ApplicationController
  def index
    @users = User.all
  end
end

Started GET "/users" for ::1 at 2016-06-10 17:10:40 +0530
Processing by UsersController#index as HTML
Completed 500 Internal Server Error in 5ms

ActionView::MissingTemplate (Missing template users/index, application/index with {:locale=>[:en], :formats=>[:html], :variants=>[], :handlers=>[:erb, :builder, :raw, :ruby]}...

Similarly, if we don’t specify response for a POST request, we will also get ActionView::MissingTemplate exception.

class UsersController < ApplicationController
  def create
    @user = User.new(user_params)
    @user.save
  end
end
Started POST "/users"
Processing by UsersController#create as HTML
  Parameters: {"utf8"=>"✓", "user"=>{"name"=>"Max"}, "commit"=>"Create User"}
   (0.1ms)  begin transaction
  SQL (2.7ms)  INSERT INTO "users" ("name", "created_at", "updated_at") VALUES (?, ?, ?)  [["name", "Max"], ["created_at", 2016-06-10 12:29:09 UTC], ["updated_at", 2016-06-10 12:29:09 UTC]]
   (0.5ms)  commit transaction
Completed 500 Internal Server Error in 5ms

ActionView::MissingTemplate (Missing template users/create, application/create with {:locale=>[:en], :formats=>[:html], :variants=>[], :handlers=>[:erb, :builder, :raw, :ruby]}...

In Rails 5, if we don’t specify response for an action then Rails returns 204: No content response by default. This change can cause some serious implications during the development phase of the app.

Let’s see what happens with the POST request without specifying the response.

class UsersController < ApplicationController
  def create
    @user = User.new(user_params)
    @user.save
  end
end
Started POST "/users"
Processing by UsersController#create as HTML
  Parameters: {"utf8"=>"✓", "user"=>{"name"=>"Max"}, "commit"=>"Create User"}
   (0.1ms)  begin transaction
  SQL (2.7ms)  INSERT INTO "users" ("name", "created_at", "updated_at") VALUES (?, ?, ?)  [["name", "Max"], ["created_at", 2016-06-10 12:29:09 UTC], ["updated_at", 2016-06-10 12:29:09 UTC]]
   (0.5ms)  commit transaction
No template found for UsersController#create, rendering head :no_content
Completed 204 No Content in 41ms (ActiveRecord: 3.3ms)

Rails happily returns with 204: No content response in this case.

This means users get the feel that nothing happened in the browser. Because Rails returned with no content and browser happily accepted it. But in reality, the user record was created in the database.

Let’s see what happens with the GET request in Rails 5.

class UsersController < ApplicationController
  def index
    @users = User.all
  end
end
ActionController::UnknownFormat (UsersController#index is missing a template for this request format and variant.

request.formats: ["text/html"]
request.variant: []

NOTE! For XHR/Ajax or API requests, this action would normally respond with 204 No Content: an empty white screen. Since you're loading it in a web browser, we assume that you expected to actually render a template, not… nothing, so we're showing an error to be extra-clear. If you expect 204 No Content, carry on. That's what you'll get from an XHR or API request. Give it a shot.):

Instead of 204: No Content, we get ActionController::UnknownFormat exception. Rails is being extra smart here and hinting that we are probably missing corresponding template for this controller action. It is smart enough to show us this message as we requested this page via browser via a GET request. But if the same request is made via Ajax or through an API call or a POST request, Rails will return 204: No Content response as seen before.

In general, this change can trip us in the development phase, as we are used to incremental steps like adding a route, then the controller action and then the template or response. Getting 204 response can give a feel of nothing happening where things have actually happened in the background. So don’t forget to respond properly from your controller actions.

Rails 5 ensures compatibility between ActionDispatch::Request::Session and Rack::Session

This blog is part of our Rails 5 series.

Before Rails 5, there were errors in running integration tests when a Rack framework like Sinatra, Grape etc. were mounted within Rails with a motive of using its session.

Problems were reported at many places including github gists and stackoverflow regarding an error which was of the following form.

NoMethodError (undefined method `each' for #<ActionDispatch::Request::Session:0x7fb8dbe7f838 not yet loaded>):
  rack (1.5.2) lib/rack/session/abstract/id.rb:158:in `stringify_keys'
  rack (1.5.2) lib/rack/session/abstract/id.rb:95:in `update'
  rack (1.5.2) lib/rack/session/abstract/id.rb:258:in `prepare_session'
  rack (1.5.2) lib/rack/session/abstract/id.rb:224:in `context'
  rack (1.5.2) lib/rack/session/abstract/id.rb:220:in `call'

As we can see, the error occurs due to absence of method each on an ActionDispatch::Request::Session object.

In Rails 5, each method was introduced to ActionDispatch::Request::Session class making it compatible with Rack frameworks mounted in Rails and hence avoiding the above mentioned errors in integration testing.

Rails 5 supports logging errors with tagged logging

This blog is part of our Rails 5 series.

We use tagged logging to better extract information from logs generated by Rails applications.

Consider a Rails 4.x application where the request id is used as a log tag by adding following in config/environments/production.rb.

config.log_tags = [:uuid]

The log generated for that application would look like:

[df88dbaa-50fd-4178-85d7-d66279ea33b6] Started GET "/posts" for ::1 at 2016-06-03 17:19:32 +0530
[df88dbaa-50fd-4178-85d7-d66279ea33b6] Processing by PostsController#index as HTML
[df88dbaa-50fd-4178-85d7-d66279ea33b6]   Post Load (0.2ms)  SELECT "posts".* FROM "posts"
[df88dbaa-50fd-4178-85d7-d66279ea33b6]   Rendered posts/index.html.erb within layouts/application (4.8ms)
[df88dbaa-50fd-4178-85d7-d66279ea33b6] Completed 500 Internal Server Error in 10ms (ActiveRecord: 0.2ms)
[df88dbaa-50fd-4178-85d7-d66279ea33b6]
ActionView::Template::Error (divided by 0):
    29: <br>
    30:
    31: <%= link_to 'New Post', new_post_path %>
    32: <%= 1/0 %>
  app/views/posts/index.html.erb:32:in `/'
  app/views/posts/index.html.erb:32:in `_app_views_posts_index_html_erb___110320845380431566_70214104632140'

As we can see the request id tag is not prepended to the lines containing error details. If we search the log file by the request id then the error details would not be shown.

In Rails 5 errors in logs show log tags as well to overcome the problem we saw above.

Please note that the log tag name for request id has changed in Rails 5. The setting would thus look like as shown below.

config.log_tags = [:request_id]

This is how the same log will look like in a Rails 5 application.

[7efb4d18-8e55-4d51-b31e-119f49f5a410] Started GET "/" for ::1 at 2016-06-03 17:24:59 +0530
[7efb4d18-8e55-4d51-b31e-119f49f5a410] Processing by PostsController#index as HTML
[7efb4d18-8e55-4d51-b31e-119f49f5a410]   Rendering posts/index.html.erb within layouts/application
[7efb4d18-8e55-4d51-b31e-119f49f5a410]   Post Load (0.9ms)  SELECT "posts".* FROM "posts"
[7efb4d18-8e55-4d51-b31e-119f49f5a410]   Rendered posts/index.html.erb within layouts/application (13.2ms)
[7efb4d18-8e55-4d51-b31e-119f49f5a410] Completed 500 Internal Server Error in 30ms (ActiveRecord: 0.9ms)
[7efb4d18-8e55-4d51-b31e-119f49f5a410]
[7efb4d18-8e55-4d51-b31e-119f49f5a410] ActionView::Template::Error (divided by 0):
[7efb4d18-8e55-4d51-b31e-119f49f5a410]     29: <br>
[7efb4d18-8e55-4d51-b31e-119f49f5a410]     30:
[7efb4d18-8e55-4d51-b31e-119f49f5a410]     31: <%= link_to 'New Post', new_post_path %>
[7efb4d18-8e55-4d51-b31e-119f49f5a410]     32: <%= 1/0 %>
[7efb4d18-8e55-4d51-b31e-119f49f5a410]
[7efb4d18-8e55-4d51-b31e-119f49f5a410] app/views/posts/index.html.erb:32:in `/'
[7efb4d18-8e55-4d51-b31e-119f49f5a410] app/views/posts/index.html.erb:32:in `_app_views_posts_index_html_erb___1136362343261984150_70232665530320'

Rails 5 makes sql statements even more colorful

This blog is part of our Rails 5 series.

In Rails 5, SQL statements have much more granular level of coloration .

INSERT statement

Font color for INSERT command is green

insert statement in green color

UPDATE and SELECT statement

Font color for UPDATE command is yellow and for SELECT it is blue.

update statement in yellow color

DELETE statement

Font color for DELETE command is red.

delete statement in red color

As you might have noticed from above that font color for transaction is cyan.

Rollback statement

Font color for Rollback transaction is red.

rollback statement in red color

Other statements

For custom SQL statements color is magenta and Model Load/exists color is cyan.

magenta