Ruby 2.5 allows creating structs with keyword arguments

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

In Ruby, structs can be created using positional arguments.

Customer = Struct.new(:name, :email)
Customer.new("John", "john@example.com")

This approach works when the arguments list is short. When arguments list increases then it gets harder to track which position maps to which value.

Here if we pass keyword argument then we won’t get any error. But the values are not what we wanted.

Customer.new(name: "John", email: "john@example.com")
=> #<struct Customer name={:name=>"John", :email=>"john@example.com"}, email=nil>

Ruby 2.5 introduced creating structs using keyword arguments. Relevant pull request is here.

However this introduces a problem. How do we indicate to Struct if we want to pass arguments using position or keywords.

Takashi Kokubun suggested to use keyword_argument as an identifier.

Customer = Struct.new(:name, :email, keyword_argument: true)
Customer.create(name: "John", email: "john@example.com")

Matz suggested to change the name to keyword_init.

So in Ruby 2.5 we can create structs using keywords as long as we are passing keyword_init.

Customer = Struct.new(:name, :email, keyword_init: true)

Customer.new(name: "John", email: "john@example.com")
=> #<struct Customer name="John", email="john@example.com">

Rails 5.2 allows mailers to use custom Active Job class

This blog is part of our Rails 5.2 series.

Rails allows sending emails asynchronously via Active Job.

Notifier.welcome(User.first).deliver_later

It uses ActionMailer::DeliveryJob as the default job class to send emails. This class is defined internally by Rails.

The DeliveryJob defines handle_exception_with_mailer_class method to handle exception and to do some housekeeping work.

def handle_exception_with_mailer_class(exception)
  if klass = mailer_class
    klass.handle_exception exception
  else
    raise exception
  end
end

One might need more control on the job class to retry the job under certain conditions or add more logging around exceptions.

Before Rails 5.2, it was not possible to use a custom job class for this purpose.

Rails 5.2 has added a feature to configure the job class per mailer.

class CustomNotifier < ApplicationMailer
  self.delivery_job = CustomNotifierDeliveryJob
end

By default, Rails will use the internal DeliveryJob class if the delivery_job configuration is not present in the mailer class.

Now, Rails will use CustomNotifierDeliveryJob for sending emails for CustomNotifier mailer.

CustomNotifier.welcome(User.first).deliver_later

As mentioned above CustomNotifierDeliveryJob can be further configured for logging, exception handling and reporting.

By default, deliver_later will pass following arguments to the perform method of the CustomNotifierDeliveryJob.

  • mailer class name
  • mailer method name
  • mail delivery method
  • original arguments with which the mail is to be sent
class CustomNotifierDeliveryJob < ApplicationJob

  rescue_from StandardError, with: :handle_exception_with_mailer_class

  retry_on CustomNotifierException

  discard_on ActiveJob::DeserializationError

  def perform(mailer, mail_method, delivery_method, *args)
    logger.log "Mail delivery started"
    klass = mailer.constantize
    klass.public_send(mail_method, *args).send(delivery_method)
    logger.log "Mail delivery completed"
  end

  def handle_exception_with_mailer_class(exception)
    if klass = mailer_class
      klass.handle_exception exception
    else
      raise exception
    end
  end
end

We can also simply inherit from the ActionMailer::DeliveryJob and override the retry logic.

class CustomNotifierDeliveryJob < ActionMailer::DeliveryJob
  retry_on CustomNotifierException
end

Rails 5.2 supports descending indexes for MySQL

This blog is part of our Rails 5.2 series.

An index is used to speed up the performance of queries on a database.

Rails allows us to create index on a database column by means of a migration. By default, the sort order for the index is ascending.

But consider the case where we are fetching reports from the database. And while querying the database, we always want to get the latest report. In this case, it is efficient to specify the sort order for the index to be descending.

We can specify the sort order by adding an index to the required column by adding a migration .

add_index :reports, [:user_id, :name], order: { user_id: :asc, name: :desc }

PostgreSQL

If our Rails application is using postgres database, after running the above migration we can verify that the sort order was added in schema.rb

create_table "reports", force: :cascade do |t|
  t.string "name"
  t.integer "user_id"
  t.datetime "created_at", null: false
  t.datetime "updated_at", null: false
  t.index ["user_id", "name"], name: "index_reports_on_user_id_and_name", order: { name: :desc }
end

Here, the index for name has sort order in descending. Since the default is ascending, the sort order for user_id is not specified in schema.rb.

MySQL < 8.0.1

For MySQL < 8.0.1, running the above migration, would generate the following schema.rb

create_table "reports", force: :cascade do |t|
  t.string "name"
  t.integer "user_id"
  t.datetime "created_at", null: false
  t.datetime "updated_at", null: false
  t.index ["user_id", "name"], name: "index_reports_on_user_id_and_name"
end

As we can see, although the migration runs successfully, it ignores the sort order and the default ascending order is added.

Rails 5.2 and MySQL > 8.0.1

MySQL 8.0.1 added support for descending indices.

Rails community was quick to integrate it as well. So now in Rails 5.2, we can add descending indexes for MySQL databases.

Running the above migration would lead to the same output in schema.rb file as that of the postgres one.

create_table "reports", force: :cascade do |t|
  t.string "name"
  t.integer "user_id"
  t.datetime "created_at", null: false
  t.datetime "updated_at", null: false
  t.index ["user_id", "name"], name: "index_reports_on_user_id_and_name", order: { name: :desc }
end

Ruby 2.5 adds Hash#transform_keys method

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

Ruby 2.4 added Hash#transform_values method to transform values of the hash.

In Ruby 2.5, a similar method Hash#transform_keys is added for transforming keys of the hash.

>> h = { name: "John", email: "john@example.com" }
=> {:name=>"John", :email=>"john@example.com"}

>> h.transform_keys { |k| k.to_s }
=> {"name"=>"John", "email"=>"john@example.com"}

The bang sibling of this method, Hash#transform_keys! is also added which changes the hash in place.

These two methods are already present in Active Support from Rails and are natively supported in Ruby now.

Rails master is already supporting using the native methods if supported by the Ruby version.

Rails 5.2 allows passing request params to Action Mailer previews

This blog is part of our Rails 5.2 series.

Rails has inbuilt feature to preview the emails using Action Mailer previews.

A preview mailer can be setup as shown here.

class NotificationMailer < ApplicationMailer
  def notify(email: email, body: body)
    user = User.find_by(email: email)
    mail(to: user.email, body: body)
  end
end

class NotificationMailerPreview < ActionMailer::Preview
  def notify
    NotificationMailer.notify(email: User.first.email, body: "Hi there!")
  end
end

This will work as expected. But our email template is displayed differently based on user’s role. To test this, we need to update the notify method and then check the updated preview.

What if we could just pass the email in the preview URL.

http://localhost:3000/rails/mailers/notification/notify?email=superadmin@example.com

In Rails 5.2, we can pass the params directly in the URL and params will be available to the preview mailers.

Our code can be changed as follows to use the params.

class NotificationMailerPreview < ActionMailer::Preview
  def notify
    email =  params[:email] || User.first.email
    NotificationMailer.notify(email: email, body: "Hi there!")
  end
end

This allows us to test our mailers with dynamic input as per requirements.

Ruby 2.5 enumerable predicates accept pattern argument

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

Ruby 2.5.0 was recently released.

Ruby has sequence predicates such as all?, none?, one? and any? which take a block and evaluate that by passing every element of the sequence to it.

if queries.any? { |sql| /LEFT OUTER JOIN/i =~ sql }
  logger.log "Left outer join detected"
end

Ruby 2.5 allows using a shorthand for this by passing a pattern argument. Internally case equality operator(===) is used against every element of the sequence and the pattern argument.

if queries.any?(/LEFT OUTER JOIN/i)
  logger.log "Left outer join detected"
end

# Translates to:

queries.any? { |sql| /LEFT OUTER JOIN/i === sql }

This allows us to write concise and shorthand expressions where block is only used for comparisons. This feature is applicable to all?, none?, one? and any? methods.

Similarities with Enumerable#grep

This feature is based on how Enumerable#grep works. grep returns an array of every element in the sequence for which the case equality operator(===) returns true by applying the pattern. In this case, the all? and friends return true or false.

There is a proposal to add it for select and reject as well.

Rails 5.2 adds bootsnap to the app to speed up boot time

This blog is part of our Rails 5.2 series.

Rails 5.2 beta 1 was recently released.

If we generate a new Rails app using Rails 5.2, we will see bootsnap gem in the Gemfile. bootsnap helps in reducing the boot time of the app by caching expensive computations.

In a new Rails 5.2 app, boot.rb will contain following content:

ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__)

require 'bundler/setup' # Set up gems listed in the Gemfile.
require 'bootsnap/setup' # Speed up boot time by caching expensive operations.

if %w[s server c console].any? { |a| ARGV.include?(a) }
  puts "=> Booting Rails"
end

This sets up bootsnap to start in all environments. We can toggle it per environment as required.

This works out of the box and we don’t have do to anything for the new app.

If we are upgrading an older app which already has bootsnap, then we need to make sure that we are using bootsnap >= 1.1.0 because new Rails apps ship with that version constraint.

If the app doesn’t contain the bootsnap gem already then we will need to add it manually since rails app:update task adds the bootsnap/setup line to boot.rb regardless of its presence in the Gemfile.

Ruby 2.5 requires pp by default

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

Ruby 2.5.0-preview1 was recently released.

Ruby allows pretty printing of objects using pp method.

Before Ruby 2.5, we had to require PP explicitly before using it. Even the official documentation states that “All examples assume you have loaded the PP class with require ‘pp’”.

>> months = %w(January February March)
=> ["January", "February", "March"]

>> pp months
NoMethodError: undefined method `pp' for main:Object
Did you mean?  p
	from (irb):5
	from /Users/prathamesh/.rbenv/versions/2.4.1/bin/irb:11:

>> require 'pp'
=> true

>> pp months
["January",
 "February",
 "March"]
=> ["January", "February", "March"]

In Ruby 2.5, we don’t need to require pp. It gets required by default. We can use it directly.

>> months = %w(January February March)
=> ["January", "February", "March"]

>> pp months
["January",
 "February",
 "March"]
=> ["January", "February", "March"]

This feature was added after Ruby 2.5.0 preview 1 was released, so it’s not present in the preview. It’s present in Ruby trunk.

Array#prepend and Array#append in Ruby 2.5

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

Ruby has Array#unshift to prepend an element to the start of an array and Array#push to append an element to the end of an array.

The names of these methods are not very intuitive. Active Support from Rails already has aliases for the unshift and push methods , namely prepend and append.

In Ruby 2.5, these methods are added in the Ruby language itself.

>> a = ["hello"]
=> ["hello"]
>> a.append "world"
=> ["hello", "world"]
>> a.prepend "Hey"
=> ["Hey", "hello", "world"]
>>

They are implemented as aliases to the original unshift and push methods so there is no change in the behavior.

Ruby 2.5 added yield_self

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

Ruby 2.5 added a new method named yield_self. It yields the receiver to the given block and returns output of the last statement in the block.

irb> "Hello".yield_self { |str| str + " World" }
  => "Hello World"

How is it different from try in Rails ?

Without a method argument try behaves similar to yield_self. It would yield to the given block unless the receiver is nil and returns the output of the last statement in the block.

irb> "Hello".try { |str| str + " World" }
  => "Hello World"

Couple of differences to note are, try is not part of Ruby but Rails. Also try’s main purpose is protection against nil hence it doesn’t execute the block if receiver is nil.

irb> nil.yield_self { |obj| "Hello World" }
  => "Hello World"

irb> nil.try { |obj| "Hello World" }
  => nil

What about tap?

tap also is similar to yield_self. It’s part of Ruby itself. The only difference is the value that is returned. tap returns the receiver itself while yield_self returns the output of the block.

irb> "Hello".yield_self { |str| str + " World" }
  => "Hello World"

irb> "Hello".tap { |str| str + " World" }
  => "Hello"

Overall, yield_self improves readability of the code by promoting chaining over nested function calls. Here is an example of both the styles.

irb> add_greeting = -> (str) { "HELLO " + str }
irb> to_upper = -> (str) { str.upcase }

# with new `yield_self`
irb> "world".yield_self(&to_upper)
            .yield_self(&add_greeting)
  => "HELLO WORLD"

# nested function calls
irb> add_greeting.call(to_upper.call("world"))
  => "HELLO WORLD"

yield_self is part of Kernel and hence it’s available to all the objects.

Rails 5.2 implements fetch_values for HashWithIndifferentAccess

This blog is part of our Rails 5.2 series.

Ruby 2.3 added fetch_values method to hash.

By using fetch_values we are able to get values for multiple keys in a hash.

capitals = { usa: "Washington DC",
             china: "Beijing",
             india: "New Delhi",
             australia: "Canberra" }

capitals.fetch_values(:usa, :india)
#=> ["Washington DC", "New Delhi"]

capitals.fetch_values(:usa, :spain) { |country| "N/A" }
#=> ["Washington DC", "N/A"]

Rails 5.2 introduces method fetch_values on HashWithIndifferentAccess. We’ll hence be able to fetch values of multiple keys on any instance of HashWithIndifferentAccess class.

capitals = HashWithIndifferentAccess.new
capitals[:usa] = "Washington DC"
capitals[:china] = "Beijing"

capitals.fetch_values("usa", "china")
#=> ["Washington DC", "Beijing"]

capitals.fetch_values("usa", "spain") { |country| "N/A" }
#=> ["Washington DC", "N/A"]

Ruby 2.5 added delete_prefix and delete_suffix methods

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

Ruby 2.4

Let’s say that we have a string Projects::CategoriesController and we want to remove Controller. We can use chomp method.

irb> "Projects::CategoriesController".chomp("Controller")
  => "Projects::Categories"

However if we want to remove Projects:: from the string then there is no corresponding method of chomp. We need to resort to sub.

irb> "Projects::CategoriesController".sub(/Projects::/, '')
  => "CategoriesController"

Naotoshi Seo did not like using regular expression for such a simple task. He proposed that Ruby should have a method for taking care of such tasks.

Some of the names proposed were remove_prefix, deprefix, lchomp, remove_prefix and head_chomp.

Matz suggested the name delete_prefix and this method was born.

Ruby 2.5.0-preview1

irb> "Projects::CategoriesController".delete_prefix("Projects::")
  => "CategoriesController"

Now in order to delete prefix we can use delete_prefix and to delete suffix we could use chomp. This did not feel right. So for symmetry delete_suffix was added.

irb> "Projects::CategoriesController".delete_suffix("Controller")
  => "Projects::Categories"

Read up on this discussion to learn more about how elixir, go, python, and PHP deal with similar requirements.

Ruby 2.5 introduces Dir.children and Dir.each_child

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

Dir.entries is a method present in Ruby 2.4. It returns the output of shell command ls -a in an array.

 > Dir.entries("/Users/john/Desktop/test")
 => [".", "..", ".config", "program.rb", "group.txt"]

We also have method Dir.foreach which iterates and yields each value from the output of ls -a command to the block.

> Dir.foreach("/Users/john/Desktop/test") { |child| puts child }
.
..
.config
program.rb
group.txt
test2

We can see that the output includes the directives for current directory and parent directory which are "." and "..".

When we want to have access only to the children files and directories, we do not need the [".", ".."] subarray.

This is a very common use case and we’ll probably have to do something like Dir.entries(path) - [".", ".."] to achieve the desired output.

To overcome such issues, Ruby 2.5 introduced Dir.children. It returns the output of ls -a command without the directives for current and parent directories.


> Dir.children("/Users/mohitnatoo/Desktop/test")
 => [".config", "program.rb", "group.txt"]

Additionally, we can use Dir.each_child method to avoid yielding current and parent directory directives while iterating,

> Dir.each_child("/Users/mohitnatoo/Desktop/test") { |child| puts child }
.config
program.rb
group.txt
test2

As noted in the discussion the names were chosen to match with existing methods Pathname#children and Pathname#each_child.

These additions seem like simple features. Well the issue was posted more than two years ago.

Higher Order Component for rendering spinner in React Native app

In one of our previous blogs, we mentioned how recompose improves both the readability and the maintainability of the code.

We also saw how branch and renderComponent functions from recompose help us in deciding which component to render based on a condition.

We can use the code from renderComponent documentation to render a spinner component when the data is being fetched in a ReactJS application.

Initial Code

// PatientsList.js

import React, { Component } from 'react';
import LoadingIndicator from './LoadingIndicator';

export default class PatientsList extends Component {

  state = {
    isLoading: true,
    patientsList: [],
  }

  componentDidMount() {
    api.getPatientsList().then(responseData => {
      this.setState({
        patientsList: responseData,
        isLoading: false,
      })
    })
  }

  render() {
    const { isLoading } = this.state;
    if (isLoading) {
      return <LoadingIndicator isLoading={isLoading} />
    } else {
      return (
        <ScrollView>
          // Some header component
          // View rendering the patients
        </ScrollView>
      )
    }
  }

In the above code, when the PatientsList component mounts, it fetches the list of patients from the API. During this time, the isLoading state is true, so we render the LoadingIndicator component.

Once the API call returns with the response, we set the isLoading state to false. This renders ScrollView component, with our list of patients.

The above code works fine, but if our app has multiple screens, which show the loading indicator and fetch data, the above way of handling it becomes repetitive and hard to maintain.

Building a higher order component

Here’s where Higher Order Components(HOC) are very useful. We can extract the logic for the ability to show the loading indicator in a HOC.

// withSpinner.js

import React from 'react';
import { ScrollView } from 'react-native';
import LoadingIndicator from './LoadingIndicator';

const withSpinner = Comp => ({ isLoading, children, ...props }) => {
  if (isLoading) {
    return <LoadingIndicator isLoading={isLoading} />
  } else {
    return (
      <Comp {...props}>
        {children}
      </Comp>
    )
  }
};

export default withSpinner;

Here, we created a HOC component which accepts a component and the isLoading prop.

If isLoading is true, we show the LoadingIndicator. If isLoading is false, we show the supplied component with its children, and pass in the props.

Now, we can use the above HOC in our PatientsList.js file. The supplied component can be any React Native component based on the use case. Here in our case, its a ScrollView.

// PatientsList.js

import { ScrollView } from 'react-native';
import withSpinner from './withSpinner';
const ScrollViewWithSpinner = withSpinner(ScrollView);

export default class PatientsList extends Component {

  state = {
    isLoading: true,
    patientsList: [],
  }

  componentDidMount() {
    api.getPatientsList().then(responseData => {
      this.setState({
        patientsList: responseData,
        isLoading: false,
      })
    })
  }

  render() {
    const { isLoading } = this.state;
    return(
      <ScrollViewWithSpinner
        isLoading={isLoading}
        // other props
      >
        // Some header component
        // View rendering the patients
      </ScrollViewWithSpinner>
    )
  }

Conclusion

Because of the above extraction of logic to a HOC, we can now use the same HOC in all our components which render a loading indicator while the data is being fetched.

The logic to show a loading indicator now resides in a HOC. This makes the code easier to maintain and less repetitive.

Rails 5.1 does not load all records on ActiveRecord::Relation#inspect

This blog is part of our Rails 5.1 series.

Let’s take a project with hundreds of users. When we call inspect on User.all, we see an array of 10 users followed by .... That means the output of #inspect method shows data only for 10 records.

> User.all.inspect
User Load (3.7ms)  SELECT  "users".* FROM "users"
=> "#<ActiveRecord::Relation [
#<User id: 1, email: \"dirbee@example.com\" >,
#<User id: 2, email: \"tee@example.com\">,
#<User id: 3, email: \"scott@example.com\">,
#<User id: 4, email: \"mark@example.com\">,
#<User id: 5, email: \"ben@example.com\">,
#<User id: 6, email: \"tina@example.com\">,
#<User id: 7, email: \"tyler@example.com\">,
#<User id: 8, email: \"peter@example.com\">,
#<User id: 9, email: \"rutul@example.com\">,
#<User id: 10, email:\"michael@example.com\">,
...]>"

We can see that the query executed in the process is fetching all the records even though the output doesn’t need all of them.

In Rails 5.1, only the needed records are loaded when inspect is called on ActiveRecord::Relation.

> User.all.inspect
User Load (3.7ms)  SELECT  "users".* FROM "users" LIMIT $1 /*application:Ace Invoice*/  [["LIMIT", 11]]

=> "#<ActiveRecord::Relation [
#<User id: 1, email: \"dirbee@example.com\" >,
#<User id: 2, email: \"tee@example.com\">,
#<User id: 3, email: \"scott@example.com\">,
#<User id: 4, email: \"mark@example.com\">,
#<User id: 5, email: \"ben@example.com\">,
#<User id: 6, email: \"tina@example.com\">,
#<User id: 7, email: \"tyler@example.com\">,
#<User id: 8, email: \"peter@example.com\">,
#<User id: 9, email: \"rutul@example.com\">,
#<User id: 10, email:\"michael@example.com\">,
...]>"

We can see in the above case that query executed has limit constraint and hence only the required number of records are loaded.

Fixing CORS issue with AWS services

While working on a client project, we started facing an issue where the JWPlayer stopped playing videos when we switched to hls version of videos. We found a CORS error in the JS console as shown below.

cors error

After researching we found that JWPlayer makes an AJAX request to load the m3u8 file. To fix the issue, we needed to enable CORS and for that we needed to make changes to S3 and Cloudfront configurations.

S3 configuration changes

We can configure CORS for the S3 bucket by allowing requests originating from specified hosts. As show in the image below we can find the CORS configuration option in Permissions tab of the S3 bucket. Here is the official documentation on configuring CORS for S3.

s3 cors configuration

S3 bucket will now allow requests originating from the specified hosts.

Cloudfront configuration changes

Cloudfront is a CDN service provided by AWS which uses edge locations to speed up the delivery of static content. Cloudfront takes content from S3 buckets and caches it at edge locations and delivers it to the end user. For enabling CORS we need to configure Cloudfront to allow forwarding of required headers.

We can configure the behavior of Cloudfront by clicking on Cloudfront Distribution’s “Distribution Settings”. Then from the “Behaviour” tab click on “Edit”. Here we need to whitelist the headers that need to be forwarded. Select the “Origin” header to whitelist which is required for CORS, as shown in the image below.

cloudfront behaviour

Ruby 2.5 allows rescue/else/ensure inside do/end blocks

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

Ruby 2.4

irb> array_from_user = [4, 2, 0, 1]
  => [4, 2, 0, 1]

irb> array_from_user.each do |number|
irb>   p 10 / number
irb> rescue ZeroDivisionError => exception
irb>   p exception
irb>   next
irb> end
SyntaxError: (irb):4: syntax error, unexpected keyword_rescue,
expecting keyword_end
rescue ZeroDivisionError => exceptio
      ^

Ruby 2.4 throws error when we try to use rescue/else/ensure inside do/end blocks.

Ruby 2.5.0-preview1

irb> array_from_user = [4, 2, 0, 1]
  => [4, 2, 0, 1]
irb> array_from_user.each do |number|
irb>   p 10 / number
irb> rescue ZeroDivisionError => exception
irb>   p exception
irb>   next
irb> end
2
5
#<ZeroDivisionError: divided by 0>
10
 => [4, 2, 0, 1]

Ruby 2.5 supports rescue/else/ensure inside do/end blocks.

Here is relevant commit and discussion.

Ruby 2.5 has removed top level constant lookup

This blog is part of our Ruby 2.5 series. Ruby 2.5 was recently released.

Ruby 2.4

irb> class Project
irb> end
=> nil

irb> class Category
irb> end
=> nil

irb> Project::Category
(irb):5: warning: toplevel constant Category referenced by Project::Category
 => Category

Ruby 2.4 returns top level constant with a warning if it is unable to find find constant in the specified scope.

This does not work well in cases where we need constants to be defined with same name at top level and also in same scope.

Ruby 2.5.0-preview1

irb> class Project
irb> end
=> nil

irb> class Category
irb> end
=> nil

irb> Project::Category
NameError: uninitialized constant Project::Category
Did you mean?  Category
	from (irb):5

Ruby 2.5 throws an error if it is unable to find constant in the specified scope.

Here is relevant commit and discussion.

Scheduling pods on nodes in Kubernetes using labels

This post assumes that you have basic understanding of Kubernetes terms like pods, deployments and nodes.

A Kubernetes cluster can have many nodes. Each node in turn can run multiple pods. By default Kubernetes manages which pod will run on which node and this is something we do not need to worry about it.

However sometimes we want to ensure that certain pods do not run on the same node. For example we have an application called wheel. We have both staging and production version of this app and we want to ensure that production pod and staging pod are not on the same host.

To ensure that certain pods do not run on the same host we can use nodeSelector constraint in PodSpec to schedule pods on nodes.

Kubernetes cluster

We will use kops to provision cluster. We can check the health of cluster using kops validate-cluster.

$ kops validate cluster
Using cluster from kubectl context: test-k8s.nodes-staging.com

Validating cluster test-k8s.nodes-staging.com

INSTANCE GROUPS
NAME              ROLE   MACHINETYPE MIN MAX SUBNETS
master-us-east-1a Master m4.large    1   1 us-east-1a
master-us-east-1b Master m4.large    1   1 us-east-1b
master-us-east-1c Master m4.large    1   1 us-east-1c
nodes-wheel-stg   Node   m4.large    2   5 us-east-1a,us-east-1b
nodes-wheel-prd   Node   m4.large    2   5 us-east-1a,us-east-1b

NODE STATUS
           NAME                ROLE   READY
ip-192-10-110-59.ec2.internal  master True
ip-192-10-120-103.ec2.internal node   True
ip-192-10-42-9.ec2.internal    master True
ip-192-10-73-191.ec2.internal  master True
ip-192-10-82-66.ec2.internal   node   True
ip-192-10-72-68.ec2.internal   node   True
ip-192-10-182-70.ec2.internal  node   True

Your cluster test-k8s.nodes-staging.com is ready

Here we can see that there are two instance groups for nodes: nodes-wheel-stg and nodes-wheel-prd.

nodes-wheel-stg might have application pods like pod-wheel-stg-sidekiq, pod-wheel-stg-unicorn and pod-wheel-stg-redis. Smilarly nodes-wheel-prd might have application pods like pod-wheel-prd-sidekiq, pod-wheel-prd-unicorn and pod-wheel-prd-redis.

As we can see the Max number of nodes for instance group nodes-wheel-stg and nodes-wheel-prd is 5. It means if new nodes are created in future then based on the instance group the newly created nodes will automatically be labelled and no manual work is required.

Labelling a Node

We will use kubernetes labels to label a node. To add a label we need to edit instance group using kops.

$ kops edit ig nodes-wheel-stg

This will open up instance group configuration file, we will add following label in instance group spec.

nodeLabels:
   type: wheel-stg

Complete ig configuration looks like this.

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-10-12T06:24:53Z
  labels:
    kops.k8s.io/cluster: k8s.nodes-staging.com
  name: nodes-wheel-stg
spec:
  image: kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28
  machineType: m4.large
  maxSize: 5
  minSize: 2
  nodeLabels:
    type: wheel-stg
  role: Node
  subnets:
  - us-east-1a
  - us-east-1b
  - us-east-1c

Similarly, we can label for instance group nodes-wheel-prod with label type wheel-prod.

After making the changes update cluster using kops rolling update cluster --yes --force. This will update the cluster with specified labels.

New nodes added in future will have labels based on respective instance groups.

Once nodes are labeled we can verify using kubectl describe node.

$ kubectl describe node ip-192-10-82-66.ec2.internal
Name:               ip-192-10-82-66.ec2.internal
Roles:              node
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=m4.large
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=us-east-1
                    failure-domain.beta.kubernetes.io/zone=us-east-1a
                    kubernetes.io/hostname=ip-192-10-82-66.ec2.internal
                    kubernetes.io/role=node
                    type=wheel-stg

In this way we have our node labeled using kops.

Labelling nodes using kubectl

We can also label node using kubectl.

$ kubectl label node ip-192-20-44-136.ec2.internal type=wheel-stg

After labeling a node, we will add nodeSelector field to our PodSpec in deployment template.

We will add the following block in deployment manifest.

nodeSelector:
  type: wheel-stg

We can add this configuration in original deployment manifest.

apiVersion: v1
kind: Deployment
metadata:
  name: test-staging-node
  labels:
    app: test-staging
  namespace: test
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: test-staging
    spec:
      containers:
      - image: <your-repo>/<your-image-name>:latest
        name: test-staging
        imagePullPolicy: Always
        - name: REDIS_HOST
          value: test-staging-redis
        - name: APP_ENV
          value: staging
        - name: CLIENT
          value: test
        ports:
        - containerPort: 80
      nodeSelector:
        type: wheel-stg
      imagePullSecrets:
        - name: registrykey

Let’s launch this deployment and check where the pod is scheduled.

$ kubectl apply -f test-deployment.yml
deployment "test-staging-node" created

We can verify that our pod is running on node type=wheel-stg.

kubectl describe pod test-staging-2751555626-9sd4m
Name:           test-staging-2751555626-9sd4m
Namespace:      default
Node:           ip-192-10-82-66.ec2.internal/192.10.82.66
...
...
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
QoS Class:       Burstable
Node-Selectors:  type=wheel-stg
Tolerations:     node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Similarly we run nodes-wheel-prod pods on nodes labeled with type: wheel-prod.

Please note that when we specify nodeSelector and no node matches label then pods are in pending state as they dont find node with matching label.

In this way we schedule our pods to run on specific nodes for certain use-cases.

SAML integration with multiple IDPs using Devise & OmniAuth

Recently, we integrated our SAML service provider(SP) with multiple identity providers(IDPs) to facilitate Single sign-on(SSO) using Devise with OmniAuth.

Before we jump into the specifics, here is SAML definition from wikipedia.

Security Assertion Markup Language (SAML, pronounced sam-el) is an open standard for exchanging authentication and authorization data between parties, in particular, between an identity provider(IDP) and a service provider(SP).

The choice of Devise with OmniAuth-SAML to build SAML SSO capabilities was natural to us, as we already had dependency on Devise and OmniAuth nicely integrates with Devise.

Here is the official overview on how to integrate OmniAuth with Devise.

After following the overview, this is how our config and user.rb looked like.

# config file
Devise.setup do |config|
  config.omniauth :saml,
    idp_cert_fingerprint: 'fingerprint',
    idp_sso_target_url: 'target_url'
end

#user.rb file
devise :omniauthable, :omniauth_providers => [:saml]

The problem with above configuration is that it supports only one SAML IDP.

To have support for multiple IDPs, we re-defined files as below.

# config file
Devise.setup do |config|
  config.omniauth :saml_idp1,
    idp_cert_fingerprint: 'fingerprint-1',
    idp_sso_target_url: 'target_url-1'
    strategy_class: ::OmniAuth::Strategies::SAML,
    name: :saml_idp1

  config.omniauth :saml_idp2,
    idp_cert_fingerprint: 'fingerprint-2',
    idp_sso_target_url: 'target_url-2'
    strategy_class: ::OmniAuth::Strategies::SAML,
    name: :saml_idp2
end

#user.rb file
devise :omniauthable, :omniauth_providers => [:saml_idp1, :saml_idp2]

Let’s go through the changes one by one.

1. Custom Providers: Instead of using standard provider saml, we configured custom providers (saml_idp1, saml_idp2) in the first line of configuration as well as in user.rb

2. Strategy Class: In case of the standard provider(saml), Devise can figure out strategy_class on its own. For custom providers, we need to explicitly specify it.

3. OmniAuth Unique Identifier: After making the above two changes, everything worked fine except OmniAuth URLs. For some reason, OmniAuth was still listening to saml scoped path instead of new provider names saml_idp1, saml_idp2.

# Actual metadata path used by OmniAuth
/users/auth/saml/metadata

# Expected metadata path
/users/auth/saml_idp1/metadata
/users/auth/saml_idp2/metadata

After digging in Devise and OmniAuth code bases, we discovered provider name configuration. In the absence of this configuration, OmniAuth falls back to strategy class name to build the path. As we could not find any code in Devise which defined name for OmniAuth that explained saml scoped path (we were expecting Devise to pass name assigning same value as provider).

After adding name configuration, OmnitAuth started listening to the correct URLs.

4. Callback Actions: Lastly, we added both actions in OmniauthCallbacksController:

class Users::OmniauthCallbacksController < Devise::OmniauthCallbacksController

  def saml_idp1
    # Implementation
  end

  def saml_idp2
    # Implementation
  end

  # ...
  # Rest of the actions
end

With these changes along with the official guide mentioned above, our SP was able to authenticate users from multiple IDPs.