San Francisco, USA

5214F Diamond Heights Blvd #553
San Francisco, CA 94131

Pune, India

203, Jewel Towers, 2nd Floor
Lane Number 5, Koregaon Park
Pune 411001, India

301 - 275 - 3997
hello@BigBinary.com

Migrating Gumroad from RequireJS to webpack

BigBinary has been working with Gumroad for a while. Following blog post has been posted with permission from Gumroad and we are very grateful to Sahil for allowing us to discuss the work in such an open environment.

This application is a JavaScript-heavy application as most consumer-oriented applications are these days. We recently changed the JavaScript build system for Gumroad from RequireJS to webpack. We’d like to talk about how we went about doing this.

Gumroad’s web application is built using Ruby on Rails. The project was started way back in 2011 as this hacker news post suggests. When we began working on the code it was building JavaScript assets through two systems Sprockets and RequireJS. From what we could tell, all the code which was using a new(at the time) frontend framework was processed by RequireJS first and then sprockets whereas the JavaScript files which are usually present under app/javascrips/assets and vendor/assets/javascripts in a typical Rails application were present as well but they were not being processed by RequireJS. Also, there were some libraries which were sourced using Bower.

We were tasked with the work of migrating the RequireJS build system over to webpack and replacing Bower with NPM. The reason behind this was that we wanted to use newer tools with wider community support. Another reason was that we wanted to be able to take advantage of all the goodies that webpack comes with though that was not a strong motivation at that point.

We decided to break down the task into small pieces which could be worked on in iterations and, more importantly, could be shipped in iterations. This would enable us to work on other tasks in the application in parallel and not be blocked on a big chunk of work. Keeping that in mind we split the task in three different steps.

Step 1: Migrate from RequireJS to webpack with the minimal amount of changes in the actual code.

Step 2: Use NPM packages in place of Bower components.

Step 3: Use NPM packages in place of libraries present under vendor/assets/javascripts.

Step 1: Migrate from RequireJS to webpack with the minimal amount of changes in the actual code

The first thing we did here was create a new webpack.config.js configuration file which would be used by webpack. We did our best to accurately translate the configuration from the RequireJS configuration file using multiple resources available online.

Here is how most JavaScript files which were to be processed by RequireJS looked like.

'use strict';

define([
      'braintree'
    , '$app/ui/product/edit'
    , '$app/data/product'
  ],

  function(Braintree, ProductEditUI, ProductData) {
    // Do something with Braintree, ProductEditUI, and ProductData
  }
);

As you can see, the code did not use the newer import statements which you’d see in comparatively newer JavaScript code. As we’ve mentioned earlier, our goal was to have minimal code changes so we did not want to change to import just yet. Luckily for us, webpack supports the define API for specifying dependencies. This meant that we would not need to change how dependencies were specified in any of the JavaScript files.

In this step we also changed the build system configuration (The webpack.config.js file in this case) to use NPM packages where possible instead of using libraries from the vendor/ directory. This meant that we would need to have aliases in place for instances where the package name was different from the names we had aliased the libraries to.

For example, this is how the ‘braintree’ alias was set earlier in order to refer to the Braintree SDK. Now all the code had to do was to mention that braintree was a dependency.

require.config({
  paths: {
    braintree: '/vendor/assets/javascripts/braintree-2.16.0'
  }
});

With the change to use the NPM package in place of the JavaScript file the dependency sourcing did not work as expected because the NPM package name was ‘braintree-web’ and the source code was trying to load ‘braintree’ which was not known to the build system(webpack). In order to avoid making changes to source code we used the “alias” feature provided by webpack as shown below.

module.exports = {
  resolve: {
    alias: {
      braintree: 'braintree-web',
    }
  }
};

We did this for all the dependencies which had been given an alias in the RequireJS configuration and we got dependency resolution to work as expected.

As a part of this step, we also created a new common chunk and used it to improve caching. You can read more about this feature here. Note that we would tweak this iteratively later but we thought it would be good to get started with the basic configuration right away.

Step 2: Use NPM packages in place of Bower components

Another goal of the migration was to remove Bower so as to make the build system simpler. The first reason behind this was that all Bower packages which we were using were available as NPM packages. The second reason was that Bower itself is recommending users to migrate to Yarn/webpack for a while now.

What we did here was simple. We removed Bower and the Bower configuration file. Then, we sourced the required Bower components as NPM packages instead by adding them to package.json. We also removed the aliases added to source them from the webpack configuration.

For example, here’s the change required to the configuration file after sourcing clipboard as an NPM package instead of a Bower component.

resolve: {
  alias: {
    // Other Code

    $app:           path.resolve(__dirname, '../../app/javascript'),
    $lib:           path.resolve(__dirname, '../../lib/assets/javascripts')
-   clipboard:      path.resolve(__dirname, '../../vendor/assets/javascripts/clipboard.min.js')
  }
}

Step 3: Use NPM packages in place of libraries present under vendor/assets/javascripts

We had a lot of javascript libraries present under vendor/assets/javascripts which were sourced in the required javascript files. We deleted those files from the project and sourced them as NPM packages instead. This way we could have better visibility and control over the versions of these packages.

As part of this migration we also did some asset-related cleanups. These included removing unused JavaScript files, including JavaScript files only where required instead of sourcing them into the global scope, etc.

We were continuously measuring the performance of the application before and after applying changes to make sure that we were not worsening the performance during the migration. In the end, we found that we had improved the page load speeds by an average of 2%. Note that this task was not undertaken to improve the performance of the application. We are now planning to leverage webpack features and try to improve on this metric further.

Rails 5 Active Record attributes API

This blog is part of our Rails 5 series.

Rails 5 was a major release with a lot of new features like Action Cable, API Applications, etc. Active Record attribute API was also one of the features of Rails 5 release which did not receive much attention.

Active Record attributes API is used by Rails internally for a long time. In Rails 5 release, attributes API was made public and allowed support for custom types.

What is attribute API?

Attribute API converts the attribute value to an appropriate Ruby type. Here is how the syntax looks like.

attribute(name, cast_type, options)

The first argument is the name of the attribute and the second argument is the cast type. Cast type can be string, integer or custom type object.

# db/schema.rb

create_table :movie_tickets, force: true do |t|
  t.float :price
end

# without attribute API

class MovieTicket < ActiveRecord::Base
end

movie_ticket = MovieTicket.new(price: 145.40)
movie_ticket.save!

movie_ticket.price   # => Float(145.40)

# with attribute API

class MovieTicket < ActiveRecord::Base
  attribute :price, :integer
end

movie_ticket.price   # => 145

Before using attribute API, movie ticket price was a float value, but after applying attribute on price, the price value was typecast as integer.

The database still stores the price as float and this conversion happens only in Ruby land.

Now, we will typecast movie release_date from datetime to date type.

# db/schema.rb

create_table :movies, force: true do |t|
  t.datetime :release_date
end

class Movie < ActiveRecord::Base
  attribute :release_date, :date
end

movie.release_date # => Thu, 01 Mar 2018

We can also add default value for an attribute.

# db/schema.rb

create_table :movies, force: true do |t|
  t.string :license_number, :string
end

class Movie < ActiveRecord::Base
  attribute :license_number,
            :string,
            default: "IN00#{Date.current.strftime('%Y%m%d')}00#{rand(100)}"
end

# without attribute API with default value on license number

Movie.new.license_number  # => nil

# with attribute API with default value on license number

Movie.new.license_number  # => "IN00201805250068"

Custom Types

Let’s say we want the people to rate a movie in percentage. Traditionally, we would do something like this.

class MovieRating < ActiveRecord::Base

  TOTAL_STARS = 5

  before_save :convert_percent_rating_to_stars

  def convert_percent_rating_to_stars
    rating_in_percentage = value.gsub(/\%/, '').to_f

    self.rating = (rating_in_percentage * TOTAL_STARS) / 100
  end
end

With attributes API we can create a custom type which will be responsible to cast to percentage rating to number of stars.

We have to define the cast method in the custom type class which casts the given value to the expected output.

# db/schema.rb

create_table :movie_ratings, force: true do |t|
  t.integer :rating
end

# app/types/star_rating_type.rb

class StarRatingType < ActiveRecord::Type::Integer
  TOTAL_STARS = 5

  def cast(value)
    if value.present? && !value.kind_of?(Integer)
      rating_in_percentage = value.gsub(/\%/, '').to_i

      star_rating = (rating_in_percentage * TOTAL_STARS) / 100
      super(star_rating)
    else
      super
    end
  end
end

# config/initializers/types.rb

ActiveRecord::Type.register(:star_rating, StarRatingType)

# app/models/movie.rb

class MovieRating < ActiveRecord::Base
  attribute :rating, :star_rating
end

Querying

The attributes API also supports where clause. Query will be converted to SQL by calling serialize method on the type object.

class StarRatingType < ActiveRecord::Type::Integer
  TOTAL_STARS = 5

  def serialize(value)
    if value.present? && !value.kind_of?(Integer)
      rating_in_percentage = value.gsub(/\%/, '').to_i

      star_rating = (rating_in_percentage * TOTAL_STARS) / 100
      super(star_rating)
    else
      super
    end
  end
end


# Add new movie rating with rating as 25.6%.
# So the movie rating in star will be 1 of 5 stars.
movie_rating = MovieRating.new(rating: "25.6%")
movie_rating.save!

movie_rating.rating   # => 1

# Querying with rating in percentage 25.6%
MovieRating.where(rating: "25.6%")

# => #<ActiveRecord::Relation [#<MovieRating id: 1000, rating: 1 ... >]>

Passing current_user by default in Sidekiq

In one of our projects we need to capture user activity throughout the application. For example when a user updates projected distance of a delivery, the aplication should create an activity for that action. To create an activity we need the currently logged in user id since we need to associate the activity with that user.

We are using devise gem for authentication which provides current_user method by default to controllers. Any business logic residing at controller level can use current_user to associate the activity with the logged in user. However, some business logics reside in Sidekiq where current_user is not available.

Passing current_user to Sidekiq job

One way to solve this issue is to pass the current_user directly to the Sidekiq job. Here’s how we can do it.

  class DeliveryController < ApplicationController
    def update
      # update attributes
      DeliveryUpdateWorker.
        perform_async(params[:delivery], current_user.login)
      # render delivery
    end
  end
  class DeliveryUpdateWorker
    include Sidekiq::Worker

    def perform(delivery, user_login)
      user = User.find_by(login: user_login)
      ActivityGenerationService.new(delivery, user) if user
    end
  end

That works. Now let’s say we add another endpoint in which we need to track when delivery is deleted. Here’s the updated code.

  class DeliveryController < ApplicationController
    def update
      # update attributes
      DeliveryUpdateWorker.
        perform_async(params[:delivery], current_user.login)
      # render delivery
    end

    def destroy
      # delete attributes
      DeliveryDeleteWorker.
        perform_async(params[:delivery], current_user.login)
      # render :ok
    end
  end
  class DeliveryDeleteWorker
    include Sidekiq::Worker

    def perform(delivery, user_login)
      user = User.find_by(login: user_login)
      ActivityGenerationService.new(delivery, user) if user
    end
  end

Again we needed to pass current_user login in the new endpoint. You can notice a pattern here. For each endpoint which needs to track activity we need to pass current_user. What if we could pass current_user info by default.

The main reason we want to pass current_user by default is because we’re tracking model attribute changes in the model’s before_save callbacks.

For this we store current_user info in Thread.current and access it in before_save callbacks of the model which generated relevant activity.

This will work fine for model attribute changes made in controllers and services where Thread.current is accessible and persisted. However, for Sidekiq jobs which changes the model attributes whose activity is generated, we need to pass the current_user manually since Thread.current is not available in Sidekiq jobs.

Again we can argue here that we don’t need to pass the current_user by default. Instead we can pass it in each Sidekiq job as an argument. This will work in simple cases, although for more complex cases this will require extra effort.

For eg. let’s say we’re tracking delivery’s cost. We’ve three sidekiq jobs, DeliveryDestinationChangeWorker, DeliveryRouteChangeWorker and DeliveryCostChangeWorker. We call DeliveryDestinationChangeWorker which changes the destination of a delivery. This calls DeliveryRouteChangeWorker which calculates the new route and calls DeliveryCostChangeWorker. Now DeliveryCostChangeWorker changes the delivery cost where the before_save callback is called.

In this example you can see that we need to pass current_user through all three Sidekiq workers and initialize Thread.current in DeliveryCostChangeWorker. The nesting can go much deeper.

Passing current_user by default will make sure if the activity is being generated in a model’s before_save callback then it can access the current_user info from Thread.current no matter how much nested the Sidekiq call chain is.

Also it makes sure that if a developer adds another Sidekiq worker class in the future which changes a model whose attribute change needs to be tracked. Then the developer need not remember to pass current_user explicitly to the Sidekiq worker.

Note the presented problem in this blog is an oversimplified version in order to better present the solution.

Creating a wrapper module to include current_user by default

The most basic solution to pass current_user by default is to create a wrapper module. This module will be responsible for adding the current_user when perform_async is invoked. Here’s an example.

  module SidekiqMediator
    def perform_async(klass, *args)
      args.push(current_user.login)
      klass.send(:perform_async, *args)
    end
  end
  class DeliveryController < ApplicationController
    include SidekiqMediator

    def update
      # update attributes
      perform_async(DeliveryUpdateWorker, params[:delivery])
      # render delivery
    end

    def destroy
      # delete attributes
      perform_async(DeliveryDeleteWorker, params[:delivery])
      # render :ok
    end
  end
  class DeliveryDeleteWorker
    include Sidekiq::Worker

    def perform(delivery, user_login)
      user = User.find_by(login: user_login)
      ActivityGenerationService.new(delivery, user) if user
    end
  end

Now we don’t need to pass current_user login in each call. However we still need to remember including SidekiqMediator whenever we need to use current_user in the Sidekiq job for activity generation. Another way to solve this problem is to intercept the Sidekiq job before it is pushed to redis. Then we can include current_user login by default.

Using Sidekiq client middleware to pass current_user by default

Sidekiq provides a client middleware to run custom logic before pushing the job in redis. We can use the client middleware to push current_user as default argument in the Sidekiq arguments. Here’s an example of Sidekiq client middleware.

  class SidekiqClientMiddleware
    def call(worker_class, job, queue, redis_pool = nil)
      # Do something before pushing the job in redis
      yield
    end
  end

We need a way to introduce current_user in the Sidekiq arguments. The job payload contains the arguments passed to the Sidekiq worker. Here’s what the job payload looks like.

  {
    "class": "DeliveryDeleteWorker",
    "jid": "b4a577edbccf1d805744efa9",
    "args": [1, "arg", true],
    "created_at": 1234567890,
    "enqueued_at": 1234567890
  }

Notice here the args key which is an array containing the arguments passed to the Sidekiq worker. We can push the current_user in the args array. This way each Sidekiq job will have current_user by default as the last argument. Here’s the modified version of the client middleware which includes current_user by default.

  class SidekiqClientMiddleware
    def call(_worker_class, job, _queue, _redis_pool = nil)
      # Push current user login as the last argument by default
      job['args'].push(current_user.login)
      yield
    end
  end

Now we don’t need to pass current_user login to Sidekiq workers in the controller. Here’s how our controller logic looks like now.

  class DeliveryController < ApplicationController
    def update
      # update attributes
      DeliveryUpdateWorker.perform_async(params[:data])
      # render delivery
    end

    def destroy
      # delete attributes
      DeliveryDeleteWorker.perform_async(params[:data])
      # render :ok
    end
  end

We don’t need SidekiqMediator anymore. The current_user will automatically be included as the last argument in every Sidekiq job.

Although there’s one issue here. We included current_user by default to every Sidekiq worker. This means workers which does not expect current_user as an argument will also have current_user as their last argument. This will raise ArgumentError: wrong number of arguments (2 for 1). Here’s an example.

  class DeliveryCreateWorker
    include Sidekiq::Worker

    def perform(data)
      # doesn't use current_user login to track activity when called
      # this will get data, current_user_login as the arguments
    end
  end

To solve this we need to extract current_user argument from job['args'] before the worker starts processing.

Using Sidekiq server middleware to extract current_user login

Sidekiq also provides server middleware which runs before processing any Sidekiq job. We used this to extract current_user from job['args'] and saved it in a global state.

This global state should persist when the server middleware execution is complete and the actual Sidekiq job processing has started. Here’s the server middleware.

  class SidekiqServerMiddleware
    def call(_worker, job, _queue)
      set_request_user(job['args'].pop)
      yield
    end

    private
    def set_request_user(request_user_login)
      RequestStore.store[:request_user_login] = request_user_login
    end
  end

Notice here we used pop to extract the last argument. Since we’re setting the last argument to current_user in the client middleware, the last argument will always be the current_user in server middleware.

Using pop also removes current_user from job['args'] which ensures the worker does not get current_user as an extra argument.

We used request_store to persist a global state. RequestStore provides a per request global storage using Thread.current which stores info as a key value pair. Here’s how we used it in Sidekiq workers to access the current_user info.

  class DeliveryDeleteWorker
    include Sidekiq::Worker

    def perform(delivery)
      user_login = RequestStore.store[:request_user_login]
      user = User.find_by(login: user_login)
      ActivityGenerationService.new(delivery, user) if user
    end
  end

Now we don’t need to pass current_user in the controller when calling the Sidekiq worker. Also we don’t need to add user_login as an extra argument in each Sidekiq worker every time we need to access current_user.

Configure server middleware for Sidekiq test cases

By default Sidekiq does not run server middleware in inline and fake mode.

Because of this current_user was being added in the client middleware but it’s not being extracted in the server middleware since it’s never called.

This resulted in ArgumentError: wrong number of arguments (2 for 1) failures in our test cases which used Sidekiq in inline or fake mode. We solved this by adding following config:

  Sidekiq::Testing.server_middleware do |chain|
    chain.add SidekiqServerMiddleware
  end

This ensures that SidekiqServerMiddleware is called in inline and fake mode in our test cases.

However, we found an alternative to this which was much simpler and cleaner. We noticed that job payload is a simple hash which is pushed to redis as it is and is available in the server middleware as well.

Instead of adding the current_user as an argument in job['args'] we could add another key in job payload itself which will hold the current_user. Here’s the modified logic.

  class SidekiqClientMiddleware
    def call(_worker_class, job, _queue, _redis_pool = nil)
      # Set current user login in job payload
      job['request_user_login'] = current_user.login if defined?(current_user)
      yield
    end
  end
  class SidekiqServerMiddleware
    def call(_worker, job, _queue)
      if job.key?('request_user_login')
        set_request_user(job['request_user_login'])
      end
      yield
    end

    private
    def set_request_user(request_user_login)
      RequestStore.store[:request_user_login] = request_user_login
    end
  end

We used a unique key request_user_login which would not conflict with the other keys in the job payload. Additionally we added a check if request_user_login key is present in the job payload. This is necessary because if the user calls the worker from console then it’ll not have current_user set.

Apart from this we noticed that we had multiple api services talking to each other. These services also generated user activity. Few of them didn’t use Devise for authentication, instead the requesting user info was passed to them in each request as param.

For these services we set the request user info in RequestStore.store in our BaseApiController and changed the client middleware to use RequestStore.store instead of current_user method.

We also initialized RequestStore.store in services where we used Devise to make it completely independent of current_user. Here’s how our client middleware looks now.

  class SidekiqClientMiddleware
    def call(_worker_class, job, _queue, _redis_pool = nil)
      # Set current user login in job payload
      if RequestStore.store[:request_user_login]
        job['request_user_login'] = RequestStore.store[:request_user_login]
      end
      yield
    end
  end

Lastly we needed to register the client and server middleware in Sidekiq.

Configuring Sidekiq middleware

To enable the middleware with Sidekiq, we need to register the client middleware and the server middleware in config/initializers/sidekiq.rb. Here’s how we did it.

Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add SidekiqClientMiddleware
  end
end

Sidekiq.configure_server do |config|
  config.client_middleware do |chain|
    chain.add SidekiqClientMiddleware
  end
  config.server_middleware do |chain|
    chain.add SidekiqServerMiddleware
  end
end

Notice that we added SidekiqClientMiddleware in both configure_server block and configure_client block, this is because a Sidekiq job can call another Sidekiq job in which case the Sidekiq server itself will act as the client.

To sum it up, here’s how our client middleware and server middleware finally looked like.

  class SidekiqClientMiddleware
    def call(_worker_class, job, _queue, _redis_pool = nil)
      # Set current user login in job payload
      if RequestStore.store[:request_user_login]
        job['request_user_login'] = RequestStore.store[:request_user_login]
      end
      yield
    end
  end
  class SidekiqServerMiddleware
    def call(_worker, job, _queue)
      if job.key?('request_user_login')
        set_request_user(job['request_user_login'])
      end
      yield
    end

    private
    def set_request_user(request_user_login)
      RequestStore.store[:request_user_login] = request_user_login
    end
  end

The controller example we mentioned initially looks like:

  class DeliveryController < ApplicationController
    def update
      # update attributes
      DeliveryUpdateWorker.perform_async(params[:delivery])
      # render delivery
    end

    def destroy
      # delete attributes
      DeliveryDeleteWorker.perform_async(params[:delivery])
      # render :ok
    end
  end
  class DeliveryDeleteWorker
    include Sidekiq::Worker

    def perform(delivery)
      user_login = RequestStore.store[:request_user_login]
      user = User.find_by(login: user_login)
      ActivityGenerationService.new(delivery, user) if user
    end
  end
  class DeliveryUpdateWorker
    include Sidekiq::Worker

    def perform(delivery)
      user_login = RequestStore.store[:request_user_login]
      user = User.find_by(login: user_login)
      ActivityGenerationService.new(delivery, user) if user
    end
  end

Now we don’t need to explicitly pass current_user to each Sidekiq job. It’s available out of the box without any changes in all Sidekiq jobs.

As an alternative we can also use ActiveSupport::CurrentAttributes.

Discuss it on Reddit

Optimize loading multiple routes on Google map using B-spline

Applications use Google maps for showing routes from point A to B. For one of our clients we needed to show delivery routes on Google maps so that user can select multiple deliveries and then consolidate them as one single delivery. This meant we needed to show around 30 to 500 deliveries on a single map.

Using Google Map polylines

We used polylines to draw individual routes on Google maps.

Polyline is composed of line segments connecting a list of points on the map. The more points we use for drawing a polyline the more detailed the final curve will be. Here’s how we added route points to the map.

// List of latitude and longitude
let path = points.map((point) => [point.lat, point.lng]);

let route_options = {
  path: path,
  strokeColor: color,
  strokeOpacity: 1.0,
  strokeWeight: mapAttributes.strokeWeight || 3,
  map: map, // google.maps.Map
};

new google.maps.Polyline(route_options);

Here’s an example of a polyline on a Google map. We used 422 latitude and longitude points to draw these routes which makes it look more contiguous.

Polyline example

We needed to show 200 deliveries in that map. On an average a delivery contains around 500 route points. This means we need to load 1,00,000 route points. Let’s measure and see how much time the whole process takes.

Loading multiple routes on a map

Plotting a single route on a map can be done in less than a second. However as we increase the number of routes to plot, the payload size increases which affects the load time. This is because we’ve around 500 route points per delivery. If we want to show 500 deliveries on the map then we need to load 500 * 500 = 2,50,000 routes points. Let’s benchmark the load time it takes to show deliveries on a map.

No. of deliveries Load Time Payload Size
500 8.77s 12.3MB
400 7.76s 10.4MB
300 6.68s 7.9MB
200 5.88s 5.3MB
100 5.47s 3.5MB

The load time is more than 5 seconds which is high. What if we could decrease the payload size and still be able to plot the routes.

Sampling route points for decreased payload size

For each delivery we’ve around 500 route points. If we drop a few route points in between on a regular interval then we’ll be able to decrease the payload size. Latitude and longitude have atleast 5 decimal points. We rounded them off to 1 decimal point and then we picked unique values.

  def route_lat_lng_points
    return '' unless delivery.route_lat_lng_points

    delivery.route_lat_lng_points.
        chunk{ |point| [point.first.round(1), point.second.round(1)] }.
        map(&:first).join(',')
  end

Now let’s check the payload size and the load time.

No. of deliveries Load Time Payload Size
500 6.52s 6.7MB
400 5.97s 5.5MB
300 5.68s 4.2MB
200 4.88s 2.9MB
100 4.07s 2.0MB

The payload size decreased by 50 percent. However since we sampled the data the routes are not contiguous anymore. Here’s how it looks now.

Sampled routes Contiguous routes

Note that we sampled route points till single decimal point. Notice that the routes in which we did sampling appears to be jagged instead of contiguous. We can solve this by using a curve fitting method to create a curve from the discrete points we have.

Curve fitting using B-spline function

B-spline or basis spline is a spline function which can be used for creating smooth curves best fitted to a set of control points. Here’s an example of a B-spline curve created from a set of control points.

Bspline example

We changed our previous example to use B-spline function to generate latitude and longitude points.

// List of latitude and longitude
let lats = points.map((point) => point.lat);
let lngs = points.map((point) => point.lng);
let path = bspline(lats, lngs);

let route_options = {
  path: path,
  strokeColor: color,
  strokeOpacity: 1.0,
  strokeWeight: mapAttributes.strokeWeight || 3,
  map: map, // instance of google.maps.Map
};

new google.maps.Polyline(route_options);
 function bspline(lats, lngs) {
    let i, t, ax, ay, bx, by, cx, cy, dx, dy, lat, lng, points;
    points = [];

    for (i = 2; i < lats.length - 2; i++) {
      for (t = 0; t < 1; t += 0.2) {
        ax = (-lats[i - 2] + 3 * lats[i - 1] - 3 * lats[i] + lats[i + 1]) / 6;
        ay = (-lngs[i - 2] + 3 * lngs[i - 1] - 3 * lngs[i] + lngs[i + 1]) / 6;

        bx = (lats[i - 2] - 2 * lats[i - 1] + lats[i]) / 2;
        by = (lngs[i - 2] - 2 * lngs[i - 1] + lngs[i]) / 2;

        cx = (-lats[i - 2] + lats[i]) / 2;
        cy = (-lngs[i - 2] + lngs[i]) / 2;

        dx = (lats[i - 2] + 4 * lats[i - 1] + lats[i]) / 6;
        dy = (lngs[i - 2] + 4 * lngs[i - 1] + lngs[i]) / 6;

        lat = (ax * Math.pow(t + 0.1, 3)) +
              (bx * Math.pow(t + 0.1, 2)) +
              (cx * (t + 0.1)) + dx;

        lng = (ay * Math.pow(t + 0.1, 3)) +
              (by * Math.pow(t + 0.1, 2)) +
              (cy * (t + 0.1)) + dy;

        points.push(new google.maps.LatLng(lat, lng));
      }
    }
    return points;
  }

Source: https://johan.karlsteen.com/2011/07/30/improving-google-maps-polygons-with-b-splines

After the change the plotted routes are much better. Here’s how it looks now.

Bspline routes Contiguous routes

The only downside here is that if we zoom in the map we’ll notice that the routes are not exactly overlapping the Google map paths. Otherwise we’re able to plot almost same routes with sampled route points. However we still need 6.5 seconds to load 500 deliveries. How do we fix that ?

Loading deliveries in batches

Sometimes users have upto 500 deliveries but they want to change only a few deliveries and then use the application. Right now the way application is setup users have no choice but to wait until all 500 deliveries are loaded and then only they would be able to change a few deliveries. This is not ideal.

We want to show deliveries as soon as they’re loaded. We added a polling mechanism that would load batches of 20 deliveries and as soon as a batch is loaded we would plot them on the map. This way user could interact with the loaded deliveries while the remaining deliveries are still being loaded.

  loadDeliveriesWindow(updatedState = {}, lastPage = 0, currentWindow = 1) {
    // windowSize: Size of the batch to be loaded
    // perPage: No of deliveries per page
    const { perPage, windowSize } = this.state;

    if (currentWindow > perPage / windowSize) {
      // Streaming deliveries ended
      this.setState($.extend(updatedState, { windowStreaming: false }));
      return;
    }

    if (currentWindow === 1) {
      // Streaming deliveries started
      this.setState({ windowStreaming: true });
    }

    // Gets delivery data from backend
    this.fetchDeliveries(currentWindow + (lastPage * windowSize), queryParams).complete(() => {
      // Plots deliveries on map
      this.loadDeliveries();

      // Load the next batch of deliveries
      setTimeout((() => {
        this.loadDeliveriesWindow(updatedState, lastPage, currentWindow + 1);
      }).bind(this, currentWindow, updatedState, lastPage), 100);
    });
  }

Here’s a comparison of how the user experience changed.

Streaming deliveries Normal loading

Notice that loaded deliveries are instantly plotted and user can start interacting with them. While if we load all the deliveries before plotting them the user has to wait for all of them to be loaded. This made the user experience much better if the user loaded more than 100 deliveries.

Serializing route points list without brackets

One more optimization we did was to change how route points are being serialized.

The route points after serialization contained opening and closing square brackets. So let’s say the route points are

[[25.57167, -80.421271], [25.676544, -80.388611], [25.820025, -80.386488],...].

After serialization they looked like

[[25.57167,-80.421271], [25.676544,-80.388611], [25.820025,-80.386488],...].

For each route point we’ve an extra opening and closing square bracket which can be avoided.

We could get rid of the brackets by concatenating the route points array and converting it to a string. After conversion it looked like this.

"25.57167,-80.421271|25.676544,-80.388611|25.820025,-80.386488|..."

At the client side we converted it back to an array. This reduced the payload size by 0.2MB for dense routes.

Note this is a trade off between client side processing and network bandwidth. In modern computers the client side processing will be negligible. For our clients, network bandwidth was a crucial resource so we optimized it for network bandwidth.

Deploying feature branches to have a review app

BigBinary has been working with Gumroad for a while. Following blog post has been posted with permission from Gumroad and we are very grateful to Sahil for allowing us to discuss the work in such an open environment.

Staging environment helps us in testing the code before pushing the code to production. However it becomes hard to manage the staging environment when more people work on different parts of the application. This can be solved by implementing a system where feature branch can have its own individual staging environment.

Heroku has Review Apps feature which can deploy different branches separately. Gumroad, doesn’t use Heroku so we built a custom in-house solution.

The first step was to build the infrastructure. We created a new Auto Scaling Group, Application Load Balancer and route in AWS for the review apps. Load balancer and route are common for all review apps, but a new EC2 instance is created in the ASG when a new review app is commissioned.

review app architecture

The main challange was to forward the incoming requests to the correct server running the review app. This was made possible using Lua in nginx and consul. When a review app is deployed, it writes its IP and port to consul along with the hostname. Each review app server runs an instance of OpenResty (Nginx + Lua modules) with the following configuration.

server {
  listen                   80;
  server_name              _;
  server_name_in_redirect  off;
  port_in_redirect         off;

  try_files $uri/index.html $uri $uri.html @app;

  location @app {
    set $upstream "";
    rewrite_by_lua '
      http   = require "socket.http"
      json   = require "json"
      base64 = require "base64"

      -- read upstream from consul
      host          = ngx.var.http_host
      body, c, l, h = http.request("http://172.17.0.1:8500/v1/kv/" .. host)
      data          = json.decode(body)
      upstream      = base64.decode(data[1].Value)

      ngx.var.upstream = upstream
    ';

    proxy_buffering   off;
    proxy_set_header  Host $host;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_redirect    off;
    proxy_pass        http://$upstream;
  }
}

It forwards all incoming requests to the correct IP:PORT after looking up in consul with the hostname.

The next task was to build a system to deploy the review apps to this infrastructure. We were already using docker in both production and staging environments. We decided to extend it to deploy branches by building docker image for every branch with deploy- prefix in the branch name. When such a branch is pushed to GitHub, a CircleCI job is run to build a docker image with the code and all the necessary packages. This can be configured using a configuration template like this.

jobs:
  build_image:
    <<: *defaults
    parallelism: 2
    steps:
      - checkout
      - setup_remote_docker:
          version: 17.09.0-ce
      - run:
          command: |
            ci_scripts/2.0/build_docker_image.sh
          no_output_timeout: 20m

workflows:
  version: 2

  web_app:
    jobs:
      - build_image:
          filters:
            branches:
              only:
                - /deploy-.*/

It also pushes static assets like JavaScript, CSS and images to an S3 bucket from where they are served directly through CDN. After building the docker image, another CircleCI job is run to do the following tasks.

  • Create a new database in RDS and configure the required credentials.
  • Scale up Review App’s Auto Scaling Group by increasing the number of desired instances by 1.
  • Run redis, database migration, seed-data population, unicorn and resque instances using nomad.

The ease of deploying a review app helped increase our productivity.

Skip devise trackable module for API calls to avoid users table getting locked

We use devise gem for authentication in one of our applications. This application provides an API which uses token authentication provided by the devise gem.

We were authenticating the user using auth token for every API call.

class Api::V1::BaseController < ApplicationController
  before_action :authenticate_user_using_x_auth_token
  before_action :authenticate_user!

  def authenticate_user_using_x_auth_token
    user_email = params[:email].presence || request.headers['X-Auth-Email']
    auth_token = request.headers['X-Auth-Token'].presence

    @user = user_email && User.find_by(email: user_email)

    if @user && Devise.secure_compare(@user.authentication_token, auth_token)
      sign_in @user, store: false
    else
      render_errors('Could not authenticate with the provided credentials', 401)
    end
  end
end

Everything was working smoothly initially, but we started noticing significant reduction in the response times during peak hours after a few months.

Because of the nature of the business, the application gets API calls for every user after every minute. Sometimes the application also get concurrent API calls for the same user. We noticed that in such cases, the users table was getting locked during the authentication process. This was resulting into cascading holdups and timeouts as it was affecting other API calls which were also accessing the users table.

After looking at the monitoring information, we found that the problem was happening due to the trackable module of devise gem. The trackable module keeps track of the user by storing the sign in time, sign in count and IP address information. Following queries were running for every API call and were resulting into exclusive locks on the users table.

UPDATE users SET last_sign_in_at = '2018-01-09 04:55:04',
current_sign_in_at = '2018-01-09 04:55:05',
sign_in_count = 323,
updated_at = '2018-01-09 04:55:05'
WHERE users.id = $1

To fix this issue, we decided to skip the user tracking for the API calls. We don’t need to track the user as every call is stateless and every request authenticates the user.

Devise provides a hook to achieve this for certain requests through the environment of the request. As we were already using a separate base controller for API requests, it was easy to skip it for all API calls at once.

class Api::V1::BaseController < ApplicationController
  before_action :skip_trackable
  before_action :authenticate_user_using_x_auth_token
  before_action :authenticate_user!

  def skip_trackable
    request.env['warden'].request.env['devise.skip_trackable'] = '1'
  end
end

This fixed the issue of exclusive locks on the users table caused by the trackable module.

Ruby 2.6 Range#cover? now accepts Range object as an argument

This blog is part of our Ruby 2.6 series. Ruby 2.6.0-preview2 was recently released.

Range#cover? returns true if the object passed as argument is in the range.

(1..10).cover?(5)
=> true

Range#cover? returns false if the object passed as an argument is non-comparable or is not in the range.

Before Ruby 2.6, Range#cover? used to return false if a Range object is passed as an argument.

>> (1..10).cover?(2..5)
=> false

Ruby 2.6

In Ruby 2.6 Range#cover? can accept a Range object as an argument. It returns true if the argument Range is equal to or a subset of the Range.

(1..100).cover?(10..20)
=> true

(1..10).cover?(2..5)
=> true

(5..).cover?(4..)
=> false

("a".."d").cover?("x".."z")
=> false

Here is relevant commit and discussion for this change.

Rails 5.2 adds DSL for configuring Content Security Policy header

This blog is part of our Rails 5.2 series.

Content Security Policy (CSP) is an added layer of security that helps to detect and mitigate various types of attacks on our web applications, including Cross Site Scripting (XSS) and data injection attacks.

What is XSS ?

In this attack, victim’s browser may execute malicious scripts because browser trusts the source of the content even when it’s not coming from the correct source.

Here is our blog on XSS written sometime back.

How CSP can be used to mitigate and report this attack ?

By using CSP, we can specify domains that are valid sources of executable scripts. Then a browser with CSP compatibility will only execute those scripts that are loaded from these whitelisted domains.

Please note that CSP makes XSS attack a lot harder but CSP does not make XSS attack impossible. CSP does not stop DOM-based XSS (also known as client-side XSS). To prevent DOM-based XSS, Javascript code should be carefully written to avoid introducing such vulnerabilities.

In Rails 5.2, a DSL was added for configuring Content Security Policy header.

Let’s check the configuration.

We can define global policy for the project in an initializer.

# config/initializers/content_security_policy.rb

Rails.application.config.content_security_policy do |policy|
  policy.default_src :self, :https
  policy.font_src    :self, :https, :data
  policy.img_src     :self, :https, :data
  policy.object_src  :none
  policy.script_src  :self, :https
  policy.style_src   :self, :https, :unsafe_inline
  policy.report_uri  "/csp-violation-report-endpoint"
end

We can override global policy within a controller as well.

# Override policy inline

class PostsController < ApplicationController
  content_security_policy do |policy|
    policy.upgrade_insecure_requests true
  end
end
# Using mixed static and dynamic values

class PostsController < ApplicationController
  content_security_policy do |policy|
    policy.base_uri :self, -> { "https://#{current_user.domain}.example.com" }
  end
end

Content Security Policy can be deployed in report-only mode as well.

Here is global setting in an initializer.

# config/initializers/content_security_policy.rb

Rails.application.config.content_security_policy_report_only = true

Here we are putting an override at controller level.

class PostsController < ApplicationController
  content_security_policy_report_only only: :index
end

Policy specified in content_security_policy_report_only header will not be enforced, but any violations will be reported to a provided URI. We can provide this violation report URI in report_uri option.

# config/initializers/content_security_policy.rb

Rails.application.config.content_security_policy do |policy|
  policy.report_uri  "/csp-violation-report-endpoint"
end

If both content_security_policy_report_only and content_security_policy headers are present in the same response then policy specified in content_security_policy header will be enforced while content_security_policy_report_only policy will generate reports but will not be enforced.

Rails 5.2 disallows raw SQL in dangerous Active Record methods preventing SQL injections

This blog is part of our Rails 5.2 series.

We sometimes use raw SQL in Active Record methods. This can lead to SQL injection vulnerabilities when we unknowingly pass unsanitized user input to the Active Record method.

class UsersController < ApplicationController
  def index
    User.order("#{params[:order]} ASC")
  end
end

Although this code is looking fine on the surface, we can see the issues looking at the example from rails-sqli.

pry(main)> params[:order] = "(CASE SUBSTR(authentication_token, 1, 1) WHEN 'k' THEN 0 else 1 END)"

pry(main)> User.order("#{params[:order]} ASC")
User Load (1.0ms)  SELECT "users".* FROM "users" ORDER BY (CASE SUBSTR(authentication_token, 1, 1) WHEN 'k' THEN 0 else 1 END) ASC
=> [#<User:0x00007fdb7968b508
  id: 1,
  email: "piyush@example.com",
  authentication_token: "Vkn5jpV_zxhqkNesyKSG">]

There are many Active Record methods which are vulnerable to SQL injection and some of these can be found here.

However, in Rails 5.2 these APIs are changed and they allow only attribute arguments and Rails does not allow raw SQL. With Rails 5.2 it is not mandatory but the developer would see a deprecation warning to remind about this.

irb(main):004:0> params[:order] = "email"
=> "email"
irb(main):005:0> User.order(params[:order])
  User Load (1.0ms)  SELECT  "users".* FROM "users" ORDER BY email LIMIT $1  [["LIMIT", 11]]
=> #<ActiveRecord::Relation [#<User id: 1, email: "piyush@example.com", authentication_token: "Vkn5jpV_zxhqkNesyKSG">]>

irb(main):008:0> params[:order] = "(CASE SUBSTR(authentication_token, 1, 1) WHEN 'k' THEN 0 else 1 END)"
irb(main):008:0> User.order("#{params[:order]} ASC")
DEPRECATION WARNING: Dangerous query method (method whose arguments are used as raw SQL) called with non-attribute argument(s): "(CASE SUBSTR(authentication_token, 1, 1) WHEN 'k' THEN 0 else 1 END)". Non-attribute arguments will be disallowed in Rails 6.0. This method should not be called with user-provided values, such as request parameters or model attributes. Known-safe values can be passed by wrapping them in Arel.sql(). (called from irb_binding at (irb):8)
  User Load (1.2ms)  SELECT  "users".* FROM "users" ORDER BY (CASE SUBSTR(authentication_token, 1, 1) WHEN 'k' THEN 0 else 1 END) ASC
=> #<ActiveRecord::Relation [#<User id: 1, email: "piyush@example.com", authentication_token: "Vkn5jpV_zxhqkNesyKSG">]>

In Rails 6, this will result into an error.

In Rails 5.2, if we want to run raw SQL without getting the above warning, we have to change raw SQL string literals to an Arel::Nodes::SqlLiteral object.

irb(main):003:0> Arel.sql('title')
=> "title"
irb(main):004:0> Arel.sql('title').class
=> Arel::Nodes::SqlLiteral

irb(main):006:0> User.order(Arel.sql("#{params[:order]} ASC"))
  User Load (1.2ms)  SELECT  "users".* FROM "users" ORDER BY (CASE SUBSTR(authentication_token, 1, 1) WHEN 'k' THEN 0 else 1 END) ASC
=> #<ActiveRecord::Relation [#<User id: 1, email: "piyush@example.com", authentication_token: "Vkn5jpV_zxhqkNesyKSG">]>

This should be done with care and should not be done with user input.

Here is relevant commit and discussion.

Ruby 2.6 adds RubyVM::AST module

This blog is part of our Ruby 2.6 series. Ruby 2.6.0-preview2 was recently released.

Ruby 2.6 added RubyVM::AST to generate Abstract Syntax Tree of code. Please note that this feature is experimental and under active development.

As of now RubyVM::AST supports two methods named as parse and parse_file.

parse method takes string as parameter and returns root node of the tree in the form of an object of RubyVM::AST::Node.

parse_file method takes file name as parameter and returns root node of the tree in the form of an object of RubyVM::AST::Node.

Ruby 2.6.0-preview2

irb> RubyVM::AST.parse("(1..100).select { |num| num % 5 == 0 }")
=> #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >

irb> RubyVM::AST.parse_file("/Users/amit/app.rb")
=> #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >

RubyVM::AST::Node has seven public instance methods - children, first_column, first_lineno, inspect, last_column, last_lineno and type.

Ruby 2.6.0-preview2

irb> ast_node = RubyVM::AST.parse("(1..100).select { |num| num % 5 == 0 }")
=> #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >

irb> ast_node.children
=> [nil, #<RubyVM::AST::Node(NODE_ITER(9) 1:0, 1:38): >]

irb> ast_node.first_column
=> 0

irb> ast_node.first_lineno
=> 1

irb> ast_node.inspect
=> "#<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >"

irb> ast_node.last_column
=> 38

irb> ast_node.last_lineno
=> 1

irb> ast_node.type
=> "NODE_SCOPE"

This module will majorly help in building static code analyzer and formatter.

Inline Installation of Firefox Extension

Inline Installation

Firefox extensions, similar to Chrome extensions, help us modify and personalize our browsing experience by adding new features to the existing sites.

Once we’ve published our extension to the Mozilla’s Add-on store(AMO), users who browse the AMO can find the extension and install it with one-click. But, if a user is already on our site where a link is provided to the extension’s AMO listing page, they would need to navigate away from our website to the AMO, complete the install process, and then return back to our site. That is a bad user experience.

The inline installation enables us to initiate the extension installation from our site. The extension can still be hosted on the AMO but users would no longer have to leave our site to install it.

We had to try out a few suggested approaches before we got it working.

InstallTrigger

InstallTrigger is an interface included in the Mozilla’s Apps API for installing extensions. Using JavaScript, the install method of InstallTrigger can be used to start the download and installation of an extension (or anything packaged in a .xpi file) from a Web page.

A XPI(pronounced as “zippy”) is similar to a zip file, which contains manifest file and the install script for the extension.

So, let’s try to install the Grammarly Extension for Firefox. To use it, we first need its .xpi file’s location. Once we have published our extension on the AMO, we can navigate to the listings page for it and get the link for the .xpi.

For our present example, here’s the listing page for Grammarly Extension.

Here, we can get the .xpi file’s location by right clicking on the + Add to Firefox button and clicking on Copy Link Location. Note that the + Add to Firefox button would only be visible if we browse the link on a Firefox browser. Otherwise, it would be replaced by a Get Firefox Now button.

Once we have the URL, we can trigger the installation via JavaScript on our web page.

InstallTrigger.install({
  'Name of the Extension': {
    URL: "url pointing to the .xpi file's location on AMO",
  },
});

Pointing to the latest version of the Extension

When we used the URL in the above code, the .xpi file’s URL was specific to the extension’s current version. If the extension has an update, the installed extensions for existing users would be updated automatically. But the URL to the .xpi on our website would be pointing to the older version. Although the old link would still work, we would always want new users to download the latest version.

To do that, we can either fetch the listing page in the background and parse the HTML to get the latest link. But that approach can break if the HTML changes.

Or we can query the Addons Services API, which returns the information for the extension in XML format.

For the Grammarly Extension, we first need its slug-id. We can get it by looking at its listing page’s URL. From https://addons.mozilla.org/en-US/firefox/addon/grammarly-1/, we can note down the slug which is grammarly-1

Using this slug id, we can now get the extension details using https://services.addons.mozilla.org/en-US/firefox/api/1.5/addon/grammarly-1. It returns the info for the Grammarly Extension. What we are particularly interested in is the value in the <install> node. That is what the desired value is for the latest version for our .xpi.

Let’s see how we can implement the whole thing using React.

import axios from 'axios';
import cheerio from 'cheerio';

const FALLBACK_GRAMMARLY_EXTENSION_URL =
  'https://addons.mozilla.org/firefox/downloads/file/1027073/grammarly_for_firefox-8.828.1757-an+fx.xpi';
const URL_FOR_FETCHING_XPI = `https://services.addons.mozilla.org/en-US/firefox/api/1.5/addon/grammarly-1`;

export default class InstallExtension extends Component {
  state = {
    grammarlyExtensionUrl: FALLBACK_GRAMMARLY_EXTENSION_URL,
  };

  componentWillMount() {
    axios.get(URL_FOR_FETCHING_XPI).then(response => {
      const xml = response.data;
      const $ = cheerio.load(xml);
      const grammarlyExtensionUrl = $('addon install').text();
      this.setState({ grammarlyExtensionUrl });
    });
  }

  triggerInlineInstallation = event => {
    InstallTrigger.install({
      Grammarly: { URL: this.state.grammarlyExtensionUrl },
    });
  };

  render() {
    return (
      <Button onClick={this.triggerInlineInstallation}>
        Install Grammarly Extension
      </Button>
    );
  }
}

In the above code, we are using the npm packages axios for fetching the xml and cheerio for parsing the xml. Also, we have set a fallback URL as the initial value in case the fetching of the new URL from the xml response fails.

Using parametrized containers for deploying Rails micro services on Kubernetes

When using micro services with containers, one has to consider modularity and reusability while designing a system.

While using Kubernetes as a distributed system for container deployments, modularity and reusability can be achieved using parameterizing containers to deploy micro services.

Parameterized containers

Assuming container as a function in a program, how many parameters does it have? Each parameter represents an input that can customize a generic container to a specific situation.

Let’s assume we have a Rails application isolated in services like puma, sidekiq/delayed-job and websocket. Each service runs as a separate deployment on a separate container for the same application. When deploying the change we should be building the same image for all the three containers but they should be different function/processes. In our case, we will be running 3 pods with the same image. This can be achieved by building a generic image for containers. The Generic container must be accepting parameters to run different services.

We need to expose parameters and consume them inside the container. There are two ways to pass parameters to our container.

  1. Using environment variables.
  2. Using command line arguments.

In this article, we will use environment variables to run parameterized containers like puma, sidekiq/delayed-job and websocket for Rails applications on kubernetes.

We will deploy wheel on kubernetes using parametrized container approach.

Pre-requisite

Building a generic container image.

Dockerfile in wheel uses bash script setup_while_container_init.sh as a command to start a container. The script is self-explanatory and, as we can see, it consists of two functions web and background. Function web starts the puma service and background starts the delayed_job service.

We create two different deployments on kubernetes for web and background services. Deployment templates are identical for both web and background. The value of environment variable POD_TYPE to init-script runs the particular service in a pod.

Once we have docker image built, let’s deploy the application.

Creating kubernetes deployment manifests for wheel application

Wheel uses PostgreSQL database and we need postgres service to run the application. We will use the postgres image from docker hub and will deploy it as deployment.

Note: For production deployments, database should be deployed as a statefulset or use managed database services.

K8s manifest for deploying PostgreSQL.

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: db
  name: db
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
      - image: postgres:9.4
        name: db
        env:
        - name: POSTGRES_USER
          value: postgres
        - name: POSTGRES_PASSWORD
          value: welcome

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: db
  name: db
spec:
  ports:
  - name: headless
    port: 5432
    targetPort: 5432
  selector:
    app: db

Create Postgres DB and the service.

$ kubectl create -f db-deployment.yml -f db-service.yml
deployment db created
service db created

Now that DB is available, we need to access it from the application using database.yml.

We will create configmap to store database credentials and mount it on the config/database.yml in our application deployments.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: database-config
data:
  database.yml: |
    development:
      adapter: postgresql
      database: wheel_development
      host: db
      username: postgres
      password: welcome
      pool: 5

    test:
      adapter: postgresql
      database: wheel_test
      host: db
      username: postgres
      password: welcome
      pool: 5

    staging:
      adapter: postgresql
      database: postgres
      host: db
      username: postgres
      password: welcome
      pool: 5

Create configmap for database.yml.

$ kubectl create -f database-configmap.yml
configmap database-config created

We have the database ready for our application, now let’s proceed to deploy our Rails services.

Deploying Rails micro-services using the same docker image

In this blog, we will limit our services to web and background with kubernetes deployment.

Let’s create a deployment and service for our web application.

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: wheel-web
  labels:
    app: wheel-web
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: wheel-web
    spec:
      containers:
      - image: bigbinary/wheel:generic
        name: web
        imagePullPolicy: Always
        env:
        - name: DEPLOY_TIME
          value: $date
          value: staging
        - name: POD_TYPE
          value: WEB
        ports:
        - containerPort: 80
        volumeMounts:
          - name: database-config
            mountPath: /wheel/config/database.yml
            subPath: database.yml
      volumes:
        - name: database-config
          configMap:
            name: database-config

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: wheel-web
  name: web
spec:
  ports:
  - name: puma
    port: 80
    targetPort: 80
  selector:
    app: wheel-web
  type: LoadBalancer

Note that we used POD_TYPE as WEB, which will start the puma process from the container startup script.

Let’s create a web/puma deployment and service.

kubectl create -f web-deployment.yml -f web-service.yml
deployment wheel-web created
service web created
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: wheel-background
  labels:
    app: wheel-background
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: wheel-background
    spec:
      containers:
      - image: bigbinary/wheel:generic
        name: background
        imagePullPolicy: Always
        env:
        - name: DEPLOY_TIME
          value: $date
        - name: POD_TYPE
          value: background
        ports:
        - containerPort: 80
        volumeMounts:
          - name: database-config
            mountPath: /wheel/config/database.yml
            subPath: database.yml
      volumes:
        - name: database-config
          configMap:
            name: database-config

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: wheel-background
  name: background
spec:
  ports:
  - name: background
    port: 80
    targetPort: 80
  selector:
    app: wheel-background

For background/delayed-job we set POD_TYPE as background. It will start delayed-job process.

Let’s create background deployment and the service.

kubectl create -f background-deployment.yml -f background-service.yml
deployment wheel-background created
service background created

Get application endpoint.

$ kubectl get svc web -o wide | awk '{print $4}'
a55714dd1a22d11e88d4b0a87a399dcf-2144329260.us-east-1.elb.amazonaws.com

We can access the application using the endpoint.

Now let’s see pods.

$ kubectl get pods
NAME                                READY     STATUS    RESTARTS   AGE
db-5f7d5c96f7-x9fll                 1/1       Running   0          1h
wheel-background-6c7cbb4c75-sd9sd   1/1       Running   0          30m
wheel-web-f5cbf47bd-7hzp8           1/1       Running   0          10m

We see that db pod is running postgres, wheel-web pod is running puma and wheel-background pod is running delayed job.

If we check logs, everything coming to puma is handled by web pod. All the background jobs are handled by background pod.

Similarly, if we are using websocket, separate API pods, traffic will be routed to respective services.

This is how we can deploy Rails micro services using parametrized containers and a generic image.

Configuring memory allocation in ImageMagick

ImageMagick has a security policy file policy.xml using which we can control and limit the execution of the program. For example, the default memory limit of ImageMagick-6 is 256 MiB.

Recently, we saw following error while processing a gif image.

convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `file.gif' @ error/cache.c/OpenPixelCache/3945.

This happens when ImageMagick cannot allocate enough memory to process the image. This can be fixed by tweaking memory configuration in policy.xml.

Path of policy.xml can be located as follows.

$ identify -list policy

Path: /etc/ImageMagick-6/policy.xml
  Policy: Resource
    name: disk
    value: 1GiB

Memory limit can be configured in the following line of policy.xml.

<policy domain="resource" name="memory" value="256MiB"/>

Increasing this value would solve the error if you have a machine with larger a memory.

Uploading files directly to S3 using Pre-signed POST request

It’s easy to create a form in Rails which can upload a file to the backend. The backend, can then take the file and upload it to S3. We can do that by using gems like paperclip or carrierwave. Or if we are using Rails 5.2, we can use Active Storage

But for applications, where Rails is used only as an API backend, uploading via a form is not an option. In this case, we can expose an endpoint which accepts files, and then Rails can handle uploading to S3.

In most of the cases, the above solution works. But recently, in one of our applications which is hosted at Heroku we faced time-out related problems while uploading large files. Here is what heroku’s docs says about how long a request can take.

The router terminates the request if it takes longer than 30 seconds to complete.

Pre-signed POST request

An obvious solution is to upload the files directly to S3. However inorder to do that, the client needs AWS credentials, which is not ideal. If the client is a Single Page Application, the AWS credentials would be visible in the javascript files. Or if the client is a mobile app, someone might be able to reverse engineer the application, and get hold of the AWS credentials.

Here’s where Pre-signed POST request comes to the rescue. Here is official docs from AWS on this topic.

Uploading via Pre-signed POST is a two step process. The client first requests a permission to upload the file. The backend receives the request, generates the pre-signed URL and returns the response along with other fields. The client can then upload the file to the URL received in the response.

Implementation

Add the AWS gem to you Gemfile and run bundle install.

gem 'aws-sdk'

Create a S3 bucket with the AWS credentials.

aws_credentials = Aws::Credentials.new(
  ENV['AWS_ACCESS_KEY_ID'],
  ENV['AWS_SECRET_ACCESS_KEY']
)

s3_bucket = Aws::S3::Resource.new(
  region: 'us-east-1',
  credentials: aws_credentials
).bucket(ENV['S3_BUCKET'])

The controller handling the request for getting the presigned URL should have following code.

def request_for_presigned_url
  presigned_url = s3_bucket.presigned_post(
    key: "#{Rails.env}/#{SecureRandom.uuid}/${filename}",
    success_action_status: '201',
    signature_expiration: (Time.now.utc + 15.minutes)
  )

  data = { url: presigned_url.url, url_fields: presigned_url.fields }

  render json: data, status: :ok
end

In the above code, we are creating a presigned url using the presigned_post method.

The key option specifies path where the file would be stored. AWS supports a custom ${filename} directive for the key option. This ${filename} directive tells S3 that if a user uploads a file named image.jpg, then S3 should store the file with the same name. In S3, we cannot have duplicate keys, so we are using SecureRandom to generate unique key so that 2 files with same name can be stored.

If a file is successfully uploaded, then client receives HTTP status code under key success_action_status. If the client sets its value to 200 or 204 in the request, Amazon S3 returns an empty document along with 200 or 204 as HTTP status code. We set it to 201 here because we want the client to notify us with the S3 key where the file was uploaded to. The S3 key is present in the XML document which is received as a response from AWS only when the status code is 201.

signature_expiration specifies when the signature on the POST will expire. It defaults to one hour from the creation of the presigned POST. This value should not exceed one week from the creation time. Here, we are setting it to 15 minutes.

Other configuration options can be found here.

In response to the above request, we send out a JSON which contains the URL and the fields required for making the upload.

Here’s a sample response.

{
  "url": "https://s3.amazonaws.com/<some-s3-url>",
  "url_fields": {
    "key": "development/8614bd40-691b-4668-9241-3b342c6cf429/${filename}",
    "success_action_status": "201",
    "policy": "<s3-policy>",
    "x-amz-credential": "********************/20180721/us-east-1/s3/aws4_request",
    "x-amz-algorithm": "AWS4-HMAC-SHA256",
    "x-amz-date": "201807021T144741Z",
    "x-amz-signature": "<hexadecimal-signature>"
  }
}

Once the client gets the above credentials, it can proceed with the actual file upload.

The client can be anything. An iOS app, android app, an SPA or even a Rails app. For our example, let’s assume it’s a node client.

var request = require("request");
function uploadFileToS3(response) {
  var options = {
    method: 'POST',
    url: response.url,
    formData: {
      ...response.url_fields,
      file: <file-object-for-upload>
    }
  }

  request(options, (error, response, body) => {
    if (error) throw new Error(error);
    console.log(body);
  });
}

Here, we are making a POST request to the URL received from the earlier presigned response. Note that we are using the spread operator to pass url_fields in formData.

When the POST request is successful, the client then receives an XMLresponse from S3 because we set the response code to be 201. A sample response can be like the following.

<?xml version="1.0" encoding="UTF-8"?>
<PostResponse>
    <Location>https://s3.amazonaws.com/link-to-the-file</Location>
    <Bucket>s3-bucket</Bucket>
    <Key>development/8614bd40-691b-4668-9241-3b342c6cf429/image.jpg</Key>
    <ETag>"32-bit-tag"</ETag>
</PostResponse>

Using the above response, the client can then let the API know about where the file was uploaded by sending the value from the Key node. Although, this can be optional in some cases, depending on the API, if it actually needs this info.

Advantages

Using AWS S3 presigned-urls has a few advantages.

  • The main advantage of uploading directly to S3 is that there would be considerably less load on your application server since the server is now free from handling the receiving of files and transferring to S3.

  • Since the file upload happens directly on S3, we can bypass the 30 seconds Heroku time limit.

  • AWS credentials are not shared with the client application. So no one would be able to get their hands on your AWS keys.

  • The generated presigned-url can be initialized with an expiration time. So the URLs and the signatures generated would be invalid after that time period.

  • The client does not need to install any of the AWS libraries. It just needs to upload the file via a simple POST request to the generated URL.

How we reduced infrastructure cost by 10% for an e-commerce project

Recently, we got an opportunity to reduce the infrastructure cost of a medium-sized e-commerce project. In this blog we discuss how we reduced the total infrastructure cost by 10%.

Changes to MongoDB instances

Depending on the requirements, modern web applications use different third-party services. For example, it’s easy and cost effective to subscribe to a GeoIP lookup service than building and maintaining one. Some third-party services get very expensive as the usage increases but people don’t look for alternatives due to legacy reasons.

In our case, our client had been paying more than $5,000/month for a third-party MongoDB service. This service charges based on the storage used and we had years of data in it. This data is consumed by a machine learning system to fight fraudulent purchases and users. We had a look at both the ML system and the data in MongoDB and found we actually didn’t need all the data in the database. The system never read data older than 30-60 days in some of the biggest mongo collections.

Since we were already using nomad as our scheduler, we wrote a periodic nomad job that runs every week to delete unnecessary data. The nomad job syncs both primary and secondary MongoDB instances to release the free space back to OS. This helped reduce monthly bill to $630/month.

Changes to MongoDB service provider

Then we looked at the MongoDB service provider. It was configured years back when the application was built. There are other vendors who provided the same service for a much cheaper price. We switched our MongoDB to mLab and now the database runs in a $180/month dedicated cluster. With WiredTiger’s compression enabled, we don’t use as much storage we used to use before.

Making use of Auto Scaling

Auto Scaling can be a powerful tool when it comes to reducing costs. We had been running around 15 large EC2 instances. This was inefficient due to following two reasons.

  1. It cannot cope up when the traffic increases beyond its limit.
  2. Resources are underused when traffic is less.

Auto Scaling solves both the issues. For web servers, we switched to smaller instances and used Target Tracking Scaling Policy to keep the average aggregate CPU utilization at 70%.

Background job workers made use of a nomad job we built. It periodically calculated the number of required instances based on the count of pending jobs and the job’s queue priority. This number was pushed to CloudWatch as a metric and the Auto Scaling group scaled based on that. This approach was effective in boosting performance and reducing cost.

Buying reserved instances

AWS has a feature to reserve instances for services like EC2, RDS, etc.. It’s often preferable to buy reserved instances than running the application using on-demand instances. We evaluated reserved instance utilization using the reporting tool and bought the required reserved instances.

Looking for cost-effective solutions

Sometimes, different solutions to the same problem can have different costs. For example, we had been facing small DDoS attack regularly and we had to rate-limit requests based on IP and other parameters. Since we had been using Cloudflare, we could have used their rate-limiting feature. Performance wise, it was the best solution but they charge based on the number of good requests. It would be expensive for us since it’s a high-traffic application. We looked for other solutions and solved the problem using Rack::Attack. We wrote a blog about it sometime back. The solution presented in the blog was effective in mitigating the DDoS attack we faced and didn’t cost us anything significant.

Requesting custom pricing

If you are a comparatively larger customer of a third-party service, it’s more likely that you don’t have to pay the published price. Instead, we could request for custom pricing. Many companies will be happy to give 20% to 50% price discounts if we can commit to a minimum spending in the year. We tried negotiating a new contract for an expensive third-party service and got the deal with 40% discount compared to their published minimum price.

Running an infrastructure can be both technically and economically challenging. But if we can look between the lines and if we are willing to update existing systems, we would be amazed in terms of how much money we can save every month.

Ruby 2.6 adds Enumerable#filter as an alias of Enumerable#select

This blog is part of our Ruby 2.6 series. Ruby 2.6.0-preview2 was recently released.

Ruby 2.6 has added Enumerable#filter as an alias of Enumerable#select. The reason for adding Enumerable#filter as an alias is to make it easier for people coming from other languages to use Ruby. A lot of other languages including Java, R, PHP etc. have filter method to filter/select records based on a condition.

Let’s take an example in which we have to select/filter all numbers which are divisible by 5 from a range.

Ruby 2.5

irb> (1..100).select { |num| num % 5 == 0 }
=> [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]

irb> (1..100).filter { |num| num % 5 == 0 }
=> Traceback (most recent call last):
2: from /Users/amit/.rvm/rubies/ruby-2.5.1/bin/irb:11:in `<main>' 1: from (irb):2 NoMethodError (undefined method`filter' for 1..100:Range)

Ruby 2.6.0-preview2

irb> (1..100).select { |num| num % 5 == 0 }
=> [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]

irb> (1..100).filter { |num| num % 5 == 0 }
=> [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]

Also note that along with Enumerable#filter, Enumerable#filter! is also added as an alias for Enumerable#select!.

Here is relevant commit and discussion.

Ruby 2.6 adds support for non-ASCII capital letter as a first character in constant name

This blog is part of our Ruby 2.6 series. Ruby 2.6.0-preview2 was recently released.

Before Ruby 2.6, constant must have a capital ASCII letter as the first character. It means class and module name cannot start with non-ASCII capital character.

Below code will raise class/module name must be CONSTANT exception.

  class Большойдвоичный
  end

We can use above non-ASCII character as a method name or variable name though.

Below code will run without any exception

  class NonAsciiMethodAndVariable
    def Большойдвоичный
      Имя = "BigBinary"
    end
  end

“Имя” is treated as a variable name in above example, even though first letter(И) is a capital non-ASCII character.

Ruby 2.6

Ruby 2.6 relaxes above mentioned limitation. We can now define constants in laguages other than English. Languages having capital letters like Russian and Greek can be used to define constant name.

Below code will run without exception in any Ruby 2.6.

  class Большойдвоичный
  end

As capital non-Ascii characters are now treated as constant, below code will raise a warning in Ruby 2.6.

  irb(main):001:0> Имя = "BigBinary"
  => "BigBinary"
  irb(main):002:0> Имя = "BigBinary"
  (irb):2: warning: already initialized constant Имя
  (irb):1: warning: previous definition of Имя was here

Above code will run without any warnings on Ruby versions prior to 2.6

Here is relevant commit and discussion for this change.

Setting up a high performance Geocoder

One of our applications uses geocoding extensively. When we started the project, we included the excellent Geocoder gem, and set Google as the geocoding backend. As the application scaled, its geocoding requirements grew and soon we were looking at geocoding bills worth thousands of dollars.

An alternative Geocoder

Our search for an alternative geocoder landed us on Nominatim. Written in C, with a PHP web interface, Nominatim was performant enough for our requirements. Once set up, Nominatim required 8GB of RAM to run and this included RAM for the PostgreSQL (+ PostGIS) as well.

The rest of the blog discusses how to setup Nominatim and the tips and tricks that we learned along the way and how it compares with the geocoding solution offered by Google.

Setting up Nominatim

We started off by looking for Amazon Machine Images with Nominatim setup and could only find one which was hosted by OpenStreetMap but the magnet link was dead.

Next, we went through the official installation document. We decided to give docker a shot and found that there are many Nominatim docker builds. We used https://github.com/merlinnot/nominatim-docker since it seemed to follow all the steps mentioned in the official installation guide.

Issues faced during Setup

Out of Memory Errors

The official documentation recommends using 32GB of RAM for initial import but we needed to double the memory to 64GB to make it work.

Also any time docker build failed, due to the large amount of data that is generated on each run, we also ran out of disk space on subsequent docker builds since docker caches layers across builds.

Merging Multiple Regions

We wanted to geocode locations from USA, Mexico, Canada and Sri Lanka. USA, Mexico and Canada are included by default in North America data extract but we had to merge data for Sri Lanka with North America to get it in a format required for initial import.

The following snippet pre-processes map data for North America and Sri Lanka into a single data.osm.pbf file that can be directly used by Nominatim installer.

RUN curl -L 'http://download.geofabrik.de/north-america-latest.osm.pbf' \
    --create-dirs -o /srv/nominatim/src/north-america-latest.osm.pbf
RUN curl -L 'http://download.geofabrik.de/asia/sri-lanka-latest.osm.pbf' \
    --create-dirs -o /srv/nominatim/src/sri-lanka-latest.osm.pbf

RUN osmconvert /srv/nominatim/src/north-america-latest.osm.pbf \
    -o=/srv/nominatim/src/north-america-latest.o5m
RUN osmconvert /srv/nominatim/src/sri-lanka-latest.osm.pbf \
    -o=/srv/nominatim/src/sri-lanka-latest.o5m

RUN osmconvert /srv/nominatim/src/north-america-latest.o5m \
    /srv/nominatim/src/sri-lanka-latest.o5m \
    -o=/srv/nominatim/src/data.o5m

RUN osmconvert /srv/nominatim/src/data.o5m \
    -o=/srv/nominatim/src/data.osm.pbf

Slow Search times

Once the installation was done, we tried running simple location searches like this one, but the search timed out. Usually Nominatim can provide a lot of information from its web-interface by appending &debug=true to the search query.

# from
https://nominatim.openstreetmap.org/search.php?q=New+York&polygon_geojson=1&viewbox=
# to
https://nominatim.openstreetmap.org/search.php?q=New+York&polygon_geojson=1&viewbox=&debug=true

We created an issue in Nominatim repository and got very prompt replies from Nominatim maintainers, especially from Sarah Hoffman .

# runs analyze on the entire nominatim database
psql -d nominatim -c 'ANALYZE VERBOSE'

PostgreSQL query planner depends on statistics collected by postgres statistics collector while executing a query. In our case, query planner took an enormous amount of time to plan queries as there were no stats collected since we had a fresh installation.

Comparing Nominatim and Google Geocoder

We compared 2500 addresses and we found that Google geocoded 99% of those addresses. In comparison Nominatim could only geocode 47% of the addresses.

It means we still need to geocode ~50% of addresses using Google geocoder. We found that we could increase geocoding efficiency by normalizing the addresses we had.

Address Normalization using libpostal

Libpostal is an address normalizer, which uses statistical natural-language processing to normalize addresses. Libpostal also has ruby bindings which made it quite easy to use it for our test purposes.

Once libpostal and its ruby-bindings were installed (installation is straightforward and steps are available in ruby-postal’s github page), we gave libpostal + Nominatim a go.

require 'geocoder'
require 'ruby_postal/expand'
require 'ruby_postal/parser'

Geocoder.configure({lookup: :nominatim, nominatim: { host: "nominatim_host:port"}})

full_address = [... address for normalization ...]
expanded_addresses = Postal::Expand.expand_address(full_address)
parsed_addresses = expanded_addresses.map do |address|
  Postal::Parser.parse_address(address)
end

parsed_addresses.each do | address |
  parsed_address = [:house_number, :road, :city, :state, :postcode, :country].inject([]) do |acc, key|
    # address is of format
    # [{label: 'postcode', value: 12345}, {label: 'city', value: 'NY'} .. ]
    key_value = address.detect { |address| address[:label] == key }
    if key_value
        acc << "#{key_value_pair[:value]}".titleize
    end
    acc
  end

  coordinates = Geocoder.coordinates(parsed_address.join(", "))
  if (coordinates.is_a? Array) && coordinates.present?
    puts "By Libpostal #{coordinates} => #{parsed_address.join(", ")}"
    break
  end
end

With this, we were able to improve our geocoding efficiency by 10% as Nominatim + Libpostal combination could geocode ~ 59% of addresses.

Debugging failing tests in puppeteer because of background tab

We have been using puppeteer in one of our projects to write end-to-end tests. We run our tests in headful mode to see the browser in action.

If we start puppeteer tests and do nothing in our laptop (just watch the tests being executed) then all the tests will pass.

However if we are doing our regular work in our laptop while tests are running then tests would fail randomly. This was quite puzzling.

Debugging such flaky tests is hard. We first suspected that the test cases themselves needed more of implicit waits for element/text to be present/visible on the DOM.

After some debugging using puppeteer protocol logs, it seemed like the browser was performing certain actions very slowly or was waiting for the browser to be active ( in view ) before performing those actions.

Chrome starting with version 57 introduced throtlling of background tabs for improving performance and battery life. We execute one test per browser meaning we didn’t make use of multiple tabs. Also tests failed only when the user was performing some other activities while the tests were executing in other background windows. Pages were hidden only when user switched tabs or minimized the browser window containing the tab.

After observing closely we noticed that the pages were making requests to the server. The issue was the page was not painting if the page is not in view. We added flag --disable-background-timer-throttling but we did not notice any difference.

After doing some searches we noticed the flag --disable-renderer-backgrounding was being used in karma-launcher. The comment states that it is specifically required on macOS. Here is the code responsible for lowering the priority of the renderer when it is hidden.

But the new flag didn’t help either.

While looking at all the available command line switches for chromium, we stumbled upon --disable-backgrounding-occluded-windows. Chromium also backgrounds the renderer while the window is not visible to the user. It seems from the comment that the flag kDisableBackgroundingOccludedWindowsForTesting is specifically added to avoid non-deterministic behavior during tests.

We have added following flags to chromium for running our integration suite and this solved our problem.

const chromeArgs = [
  '--disable-background-timer-throttling',
  '--disable-backgrounding-occluded-windows',
  '--disable-renderer-backgrounding'
];

References

Using Kubernetes ingress controller for authenticating applications

Kubernetes Ingress has redefined the routing in this era of containerization and with all these freehand routing techniques the thought of “My router my rules” seems real.

We use nginx-ingress as a routing service for our applications. There is a lot more than routing we can do with ingress. One of the important features is setting up authentication using ingress for our application. As all the traffic goes from ingress to our service, it makes sense to setup authentication on ingress.

As mentioned in ingress repository there are different types of techniques available for authentication including:

  • Basic authentication
  • Client-certs authentication
  • External authentication
  • Oauth external authentication

In this blog, we will set up authentication for the sample application using basic ingress authentication technique.

Pre-requisites

First, let’s create ingress resources from upstream example by running the following command.

$ kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
namespace "ingress-nginx" created
deployment "default-http-backend" created
service "default-http-backend" created
configmap "nginx-configuration" created
configmap "tcp-services" created
configmap "udp-services" created
serviceaccount "nginx-ingress-serviceaccount" created
clusterrole "nginx-ingress-clusterrole" created
role "nginx-ingress-role" created
rolebinding "nginx-ingress-role-nisa-binding" created
clusterrolebinding "nginx-ingress-clusterrole-nisa-binding" created
deployment "nginx-ingress-controller" created

Now that ingress controller resources are created we need a service to access the ingress.

Use following manifest to create service for ingress.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: http
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: ingress-nginx
  type: LoadBalancer

Now, get the ELB endpoint and bind it with some domain name.

$kubectl create -f ingress-service.yml
service ingress-nginx created

$ kubectl -n ingress-nginx get svc  ingress-nginx -o wide
NAME            CLUSTER-IP      EXTERNAL-IP                                                               PORT(S)                      AGE       SELECTOR
ingress-nginx   100.71.250.56   abcghccf8540698e8bff782799ca8h04-1234567890.us-east-2.elb.amazonaws.com   80:30032/TCP,443:30108/TCP   10s       app=ingress-nginx

Let’s create a deployment and service for our sample application kibana. We need elasticsearch to run kibana.

Here is manifest for the sample application.

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: kibana
  name: kibana
  namespace: ingress-nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
       - image: kibana:latest
         name: kibana
         ports:
           - containerPort: 5601
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: kibana
  name: kibana
  namespace: ingress-nginx

spec:
  ports:
  - name: kibana
    port: 5601
    targetPort: 5601
  selector:
    app: kibana
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: elasticsearch
  name: elasticsearch
  namespace: ingress-nginx
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
       - image: elasticsearch:latest
         name: elasticsearch
         ports:
           - containerPort: 5601
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: elasticsearch
  name: elasticsearch
  namespace: ingress-nginx
spec:
  ports:
  - name: elasticsearch
    port: 9200
    targetPort: 9200
  selector:
    app: elasticsearch

Create the sample application.

kubectl apply -f kibana.yml
deployment "kibana" created
service "kibana" created
deployment "elasticsearch" created
service "elasticsearch" created

Now that we have created application and ingress resources, it’s time to create an ingress and access the application.

Use the following manifest to create ingress.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
  name: kibana-ingress
  namespace: ingress-nginx
spec:
  rules:
    - host: logstest.myapp-staging.com
      http:
        paths:
          - path: /
            backend:
              serviceName: kibana
              servicePort: 5601
$kubectl -n ingress-nginx create -f ingress.yml
ingress "kibana-ingress" created.

Now that our application is up, when we access the kibana dashboard using URL http://logstest.myapp-staging.com We directly have access to our Kibana dashboard and anyone with this URL can access logs as shown in the following image.

Kibana dashboard without authentication

Now, let’s set up a basic authentication using htpasswd.

Follow below commands to generate the secret for credentials.

Let’s create an auth file with username and password.

$ htpasswd -c auth kibanaadmin
New password: <kibanaadmin>
New password:
Re-type new password:
Adding password for user kibanaadmin

Create k8s secret.

$ kubectl -n ingress-nginx create secret generic basic-auth --from-file=auth
secret "basic-auth" created

Verify the secret.

kubectl get secret basic-auth -o yaml
apiVersion: v1
data:
  auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK
kind: Secret
metadata:
  name: basic-auth
  namespace: ingress-nginx
type: Opaque

Use following annotations in our ingress manifest by updating the ingress manifest.

kubectl -n ingress-nginx edit ingress kibana ingress

Paste the following annotations

nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-realm: "Kibana Authentication Required - kibanaadmin"

Now that ingress is updated, hit the URL again and as shown in the image below we are asked for authentication.

Kibana dashboard without authentication