During the development of a chrome extension, debugging was difficult because line number of minified JavaScript file is of no use without a source map. Previously, Honeybadger could only download the source map files which were public and our source maps were inside the .crx package which was inaccessible to honeybadger.

Recently, Honeybadger released a new feature to upload the source maps to Honeybadger. We have written a grunt plugin to upload the source maps to Honeybadger.

Here is how we can upload source map to Honeybadger.

First, install the grunt plugin.

npm install --save-dev grunt-honeybadger-sourcemaps

Configure the gruntfile.

grunt.initConfig({
  honeybadger_sourcemaps: {
    default_options:{
      options: {
        appId: "xxxx",
        token: "xxxxxxxxxxxxxx",
        urlPrefix: "http://example.com/",
        revision: "<app version>"
        prepareUrlParam: function(fileSrc){
          // Here we can manipulate the filePath
          return filesrc.replace('built/', '');
        },
      },
      files: [{
        src: ['@path/to/**/*.map']
      }],
    }
  },
});
grunt.loadNpmTasks('grunt-honeybadger-sourcemaps');
grunt.registerTask('upload_sourcemaps', ['honeybadger_sourcemaps']);

We can get the appId and token from Honeybadger project settings.

grunt upload_sourcemaps

Now, we can upload the source maps to Honeybadger and get better error stack trace.

Testing

Clone the following repo.

git clone https://github.com/bigbinary/grunt-honeybadger-sourcemaps

Replace appId and token in Gruntfile.js and run grunt test. It should upload the sample source maps to your project.

This blog is part of our Ruby 2.6 series. Ruby 2.6.0 was released on Dec 25, 2018.

Ruby 2.5

If we use else without rescue inside begin..end block in Ruby 2.5, it gives a warning.

  irb(main):001:0> begin
  irb(main):002:1>    puts "Inside begin block"
  irb(main):003:1>  else
  irb(main):004:1>    puts "Inside else block"
  irb(main):005:1> end
  (irb):5: warning: else without rescue is useless

This warning is present as code inside else block will never get executed

Ruby 2.6

In Ruby 2.6 it will raise an exception if we use else without rescue in begin..end block. This commit changed warning into exception in Ruby 2.6. Changes made in the commit are experimental.

  irb(main):001:0>  begin
  irb(main):002:1>    puts "Inside begin block"
  irb(main):003:1>  else
  irb(main):004:1>    puts "Inside else block"
  irb(main):005:1>  end
  Traceback (most recent call last):
        1: from /usr/local/bin/irb:11:in `<main>'
  SyntaxError ((irb):3: else without rescue is useless)

The Chinese version of this blog is available here.

In one of our earlier posts we talked about how we set up prettier and rubocop to automatically format our JavaScript and Ruby code on git commit.

Recently we started working with Elm in a couple of our projects - APISnapshot and AceHelp.

Tools like prettier and rubocop have really helped us take a load off our mind with regards to formatting code. And one of the very first things we wanted to sort out when we started doing Elm was pretty printing our Elm code.

elm-format created by Aaron VonderHaar formats Elm source code according to a standard set of rules based on the official Elm Style Guide.

Automatic code formatting

Let’s setup git hook to automatically take care of code formatting. We can acheive this much like how we did it in our previous post, using Husky and Lint-staged.

Let’s add Husky and lint-staged as dev dependencies to our project. And for completeness also include elm-format as a dev dependency.

npm install --save-dev husky lint-staged elm-format

Husky makes it real easy to create git hooks. Git hooks are scripts that are executed by git before or after an event. We will be using the pre-commit hook which is run after you do a git commit command but before you type in a commit message.

This way we can change and format files that’s about to be commited by running elm-format using Husky.

But there is one problem here. The changed files do not get added back to our commit.

This is where Lint-staged comes in. Lint-staged is built to run linters on staged files. So instead of running elm-format on a pre-commit hook we would run lint-staged. And we can configure lint-staged such that elm-format is run on all staged elm files.

We can also include Prettier to take care of all staged JavaScript files too.

Lets do this by editing our package.json file.

{
  "scripts": {
   "precommit": "lint-staged"
  },
  "lint-staged": {
    "*.elm": [
      "elm-format --yes",
      "git add"
    ],
    "*.js": [
      "prettier --write",
      "git add"
    ]
  }
}

All set and done!

Now whenever we do a git commit command, all our staged elm and JavaScript files will get properly formatted before the commit goes in.

This blog is part of our Ruby 2.6 series. Ruby 2.6.0 was released on Dec 25, 2018.

Before Ruby 2.6, if we want endless loop with index, we would need to use Float::INFINITY with upto or Range, or use Numeric#step.

Ruby 2.5.0

irb> (1..Float::INFINITY).each do |n|
irb*   # logic goes here
irb> end

OR

irb> 1.step.each do |n|
irb*   # logic goes here
irb> end

Ruby 2.6.0

Ruby 2.6 makes infinite loop more readable by changing mandatory second argument in range to optional. Internally, Ruby changes second argument to nil if second argument is not provided. So, both (0..) and (0..nil) are same in Ruby 2.6.

Using endless loop in Ruby 2.6

irb> (0..).each do |n|
irb*   # logic goes here
irb> end

irb> (0..nil).size
=> Infinity
irb> (0..).size
=> Infinity

In Ruby 2.5, nil is not an acceptable argument and (0..nil) would throw ArgumentError.

irb> (0..nil)
ArgumentError (bad value for range)

Here is the relevant commit and discussion for this change.

The Chinese version of this blog is available here.

This blog is part of our Rails 5.2 series.

Before 5.2 it was not possible to write multiple entries to cache store in one shot even though cache stores like Redis has MSET command to set multiple keys in a single atomic operation. However we were not able to use this feature of Redis because of the way Rails had implemented caching.

Rails has implemented caching using an abstract class ActiveSupport::Cache::Store which defines the interface that all cache store classes should implement. Rails also provides few common functionality that all cache store classes will need.

Prior to Rails 5.2 ActiveSupport::Cache::Store didn’t have any method to set multiple entities at once.

In Rails 5.2, write_multi was added . Each cache store can implement this method and provide the functionality to add multiple entries at once. If cache store does not implement this method, then the default implementation is to loop over each key value pair and sets it individually using write_entity method.

Multiple entities can be set as shown here.

Rails.cache.write_multi name: 'Alan Turning', country: 'England'

redis-rails gem provides redis as cache store. However it does not implement write_multi method.

However if we are using Rails 5.2, then there is no point in using redis-rails gem, as Rails 5.2 comes with built in support for redis cache store, which implements write_multi method. It was added by this PR.

We need to make following change.

# before
config.cache_store = :redis_store

# after
config.cache_store = :redis_cache_store

redis-rails repo has a pull request to notify users that development of this gem is ceased. So it’s better to use redis cache store that comes with Rails 5.2.

We have recently worked on many chrome extensions. Releasing new chrome extensions manually gets tiring afer a while.

So, we thought about automating it with CircleCI, similar to continuous deployment.

We are using the following configuration in circle.yml to continuously release chrome extensions from the master branch.

workflows:
  version: 2
  main:
    jobs:
      - test:
          filters:
            branches:
              ignore: []
      - build:
          requires:
            - test
          filters:
            branches:
              only: master
      - publish:
          requires:
            - build
          filters:
            branches:
              only: master

version: 2
jobs:
  test:
    docker:
      - image: cibuilds/base:latest
    steps:
      - checkout
      - run:
          name: "Install Dependencies"
          command: |
            apk add --no-cache yarn
            yarn
      - run:
          name: "Run Tests"
          command: |
            yarn run test
  build:
    docker:
      - image: cibuilds/chrome-extension:latest
    steps:
      - checkout
      - run:
          name: "Install Dependencies"
          command: |
            apk add --no-cache yarn
            apk add --no-cache zip
            yarn
      - run:
          name: "Package Extension"
          command: |
            yarn run build
            zip -r build.zip build
      - persist_to_workspace:
          root: /root/project
          paths:
            - build.zip

  publish:
    docker:
      - image: cibuilds/chrome-extension:latest
    environment:
      - APP_ID: <APP_ID>
    steps:
      - attach_workspace:
          at: /root/workspace
      - run:
          name: "Publish to the Google Chrome Store"
          command: |
            ACCESS_TOKEN=$(curl "https://accounts.google.com/o/oauth2/token" -d "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&refresh_token=${REFRESH_TOKEN}&grant_type=refresh_token&redirect_uri=urn:ietf:wg:oauth:2.0:oob" | jq -r .access_token)
            curl -H "Authorization: Bearer ${ACCESS_TOKEN}" -H "x-goog-api-version: 2" -X PUT -T /root/workspace/build.zip -v "https://www.googleapis.com/upload/chromewebstore/v1.1/items/${APP_ID}"
            curl -H "Authorization: Bearer ${ACCESS_TOKEN}" -H "x-goog-api-version: 2" -H "Content-Length: 0" -X POST -v "https://www.googleapis.com/chromewebstore/v1.1/items/${APP_ID}/publish"

We have created three jobs named as test, build and publish and used these jobs in our workflow to run tests, build the extension, and publish them to the chrome store, respectively. Every step requires the previous step to run successfully.

Let’s check each job one by one.

test:
  docker:
    - image: cibuilds/base:latest
  steps:
    - checkout
    - run:
        name: "Install Dependencies"
        command: |
          apk add --no-cache yarn
          yarn
    - run:
        name: "Run Tests"
        command: |
          yarn run test

We use cibuilds docker image for this job. First, we do a checkout to the branch and then use yarn to install dependencies. Alternatively, we can use npm to install dependencies as well. Then, as the last step, we are use yarn run test to run tests. We can skip this step if running tests is not needed.

build:
  docker:
    - image: cibuilds/chrome-extension:latest
  steps:
    - checkout
    - run:
        name: "Install Dependencies"
        command: |
          apk add --no-cache yarn
          apk add --no-cache zip
          yarn
    - run:
        name: "Package Extension"
        command: |
          yarn run build
          zip -r build.zip build
    - persist_to_workspace:
        root: /root/project
        paths:
          - build.zip

For building chrome extensions, we use the chrome-extension image. Here, we also first do a checkout and then, install dependencies using yarn. Note, we are install zip utility along with yarn because we need to zip our chrome extension before publishing it in next step. Also, we are not generating version numbers on our own. The version number will be picked from the manifest file. This step assumes that we have a task named build in package.json to build our app.

The Chrome store rejects multiple uploads with the same version number. So, we have to make sure to update the version number, which should be unique in the manifest file before this step.

In the last step, we use persist_to_workspace to make build.zip available for the next step, publishing.

publish:
  docker:
    - image: cibuilds/chrome-extension:latest
  environment:
    - APP_ID: <APP_ID>
  steps:
    - attach_workspace:
        at: /root/workspace
    - run:
        name: "Publish to the Google Chrome Store"
        command: |
          ACCESS_TOKEN=$(curl "https://accounts.google.com/o/oauth2/token" -d "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&refresh_token=${REFRESH_TOKEN}&grant_type=refresh_token&redirect_uri=urn:ietf:wg:oauth:2.0:oob" | jq -r .access_token)
          curl -H "Authorization: Bearer ${ACCESS_TOKEN}" -H "x-goog-api-version: 2" -X PUT -T /root/workspace/build.zip -v "https://www.googleapis.com/upload/chromewebstore/v1.1/items/${APP_ID}"
          curl -H "Authorization: Bearer ${ACCESS_TOKEN}" -H "x-goog-api-version: 2" -H "Content-Length: 0" -X POST -v "https://www.googleapis.com/chromewebstore/v1.1/items/${APP_ID}/publish"

For publishing of the chrome extension, we use the chrome-extension image.

We need APP_ID, CLIENT_ID, CLIENT_SECRET and REFRESH_TOKEN/ACCESS_TOKEN to publish our app to the chrome store.

APP_ID needs to be fetched from Google Webstore Developer Dashboard. APP_ID is unique for each app whereas CLIENT_ID, CLIENT_SECRET and REFRESH_TOKEN/ACCESS_TOKEN can be used for multiple apps. Since APP_ID is generally public, we specify that in the yml file. CLIENT_ID, CLIENT_SECRET and REFRESH_TOKEN/ACCESS_TOKEN are stored as private environment variables using CircleCI UI. For cases when our app is unlisted in the chrome store, we need to store APP_ID as a private environment variable.

CLIENT_ID and CLIENT_SECRET need to be fetched from Google API console. There, we need to select a project and then click on the credentials tab. If there is no project, we need to create one and then access the credentials tab.

REFRESH_TOKEN needs to be fetched from Google API. It also defines the scope of access for Google APIs. We need to refer to Google OAuth2 for obtaining the refresh token. We can use any language library.

In the first step of the publish job, we are attaching a workspace to access build.zip, which was created previously. Now, by using all the required tokens obtained previously, we need to obtain an access token from Google OAuth API, which must be used to push the app to the chrome store. Then, we make a PUT request to the Chrome store API to push the app, and then use the same API again, to publish the app.

Uploading via API has one more advantage over manual upload. Manual upload generally takes up to 1 hour to show the app in the chrome store. Whereas uploading using Google API generally reflects the app within 5-10 minutes, considering app does not go for a review by Google.

This blog is part of our Rails 5.2 series.

Before Rails 5.2, AES-256-CBC authenticated encryption was the default cipher for encrypting messages.

It was proposed to use AES-256-GCM authenticated encryption as the default cipher for encrypting messages because of following reasons:

• It produces shorter ciphertexts and performs quick encryption and decryption.
• It is less error prone and more secure.

So, AES-256-GCM became default cipher for encrypting messages in Rails 5.2 .

If we do not want AES-256-GCM as default cipher for encrypting messages in our rails application, then we can disable it.

Rails.application.config.active_support.use_authenticated_message_encryption = false

Default Encryption for cookies and sessions was also updated to use AES-256-GCM in this pull request.

If we do not want AES-256-GCM as default encryption of cookies and sessions, then we can disable it too.

Rails.application.config.active_support.use_authenticated_cookie_encryption = false

Apple announced iOS 12 at WWDC 2018 a few days back. Being honest, it was a bit disappointing to see some of the most requested features not being there in iOS 12. Like users have been asking for Dark mode since before iOS 11, and the ability to set default apps. It’s more of an update focussed on performance improvements, usability, and compatibility. The fact that iOS 12 is also available for the iPhones Apple released 5 years back is a great effort from Apple to keep users happy. And unlike last couple of years, this time we decided to calm our curiosities and installed iOS 12 beta 1 on our devices right away after Apple released it for developers. This blog is based on our experience with iOS 12 on iPhone 8 Plus.

Installing iOS 12 on your iPhone

First things first, make sure you have iPhone 5s or newer. And before getting started, plug-in your phone to iTunes and take a full-backup in case your phone gets bricked while installing iOS 12, which is very unlikely.

Once done, download and install beta profile for iOS 12 and then download and update from “Software Update” section just like you install a regular iOS update. It’s a straightforward OTA update process which you’re already familiar with.

Note: This beta profile is from a third-party developer and is not officially from Apple. Apple will officially release public beta in around a month.

We’ve been running iOS 12 since last week now and here are our thoughts on the additions and changes introduced in iOS 12.

iOS 12 is fast

The performance improvements are significant and definitely noticeable. Previously on iOS 11, while accessing spotlight by swiping down from home screen, it used to lag. And not just that, sometimes keyboard didn’t even used to show up and we had to repeat the same action to make it work. But things are faster and better in iOS 12. The keyboard gets up as soon as spotlight shows up.

Another thing that we’ve noticed is the multitasking shortcut for switching to the last app by 3d touching on left wasn’t that reliable in iOS 11, so much that it was easy to ignore the feature altogether than to use it, but in iOS 12 the same shortcut is very smooth. Although there are times when it still doesn’t work well, but that’s very rare.

Spotlight widgets load faster in iOS 12 than they used to in iOS 11. Apart from this, 3d touch feels pretty smooth too. It’s good to see Apple squeezing out the power to improve the already good performance in iOS.

Notifications

Notifications in iOS 11 are a mess, there’s no grouping, there’s no way to quickly control notifications, there’s no way to set priorities. Apple has added notifications grouping and better notification management in iOS 12, so now notifications from the same app are grouped together and you can control how often you want to get notifications from the apps right from the notification.

We think the implementation can be a whole lot better. For the notifications that are grouped, you get to see only the last notification from that app, a better way would’ve been to show two or three notifications and cut the rest. There’s no Notification pinning or snoozing which could’ve been very useful features.

Screen time and App limits

There’s a new feature in iOS 12 called Screen Time which is more like bird’s-eye view of your phone usage. There’s a saying that you can’t improve something that you can’t measure. Screen Time is a feature that’s going to be very useful for everyone who wants to cut down time on Social apps or overall phone usage. It shows you every little detail of how many apps you use and for how much time and at what times. Not only this, it also keeps track of how many times you picked up your phone, and how many notifications you receive from the apps you have on your phone.

Other useful sub-feature of Screen time is App limits, which allows you to set limit on app usage based on app or category. So let’s say you don’t want to use WhatsApp for more than 30 mins a day, you can do that through App limits. It works for app categories including Games, Social Networking, Entertainment, Creativity, Productivity, Education, Reading & Reference, Health & Fitness. So you can limit it via category which works across apps. Plus, it syncs across your other iOS devices, so you can’t cheat that way.

Siri and Shortcuts app

In iOS 12, you can assign custom shortcuts to Siri to trigger specific actions, which not only works for System apps but also with third-party apps. So now if you want to send a specific message to someone on WhatsApp, you can assign a command for that to Siri and you can trigger that action just from Siri using that command.

Apple has also introduced a new app in iOS 12 called Shortcuts. Shortcuts app lets you group actions and run those actions quickly. Although Shortcuts app isn’t there in iOS 12 beta 1 but we think it’s one of best addition in iOS 12.

Updated Photos app

Photos app now has new section called “For you”, where it shows all the new Albums, your best moments, share suggestions, photos and effect suggestions. This is more like the Assistant tab of Google Photos app. Also you can share selected photos or albums with your friends right from the Photos app.

The Album tab in Photos app is redesigned for easier navigation. Also there’s a new tab for Search which has been advanced, so you can now search photos using tags like “surfing” and “vacation”.

It’s good to see Apple paying attention to up the Photos app but we still think Google Photos is a better option for average person considering it lets you store photos in Cloud for free. Also photo organization in Google Photos is much better than in new Photos app in iOS 12.

Enhanced Do Not Disturb Mode

Do Not Disturb in iOS 12 is enhanced to be more flexible. You can now enable Do Not Disturb mode to end automatically in an hour, or at night, or according to your Calendar events, or even based on your location.

Not just that, Do Not Disturb has a new Bedtime mode enabling which will shut all your notifications during bed time and dim your display. And when you will wake up, it’ll show you a welcome back message along with weather details on the lock screen.

Conclusion

There are other updates and under the hood improvements as well, like new Measure app, redesigned iBooks app, tracking prevention, group FaceTime etc. Overall, we think it’s an okay update considering there are not as many bugs as there should be according to Apple standards. The force touch in the keyboard to drag cursor doesn’t work, Skype and some other apps crash, but for the most part, it’s good enough to be installed on your primary device.

Concurrent Ruby is a concurrency toolkit that builds on a lot of interesting ideas from many functional languages and classic concurrency patterns. When it comes to writing threaded code in Rails applications, look no further since concurrent ruby is already included in Rails via Active Support.

Using Concurrent::Future

In one of our applications, to improve performance we added threaded code using Concurrent::Future. It worked really well for us until one day it stopped working.

“Why threads?” one might ask. The code in question was a textbook threading use case. It had a few API calls, some DB requests and finally an action that was performed on all the data that was aggregated.

Let us look at what this code looks like.

Non threaded code

selected_shipping_companies.each do | carrier |
  # api calls
  distance_in_miles = find_distance_from_origin_to_destination
  historical_average_rate = historical_average_for_this_particular_carrier

  # action performed
  build_price_details_for_this_carrier(distance_in_miles,
                                       historical_average_rate)
end

Converting the above code to use Concurrent::Future is trivial.

futures = selected_shipping_companies.map do |carrier|
  Concurrent::Future.execute do
    # api calls
    distance_in_miles = find_distance_from_origin_to_destination
    historical_average_rate = historical_average_for_this_particular_carrier

    # action performed
    build_price_details_for_this_carrier(distance_in_miles,
                                         historical_average_rate)
  end
end

futures.map(&:value)

A bit more about Concurrent::Future

It is often intimidating to work with threads. They can bring in complexity and can have unpredictable behaviors due to lack of thread-safety. Ruby, being a language of mutable references, we often find it difficult to write 100% thread-safe code.

Inspired by Clojure’s Future function, Concurrent::Future is a primitive that guarantees thead safety. It takes a block of work and performs the work asynchronously using Concurrent Ruby’s global thread-pool. Once a block of work is scheduled, Concurrent Ruby gives us a handle to this future work, on which when #value (or #deref) is called block’s value is returned.

The Bug

Usually, when an exception occurs in the main thread, the interpreter stops and gathers the exception data. In the case of Ruby Threads, any unhandled exceptions are reported only when Thread#join is called. Setting Thread#abort_on_exception to true, is an better alternative which will cause all threads to exit when an exception is raised in any running thread. We published a blog recently which talks about this in great detail.

Exception handling in Concurrent Ruby

future = Concurrent::Future.execute {
            raise StandardError.new("Boom!")
          }

sleep(0.1) # giving arbitrary time for future to execute

future.value     #=> nil

Where did the exception go? This code fails silently and swallows the exceptions. How can we find out if the code executed successfully?

future = Concurrent::Future.execute {
              raise StandardError.new("Boom!")
          }

sleep(0.1) # giving arbitrary time for future to execute

future.value     #=> nil

future.rejected? #=> true
future.reason    #=> "#<StandardError: Boom!>"

How we fixed our issue

We found places in our application where Concurrent::Future was used in a way that would swallow exceptions. It is also a possibility that people might overlook the explicit need to manually report exception. We addressed these concerns with the following wrapper class.

module ConcurrentExecutor
  class Error < StandardError
    def initialize(exceptions)
      @exceptions = exceptions
      super
    end

    def message
      @exceptions.map { | e | e.message }.join "\n"
    end

    def backtrace
      traces = @exceptions.map { |e| e.backtrace }
      ["ConcurrentExecutor::Error START", traces, "END"].flatten
    end
  end

  class Future
    def initialize(pool: nil)
      @pool = pool || Concurrent::FixedThreadPool.new(20)
      @exceptions = Concurrent::Array.new
    end

    # Sample Usage
    # executor = ConcurrentExecutor::Future.new(pool: pool)
    # executor.execute(carriers) do | carrier |
    #   ...
    # end
    #
    # values = executor.resolve

    def execute array, &block
      @futures = array.map do | element |
        Concurrent::Future.execute({ executor: @pool }) do
          yield(element)
        end.rescue do | exception |
          @exceptions << exception
        end
      end

      self
    end

    def resolve
      values = @futures.map(&:value)

      if @exceptions.length > 0
        raise ConcurrentExecutor::Error.new(@exceptions)
      end

      values
    end
  end
end

Please note that using Concurrent Ruby Futures caused segmentation fault while running specs in Circle CI. As of this writing, we are using normal looping instead of Futures in Circle CI until the reason for the segfault is isolated and fixed.

Update

Concurrent::Future also gives us another API which not only returns the value of the block but also posts/raises any exceptions that occur into the main thread.

thread_pool = Concurrent::FixedThreadPool.new(20)
executors = [1, 2, 3, 4].map do |random_number|
  Concurrent::Future.execute({ executor: thread_pool }) do
    random_number / (random_number.even? ? 0 : 1)
  end
end

executors.map(&:value)
=> [1, nil, 3, nil]

executors.map(&:value!)

> ZeroDivisionError: divided by 0
> from (pry):4:in `/'

We thank Jonathan Rochkind for pointing us to this undocumented api in his reddit post.

We recently made ApiSnapshot open source. As mentioned in that blog we ported code from React.js to Elm.

One of the features of ApiSnapshot is support for Basic Authentication.

While we were rebuilding the whole application in Elm, we had to port the “Add Basic Authentication” feature. This feature can be accessed from the “More” drop-down on the right-hand side of the app and it lets user add username and password to the request.

Let’s see how the Model of our Elm app looks.

type alias Model =
    { request : Request.MainRequest.Model
    , response : Response.MainResponse.Model
    , route : Route
    }

Here is the Model in Request.MainRequest module.

type alias APISnapshotRequest =
    { url : String
    , httpMethod : HttpMethod
    , requestParameters : RequestParameters
    , requestHeaders : RequestHeaders
    , username : Maybe String
    , password : Maybe String
    , requestBody : Maybe RequestBody
    }

type alias Model =
    { request : APISnapshotRequest
    , showErrors : Bool
    }

username and password fields are optional for the users so we kept them as Maybe types.

Note that API always responds with username and password whether user clicked to add Basic Authentication or not. The API would respond with a null for both username and password when a user tries to retrieve a snapshot for which user did not fill username and password.

Here is a sample API response.

{
"url": "http://dog.ceo/api/breed/affenpinscher/images/random",
"httpMethod": "GET",
"requestParams": {},
"requestHeaders": {},
"requestBody": null,
"username": "alanturning",
"password": "welcome",
"assertions": [],
"response": {
    "response_headers": {
        "age": "0",
        "via": "1.1 varnish (Varnish/6.0), 1.1 varnish (Varnish/6.0)",
        "date": "Thu, 03 May 2018 09:43:11 GMT",
        "vary": "",
        "cf_ray": "4151c826ac834704-EWR",
        "server": "cloudflare"
    },
    "response_body": "{\"status\":\"success\",\"message\":\"https:\\/\\/images.dog.ceo\\/breeds\\/affenpinscher\\/n02110627_13221.jpg\"}",
    "response_code": "200"
  }
}

Let’s look at the view code which renders the data received from the API.

view : (Maybe String, Maybe String) -> Html Msg
view usernameAndPassword =
    case usernameAndPassword of
        (Nothing, Nothing) -> text ""
        (Just username, Nothing) -> basicAuthenticationView username ""
        (Nothing, Just password) -> basicAuthenticationView "" password
        (Just username, Just password) -> basicAuthenticationView username password


basicAuthenticationView : String -> String -> Html Msg
basicAuthenticationView username password =
    [ div [ class "form-row" ]
        [ input
            [ type_ "text"
            , placeholder "Username"
            , value username
            , onInput (UpdateUsername)
            ]
            []
        , input
            [ type_ "password"
            , placeholder "Password"
            , value password
            , onInput (UpdatePassword)
            ]
            []
        , a
            [ href "javascript:void(0)"
            , onClick (RemoveBasicAuthentication)
            ]
            [ text "×" ]
        ]
    ]

To get the desired view we apply following rules.

1. Check if both the values are string.
2. Check if either of the values is string.
3. Assume that both the values are null.

This works but we can do a better job of modelling it.

What’s happening here is that we were trying to translate our API responses directly to the Model . Let’s try to club username and password together into a new type called BasicAuthentication.

In the model add a parameter called basicAuthentication which would be of type Maybe BasicAuthentication. This way if user has opted to use basic authentication fields then it is a Just BasicAuthentication and we can show the input boxes. Otherwise it is Nothing and we show nothing!

Here is what the updated Model for Request.MainRequest would look like.

type alias BasicAuthentication =
    { username : String
    , password : String
    }


type alias APISnapshotRequest =
    { url : String
    , httpMethod : HttpMethod
    , requestParameters : RequestParameters
    , requestHeaders : RequestHeaders
    , basicAuthentication : Maybe BasicAuthentication
    , requestBody : Maybe RequestBody
    }


type alias Model =
    { request : APISnapshotRequest
    , showErrors : Bool
    }

Elm compiler is complaining that we need to make changes to JSON decocding for APISnapshotRequest type because of this change.

Before we fix that let’s take a look at how JSON decoding is currently being done.

import Json.Decode as JD
import Json.Decode.Pipeline as JP

decodeAPISnapshotRequest : Response -> APISnapshotRequest
decodeAPISnapshotRequest hitResponse =
    let
        result =
            JD.decodeString requestDecoder hitResponse.body
    in
        case result of
            Ok decodedValue ->
                decodedValue

            Err err ->
                emptyRequest


requestDecoder : JD.Decoder APISnapshotRequest
requestDecoder =
    JP.decode Request
        |> JP.optional "username" (JD.map Just JD.string) Nothing
        |> JP.optional "password" (JD.map Just JD.string) Nothing

Now we need to derive the state of the application from our API response .

Let’s introduce a type called ReceivedAPISnapshotRequest which would be the shape of our old APISnapshotRequest with no basicAuthentication field. And let’s update our requestDecoder function to return a Decoder of type ReceivedAPISnapshotRequest instead of APISnapshotRequest.

type alias ReceivedAPISnapshotRequest =
    { url : String
    , httpMethod : HttpMethod
    , requestParameters : RequestParameters
    , requestHeaders : RequestHeaders
    , username : Maybe String
    , password : Maybe String
    , requestBody : Maybe RequestBody
    }


requestDecoder : JD.Decoder ReceivedAPISnapshotRequest

We need to now move our earlier logic that checks to see if a user has opted to use the basic authentication fields or not from the view function to the decodeAPISnapshotRequest function.

decodeAPISnapshotRequest : Response -> APISnapshotRequest
decodeAPISnapshotRequest hitResponse =
    let
        result =
            JD.decodeString requestDecoder hitResponse.body
    in
        case result of
            Ok value ->
                let
                    extractedCreds =
                        ( value.username, value.password )

                    derivedBasicAuthentication =
                        case extractedCreds of
                            ( Nothing, Nothing ) ->
                                Nothing

                            ( Just receivedUsername, Nothing ) ->
                                Just { username = receivedUsername, password = "" }

                            ( Nothing, Just receivedPassword ) ->
                                Just { username = "", password = receivedPassword }

                            ( Just receivedUsername, Just receivedPassword ) ->
                                Just { username = receivedUsername, password = receivedPassword }
                in
                    { url = value.url
                    , httpMethod = value.httpMethod
                    , requestParameters = value.requestParameters
                    , requestHeaders = value.requestHeaders
                    , basicAuthentication = derivedBasicAuthentication
                    , requestBody = value.requestBody
                    }

            Err err ->
                emptyRequest

We extract the username and password into extractedCreds as a Pair from ReceivedAPISnapshotRequest after decoding and construct our APISnapshotRequest from it.

And now we have a clean view function which just takes a BasicAuthentication type and returns us a Html Msg type.

view : BasicAuthentication -> Html Msg
view b =
    [ div [ class "form-row" ]
        [ input
            [ type_ "text"
            , placeholder "Username"
            , value b.username
            , onInput (UpdateUsername)
            ]
            []
        , input
            [ type_ "password"
            , placeholder "Password"
            , value b.password
            , onInput (UpdatePassword)
            ]
            []
        , a
            [ href "javascript:void(0)"
            , onClick (RemoveBasicAuthentication)
            ]
            [ text "×" ]
        ]
    ]

We now have a Model that better captures the business logic. And should we change the logic of basic authentication parameter selection in the future, We do not have to worry about updating the logic in the view .

Monitoring and Logging are important aspects of deployments. Centralized logging is always useful in helping us identify the problems.

EFK (Elasticsearch, Fluentd, Kibana) is a beautiful combination of tools to store logs centrally and visualize them on a single click. There are many other open-source logging tools available in the market but EFK (ELK if Logstash is used) is one of the most widely used centralized logging tools.

This blog post shows how to integrate Logtrail which has a papertrail like UI to tail the logs. Using Logtrail we can also apply filters to tail the logs centrally.

As EFK ships as an addon with Kubernetes, all we have to do is deploy the EFK addon on our k8s cluster.

Pre-requisite:

• Access to working kubernetes cluster with kubectl configuration.

• All our application logs should be redirected to STDOUT, so that Fluentd forwards them to Elasticsearch.

• Understanding of Kubernetes terms like pods, deployments, services, daemonsets, configmap and addons.

Installing EFK addon from kubernetes upstream is simple. Deploy EFK using following command.

$ kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/logging-elasticsearch/v1.6.0.yaml
serviceaccount "elasticsearch-logging" created
clusterrole "elasticsearch-logging" created
clusterrolebinding "elasticsearch-logging" created
serviceaccount "fluentd-es" created
clusterrole "fluentd-es" created
clusterrolebinding "fluentd-es" created
daemonset "fluentd-es" created
service "elasticsearch-logging" created
statefulset "elasticsearch-logging" created
deployment "kibana-logging" created
service "kibana-logging" created

Once k8s resources are created access the Kibana dashboard. To access the dashboard get the URL using kubectl cluster-info

$ kubectl cluster-info | grep Kibana
Kibana is running at https://api.k8s-test.com/api/v1/proxy/namespaces/kube-system/services/kibana-logging

Now goto Kibana dashboard and we should be able to see the logs on our dashboard.

Above dashboard shows the Kibana UI. We can create metrics and graphs as per our requirement.

We also want to view logs in tail style. We will use logtrail to view logs in tail format. For that, we need docker image having logtrail plugin pre-installed.

Note: If upstream Kibana version of k8s EFK addon is 4.x, use kibana 4.x image for installing logtrail plugin in your custom image. If addon ships with kibana version 5.x, make sure you pre-install logtrail on kibana 5 image.

Check the kibana version for addon here.

We will replace default kibana image with kubernetes-logtrail image.

To replace docker image update the kibana deployment using below command.

$ kubectl -n kube-system set image deployment/kibana-logging kibana-logging=rahulmahale/kubernetes-logtrail:latest
deployment "kibana-logging" image updated

Once the image is deployed go to the kibana dashboard and click on logtrail as shown below.

After switching to logtrail we will start seeing all the logs in real time as shown below.

This centralized logging dashboard with logtrail allows us to filter on several parameters.

For example let’s say we want to check all the logs for namespace myapp. We can use filter kubernetes.namespace_name:"myapp". We can user filter kubernetes.container_name:"mycontainer" to monitor log for a specific container.

RubyKaigi is happening at Sendai, Japan from 31st May to 2nd June. It is perhaps the only conference where one can find almost all the core Ruby team members in attendance.

This is Prathamesh. I bring you live details about what is happening at the Kaigi over the next two days. If you are at the conference please come and say “Hi” to me.

Check out what happened on day 1 .

Faster Apps, No Memory Thrash: Get Your Memory Config Right by Noah Gibbs

Noah gave an awesome talk on techniques to manage the memory used by Ruby applications. One of the main point while dealing with GC is to make it run less, which means don’t create too many objects. He also mentioned that if application permits then destructive operations such as gsub! or concat should be used since they save CPU cycles and memory. Ruby allows setting up environment variables for managing the heap memory but it is really hard to choose values for these environment variables blindly.

Noah has built a tool which uses GC.stat results from applications to estimate the values of the memory related environment variables. Check out the EnvMem gem.

In the end, he discussed some advanced debugging methods like checking fragmentation percentage. The formula is prepared by Nate Berkopec.

s = GC.stat
used_ratio = s[:heap_live_slots].to_f / (s[:heap_eden_pages] * 408)
fragmentation = 1 - used_ratio

We can also use GC::Profiler to profile the code in real time to see how GC is behaving.

Benchmark used for this talk can be found here. Slides for this talk can be found here.

Guild prototype

Next I attended talk by Koichi Sasada on Guild prototype. He discussed the proposed design spec for Guild with a demo of Fibbonacci number program with 40 core CPU and 40 guilds. One of the interesting observations is that performance drops as number of guilds increases because of the global locking.

He discussed the concept of sharable and non-sharable objects. Sharable objects can be shared across multiple Guilds whereas non-sharable objects can only be used one Guild. This also means, you can’t make thread unsafe program with Guilds “by design”. He discussed about the challanges in specifying the sharable objects for Guilds.

Overall, there is stil a lot of work left to be done for Guilds to become a part of Ruby. It includes defining protocols for sharable and non-sharable objects, making sure GC runs properly in case of Guilds, synchronization between different Guilds.

The slides for this talk can be found here.

Ruby programming with type checking

Soutaro from SideCI gave a talk on Steep, a gradual type checker for Ruby.

In the past, Matz has said that he doesn’t like type definitions to be present in the Ruby code. Steep requires type definitions to be present in separate files with extension .rbi. The Ruby source code needs little amount of annotations. Steep also has a scaffold generator to generate the basic type definitions for existing code.

As of now, Steep runs slower than Sorbet which was discussed yesterday by Stripe team. Soutaro also discssued issues in type definitions due to meta programming in libraries such as Active Record. That looks like a challange for Steep as of now.

Web console

After the tea break, I attended talk by Genadi on how web console works.

He discussed the implementation of web-console in detail with references to Ruby internals related to bindings. He compared the web-console interface with IRB and pry and explained the difference. As of now, web console has to monkey patch some of the Rails internals. Genadi has added support for registering interceptors which will prevent this monkey patching in Rails 6. He is also mentoring a Google summer of code student to work on Actionable errors project where the user can take actions like running pending migrations via the webpage itself when the error is shown.

Ruby committers v/s the World

RubyKaigi offers this unique event where all the Ruby committers come on stage and face the questions from the audience. This year the format was slightly different and it was run in the style of the Ruby core developer meeting. The agenda was predecided with some questions from the people and some tickets to discuss. The session started with discussion of features coming up in Ruby 2.6.

After that, the questions and the tickets in the agenda were discussed. It was good to see how the Ruby team takes decisions about features and suggestions.

Apart from this, there were talks on SciRuby, mRuby, linting, gem upgrades, c extensions and more which I could not attend.

That’s all for the day two. Looking forward to the day three already!

Oh and here is the world map of all the attendees from RubyKaigi.

RubyKaigi is happening at Sendai, Japan from 31st May to 2nd June. It is perhaps the only conference where one can find almost all the core Ruby team members in attendance.

This is Prathamesh. I bring you live details about what is happening at the Kaigi over the next three days. If you are at the conference please come and say “Hi” to me.

Matz’s keynote

RubyKaigi started with Matz’s keynote. He used lot of proverbs applying them to the Ruby language and software development.

He talked about one of the hardest problems in programming - naming with an example of yield_self. Matz added alias then to the yield_self method yesterday. He also discussed about googlability of the names. Ironically, Ruby was named in 1993 which was before Google had started.

Matz also touched upon JIT option being introduced in Ruby 2.6 and guild as the ways the language continues to improve in performance and concurrency. There is a talk on Guild by Koichi Sasada on second day of RubyKaigi which will have further details about it.

Matz ended the keynote talking about the need of maintaining backward compatibility and not running into the situation like Ruby 1.9 or Python 3 where the compatibility was not maintained. He also stressed upon the community aspect of the Ruby language and its importance in the success of Ruby.

ETL processing in Ruby using Kiba

Thibaut Barrère gave a talk on Kiba - a data processing ETL framework for Ruby. He discussed about the design decisions that went into the version 1 and how it evolved to version 2 which was recently released.

Kiba provides programmatic API which can be used in the background jobs instead of shelling out. It also has support for multistep batch processing.

Thibaut also explained how it can be used for data migration, reusing the components and big rewrites. He observed that the performance has been gradually increasing with each Ruby release over the years.

The slides for this talk can be found here.

Architecture of Hanami applications

Next I attended talk from Anton Davydov on architecture patterns in Hanami apps. He discussed about the problems typical Rails applications face and how abstractions can address those issues. He explained how Hanami tries to achieve business logic isolation, avoid global state, sequential logic and test coverage. Functional callable objects, containers, dry-containers, dry-inject and event sourcing are some of the abstractions that can be used in Hanami apps that help in achieving this.

Lightning talks

The last session of the day was lightning talks.

The talk on Rib(wordplay on IRB) was an interesting one. Rib is yet another interactive Ruby shell but lightweight compared to IRB and pry. It has some nice features like auto indent, multiline history, filtering of callers. It can also beep when the console starts, so you know it is time to get back to work.

I liked another talk where Watson had worked on improving the performance of JSON gem. He achieved this by using CRuby API whereever applicable and avoiding heavy calls like rbfuncall. Check these two pull requests for benchmark and more discussions.

Apart from these talks, there were lot of other talks as well which I could not attend. Stripe team is building a type checker for Ruby which looks very interesting and is is extremely fast.

Bozhidar Batsov gave a talk on Rubocop project and how it has evolved over the years. There was also a talk on Karafka - event driven architecture in Ruby. This talk was a good precursor to the Hanami talk where event driven architecture was mentioned again.

Other talks from day one ranged from memory management, playing with Ruby synatx, code highlighter, deep learning, C extensions to Rubygems.

That’s all for the day one. Looking forward to the day two already!

Rails 5.0 had introduced redirect_back method to perform redirection to path present in HTTP_REFERER. If there is no HTTP_REFERER present, then site is redirected to fallback_location.

Now consider the following scenario.

In one of the searches on google.com, we see a link to bigbinary.com. On clicking the link, we are navigated to bigbinary.com.

When somebody gets redirected to bigbinary.com from google.com, the HTTP REFERER is set to google.com

If bigbinary.com uses redirect_back in its code then the user will get redirected to google.com which might be undesired behavior for some applications.

To avoid such cases, Rails 5.2 has added a flag allow_other_host to not allow redirecting to a different host other than the current site.

By default, allow_other_host option is set to true. So if you do not want users to go back to google.com then you need to explicitly set allow_other_host: false.

> request.host
#=> "http://www.bigbinary.com"

> request.headers["Referer"]
#=> "http://www.google.com"

# This will redirect back to google.com
redirect_back(fallback_path: "/")

# This will not redirect back to google.com
redirect_back(fallback_path: "/", allow_other_host: false)

There are various ways to generate JSON in Rails. There is to_json method built in Rails. We can also use jbuilder gem or active model serializer gem which can help us achieve the same.

As the number of records in the database grow, Rails can take a very long time to generate a response. The bottleneck can generally be traced back to JSON generation.

Recently, in one of our applications, we faced this issue where a page was taking too long to load. The load time was critical for us as this was the most visited page on the site. The page was loading a race, its racers and their lap details.

The page was loading fine for short races, with 10-15 racers and each racer having 30-50 laps. But for endurance races, with around 50-80 racers and each racer having around 700-800 laps, we were hitting the bottleneck with load times.

After benchmarking, JSON generation at the backend was found to be the culprit.

Looking out for solutions to fix the problem, we came across PostgreSQL JSON functions.

PostgreSQL 9.2 and above have built in support for generating JSON using functions row_to_json and array_to_json. Let’s look into both of them in detail.

row_to_json

row_to_json returns each of the rows as JSON object.

select row_to_json(laps) from laps;

{"id":1,
 "number":1,
 "position":4,
 "time":"628.744",
 "flag_type":"Green"
}
.
.
.

We could use a subquery to only fetch the attributes/columns which we require.

select row_to_json(lap)
from (
  select id, number, position, time, flag_type from laps
) lap;

{"id":1,"number":1,"position":4,"time":"628.744","flag_type":"Green"}
{"id":2,"number":2,"position":4,"time":"614.424","flag_type":"Green"}
.
.
.

array_to_json

To understand array_to_json function, we must first look into array_agg. array_agg is an aggregate function. Aggregate functions compute a single result from a set of input values. sum, min, max are some other examples of aggregate functions. array_agg concatenates all the input values into a PostgreSQL array.

select array_agg(lap)
from (
  select id, number, position, time, flag_type from laps
) lap;

{"(1,1,4,\"628.744\",\"Green\")","(2,2,4,\"614.424\",\"Green\")", ... }

To convert this PostgreSQL array into JSON, we can use the array_to_json function.

select array_to_json(array_agg(lap))
from (
  select id, number, position, time, flag_type from laps
) lap;

[{"id":1,
  "number":1,
  "position":4,
  "time":"628.744",
  "flag_type":"Green"},
  ...]

A more complex example

We can use the above two functions together to generate custom JSON response.

select row_to_json(u)
from (
  select first_name, last_name,
    (
      select array_to_json(array_agg(b))
      from (
        select number, position, time, flag_type
        from laps
        inner join racer_laps
        on laps.id = racer_laps.lap_id
        where racer_laps.racer_id = racers.id
      ) b
    ) as laps
  from racers
  where first_name = 'Jack'
) u;

{
  "first_name": "Jack",
  "last_name": "Altenwerth",
  "laps": [
    {
      "number": 1,
      "position": 4,
      "time": "628.744",
      "flag_type": "Green"
    },
    {
      "number": 2,
      "position": 4,
      "time": "614.424",
      "flag_type": "Green"
    },
    ...
  ]
}

Using the functions in Rails

We can use the above mentioned functions in Rails as shown here.

query = <<~EOQ
select row_to_json(u)
from (
  select first_name, last_name,
    (
      select array_to_json(array_agg(b))
      from (
        select number, position, time, flag_type
        from laps
        inner join racer_laps
        on laps.id = racer_laps.lap_id
        where racer_laps.racer_id = racers.id
      ) b
    ) as laps
  from racers
  where first_name = 'Jack'
) u;
EOQ

generated_json = ActiveRecord::Base.connection.execute(query).values;
puts generated_json

{
  "first_name": "Jack",
  "last_name": "Altenwerth",
  "laps": [
    {
      "number": 1,
      "position": 4,
      "time": "628.744",
      "flag_type": "Green"
    },
    {
      "number": 2,
      "position": 4,
      "time": "614.424",
      "flag_type": "Green"
    },
    ...
  ]
}

Although the code to generate the JSON using the above way is more verbose and less readable compared to other ways to generate JSON in Rails, it is more performant.

Observations

On the racer’s page, generating the JSON using the PostgreSQL functions, gave us the following improvements.

For short races (10-15 racers and each racer having 30-50 laps), the average API response time decreased from 40ms to 15ms.

For endurance races (50-80 racers and each racer having around 700-800 laps), the average API response time decreased from 1200ms to 20ms.

Conclusion

Use Rails way of generating JSON as long as you can. If performance starts to be an issue then don’t be afraid of using the features made available by the database. In this case we would be trading performance for complexity in code. However sometimes this trade is worth it.

APISnapshot is built using Elm and Ruby on Rails. Today we’re happy to announce that the code is publicly available on GitHub.

We built APISnapshot for two reasons.

We wanted to work with Elm.

We wanted to have a tool that is easy to use and that will help us capture what response we are getting from the API in a format that is easy to share in github issue, in slack or in an email. As a consulting company we work with various teams around the world and during development phase either API is unstable or they do not do what they should be doing.

We originally built this tool using React. Elm compiler is quite strict and forced us to take into consideration all possibilities. This lead us to notice a few bugs which were still present in React code. In this way Elm compiler helped us produce “correct” software by eliminating some of the bugs that we would have found later.

JSON encoding/decoding is a hard problem in Elm in general. In most of the cases we know the shape of the API response we are going to get.

In the case of APISnapshot we do not know the shape of the JSON response we will get. Because of that it took us a bit longer to build the application. However, this forced us to really dig deep into JSON encoding/decoding issue in Elm and we learned a lot.

We would like to thank all the contributors to the project. Special shout out goes to Jasim for the discussion and the initial work on parsing the JSON file.

We like the combination of Elm and Ruby on Rails so much so that we are building AceHelp using the same technologies. AceHelp is open source from day one.

We are writing a lot of automation tests using Puppeteer.

Since puppeteer scripts execute so fast certain tests fail when they should be passing. Debugging those tests can be a challenge.

Chrome devtools comes with filmstrip feature. In “Performance” tab we can see screenshots of the site as they change over time.

We wanted puppeteer to generate similar filmstrip so that we can visually identify the source of the problem.

It turns out that puppeteer makes it very easy. Here is the full code.

import puppeteer from 'puppeteer';

(async () => {
  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();
  await page.setViewport({ width: 1280, height: 1024 });

  await page.tracing.start({path: 'trace.json', screenshots: true});

  await page.goto('https://www.bigbinary.com');
  await Promise.all([
    page.waitForNavigation(),
    page.click('#navbar > ul > li:nth-child(1) > a')
  ]);

  await page.tracing.stop();

  await browser.close();
})();

If we execute this script then it will generate a file called trace.json. This file has images embedded in it which are base64 encoded.

To see the the filmstrip drag the trace.json file to “Performance” tab in the Chrome devtool. Here is a quick video explaining this.

This blog is part of our Ruby 2.5 series.

irb> def greet
irb>   yield
irb> end
  => :greet
irb>
irb> def greet_with_welcome(&block)
irb>   puts 'Welcome'
irb>   greet(&block)
irb> end
  => :greet_with_welcome
irb> greet_with_welcome { p 'BigBinary' }
Welcome
"BigBinary"
  => "BigBinary"

In Ruby 2.4 when we pass a block to a method, which further passes to another method, Ruby creates a new Proc object by the given block before passing this proc to the another method.

This creates unnecessary objects even when the block parameter is not accessed. It also creates a chain of Proc objects when the block parameter is passed through various methods.

Proc creation is one a heavyweight operation because we need to store all local variables (represented by Env objects in MRI internal) in the heap.

Ruby 2.5 introduced a lazy proc allocation. Ruby 2.5 will not create a Proc object when passing a block to another method. Insead, it will pass the block information. If the block is accessed somewhere else, then it creates a Proc object by the given block.

This results in lesser memory allocation and faster execution.

Ruby 2.4

irb> require 'benchmark'
  => true
irb> def greet
irb>   yield
irb> end
  => :greet
irb>
irb> def greet_with_welcome(&block)
irb>   puts 'Welcome'
irb>   greet(&block)
irb> end
  => :greet_with_welcome
irb>
irb> Benchmark.measure { 1000.times { greet_with_welcome { 'BigBinary' } } }
Welcome
Welcome
...
...
...
  => #<Benchmark::Tms:0x007fe6ab929de0 @label="", @real=0.022295999999187188, @cstime=0.0, @cutime=0.0, @stime=0.01, @utime=0.0, @total=0.01>

Ruby 2.5

irb> require 'benchmark'
  => true
irb> def greet
irb>   yield
irb> end
  => :greet
irb>
irb> def greet_with_welcome(&block)
irb>   puts 'Welcome'
irb>   greet(&block)
irb> end
  => :greet_with_welcome
irb>
  irb> Benchmark.measure { 1000.times { greet_with_welcome { 'BigBinary' } } }
Welcome
Welcome
...
...
...
  => #<Benchmark::Tms:0x00007fa4400871b8 @label="", @real=0.004612999997334555, @cstime=0.0, @cutime=0.0, @stime=0.001524000000000001, @utime=0.0030690000000000023, @total=0.004593000000000003>

As we can see, there is considerable improvement in execution time when a block param is passed in Ruby 2.5.

Here is the relevant commit and discussion.

Prior to Rails 5.2, MySQL and PostgreSQL adapters had select_value, select_values & select_rows select_{value,values,rows} methods. They improve the performance by not instantiating ActiveRecord::Result.

However these methods broke query caching of ActiveRecord::FinderMethods#exists? method. Let’s check the issue.

>> User.cache do
>>   2.times { User.exists?(1) }
>> end

User Exists (2.1ms)  SELECT  1 AS one FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]

User Exists (2ms)  SELECT  1 AS one FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]

As we can see, query was not cached and sql was executed second time.

From Rails 5.2, MySQL and PostgreSQL adapters are no longer override select_{value,values,rows} methods which fix this query caching issue.

Also, the performance improvement provided by these methods was marginal and not a hotspot in Active Record, so this change was accepted.

Let’s check query caching of ActiveRecord::FinderMethods#exists? after the change.

>> User.cache do
>>   2.times { User.exists?(1) }
>> end

User Exists (2.1ms)  SELECT  1 AS one FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]

CACHE User Exists (0.0ms)  SELECT  1 AS one FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]

Now, query has been cached as expected.

This change has been backported in rails 5.1 from version 5.1.2 as well.

Recently, we faced a DDoS attack in one of the clients’ projects. There were many requests from different IPs to root and login paths, and we were running thrice the usual number of servers to keep the system alive.

We are using Cloudflare’s HTTP proxy and it was doing a great job preventing malicious requests, but we wanted to check if we can avoid the loading/captcha pages which Cloudflare uses to filter requests. We came to a conclusion that we would be able to mitigate the ongoing attack if we could throttle requests by IP.

Cloudflare has an inbuilt Rate Limiting feature to throttle requests, but it would be a little expensive in our case since Cloudflare charges by the number of good requests and it was a high traffic website. On further analysis, we found that throttling at application level would be enough in that situation and the gem Rack::Attack helped us with that.

Rack::Attack is a Rack middleware from Kickstarter. It can be configured to throttle requests based on IP or any other parameter.

To use Rack::Attack, include the gem in Gemfile.

gem "rack-attack"

After bundle install, configure the middleware in config/application.rb:

config.middleware.use Rack::Attack

Now we can create the initializer config/initializers/rack_attack.rb to configure Rack::Attack.

By default, Rack::Attack uses Rails.cache to store requests information. In our case, we wanted a separate cache for Rack::Attack and it was configured as follows.

redis_client = Redis.connect(url: ENV["REDIS_URL"])
Rack::Attack.cache.store = Rack::Attack::StoreProxy::RedisStoreProxy.new(redis_client)

If the web server is behind a proxy like Cloudflare, we have to configure a method to fetch the correct remote_ip address. Otherwise, it would block based on proxy’s IP address and would result in blocking a lot of legit requests.

class Rack::Attack
  class Request < ::Rack::Request
    def remote_ip
      # Cloudflare stores remote IP in CF_CONNECTING_IP header
      @remote_ip ||= (env['HTTP_CF_CONNECTING_IP'] ||
                      env['action_dispatch.remote_ip'] ||
                      ip).to_s
    end
  end
end

Requests can be throttled based on IP address or any other parameter. In the following example, we are setting a limit of 40rpm/IP for “/” path.

class Rack::Attack
  throttle("req/ip", :limit => 40, :period => 1.minute) do |req|
    req.remote_ip if req.path == "/"
  end
end

The downside of this configuration is that after the 1 minute period, the attacker can launch another 40 requests/IP simultaneously and it would exert pressure on the servers. This can be solved using exponential backoff.

class Rack::Attack
  # Exponential backoff for all requests to "/" path
  #
  # Allows 240 requests/IP in ~8 minutes
  #        480 requests/IP in ~1 hour
  #        960 requests/IP in ~8 hours (~2,880 requests/day)
  (3..5).each do |level|
    throttle("req/ip/#{level}",
               :limit => (30 * (2 ** level)),
               :period => (0.9 * (8 ** level)).to_i.seconds) do |req|
      req.remote_ip if req.path == "/"
    end
  end
end

If we want to turn off throttling for some IPs (Eg.: Health check services), then those IPs can be safelisted.

class Rack::Attack
  class Request < ::Rack::Request
    def allowed_ip?
      allowed_ips = ["127.0.0.1", "::1"]
      allowed_ips.include?(remote_ip)
    end
  end

  safelist('allow from localhost') do |req|
    req.allowed_ip?
  end
end

We can log blocked requests separately and this is helpful for analyzing the attack.

ActiveSupport::Notifications.subscribe('rack.attack') do |name, start, finish, request_id, req|
  if req.env["rack.attack.match_type"] == :throttle
    request_headers = { "CF-RAY" => req.env["HTTP_CF_RAY"],
                        "X-Amzn-Trace-Id" => req.env["HTTP_X_AMZN_TRACE_ID"] }

    Rails.logger.info "[Rack::Attack][Blocked]" <<
                      "remote_ip: \"#{req.remote_ip}\"," <<
                      "path: \"#{req.path}\", " <<
                      "headers: #{request_headers.inspect}"
  end
end

A sample initializer with these configurations can be downloaded from here.

Application will now throttle requests and will respond with HTTP 429 Too Many Requests response for the throttled requests.

We now block a lot of malicious requests using Rack::Attack. Here’s a graph with % of blocked requests over a week.

EDIT: Updated the post to add more context to the situation.