Make Outbound Calls From the Browser to Phone using Twilio and Rails

In this blog post we will see how to make outbound phone calls from the browser to a phone using Twilio . We will make use of the Twilio-JS library and Twilio-Ruby gem.

The Rails App we will be creating, is based on the Twilio Client Quick-start tutorial. That Twilio tutorial makes use of Sinatra. We will see how we can achieve this in a Rails application.

Step 1. Setup Twilio Credentials and TwiML App

We need to setup twilio credentials. We can find account ID and auth token from our account information.

When the call is made using browser then the phone that is receiving the call has to see a number from which the call is coming. So now we need to setup a Twilio verified number. This number will be used to place the outgoing calls from. How to setup a verified number can be found here.

When our app make a call from the browser using twilio-js client, Twilio first creates a new call connection from our Browser to Twilio. It then sends a request back to our server to get information about what to do next. We can respond by asking twilio to call a number, say something to the person after a call is connected, record a call etc.

Sending of this instructions is controlled by setting up a TwiML application. This application provides information about the end point on our server, where twilio should send the request to fetch instructions. TwiML is a set of instructions, that we can use to tell Twilio what to do in different cases like when an outbound phone call is made or when an inbound SMS message is received.

Given below is an example that will say a short message How are you today? in a call.

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Say voice="woman">How are you today?</Say>
</Response>

The TwiML app can be created here. Once the app is configured then we will get appsid.

We need to configure following information in our Rails Application:

twilio:
  verified_number: <%= ENV['TWILIO_VERIFIED_NUMBER']%>
  account_sid: <%= ENV['TWILIO_ACCOUNT_SID'] %>
  auth_token: <%= ENV['TWILIO_AUTH_TOKEN'] %>
  call_app_sid: <%= ENV['TWILIO_CALL_APP_SID'] %>

Step 2. Generate capability token to be used by twilio-js

After we have the config setup, we will proceed to create the capability token. This token will be generated using the ruby gem, and passed to the javascript SDK. The token helps the twilio-js client determine, what permissions the application has like making calls, accepting calls, sending SMS, etc.

We define a TwilioTokenGeneratorService for this purpose.

class TwilioTokenGeneratorService

  def process
    capability = twilio_capability()
    capability.generate
  end

  private

  def twilio_capability
    capability ||= Twilio::Util::Capability.new Settings.twilio.account_sid, Settings.twilio.auth_token
    capability.allow_client_outgoing Settings.twilio.call_app_sid
    capability
  end

end

As you can see, we first define a new Twilio::Util::Capability instance and pass credentials to it. We then call allow_client_outgoing method and pass the client Sid to it. This is the identifier for the TwiML application we have previously created on Twilio. Calling allow_client_outgoing gives permission to the client to make outbound calls from Twilio. Finally we call the generate method to create a token from the capability object.

Step 3. Define view elements and pass token to it

The generated token will now be passed to the Twilio JS client for connecting with Twilio. In our App we define CallController, and index action in this controller. This action takes care of setting the capability token. Our index view consists of two buttons- to place and hangup a call, a number input field, call logs, and data field to pass capability token to the javascript bindings. We import the Twilio-JS library in the view. The css styling being used is from the Twilio example.

<div id="twilioToken" data-token="<%= @twilio_token %>"></div>

<button id="caller" class="call">Call</button>
<button id="hangup" class="hangup">Hangup</button>

<input type="text" id="phoneNumber" placeholder="Enter a phone number to call"/>

<div id="log">Loading pigeons...</div>

<script type="text/javascript" src="//static.twilio.com/libs/twiliojs/1.2/twilio.min.js"></script>

Step 4. Define coffeescript bindings to handle TwilioDevice connection to Twilio

Next we setup coffeescript bindings to handle initialization of TwilioDevice and making use of entered number to place calls Twilio. We are taking care of various events like connect, disconnect, ready, etc. on TwilioDevice instance. More information about TwilioDevice usage can be found here.

class TwilioDevice
  constructor: ->
    @initTwilioDeviceBindings()
    @initFormBindings()

  initTwilioDeviceBindings: ->
    twilio_token = $('#twilioToken').data('token')
    twilio_device = Twilio.Device

    # Create the Client with a Capability Token
    twilio_device.setup(twilio_token, {debug: true});

    #/* Let us know when the client is ready. */
    twilio_device.ready ->
      $("#log").text("Ready")

    #/* Report any errors on the screen */
    twilio_device.error (error) ->
      $("#log").text("Error: " + error.message)

    #/* Log a message when a call connects. */
    twilio_device.connect (conn) ->
      $("#log").text("Successfully established call")

    #/* Log a message when a call disconnects. */
    twilio_device.disconnect (conn) ->
      $("#log").text("Call ended")

  initFormBindings: ->
    $('#caller').bind "click", (event) ->
      params = {"phone_number": $('#phoneNumber').val()}
      Twilio.Device.connect(params)

    $('#hangup').bind "click", (event) ->
      Twilio.Device.disconnectAll()

$ ->
  new TwilioDevice()

If we now load this page, we should be able to see our app saying its ready to take calls.

Twilio Rails App Ready

Step 5. Define TwiML Response Generator Service

The final step before we place calls from our App is to handle callbacks from Twilio and return TwiML response. For this we are going to define TwilioCallTwiMLGeneratorService which takes care of generating this response. More information about how we need to define the response and individual fields can be found from Twilio’s docs.

What we need to define is a response as below:

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Dial callerId="+15005550000">
    <Number>+15005550001</Number>
  </Dial>
</Response>

We are making use of two elements here - Dial, which makes Twilio place a call using defined callerId value, as the number from which the call is made, which is displayed on the callee’s phone. Note that this is the same verified number that we had specified before. Then we specify Number which is the number, to which we want to place the call to. This number is passed first by the javascript client to Twilio, and then back to our application by Twilio, which we use to generate the response as above.

We define our TwilioCallTwiMLGeneratorService to take in a phone number as parameter. It creates an instance of Twilio::TwiML::Response and tapping on this instance we provide Dial element with a :callerId value, and the Number to place the call to. We validate the number before passing it back, and return an error if the number is invalid.

class TwilioCallTwiMLGeneratorService
  attr_reader :phone_number
  VALID_PHONE_NUMBER_REGEX = /^[\d\+\-\(\) ]+$/ # Matches valid phone numbers acceptable to Twilio

  def initialize phone_number
    @phone_number = phone_number
  end

  def process
    Twilio::TwiML::Response.new do |r|
      if VALID_PHONE_NUMBER_REGEX.match(phone_number)
        r.Dial :callerId => Settings.twilio.verified_number do |d| # callerId is number from which call is made.
          d.Number(CGI::escapeHTML phone_number) # The number to call
        end
      else
        r.Error("Invalid number!")
      end
    end.text.strip
  end

end

Step 6. Send TwiML response on Twilio callback

We are now set to define twilio’s callback handler. This will be handled by the create_call action in CallController. Twilio will be sending this endpoint a POST request along with some information specified here. We make use of phone_number being passed to us by Twilio and pass it along to the TwilioCallTwiMLGeneratorService, which return us with valid TwiML response. Since TwiML is a flavour of XML, we make using render xml to return the response.

def create_call
  response_to_twilio_callback = TwilioCallTwiMLGeneratorService.new(call_params[:phone_number]).process
  render xml: response_to_twilio_callback
end

def call_params
  params.permit(:phone_number)
end

Finally we need to specify the callback url in our TwiML App on Twilio. For testing this locally, we can make use of a service like https://ngrok.com/, to expose this endpoint.

Twilio TwiML App setting

Our service is now ready to place calls. The complete Rails application code that we have created can be found here.

Happy calling everyone!

Tricks and Tips for using Fixtures effectively in Rails

Recently I gave a talk at RubyKaigi 2014 about Rails fixtures. In this blog post I will be discussing some of the tips and tricks for using fixtures effectively.

You can also the see the video of this talk here.

Don’t use ids.. unless required

In fixtures, we can specify id of a fixture.

john:
  id: 1
  email: john@example.com

I would recommend not to specify the id. Rails generates the id automatically if we don’t explicitly specify it. Moreover there are a few more advantages of not specifying the id.

Stable ids for every fixture.

Rails will generate the id based on key name. It will ensure that the id is unique for every fixture. It can also generate ids for uuid primary keys.

Labeled references for associations like belongs_to, has_many.

Lets say we have a users table. And a user has many cars.

Car ferrari belongs to john. So we have mentioned user_id as 1.

ferrari:
  name: ferrari
  make: 2014
  user_id: 1

When I’m looking at cars.yml I see user_id as 1. But now I to lookup to see which user has id as 1.

Here is another implementation.

john:
  name: john
  email: john@example.com

ferrari:
  name: ferrari
  make: 2014
  user: john

Notice that I no longer specify user_id for John. I have mentioned name. And now I can reference that name in cars.yml to mention that ferrari belongs to john.

How to set a value to nil from fixture

Let’s say that I have a boolean column which is false by default. But for an edge case, I want it to be nil. I can obviously mutate the data generated by fixture before testing. However I can achieve this in fixtures also.

Specify null to make the value nil
require 'yaml'
YAML.load "--- \n:private: null\n")
=> {:private=>nil}

As you can see above if the value is null then YAML will treat it as nil.

john:
  name: john
  email: john@example.com
  private: null
Leave the value blank to make the value nil
require 'yaml'
YAML.load "--- \n:private: \n")
=> {:private=>nil}

As you can see above if the value is blank then YAML will treat it as nil.

john:
  name: john
  email: john@example.com
  private:
When model name and table name does not match

Generally in Rails, the model name and table name follow a strict convetion. Post model will have table name posts. Using this convention, the fixture file for Post models is obviously fixtures/posts.yml.

But sometimes models do not match directly with the table name. This could be becuase of legacy reason or because of namespacing of models. In such cases automaatic detection of fixture files becomes difficult.

Rails provides set_fixture_class method for this purpose. This is a class method which accepts a hash where key should be name of the fixture or relative path to fixture file and value should be model class.

I can use this method inside test_helper.rb in any class inheriting from ActiveSupport::TestCase.

# test_helper.rb
class ActiveSupport::TestCase

  # table name is "morning_appts". It is being mapped to model "MorningAppointment".
  self.set_fixture_class morning_appts: MorningAppointment

  # in this case fixture is namespaced
  self.set_fixture_class '/legacy/users' => User

  # in this case the model is namespaced.
  self.set_fixture_class outdoor_games: Legacy::OutdoorGame
end
values interpolation using $LABEL

Rails provides many ways to keep our fixtures DRY. Label interpolation is one of them. It allows the use of key of fixture as a value in the fixture. For example:

john:
  name: john
  email: john@example.com

becomes:

john:
  name: $LABEL
  email: john@example.com

$LABEL is not a global variable here. Its just a placeholder. $LABEL is replaced by the key of the fixture. And as discussed earlier the key of the fixture in this case is john. So $LABLE has value john.

Before this PR, I could only use this feature if the value is exactly $LABEL. So if the email is john@example.com I could not use the $LABEL@example.com. But after this PR, I can $LABEL anywhere in the string, and Rails will replace it with the key.

So the earlier example becomes:

john:
  name: $LABEL
  email: $LABEL@example.com
YAML defaults

I use YAML defaults in database.yml for drying it up and keeping common configuration at one place.

defaults: &defaults
  adapter: postgresql
  encoding: utf8
  pool: 5
  host: localhost
  password:

development:
  <<: *defaults
  database: wheel_development

test:
  <<: *defaults
  database: wheel_test

production:
  <<: *defaults
  database: wheel_production

I can use it for drying up fixtures too for extracting common part in our fixtures.

defaults: &defaults
  company: BigBinary
  website: bigbinary.com
  blog: blog.bigbinary.com

john:
  <<: *defaults
  name: John Smith
  email: john@bigbinary.com

prathamesh:
  <<: *defaults
  name: Prathamesh Sonpatki
  email: prathamesh@bigbinary.com
Database specific tricks

Fixtures bypass the normal Active Record object creation process. After reading them from YAML file, they are inserted into database directly using insert query. So they skip callbacks and validations check. This also has an interesting side-effect which can be used for drying up fixtures.

Suppose we have fixture with timestamp:

john:
  name: John Smith
  email: john@example.com
  last_active_at: <%= Time.now %>

If I are using PostgreSQL, I can replace the last_active_at value with now:

john:
  name: John Smith
  email: john@example.com
  last_active_at: now

now is not a keyword here. It is just a string. The actual query looks like this:

INSERT INTO "users"
("name", "email", "last_active_at", "id")
VALUES
('John Smith', 'john@example.com', 'now',1144934)

So the value for last_active_at is still just now when the query is executed.

The magic starts as PostgreSQL starts reading the values. now is a shorthand for the current timestamp . As soon as Postgres reads it, it replaces now with the current timestamp and the column last_active_at gets populated with current timestamp.

I can also use the now() function instead of just now.

This function is available in PostgreSQL as well as MySQL. So the usage of now() works in both of these databases.

Selenium IDE - How I reduced the running time from 58 minutes to 15 minutes

I wrote a bunch of selenium tests using Selenium IDE for a project. The selenium tests have proven to be very useful. However the tests take around 58 minutes to complete the full run.

Here are the specific steps I took which brought the running time to under 15 minutes.

Set to run at the maximum speed

Command Target Value
setSpeed 0

setSpeed command takes Target value in milliseconds. By setting the value to zero, I set the speed to maximum and the tests indeed ran fast. However, now I had lots of tests failing which were previously passing.

What happened.

In our tests real firefox browser is fired up and real elements are clicked. The application does make round trip to the rails server hosted on heroku.

By setting the selenium tests to the maximum speed the selenium tests started asserting for elements on the page even before the pages were fully loaded by the browser.

I needed sets of instructions using which I could tell selenium how long to wait for before asserting for elements.

Selenium provides a wonderful suite of commands which helped me fine tune the test run. Here I’m discussing some of those commands.

waitForVisible

This command is used to tell selenium to wait until the specified element is visible on the page.

In the below mentioned case, the Selenium IDE will wait untill the element css=#text-a is visible on the page.

Command Target Value
waitForVisible css=#text-a

waitForText

This command is used to tell selenium to wait until a particular text is visible in the specified element.

In the case mentioned below, Selenium IDE will wait until the text violet is displayed in the element css=#text-a.

Command Target Value
waitForText css=#text-a violet


The difference between waitForVisible and waitForText is that waitForVisible waits until the specified element is visible on the page while waitForText waits until a particular text is visible in the specified element on the page.

waitForElementPresent

This command is used to tell Selenium to wait until the specified element is displayed on the page.

In the below mentioned case, the Selenium IDE will wait until the element css=a.button is displayed on the page.

Command Target Value
waitForElementPresent css=a.button

waitForVisible and waitForElementPresent seem very similar. It seems both of these commands do the same thing. There is a subtle difference though.

waitForVisible waits until the specified element is visible. Visibility of an element is manipulated by the settings of CSS properties. For example using display none; one can make an element not be visible at all.

In contrast the command waitForElementPresent waits until the specified element is present on the page in the form of html markup. This command does not give consideration to css setttings.

refreshAndWait

This command is used to tell Selenium to wait until the page is refreshed and the targeted element is displayed on the web page.

In the example mentioned below, the Selenium IDE will wait until the page is refreshed and the targeted element css=span.button is displayed on the page.

Command Target Value
refreshAndWait css=span.button

clickAndWait

This command is used to tell selenium to wait until a particular button is clicked for submitting the form and the page starts reloading. The subsequent commands are paused until, the page is reloaded after the element is clicked on the page.

In the case mentioned below, Selenium IDE will wait until the page is reloaded after the specified element css=input#edit is clicked.

Command Target Value
clickAndWait css=input#edit

Selenium IDE commands used above and more are available at Selenium documentation .

DeccanRubyConf 2014. Hou De!

deccanrubyconflogo

I attended it, enjoyed it and took part as a volunteer in Pune’s first RubyConf - DeccanRubyConf 2014. As Hou De (let it be) name said, event went in the same way.

The day before conference, in the morning Vipul (one of the event organizers) and I picked up our guest speaker Koichi Sasada from the Pune Airport. Koichi is a Ruby core member and works for Heroku. We welcomed him and went to Hyatt Regency hotel where the event is taking place. Our guest checked into hotel and then we decided to go for a lunch at Malaka Spice restaurant.

We reached there and Koichi told us that he wanted non-spicy food (Safe food). We ordered non spicy food, but food was still too spicy for Koichi. However we enjoyed the food and had very good discussion about the Ruby internals, concurrency-parallelism, debugging in Ruby, Japanese culture and the Indian culture.

After lunch, we dropped Koichi off at the hotel and we left for our home.

Next morning was the event day and I woke up early and went to the event place. As part of volunteering team, I and other volunteers had tasks like giving pens, badges, stickers, T-shirts and coupon for night party to the attendees.

Attendees had started to come in slowly. Some attendees asked me about T-shirt size as I wore one of the conference T-shirts and from my T-shirt’s size they decided their T-shirt size. It was a great experience meeting with different kind of people from around the India.

Keynote by Koichi kicked the event off and he talked about Ruby 2.1 features like,

  • Required keyword parameter,
  • Rational number literal,
  • def returns symbol of method name
  • Runtime new features (String#scrub, Binding#local_time and etc.)

Then he talked about performance improvements, Ruby 2.2 and how to speed up Ruby Interpreter . Click here for more details about his talk.

koichi

attendees

In between the talks, some new attendees had come for the conference who had not registered for the conference. They told me that they thought it’s a regular Pune’s local Ruby Meetup. There was some miss-understandings but they seemed interested in attending the event. I contacted Gautam as he was one of the organizers and told him about the issue.

Attendees kept coming till the afternoon.

After Koichi’s talk, two sections had opened. One for talks and other for the workshops. TDD workshop was conducted by Sidu Ponnappa. I saw lots of attendees in this workshop and heard that it went very well.

tddworkshop

The next talk was on Requiem for a dream by Arnab Deka. He talked about various tips and tricks including using “Higher order functions and Concurrency” in Ruby and other programming languages like Clojure and Elixir.

After that, Rishi Jain talked about Game Development - The Ruby Way. He discussed how to build a game in Ruby using Gosu library . It was a very useful session for game developers. You can find out more about it here

Next talk was on Programming Ruby in Marathi by Ratnadeep Deshmane & his friend Aniket Awati. This was one of the best talks of the event. The way they used the similar words from Marathi for Ruby’s keywords and the examples made this talk remarkable. Their presentation style was nice too. Almost all attendees enjoyed this talk and they laughed a lot.

After this talk there was tea break for 15 minutes. Staffs from the Hyatt hotel were very helpful. There were serving tea and coffees to the attendees and overall did a good job of ensuring the the event cruised along smoothly. This is in sharp contrast to the service RubyConfIndia received from Lalit Resort.

After tea break, I didn’t get chance to attend other talks as attendees were still coming in and I was assisting them. But I heard almost all talks went very well.

In meantime, I was passing through main passage and I saw lighting talks board and decided to give lighting talk on my Ruby gem. Lighting talks is a short presentation that you can give about your achievement. You can also share your ideas and promote your library or any other projects.

Then we all had our lunch, lunch was good with lots of varieties with dessert.

After lunch I went for workshop on Deliver projects 30% faster, know your CSS by Aakash Dharmadhikari. Wanted to attend it fully, but some of the attendees had difficulties in internet connection, so I left the room to look into it.

Lightning talks were going to start so I took sometime to prepare for my presentation.

In the lighting talks, girls from Rails Girls Summer of Code, talked about their project and their progress on it. After that Prathamesh talked about RubyIndia.org and asked people to subscribe the newsletter. After that I gave talk on my Ruby gem RubySimpleSearch and you can find more on it here. The next speaker Rahul Mahale from Nashik asked people to help him in growing Ruby community in Nashik. All other lighting talks went very well.

After lighting talks, there was closing keynote On Solving Problems by Baishampayan Ghose. This talk made us think about how we write application in our daily routine. He talked about the architecture and he also explained the future is a function of past future = f(past). He also suggested that we should first understand the problem thorougly and we should build the software. Talk was very informative and went very well.

After that Gautam came on to the stage and congratulated all the sponsors, organizers and volunteers. He also told that this event got large number of girls attendees than he had ever seen in any other conference.

After this event, there was futsal and party in Irish Village hotel. Me and my friends, we all went to the party. Party was superb and we all enjoyed it.

Thanks to all sponsors and organizers who made this event fun and enjoyable.

You can checkout more pics of the conference from here.

Note: Photos are copyrighted by respective photo owners.

Flash access changes due to new serialization mechanism in Rails 4.1

Prior to upgrading to Rails 4.1 we had a helper to display flash messages and to add css class to the message based on flash type. Here is the code.

module FlashHelper
  ALERT_TYPES = [:success, :info, :warning, :danger]

  def bootstrap_flash
    flash_messages = []
    flash.each do |type, message|
      next if message.blank?

      type = :success if type == :notice
      type = :danger  if type == :alert
      type = :danger  if type == :error

      next unless ALERT_TYPES.include?(type)

      ....
      Array(message).each do |msg|
        text = content_tag(:div, msg.html_safe, :class => "alert fade in alert-#{type} ")
        flash_messages << text if msg
      end
    end
    flash_messages.join("\n").html_safe
  end
end

After upgrading to Rails 4.1, we started using the new Cookies serializer. Following code was added to an initializer.

Rails.application.config.action_dispatch.cookies_serializer = :json

Soon after this our flash helper started misbehaving and all flash messages disappeared from the application.

JSON Cookies Serializer

Before we move ahead, a word on the new JSON Cookies Serializer. Applications created before Rails 4.1 uses Marshal to serialize cookie values into the signed and encrypted cookie jars.

Commits like this and this made it possible to have Cookies serializer and defaulted from Marshal Serializer to a secure Serializer using JSON.

The JSON Serializer works on JSON objects. Thus objects like Date and Time will be stored as strings. Hash keys will be stored as strings.

JSON serizlizer makes the application much safer since it is safer to pass around strings compare to passing around arbitrary values which is what was happens when values are marshalled and passed around.

Coming back to our problem, change Stringify the incoming hash in FlashHash coupled with above serialization changes meant that even if we put a symbol as a key in the flash we have to retrieve it as “string” since the keys are internally being converted into strings.

The difference is clearly illustrated below.

flash["string"] = "a string"
flash[:symbol] = "a symbol"

# Rails < 4.1
flash.keys # => ["string", :symbol]

# Rails >= 4.1
flash.keys # => ["string", "symbol"]

Solution

Now that we know the root cause of the problem the fix was simple. Instead of relying on symbols use “string” to access value from flash.

module BootstrapFlashHelper
  ALERT_TYPES = ['success', 'info', 'warning', 'danger']

  def bootstrap_flash
    flash_messages = []
    flash.each do |type, message|
      compare_type = type.to_s # Notice the stringifying of keys here, to make this work even with symbols.
      next if message.blank?

      compare_type = 'success' if compare_type == 'notice'
      compare_type = 'danger'  if compare_type == 'alert'
      compare_type = 'danger'  if compare_type == 'error'

      next unless ALERT_TYPES.include?(compare_type)

      Array(message).each do |msg|
        text = content_tag(:div, msg.html_safe, :class => "alert fade in alert-#{compare_type} ")
        flash_messages << text if msg
      end
    end
    flash_messages.join("\n").html_safe
  end
end

Issue with Delayed Job lifecyle and Postgres Errors

The Problem

Recently in one of our projects, we experienced some strange errors from Delayed::Job. Delayed::Job workers started successfully, but when they were starting to lock the jobs, workers failed with PG::Error: no connection to server or PG::Error: FATAL: invalid frontend message type 60errors.

After some search, we found there had been such issues already experienced by others, but without much success or help, everyone pointing out issues with Serialization, or lifecyle hooks.

We started isolating the problem and digging through the recent changes we had made to the project. Since the last release the only significant modification had been made to internationalization. We had started using I18n-active_record backend to grant privilege to modify translations not only to developers but also to users in general. In this change we had added loading of I18n config as below

require 'i18n/backend/active_record'
Translation  = I18n::Backend::ActiveRecord::Translation

 if (ActiveRecord::Base.connected? && Translation.table_exists?) or in_delayed_job_process?
    I18n.backend = I18n::Backend::ActiveRecord.new
    I18n::Backend::ActiveRecord.send(:include, I18n::Backend::Memoize)
    I18n::Backend::ActiveRecord.send(:include, I18n::Backend::Flatten)
    I18n::Backend::Simple.send(:include, I18n::Backend::Memoize)
    I18n::Backend::Simple.send(:include, I18n::Backend::Pluralization)
    I18n.backend = I18n::Backend::Chain.new(I18n::Backend::Simple.new, I18n.backend)
  end

for Delayed Job we had extra check as

def in_delayed_job_process?
  executable_name = File.basename $0
  arguments = $*
  rake_args_regex = /\Ajobs:/
  (executable_name == 'delayed_job') || (executable_name == 'rake' && arguments.find{ |v| v =~ rake_args_regex })
end

After some serious searching and digging through both Delayed::Job source code and how we were using to setup its config, we started noticing some issues.

The first thing we found was that the problem did not turn up when delayed job workers were started using rake jobs:work task.

After looking at DelayedJob internals we found that the main difference between a rake task and a binstub was in the fork method that was invoked in the binstub version. The binstub version was being executed seamlessly using Daemons#run_process method and had a slightly different lifecycle of execution.

DelayedJob lifecycle

Let’s take a look into DelayedJob internals before proceeding. DelayedJob has systems of the hooks that can be used by plugin-writers and in our applications. All this events functionality is hidden in Delayed::Lifecycle class. Each worker has its own instance of that class.

So, which events exactly do we have here?

Job-related events:

:enqueue
:perform
:error
:failure
:invoke_job

Worker-related events:

:execute
:loop
:perform
:error
:failure

You can setup callbacks to be run on before, after or around events simply using Delayed::Worker.lifecycle.before, Delayed::Worker.lifecycle.after and Delayed::Worker.lifecycle.around methods.

The Solution

Let’s move on to our problem. It turned out that delayed job active record gem was closing all database connections in before_fork hook and reestablishing them in after_fork hook. It was clear that I18n-active-record did not play well with this, causing the issue at hand.

We looked into DelayedJob lifecycle and chose before :execute hook, which was executed after all DelayedJob ActiveRecord backend connections manipulations.

Finally the locales initializer for delayed_job workers was changed to match as below:

require 'i18n/backend/active_record'
Translation  = I18n::Backend::ActiveRecord::Translation

Delayed::Worker.lifecycle.before :execute do
  if (ActiveRecord::Base.connected? && Translation.table_exists?) || in_delayed_job_process?
    I18n.backend = I18n::Backend::ActiveRecord.new

    I18n::Backend::ActiveRecord.send(:include, I18n::Backend::Memoize)
    I18n::Backend::ActiveRecord.send(:include, I18n::Backend::Flatten)
    I18n::Backend::Simple.send(:include, I18n::Backend::Memoize)
    I18n::Backend::Simple.send(:include, I18n::Backend::Pluralization)

    I18n.backend = I18n::Backend::Chain.new(I18n::Backend::Simple.new, I18n.backend)
  end
end

This helped us to mitigate the connection errors, and connections stopped dying abruptly.

RedDotRubyConf 2014

reddotrubyconf2014_logo

I and Vipul recently gave a talk at RedDotRubyConf on ActiveRecord can’t do it? Arel can!. It was our first trip to Singapore and we enjoyed the conference as well as Singapore a lot.

T.L.D.R.

RDRC2014 was awesome :)

Day 1

We reached the venue in time for Koichi’s keynote on Ruby.Inspect. He talked about various things related to development of Ruby including Ruby team at Heroku, recent releases of Ruby and new syntax introduced in Ruby 2.1. He also talked about performance improvements including Generational GC - RGenGC and upcoming features in Ruby 2.2.

After that in second part of the talk, he talked about inspection tools available in Ruby. It was a deep technical part for me and something to learn about. The message he gave from the talk was to become low level engineer.

Second talk of the conf was from T.J. Schuck about solving one of the hardest problems. Storing and retrieving passwords in a secure way. He talked about how increasing improvements in hardware pose a challenge as even if you use proper algorithm it can be cracked with high computing machines. It was interesting to know about internals of storing passwords. I had never cared too much about it :)

After the coffee break, Brandon Keepers from Github gave talk on Tending Your Open Source Garden. Github is still on Rails 2.3 and Brandon is working on bringing it up to new version. His talk was a great advice for those who want to contribute to open source and community. I think this talk resonated well with the audience as most of the crowd was new and interested in open source contributions.

Gautam Rege from Josh Software gave talk on Dark Side of Ruby. We had attended this talk at GCRC so we left the hall after some time and did our one last practice. But i heard the feedback was very well for this talk.

After the lunch, Keith Pitt talked about Guide to Continuous Deployment with Rails. He talked about keeping everything related to deployment from CI to migrations in sync. One of the interesting thing that i came to know from this talk was how to enable zero downtime deployments on Heroku using preboot feature.

Benjamin Tan gave talk on Ruby + Elixir: Polyglotting FTW! after that. He talked about Elixir. This talk was about looking beyond Ruby and adding another tool to our skills. Benjamin also gave some demos including the last one in which he used sidekiq with Elixir. The actual work was done by Elixir workers. I will definitely give a shot to Elixir in the coming days.

After that we gave our talk on Arel. I was a bit nervous as it was my first talk. But it went well. We finished a bit early than expected. But there was tea break after our talk :). We got some good feedback from the attendees and especially beginners who had not used Arel before. Our slides are here

After the tea break, lightning talks started. First talk was by Hiroshi Shibata about how anyone can contribute to Ruby to make it better. He talked about how to submit issues, feature requests using Redmine. After that William Notowidagdo gave talk on Building REST API using Grape. With Rails and Rails API gem it has become easy to generate an API. But we have grape also as a lightweight tool. Sayani Basu talked about how to make a podcast with Jekyll and other tools in 5 minutes.

We are planning to start a podcast here in India about Ruby community so it was good to know about it.

Update: Checkout our podcast .

After these awesome lightning talks, the last session of Day 1 started. There were talks on Fluentd and Domain driven design. Both were good to know as something outside of daily routine. Konstantin Haase’s last talk of the day was Meta talk. He talked on abstraction and how it happens in our mind. Our mind affects what we see, like we see magenta color. Similarly abstraction happens in mind. I had to concentrate a lot in this talk to understand it. But it was worth it.

Andddd that ended the first day of the conf. It was exciting and we were looking forward to second day.

Day 2

Day 2 started with Brian Helmkamp’s talk on Docker. We missed the initial part of the talk. He talked about basics of Docker, how to deploying in container environment. He also discussed about deploying a Rails app using docker and how it makes very easy to deploy different parts of the system using docker very easily.

Zachary Scott gave next talk introducing Ruby Core team and how it works, how it collaborates, developer meetings, how anyone can contribute to MRI. We also had a Friday hug in this talk :) This talk combined with Hiroshi’s lightning talk on the first day was great insight into CRuby development.

After the break, Pioter Solnica gave an excellent talk on Convenience vs Simplicity. He talked about convenience offered by ActiveRecord may not be simple to understand. The things such as input conversion, validation are convenient to use as a developer but not necessarily simple to understand. He also discussed presenters, immutable data structures, Adamantium for creating immutable objects in Ruby. In the second part of the talk, he talked about relations and how they can be used in composing queries. He exaplained this idea using Ruby Object Mapper. It uses Axiom as underlying relational algebra instead of Arel. Its an interesting project to checkout.

After that our very own Anil Wadghule talked on Solid Design Principles in Ruby. His emphasis was on following designs than patterns. He also showed code examples and refactored them after applying principles. His talk was good insight into understanding what are these Solid principles and how they can be applied in real life.

We skipped the session after lunch and roamed around talking with people. We had an interesting discussion about hiring Ruby on Rails developers, interview processes etc.

Then lightning talks started. Sheng-Loong-Su talked first on using Algorithms for Trading. He talked about collecting data using feeder, preparing trading signals using strategy and making decision based on trading signals. One of the best talks of day 2 was by Grzegorz Witek on how he is traveling the world without getting burned out and still happily programming. He talked about his experiences in different countries being a Nomadic Programmer. It was one of the best inspirational talks according to me. The last lightning talk was about Using Vagrant for setting up Dev environment by Shuwei and Arathi.

Nicholas Simmons talked on experience of building a Single page web app and back again to normal app. He gave real life metrics from Shopify and showed problems faced with single page apps, batman.js and how moving back to a normal app helped them.

Then chocolate man from Belgium, Christophe Philemotte, gave talk on Safety Nets: Learn to code with confidence. His talk was about how we can prevent code in long term using testing, static analysis using tools such as flog, flay, rubocop for removing duplication, reducing complexity, fixing warnings. He also talked about importance of code review. His code is present here. He also gave us excellent chocolates from Belgium.

And the last keynote by Aaron Patterson. As always, it was full of everything - tech stuff, jokes, puns.

He talked on how he is making performance improvements in Active Record, link generations. He showed some graphs with performance of various database adapters tested on Rails versions ranging from 2.3 to 4 to master. He urged everyone to report performance issues to the core team so that they are addressed quickly. This is the app used for doing performance testing by him.

And that ended talks at RDRC. We had an awesome after party where we discussed with lots of people about Ruby, Rails as well as other stuff. We would like to thank Winston for inviting us to RedDotRubyConf.

Checkout some of the pics from the conference here by Nestor.

I am already looking at RDRC 2015. :-)