This is a story of how we found and fixed memory leak in
We use shoryuken to process SQS messages inside of docker containers. A while
back we noticed that memory was growing without bound. After every few days, we
had to restart all the docker containers as a temporary workaround.
Since the workers were inside of a docker container we had limited tools.
So we went ahead with the UNIX way of investigating the issue.
First we noticed that the number of threads inside the worker was high, 115 in
our case. shoryuken boots up all the worker threads at
The proc filesystem exposes a lot of useful information of all the running
processes. ``/proc/[pid]/task` directory has information about all the threads
of a process.
Some of the threads with lower ID’s were executing syscall 23
These threads were waiting
for a message to arrive in the SQS queue, but most of the threads were
executing syscall 202
At this point we had an idea about the root cause of the memory leak -
it was due to the worker
starting a lot of threads which were not getting terminated.
We wanted to know
how and when these threads are started.
Ruby 2.0.0 introduced tracepoint,
which provides an interface to a lot of
internal ruby events like when a exception is raised, when a method is called or
when a method returns, etc.
We added the following code to our workers.
Executing the ruby workers with tracing enabled revealed that a new
Celluloid::Thread was being created before each method was processed and that
thread was never terminated. Hence the number of zombie threads in the worker
was growing with the number messages processed.
Unfortunately tracepoint didn’t pinpoint the place where the thread was started,
hence we added a couple of puts statements to investigate the issue futher.
After a lot of debugging, we were able to find that a new thread was started to
increase the visibility time
of the SQS message in a shoryuken middleware when
auto_visibility_timeout was true.
The fix was to
the thread after the work is done.
Let’s take a look at how sum method
fares on some of the enumerable objects
in Ruby 2.4.
As we can see,
the behavior of Enumerable#sum from Ruby 2.4 is same
as that of Active Support in case of numbers but not the same
in case of string or array concatenation.
Let’s see what is the difference and how we can make it work
in Ruby 2.4 as well.
Understanding addition/concatenation identity
The Enumberable#sum method
takes an optional argument
which acts as an accumulator. Both Active Support
and Ruby 2.4 accept this argument.
When identity argument is not passed,
0 is used as default accumulator in Ruby 2.4 whereas
Active Support uses nil as default accumulator.
Hence in the cases of string and array concatenation,
the error occurred in Ruby because the code attempts to
add a string and array respectively
To overcome this,
we need to pass proper addition/concatenation identity
as an argument to the sum method.
The addition/concatenation identity of an object
can be defined as the value with which calling + operation
on an object returns the same object.
What about Rails ?
As we have seen earlier,
Ruby 2.4 implements Enumerable#sum
favouring numeric operations whereas
also supporting non-numeric callers
with the identity element.
This behavior is not entirely same
as that of Active Support.
But still Active Support can make use
of the native sum method whenever
There is already a
open which uses
Enumerable#sum from Ruby
This will help gain some performance boost
as the Ruby’s method is implemented
natively in C
whereas that in Active Support is
implemented in Ruby.
we use #concat to append a string to another string or an element to the array.
We can also use #prepend to add a string at the beginning of a string.
String#concat and Array#concat
Before Ruby 2.4,
we could pass only one argument
to these methods.
So we could not add multiple
items in one shot.
Changes with Ruby 2.4
In Ruby 2.4,
we can pass multiple arguments
and Ruby processes each argument
one by one.
String#concat and Array#concat
These methods work
even when no argument is passed
unlike in previous versions of Ruby.
Difference between concat and shovel << operator
Though shovel << operator can be used interchangably with
concat when we are calling it once, there is a difference
in the behavior when calling it multiple times.
So concat behaves as appending present content to the caller twice.
Whereas calling << twice is just sequence of binary operations.
So the argument for the second call is output of the first << operation.
will have these 2 methods
in the languageitself,
so even those
not using Rails
or Active Support
will be able to use them.
it will also give performance boost
over the Active Support versions
now these methods
are implemented in C natively
the Active Support versions
are in Ruby.
There is already a pull request open
in Rails to use
the native versions of these methods
from Ruby 2.4 whenever available
so that we will be able to use
the performance boost.
Between October 31,2015 and Sep 5, 2016
we wrote 80 blogs on changes in
Producing a blog every 4 days consistently over
310 days takes persistence and time - lots of it.
We needed to go through
all the commits
pick the ones which are worth writing about
then write about it.
Going into this I knew it would be a hard task.
Ruby on Rails is now a well crafted machine.
In order to fully undertand what’s going on in the code
base we need to spend sufficient time on it.
However I was surprised by the thing that turned to be the
hardest - telling story of the code change.
Every commit has a story.
There is a reason for it.
The commit itself
might be minor but that code change in itself does not tell the full story.
For example take this commit.
This commit is so simple that
you might think it is not worth writing about.
However in order to fully understand what it does we need to
tell the full story which was captured in
Now you might say that I have cherry picked
blog posts that favor my case.
So let’s pick a blog
which is simple.
You might wonder what could go wrong with a blog like this.
As it turns out, plenty.
That’s because writing a blog also requires
defining the boundary of the blog.
Deciding what to include and what to leave out is hard.
One gets a feel for it only after writing it.
And after having typed the words on screen,
pruning is hard.
A good written article is simple writing.
The problem with article which are simple to readers
is that - well it is simple.
So it feels to readers that writing it
must be simple.
Nothing can be further from the truth.
It takes a lot of hard work to produce
anything simple. It’s true in writing.
And it’s true in producing software.
Coming back to the “Skipping Mailer” blog,
it took quite a bit
of back and forth to bring the blog to its essence.
So yes the final output is quite short but that does not mean
that it took short amount of time to produce it.
Tell a story even if you have 10 seconds
John Lasseter was working as an animator at Disney in 1984.
He was just fired from Disney for promoting computer animations at Disney.
Lasseter joins Lucasfilm.
Lucasfilm renamed itselfs to Pixar Graphics Group and sold itself
to Steve Jobs for $5 million.
Lasseter was tasked with producing a short film that would show the power
of what computer animations could do so that Pixar Graphics Group can
some projects like producing TV commercials with cartoon characters and
earn some money.
Lasseter needed to produce a short film for the upcoming computer
graphics animation conference.
His initial idea was to have a short movie having
a plotless character.
He presented this idea
to a conference in Brussels.
There Belgian animator Raoul Servais
commented in slightly harsh tone that
No matter how short it is, it should have a beginning, a middle, and an end. Don’t forget the story.
Lasseter complained that it’s a pretty short movie and there
might not be time to present a story.
Raoul Servais replied
You can tell a story in ten seconds.
Lasseter started developing a character.
He came up
with the idea of Luxo Jr.
Luxo Jr. was a major hit at the conference.
Crowd was on its feet in
applause even before the two minutes film was over.
Remember this is
1986 and Computer Animation was not much advanced at that time
this was the first movie ever made with the use
of just computer graphics.
Lasseter later said that when audience was
watching the movie they forgot that they were watching a computer
animated film because the story took over them.
He learned the lesson that technology should enable
better story telling and
technology in itself divorced from
story telling would not advance the cause
Later John Lasseter went on to produce hits like Toy Story, A bug’s
life, Toy Story 2, Cars, Cars 2, Monsters Inc, Finding Nemo and many
So you see even a great John Lasseter had to be reminded to tell a
Actual content over bullet points
is so focused on knowing the full story that he banned usage
of power point in internal meetings and discussions. As per him it is
easy to hide behind bullet points in a power point presentation.
He insisted on writing the full story in word document and
distribute it to meeting attendees.
The meetings starts with everyone head down reading the document.
He is also known for saying that if we are building a feature then we first
need to know how it would be presented to the consumers when it is
unveiled. We need to know the story we are going to tell them. Without
the story we won’t have full picture of what we are going to build.
Learning to tell story is a journey
I’m glad that during the last 310 days 16 people contributed to the
The process of writing the posts at times was frustrating for a bunch of
They had done the work of digging into the code and had posted their
Continuously getting feedback to
edit the blog to build a nice coherent story where
each paragraph is a extension of the previous paragraph
is a downer.
Some were dismayed at why we are spending so much energy on
a technical blog.
However in the end we all are happy that we underwent
We could see the initial draft of the blog and the final
version and we all could see the difference.
By no means we have mastered the art of storytelling.
It’s a long journey.
However we believe we are on the right path.
Hopefully in coming months and years we at BigBinary would be able to
bring to you more stories from changes in Rails and other places.
Here is an example which
demonstrates an example on how to use it.
Now CurrentScope.user_permissions will be available till the lifetime of
currently executing thread and all the code after this point
can use this variable.
For example, we can access this variable in any of the models
without explicitly passing current_user from the controller.
It internally uses Thread.current#= method,
so all the variables are scoped to the thread currently executing.
It will also take care of namespacing these variables per class or module
CurrentScope.user_permissions and RequestScope.user_permissions will not conflict
with each other.
If you have used PerThreadRegistry before for managing global variables,
thread_mattr_* & thread_cattr_* methods can be used in place of it starting from Rails 5.
Globals are generally bad and should be avoided but this change provides nicer API
if you want to fiddle with them anyway!
we need to make sure that
are loaded when application boots.
The concept of loading all the constants even before they are actually
needed is called
In a way it is opposite of
In the case of “Autoloading”
the application does not load
the constant until it is needed.
Once a class is needed
and it is missing
then the application starts looking
in “autoloading paths” to load
the missing class.
eager_load_paths contains a list of directories.
When application boots
the application loads all constants
found in all directories listed in
We can add directories to
as shown below.
In Rails 5 autoloading is disabled for production environment by default
With this commit
Rails will no longer do Autoloading in production after it has booted.
Rails will load all the constants from eager_load_paths
but if a constant is missing then it will not look in
autoload_paths and will not attempt to load the missing constant.
This is a breaking change for some applications.
For vast majority of the applications
this should not be an issue.
In the rare situation
where our application
still needs autoloading
in the production environment,
we can enable it
by setting up enable_dependency_loading to true
Adding HTTPS support is one of the first steps towards enhancing the security of a web application.
Even when a web app is available over https,
some users may end up visiting the http version of the app,
losing the security https provides.
It is important to redirect users
to the https URLs whenever possible.
Forcing HTTPS in Rails
We can force users to use HTTPS
by setting config.force_ssl = true.
If we look at Rails source code,
we can see that
when we set config.force_ssl = true,
a middleware ActionDispatch::SSL,
is inserted into our app’s middleware stack :
This middleware, ActionDispatch::SSL
is responsible for doing three things :
Redirect all http requests to their https equivalents.
Set secure flag on cookies
to tell browsers that
these cookies must not be sent for http requests.
Add HSTS headers to response.
Let us go through each of these.
Redirect all http requests to their https equivalents
In Rails 5, we can configure
the behavior of redirection
using the redirect key
in the config.ssl_options configuration.
In previous versions of Rails,
whenever an http request was redirected to https request,
it was done with an HTTP 301 redirect.
Browsers cache 301 redirects.
When forcing https redirects,
if at any point
we want to test the http version of the page,
it would be hard to browse it,
since the browser would redirect to the https version.
Although this is the desired behavior,
this is a pain during testing and deploying.
We can specify the options for redirection in Rails 5 as follows :
If a redirect status is not specified,
requests are redirected with a 301 status code.
There is an upcoming change
to make the status code used for
redirecting any non-GET, non-HEAD http requests
to 307 by default.
Other options accepted by ssl_options under redirect key are host and body .
Set secure flags on cookies
By setting the Secure flag on a cookie,
the application can instruct the browser
not to send the cookie in clear text.
Browsers which support this flag
will send such cookies
only through HTTPS connections.
In case of a “man in the middle” attack,
the attacker places oneself between the user and the server.
By doing this, attacker aims to collect cookies which
are sent from user to server on every request.
However, if we mark the cookies with sensitive information as Secure,
those cookies won’t be sent on http requests.
This ensures that the browser
never sends cookies to an attacker
who was impersonating the webserver
at an http end point.
Upon enabling config.force_ssl = true,
the ActionDispatch::SSL middleware sets the Secure flag on all cookies by default.
Set HSTS Headers on Responses
“HTTP Strict Transport Security”
is a security enhancement
applications can specify themselves as HTTPS-only to complying browsers.
HSTS capabilities of a browser can be used by sending appropriate response headers from the server.
When a domain is added to the HSTS list of a browser,
the browser redirects to the https version of the URL without the help of the server.
Chrome maintains an HSTS Preload List
with a list of domains which are hardcoded into chrome as HTTPS only.
This list is also used by Firefox and Safari.
Rails 5 has a configuration flag
to set the preload directive
in the HSTS header and can be used as follows :
We can also specify a max-age for the HSTS header.
Rails 5 by default sets the max-age of HSTS header to 180 days,
which is considered as the lower bound by SSL Lab’s SSL Test .
This period is also above the 18 week requirement
for HSTS max-age mandated for inclusion in browser preload list.
We can specify a custom max-age by :
In Rails 5, if we disable HSTS by setting :
Rails 5 will set the value of expires header to 0,
so that browsers immediately stop treating the domain as HTTPS-only.
With custom redirect status and greater control over the HSTS header,
Rails 5 lets us roll out HTTPS in a controlled manner,
and makes rolling back of these changes easier.
For new apps, it will be set to true and for older apps upgraded to Rails 5, it
will be set to false. Once we are ready to use this feature in our upgraded app,
we just need to change it to true.
Enable HTTP Origin Header checking for CSRF mitigation
For additional defense against CSRF attacks, Rails 5 has a feature to check HTTP Origin header against the site’s origin.
This will be disabled by default in upgraded apps using the following configuration option:
We can set it to true to enable HTTP origin header check when we are ready to use this feature.
Make Ruby 2.4 preserve the timezone of the receiver
We can use the new behavior of not halting the callback chain after making sure that our code
does not break due to this change and changing the value of this config to false.
Configure SSL options to enable HSTS with subdomains
HTTP Strict Transport Security or HSTS, is a web security policy mechanism which helps to protect websites against protocol downgrade attacks and cookie hijacking.
Using HSTS, we can ask browsers to make connections using only HTTPS. In upgraded apps, HSTS is not enabled on subdomains. In new apps HSTS is enabled using the following option :
Having all these backward incompatible features which can be turned on one by one after the upgrade, in one file, eases the upgrade process. This initializer also has helpful comments explaining the features!
Previously we could only pass a single record to these methods but now Rails 5 adds support for accepting a collection of records as well. For example,
or simply written as,
This works with stale? method too, we can pass a collection of records to it. For example,
To see this in action, let’s begin by making a request at /posts.
In the second request, we would send the ETag in If-None-Match header to check if the data has changed.
Since there’s no change, the server returned HTTP/1.1 304 Not Modified. If these requests were made from a browser, it would automatically use the version in its cache on the second request.
The second request was obviously faster as the server was able to save the time of fetching data and rendering it. This can be seen in Rails log,
Cache expires when collection of records is updated. For example, an addition of a new record to the collection or a change in any of the records (which changes updated_at) would change the ETag.
Now that Rails 5 supports collection of records in fresh_when and stale?, we have an improved system to cache resources and make our applications faster. This is more helpful when we have controller actions with time consuming data processing logic.
In Rails, for setting up test data we use fixtures.
Fixtures are written as YAML files
placed in test/fixtures directory in the Rails app.
The model name of a fixture is automatically picked up
from the fixture file name.
Generally in Rails, the model name and table name follow a strict convention.
The table for User model will be users.
By this convention, the fixture file for User model is test/fixtures/users.yml.
But sometimes model names do not match directly with the table name.
When we are building on top of a legacy application
or we have namespacing of models, we might run into this scenario.
In such cases detection of model name
from fixture file name becomes difficult.
When automatic detection of model name from fixture file name fails,
we can specify the table name using the
Take a look at our older
for an example of how to do this.
One drawback of using this approach is that,
the model name set using set_fixture_class
is available only in the context of tests.
When we run rake db:fixtures:load to load the fixtures,
the tests are not run,
and the fixture file is not associated with the model name we set using set_fixture_class.
This will cause failure to load the fixtures correctly.
The Rails 5 way
In Rails 5 a new key is added
to specify the model name for a fixture file.
Let us consider the example where our table was named morning_appts,
and we used a more appropriately named model MorningAppointment
to represent this table.
We can now set the table name in our fixture file test/fixtures/morning_appts.yml
as follows :
The special key _fixture in the fixture file
is now used to store metadata about the fixture.
model_class is the key we can use to specify the model name for the fixture.
We can now use this fixture to load test data using the rake task rake db:fixtures:load as well.
Recently we worked with a client where we had to run a part of their multi-threaded code in JRuby for performance reasons. They have been using CircleCI with MRI for running tests. In this post I will explain how we configured CircleCI to run the same tests using both JRuby and MRI.
CircleCI uses circle.yml file for configuration. Before configuring JRuby, this is how it looked like:
Here are the steps to enable JRuby in CircleCI.
Specify the JDK version
We need to specify a JDK version before using JRuby.
Install proper dependencies
We needed to use JRuby 22.214.171.124 but the version of JRuby that came with Ubuntu 12.04 image of CircleCI was different. We added rvm install command as follows to install specific version that we wanted. Also we can configure any script (like bundle install) that needs to run before running tests.
We used rvm-exec to set JRuby for running tests for this particular component in the test section. Otherwise by default it picks up MRI.
Improving test runs on JRuby
Once we started running tests with JRuby, we observed it was taking comparatively slower to finish all tests. Most of the time was spent in starting the JVM. We made it faster by setting --dev parameter in JRUBY_OPTS environment variable. This parameter improves JRuby boot time and it shaved more than a minute time for us.