Recently, we faced a DDoS attack in one of the clients’ projects. There were many requests from different IPs to root and login paths, and we were running thrice the usual number of servers to keep the system alive.
We are using Cloudflare’s HTTP proxy and it was doing a great job preventing malicious requests, but we wanted to check if we can avoid the loading/captcha pages which Cloudflare uses to filter requests. We came to a conclusion that we would be able to mitigate the ongoing attack if we could throttle requests by IP.
Cloudflare has an inbuilt Rate Limiting feature to throttle requests, but it would be a little expensive in our case since Cloudflare charges by the number of good requests and it was a high traffic website. On further analysis, we found that throttling at application level would be enough in that situation and the gem Rack::Attack helped us with that.
Rack::Attack is a Rack middleware from Kickstarter. It can be configured to throttle requests based on IP or any other parameter.
To use Rack::Attack, include the gem in Gemfile.
bundle install, configure the middleware in
Now we can create the initializer
config/initializers/rack_attack.rb to configure Rack::Attack.
By default, Rack::Attack uses
Rails.cache to store requests information.
In our case, we wanted a separate cache for
Rack::Attack and it was configured as follows.
If the web server is behind a proxy like Cloudflare, we have to configure a method to fetch the correct
remote_ip address. Otherwise, it would block based on proxy’s IP address and would result in blocking a lot of legit requests.
Requests can be throttled based on IP address or any other parameter. In the following example, we are setting a limit of 40rpm/IP for “/” path.
The downside of this configuration is that after the 1 minute period, the attacker can launch another 40 requests/IP simultaneously and it would exert pressure on the servers. This can be solved using exponential backoff.
If we want to turn off throttling for some IPs (Eg.: Health check services), then those IPs can be safelisted.
We can log blocked requests separately and this is helpful for analyzing the attack.
A sample initializer with these configurations can be downloaded from here.
Application will now throttle requests and will respond with
HTTP 429 Too Many Requests response for the throttled requests.
We now block a lot of malicious requests using Rack::Attack. Here’s a graph with
% of blocked requests over a week.
EDIT: Updated the post to add more context to the situation.