In our last blog, we explained how to handle rolling deployments of Rails applications with no downtime.
In this article we will walk you through how to handle graceful shutdown of processes in Kubernetes.
When we deploy Rails applications on kubernetes it stops existing pods and spins up new ones. When old pod is terminated by Replicaset, then active Sidekiq processes are also terminated. We run our batch jobs using sidekiq and it is possible that sidekiq jobs might be running when deployment is being performed. Terminating old pod during deployment can kill the already running jobs.
As per default pod termination policy of kubernetes, kubernetes sends command to delete pod with a default grace period of 30 seconds. At this time kubernetes sends TERM signal. When the grace period expires, any processes still running in the Pod are killed with SIGKILL.
We can adjust the
terminationGracePeriodSeconds timeout as per our need and can change it from
30 seconds to 2 minutes.
However there might be cases where we are not
sure how much time a process takes to gracefully shutdown.
In such cases we should consider using
PreStop hook which is our next solution.
Kubernetes provides many Container lifecycle hooks.
PreStop hook is called immediately before a container is terminated.
It is a blocking call. It means it is synchronous.
It also means that this hook must be completed before
the container is terminated.
Note that unlike solution1 this solution is not time bound.
Kubernetes will wait as long as it takes for
to finish. It is never a good idea to have a process which takes more
than a minute to shutdown but in real world there are cases
where more time is needed. Use
PreStop for such cases.
We decided to use
preStop hook to stop Sidekiq because we had some really long running processes.
Using PreStop hooks in Sidekiq deployment
This is a simple deployment template which terminates Sidekiq process when pod is terminated during deployment.
Next we will use
PreStop lifecycle hook to stop
Sidekiq safely before pod termination.
We will add the following block in deployment manifest.
PreStop hook stops all the
Sidekiq processes and does graceful shutdown of Sidekiq
before terminating the pod.
We can add this configuration in original deployment manifest.
Let’s launch this deployment and monitor the rolling deployment.
We can confirm that existing Sidekiq jobs are completed before termination of old pod during the deployment process. In this way we handle a graceful shutdown of Sidekiq process. We can apply this technique to other processes as well.