Komodor is a Kubernetes management platform that empowers everyone from Platform engineers to Developers to stop firefighting, simplify operations and proactively improve the health of their workloads and infrastructure.
Proactively detect & remediate issues in your clusters & workloads.
Easily operate & manage K8s clusters at scale.
Reduce costs without compromising on performance.
Empower developers with self-service K8s troubleshooting.
Simplify and accelerate K8s migration for everyone.
Fix things fast with AI-powered root cause analysis.
Explore our K8s guides, e-books and webinars.
Learn about K8s trends & best practices from our experts.
Listen to K8s adoption stories from seasoned industry veterans.
The missing UI for Helm – a simplified way of working with Helm.
Visualize Crossplane resources and speed up troubleshooting.
Validate, clean & secure your K8s YAMLs.
Navigate the community-driven K8s ecosystem map.
Kubernetes 101: A comprehensive guide
Expert tips for debugging Kubernetes
Tools and best practices
Kubernetes monitoring best practices
Understand Kubernetes & Container exit codes in simple terms
Exploring the building blocks of Kubernetes
Cost factors, challenges and solutions
Kubectl commands at your fingertips
Understanding K8s versions & getting the latest version
Rancher overview, tutorial and alternatives
Kubernetes management tools: Lens vs alternatives
Troubleshooting and fixing 5xx server errors
Solving common Git errors and issues
Who we are, and our promise for the future of K8s.
Have a question for us? Write us.
Come aboard the K8s ship – we’re hiring!
Hear’s what they’re saying about Komodor in the news.
SIGTERM (signal 15) is used in Unix-based operating systems, such as Linux, to terminate a process. The SIGTERM signal provides an elegant way to terminate a program, giving it the opportunity to prepare to shut down and perform cleanup tasks, or refuse to shut down under certain circumstances. Unix/Linux processes can handle SIGTERM in a variety of ways, including blocking and ignoring.
SIGTERM is the default behavior of the Unix/Linux kill command – when a user executes kill, behind the scenes, the operating system sends SIGTERM to the process. If the process is not
In Docker containers, a container that is terminated via a SIGTERM signal shows exit code 143 in its logs. If you are a Kubernetes user, this article will help you understand what happens behind the scenes when Kubernetes terminates a container, and how to work with the SIGTERM signal in Kubernetes.
This is part of a series of articles about Exit Codes.
SIGTERM (Unix signal 15) is a “polite” Unix signal that kills the process by default, but can be handled or ignored by the process. This gives the process a chance to complete essential operations or perform cleanup before shutting down. The purpose is to kill the process regardless of whether it ended successfully or not, but to give it a chance to clean up the process first.
SIGKILL (Unix signal 9) is a “brutal” Unix signal that kills the process immediately. It is not possible to handle or ignore SIGKILL, so the process does not have an opportunity to clean up. SIGKILL should be used by Unix/Linux users as a last resort, because it can lead to errors and data corruption.
In some cases, even if SIGKILL is sent, the kernel may not be able to terminate the process. If a process is waiting for network or disk I/O, and the kernel cannot stop it, it becomes a zombie process. A restart is required to clear zombie processes from the system.
Exit codes 143 and 137 are parallel to SIGTERM and SIGKILL in Docker containers:
Itiel Shwartz
Co-Founder & CTO
In my experience, here are tips that can help you better manage and resolve SIGTERM (Signal 15, Exit Code 143) in Kubernetes:
Define `preStop` hooks to gracefully handle cleanup tasks before SIGTERM is sent.
Adjust this parameter to allow sufficient time for your application to shut down gracefully.
Implement logic to properly handle SIGTERM, ensuring orderly shutdown processes.
Add logging for SIGTERM handling to help diagnose shutdown issues.
Use Kubernetes events to track and respond to pod terminations.
The most common way to end a process in Unix/Linux is to use the kill command, like this: kill [ID]
kill [ID]
By default, the kill command sends a SIGTERM signal to the process.
kill
To discover [ID], the process ID, use the command ps -aux, which lists all running processes.
[ID]
ps -aux
In extreme cases, you may need to immediately terminate a process using SIGKILL. Use this command to send SIGKILL: kill -9 [ID]
kill -9 [ID]
When you list running processes, you may find processes that show defunct in the CMD column. These are zombie processes that did not terminate correctly. A zombie process is:
defunct
Zombie processes will appear in the process table until their parent process shuts down, or the operating system restarts. In many cases, zombie processes can accumulate in the process table, because multiple child processes were forked by a parent process and were not successfully killed. To avoid this situation, ensure that your application’s sigaction routine ignores the SIGCHLD signal.
sigaction
If you are a Kubernetes user, you can send a SIGTERM to a container by terminating a pod. Whenever a pod is terminated, by default, Kubernetes sends the containers in the pod a SIGTERM signal.
Pods are often terminated automatically as a result of scaling or deployment operations. To terminate a pod manually, you can send a kubectl delete command or an API call to terminate the pod.
kubectl delete
Note that after a grace period, which is 30 seconds by default, Kubernetes sends a SIGKILL to terminate the container immediately.
Kubernetes manages clusters of containers, performing many automated operations on your applications. For example, it can scale applications up or down, update them, and remove applications from the cluster. Therefore, there are many cases in which Kubernetes needs to shut down a pod (with one or more containers), even if they are functioning properly.
There are also cases in which Kubernetes will shut down pods because they are malfunctioning, or because there are insufficient resources on the host machine (known as eviction). Whenever Kubernetes needs to terminate a pod, for any reason, it sends SIGTERM to the containers running in the pod.
Here is the full process that occurs when Kubernetes wants to terminate a pod:
terminationGracePeriod
To ensure that pod termination does not interrupt your applications and impact end-users, you should handle pod termination.
Practically speaking, this means ensuring your application handles the SIGTERM signal and performs an orderly shutdown process when it receives it. This should include completing transactions, saving transient data, closing network connections, and erasing unneeded data.
Note that unlike in a regular Linux system, in Kubernetes, SIGTERM is followed by SIGKILL after a grace period. So you must prepare the container to shut down, and cannot simply ignore it.
Another option for handling graceful termination is the preStop hook – this lets you perform shutdown processes without code changes to the application. If you use the preStop hook, make sure that the actions performed do not duplicate, or conflict, with the actions the application performs when it receives the SIGTERM signal. It is usually best to handle either SIGTERM or preStop, to avoid conflicts.
Any Kubernetes error that results in a pod shutting down will result in a SIGTERM signal sent to containers within the pod:
kubectl describe pod
One exception is the OOMKilled error. This is a Kubernetes error that occurs because a container or pod exceeds the memory allocated to them on the host. When a container or pod is terminated due to OOMKilled, Kubernetes immediately sends a SIGKILL signal, without using SIGTERM and without using a grace period.
When running applications on Kubernetes, you must ensure that ingress controllers do not experience downtime. Otherwise, whenever the controller restarts or is redeployed, users will experience a slowdown or service interruption. If an ingress pod is terminated, this can result in dropped connections – this must be avoided in production.
If you are using the official NGINX Ingress Controller, when the controller pod is terminated, Kubernetes sends a SIGTERM signal as usual.
However, the NGINX controller does not handle SIGTERM in the way Kubernetes expects:
As we discussed in the Handling SIGTERM and preStop section above, Kubernetes provides a second option for handling graceful termination – the preStop hook. You can use the preStop hook to send a SIGQUIT signal to NGINX, just before SIGTERM is sent. This avoids NGINX shutting down abruptly, and gives it the opportunity to terminate gracefully.
As a Kubernetes administrator or user, pods or containers terminating unexpectedly can be a pain, and can result in severe production issues. The troubleshooting process in Kubernetes is complex and, without the right tools, can be stressful, ineffective and time-consuming.
Some best practices can help minimize the chances of SIGTERM or SIGKILL signals affecting your applications, but eventually something will go wrong—simply because it can.
This is the reason why we created Komodor, a tool that helps dev and ops teams stop wasting their precious time looking for needles in (hay)stacks every time things go wrong.
Acting as a single source of truth (SSOT) for all of your k8s troubleshooting needs, Komodor offers:
If you are interested in checking out Komodor, use this link to sign up for a Free Trial.
Share:
How useful was this post?
Click on a star to rate it!
Average rating 4.5 / 5. Vote count: 8
No votes so far! Be the first to rate this post.
and start using Komodor in seconds!