Deleting a pod in K8s : Behind the magic

[Chetan_Tiwary_]

Kubernetes may terminate pods under various circumstances, even if they are functioning correctly. These scenarios include rolling updates, node draining, or resource constraints. To ensure smooth operations and minimal disruption, it's crucial for applications to handle termination gracefully.

When a pod is scheduled for termination, Kubernetes sends a SIGTERM signal to the main container process. This signal prompts the application to initiate a controlled shutdown. The application should gracefully handle this signal by saving any necessary data, closing open connections, and completing ongoing tasks before exiting.

Process Breakdown:

1. Pod Deletion Initiation:

   • You issue oc delete pod test
   • The API Server updates the pod record in ETCD, adding deletionTimestamp and terminationGracePeriodSeconds (default: 30 seconds).
   • The pod enters the Terminating state.
   • 

2. Parallel Actions:

   • Endpoint Removal:

     • The endpoint controller notices the terminating pod and removes its endpoint from the associated service, preventing external traffic.
     • This removal happens asynchronously, propagating to Kube-proxy, iptables, ingress, CoreDNS, etc.

   • Pod Shutdown:

     • Kubelet receives the pod update and triggers two actions:
       • preStop Hook (if defined): Executed first, allowing for graceful pre-shutdown tasks.
       • SIGTERM Signal: Sent to the main container after the preStop hook finishes or the grace period expires.
     • If no preStop hook exists, SIGTERM is sent immediately.
     • The container attempts graceful shutdown for the specified terminationGracePeriodSeconds

     • After the specified grace period elapses, any remaining containers are forcefully terminated with a SIGKILL signal. This ensures their immediate removal from the system. Subsequently, all Kubernetes objects associated with the terminated pod, such as services and volumes, are cleaned up.

       .

3. Finalization:

   • After the grace period or upon forceful termination, the API Server removes the pod record from ETCD.

Note : The Pod is always able to request API Server during the graceful shutdown.

Sometimes, the pod is not deleted generally due to it being stuck in terminating status :

[student@workstation ~]$ oc get pods
NAME         READY   STATUS         RESTARTS  AGE
test-9mzm2    1/1     Running        0         8m
test-vmzmz    1/1     Terminating    37        30m
raj-pflxc     1/1     Running        0         8d

In this case, you can try oc delete command with the --force flag and the --grace-period=0 option.

[student@workstation ~]$ oc delete pod test-vmzmz --force --grace-period=0

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "test-vmzmz" force deleted

https://kubernetes.io/docs/reference/kubectl/generated/kubectl_delete/