Chetan_Tiwary_
Moderator
Moderator
  • 2,540 Views

CrashLoopBackOff in K8s

CrashLoopBackOff is a common status that a pod can enter in Kubernetes when there is an issue with the containers running within the pod. This status indicates that the pod has crashed and is continuously attempting to restart, but fails repeatedly.

Let us see if we can understand this through the below example :

I deployed an ephemeral database server ( just for demo - The underlying intricacies are avoided in order to be crisp and focused).


Chetan_Tiwary__0-1690315941833.png

I got a crashloopbackoff “error” status of my pod : 

 

Chetan_Tiwary__1-1690315941784.png

Here, you can see 

  • mysql Pod is not in READY condition (as shown as 0/1).
  • The STATUS column displays CrashLoopBackOff.
  • RESTARTS  column displays restarts , here 4.

CrashLoopBackOff is not an error in itself but indicates that there's an error happening that prevents a Pod from starting correctly.

In simple words : pods are failing, and they are being restarted. The grace period , in between the consecutive restarts is what is represented as CrashLoopBackOff.

Crash :  a container running in a pod has terminated unexpectedly

Loop :  K8s will attempt to restart the crashed container automatically (as desired)

BackOff : "I am gonna increase the time between successive restart attempts with each failure to give the container some time to recover."

 

Please note that crashloopback is NOT an error in itself , it is NOT the real culprit which is crashing your pod.

You need to find out the real culprit which is causing this status which can be due to ( but not limited to)

  1. Memory limits -> initiating OOM kill.
  2. Lack of permissions issue
  3. Error in app configuration like missing env variables 
  4. Runtime errors not properly handled
  5. Any external dependencies
  6. Error in liveness probe reporting the status of pod
  7. Network error such as a port binding issue.

 

Now how could you debug the issue : These below commands output should give you a fair idea about what is preventing the pods to be in running state :

oc get events

oc describe pod <pod-name>

oc logs <pod-name>

oc describe deployment <deploymentname>

Chetan_Tiwary__2-1690315942957.png

 

 

Chetan_Tiwary__3-1690315942895.png

 

Some troubleshooting based on the error causing the crashloopbackoff ( not in the objective of this post)

(So, in this case - we will use mysql secret ( pre-configured)  to initialize environment variables on the mysql deployment. The deployment needs the MYSQL_USER, MYSQL_PASSWORD, and MYSQL_DATABASE environment variables for a successful initialization. The secret has the user, password, and database keys that can be assigned to the deployment as environment variables, adding the prefix MYSQL_.)

 

Chetan_Tiwary__4-1690315942738.png

Chetan_Tiwary__5-1690315942040.png

 

So as soon as the missng configuration was supplied, the pod goes into RUNNING state and your pod does not have to Crash and Loop and BackOff !  

 

 

0 Replies
Join the discussion
You must log in to join this conversation.