Hello All,
Can anyone recommend a set of steps or other troubleshooting ideas for fixing Pods in Pending status?
Thanks,
Eric
Hello @emsecrist !
Thanks for reaching out !
PENDING status of a pod means : It is either not being able to be scheduled on the node or is waiting to be scheduled or one of the containers has not been initialized.
A pod's pending status can be due to n number of issues -> which majorly can be summed up in to the following categories :
1. Configuration dependencies : like configmap, persistent volume , secret etc not working or missing configurations.
2. Image issues : pulling from wrong repo , wrong image name , incorrect tag or authentication issues which prevents the image usage etc.
3. Scheduling or Node issues : may be nodes are not ready or they are cordoned off ( unschedulable ) , node selectors/labels issues , taints and toleration issues , affinity and anti-affinity rules , resource limits like node pressures ( disk/ cpu / memory ) which can prevent pods to be scheduled.
The major tools OpenShift admin has to troubleshoot these issues are as follows :
1. Check the logs ( oc log <pod-name> )
2. Check the cluster wide events ( oc get events )
3. Check the pod specs ( oc describe pod <pod-name> )
4. Check the deployment spec ( oc describe deployment <deploymentname> )
5. Check the node spec ( oc describe node <node-name> )
6. Check the taints in the node ( oc describe node <node-name> | grep -i taints )
7. Check the node labels ( oc get nodes -o wide or oc get nodes -L <labelname> )
Once you have got the idea about the issue throught the help of the above commands , you can then resolve it by applying the corresponding solution like correcting the image name or tag , applying / providing the correct labels or node selectors or providing the correct pv or configmap or secret etc.
Refer here for node related stuffs : https://docs.openshift.com/container-platform/4.10/nodes/index.html
Debugging pod K8 documentation : https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/
Hello @emsecrist !
Thanks for reaching out !
PENDING status of a pod means : It is either not being able to be scheduled on the node or is waiting to be scheduled or one of the containers has not been initialized.
A pod's pending status can be due to n number of issues -> which majorly can be summed up in to the following categories :
1. Configuration dependencies : like configmap, persistent volume , secret etc not working or missing configurations.
2. Image issues : pulling from wrong repo , wrong image name , incorrect tag or authentication issues which prevents the image usage etc.
3. Scheduling or Node issues : may be nodes are not ready or they are cordoned off ( unschedulable ) , node selectors/labels issues , taints and toleration issues , affinity and anti-affinity rules , resource limits like node pressures ( disk/ cpu / memory ) which can prevent pods to be scheduled.
The major tools OpenShift admin has to troubleshoot these issues are as follows :
1. Check the logs ( oc log <pod-name> )
2. Check the cluster wide events ( oc get events )
3. Check the pod specs ( oc describe pod <pod-name> )
4. Check the deployment spec ( oc describe deployment <deploymentname> )
5. Check the node spec ( oc describe node <node-name> )
6. Check the taints in the node ( oc describe node <node-name> | grep -i taints )
7. Check the node labels ( oc get nodes -o wide or oc get nodes -L <labelname> )
Once you have got the idea about the issue throught the help of the above commands , you can then resolve it by applying the corresponding solution like correcting the image name or tag , applying / providing the correct labels or node selectors or providing the correct pv or configmap or secret etc.
Refer here for node related stuffs : https://docs.openshift.com/container-platform/4.10/nodes/index.html
Debugging pod K8 documentation : https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/
Red Hat
Learning Community
A collaborative learning environment, enabling open source skill development.