shcmzzj
Cadet
Cadet
  • 3,260 Views

Pod can not tolerate to nods - EX280

Hello everyone,

After the task of manual scaling, the pods was stacked at pending status.
After the task of autoscaling, the pods was stacked at pending status.
Even in the tasks that the deploymentconfig has been prepared, the pods was just dead(timeout) there.
So you can not understand what happened there and what to do for the exam.
When I check the events I found the error of the pods like this:  (sorry,but I can't remember all the words)

0/5 nodes are available, 2 nodes taint {node.worker} that can not tolerate, 3 nodes taint {node-role.kubernetes.io/master} that can not tolerate.


What should I do for this? I'm sure that I won't pass the exam if I can't solove the problem.
I have taken the DO280 and I think I didn't see the trouble in this course.
Regards.

Labels (1)
Tags (1)
9 Replies
dgaona77
Cadet
Cadet
  • 3,232 Views

Hello shcmzzj

I had exactly the same problem during my exam, which I also didn't pass and I think this "tolerations" contributed.

After revisited the "tolerations and taints" topic, I can see now that, either you need to modify the deploymentConfig to include the correct "tolerations" or remove the taints from the nodes.

Which is the correct? still I don't know.

shcmzzj
Cadet
Cadet
  • 3,169 Views

Thank you, dgaona77

I checked the doc of "tolerations and taints" too, and I think both of the ways can resolve this problem.

I also discussed this with my colleague who passed this exam. His choise was modifying all the DeploymentConfig. 

I think I will use the same solution with him in my next exam, though I think that removing the taints should be a faster way.

Overdoes
Cadet
Cadet
  • 402 Views

Tolerations are defined at pod level not at dc or deployment. If you add toleration to a pod then try to scale it up, all new replicas won't have the same toleration so they will be in pending state

0 Kudos
e719
Mission Specialist
Mission Specialist
  • 2,653 Views

same thing. I have read DO280, and nothing useful were there (( Only one string in theoretic part of Pod Scheduling chapter - that taints and tolerations are exist in Openshift.
But in prerequisites for exam were said DO280 is the course for that exam. Can't understand how could this happen. Maybe it was "a challenge"?

But what should you do if for example you can't change dc by task conditions?
There is only one way to remove taints

0 Kudos
joseph_joy
Mission Specialist
Mission Specialist
  • 2,646 Views

chapter 6 Controlling Pod Placement,  beyond taints and tolarant  node label can also cause the pod status in pending status.. untill both matches ..

 

 

freddyflo
Cadet
Cadet
  • 2,068 Views

@joseph_joy you are right about node labeling, it can also be a likely cause of the pod being stuck in a pending state.

@shcmzzj I recently failed the exam and I had similar questions. I did change the label on the node to match that of the DeploymentConfig. I also added the toleration in the DeploymentConfig since the nodes were tainted. However, I had an error relating to tolerationSeconds because the taint on the node was NoSchedule. I found an article on this here -> https://medium.com/kubernetes-tutorials/making-sense-of-taints-and-tolerations-in-kubernetes-446e750...

I think the easier thing to do is to remove the taint on the node. I don't know whether this will have an impact on the subsequent questions in the exam

 

0 Kudos
dkcbk
Flight Engineer
Flight Engineer
  • 1,824 Views

tolerationSeconds is suitable only for effect=NoExecute. If your toleration has effect=NoSchedule, you cant have tolerationSeconds in it (only key, operator, value, effect).

0 Kudos
ericbos
Flight Engineer
Flight Engineer
  • 1,529 Views

Applications are supposed to run on worker nodes, so I would probably just remove the taints from the worker nodes unless the question tells me not to. I usually don't see any taints on worker nodes in the production clusters I work with.

ericbos
Flight Engineer
Flight Engineer
  • 1,272 Views

It would be good to add some taints and tolerations labs to the DO280 training material.

Join the discussion
You must log in to join this conversation.