alexcorcoles
Flight Engineer
Flight Engineer
  • 3,740 Views

About pod security standards and warnings

(This material has been created with the inputs of instructors and other people. We are trying to get this material published as soon as possible to address some concerns with DO180 and DO280.)

With the default configuration, any namespace that you create on OpenShift 4.12 has the following labels:

 

 

 

 

 

pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: v1.24
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: v1.24

 

 

 

 

 

These labels activate auditing and warning of the restricted pod security standard. The Kubernetes documentation describes the restricted pod security standard. Pod security standards such as restricted are not directly related to security context constraints such as the restricted SCC.

Kubernetes uses an admission controller to reject pods that do not comply with pod security standards. An admission controller is a Kubernetes component that inspects every Kubernetes resource before its creation, and acts before Kubernetes creates the resource. This action can include rejecting the creation of the resource, or modifying the resource.

The pod security standards are policies that include properties that workloads must follow. For example, the restricted standard requires that containers set the securityContext.allowPrivilegeEscalation field to false.

Pod security standards apply both to workloads, such as deployments and jobs, and to pods.

When a pod is created, the admission controller inspects the container specifications and enumerates the fields that violate the pod security standard of the namespace. The admission controller inspects the pod-security labels in the namespace to decide which pod security standard to apply (restricted, baseline, or privileged), and which action to take on violations. Namespaces can audit, warn, or enforce pod security standard violations. When the pod-security.kubernetes.io/audit annotation is set to restricted, the admission controller prevents the creation of pods that violate the security standard. When the pod-security.kubernetes.io/audit and pod-security.kubernetes.io/warn annotations are defined in the project, the admission controller does not block the creation of pods, but registers audit events or warnings.

The admission controller does not enforce pod security standards for workloads. If the namespace is configured to enforce a pod security standard, then you can create a violating workload. The admission controller enforces only pod security policies on pods.

Cluster administrators can use pod security standards to ensure that pods and workloads in namespaces limit their privileges. By limiting privileges, cluster administrators can mitigate what malicious pods can do.

The default configuration of user namespaces in OpenShift does not restrict any workload; it only notifies users when they create workloads that do not follow pod security standards that might be enforced in future OpenShift versions.

Security Context Constraints

OpenShift introduced security context constraints (SCC) before the introduction of pod security standards in Kubernetes. SCCs automatically make security modifications to pods when they are created. By default, the system:openshift:scc:restricted-v2 cluster role binding applies to all authenticated users, so regular user workloads use the restricted-v2 SCC. For example, when you execute the following command to create a workload without any security configuration, the SCC adds security configuration to the pods:

 

 

 

 

 

[user@host ~]$ oc create deployment test --image registry.ocp4.example.com:8443/redhattraining/hello-world-nginx

 

 

 

 

 

(You can add the --dry-run=client -o yaml options to view the deployment manifest. The deployment manifest does not contain the security properties, which are described later.)

If you inspect the pods that the deployment created, the SCC makes the following changes:

 

 

 

 

 

[user@host ~]$ oc get pod test-b8d5658b6-7bxfs -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
... output omitted...
	openshift.io/scc: restricted-v2
	seccomp.security.alpha.kubernetes.io/pod: runtime/default
...output omitted...
spec:
  containers:
  - image: registry.ocp4.example.com:8443/redhattraining/hello-world-nginx
	imagePullPolicy: Always
	name: hello-world-nginx
	resources: {}
	securityContext:
  	  allowPrivilegeEscalation: false
  	  capabilities:
    	    drop:
    	    - ALL
  	  runAsNonRoot: true
  	  runAsUser: 1000690000
... output omitted...
  securityContext:
	fsGroup: 1000690000
	seLinuxOptions:
  	level: s0:c26,c20
	seccompProfile:
  	type: RuntimeDefault
... output omitted...

 

 

 

 

 

These restrictions satisfy the restricted pod security policy.

The result of these defaults means that when creating a workload (such as a deployment) in a namespace without any customization, the restricted pod security standard applies, but the admission controller only warns and audits. If the workload does not satisfy the pod security standard, then the admission controller allows the workload to be created, but prints a message such as the following warning:

 

 

 

 

 

Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "example" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "example" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "database" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "example" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

 

 

 

 

 

However, by default Kubernetes creates the deployment pods with the restricted-v2 SCC, with the restrictions that the previous warning included. Due to the following factors, workloads generate only a warning, and the pod security standard admission controller allows their pods:

  • The pod security standard admission controller never enforces pod security standards on workloads (only on pods).
  • By default, namespaces do not enforce pod security standards on pods.
  • By default, regular workloads create pods by using the restricted-v2 SCC, which satisfies the restricted pod security standard.

Even if future versions of OpenShift change the default namespace configuration to enforce the restricted pod security standard, users will continue to be able to create workloads without explicitly declaring the restrictions that the pod security standard requires, because the restricted-v2 SCC will apply such restrictions automatically.

You can take the following approaches to handling pod security:

  • For most workloads, the restricted-v2 SCC limits pod privileges adequately. When creating a workload, you receive the warning, but the workload creates pods and works correctly.
  • You can also create your workloads by providing a full specification of the security constraints to apply, without relying on the SCC to provide the configuration. By following this approach, privileges are more explicit, and you have greater control, so you can adjust more precisely the specific privileges that are granted to workloads.

When the restricted-v2 SCC and the default restricted pod security standard are not adequate for your workload, because either you require further privileges, or you need further restrictions, you must use an existing SCC, or create a custom SCC, and provide explicit security specifications. You can also adjust the namespace labels to choose a different pod security standard, and decide to audit, log, or enforce the standard.

Labels (1)
3 Replies
Chetan_Tiwary_
Moderator
Moderator
  • 3,826 Views

Thanks for this detailed info  @alexcorcoles !!

0 Kudos
Jupiter
Cadet
Cadet
  • 3,472 Views

@alexcorcoles , thanks for the great write up and the background on this situation.  I have gone through DO180 and I'm halfway through DO280 now and I kept thinking how we were being told to ignore the warnings without a ton of explanation behind it.

I worked for years doing cybersecurity engineering until recently become a Red Hat Consultant, so that part of me cringes a little bit when I see training telling users to ignore security warnings.  I know from experience that once users normalize ignoring warnings, even during training, that it can bleed over into their real work.  This is my first deep dive into Kubernetes and OCP and I'm already feeling like ignoring warnings is a normal thing.  I even stopped reading them after a while, which means a different more important warning could easily slip by.

I think this is a good post and I read it a couple of days ago.  Today I see it is mentioned in the DO280 class and they're asking for feedback, so from that perspective I have a few comments.

At what point in the course do you guys see this being mentioned?  I had to read it a couple of times just to make sure I understood exactly what was being said.  I'm halfway through DO280 and I admittedly rushed through DO180 because I'm taking the DO280 exam soon, but I feel like some of the explanation would go over the head of some of the people taking the class.  Even after reading through it a couple of times I still wasn't exactly sure of what the takeaway is here.  Is it that the SCC has my back so don't worry about those Kubernetes warnings or is it that the Kubernetes warnings are not enforced in some specific circumstances (namespace vs. pod security encforcement), etc.  Now this may be because a lot of this is new to me, but of course, that would be true for a lot of people taking the course.  When I read this paragraph below I was a little confused.  You say namespaces can audit, warn or enforce and then in the first highlighted section you say that if the audit annotation is set to restrited then the admission controller blocks the creation of pods.  If it's set to audit, warn, or enforce then why is an audit setting causing the prevention of a pod creation?  Am I not understanding what audit means in this context?  The second highlighted section makes sense to me, but it seems to conflict with the other sentence.

 

Jupiter_0-1683550040423.png

I'm not a big fan of prepending something like this with "TL;DR" because it kind of tells the reader that they don't need to read the whole thing when I think they do, but when I worked in the DoD we had BLUF, which is Bottom Line Up Front which served a similar purpose.  I think this post is in need of a BLUF.  It's great info but like I mentioned earlier, if it's going to be added to the DO280 course then, at least for me, it needs to be more clear on what you're trying to tell the reader.  I learned a lot reading it, but if it were in the course I would have a hard time telling you specifically what the purpose of this note would be besides giving me more background on the situation.  I'm still left wondering, so...are these warnings something I would ignore outside of this course, is SCC the thing I really need to worry about, etc.

Tags (3)
alexcorcoles
Flight Engineer
Flight Engineer
  • 3,468 Views

Oh, thanks for your feedback- it's definitely useful. We are still working out what modifications we will make (probably to DO280), and we still desire a bit more clarity about what will the future hold for future versions of OCP and Kubernetes.

I think you have found a few errors in the document; from the K8S docs:

https://kubernetes.io/docs/concepts/security/pod-security-admission/

------
# MODE must be one of `enforce`, `audit`, or `warn`.
# LEVEL must be one of `privileged`, `baseline`, or `restricted`.
pod-security.kubernetes.io/<MODE>: <LEVEL>
------

, so to *prevent* infringing workloads from being created, you'd set an annotation like "pod-security.kubernetes.io/enforce: restricted", so we need to make a few corrections on the paragraph you highlight.

About the other comments, we are collecting this feedback and will take your input for consideration- thanks.

Join the discussion
You must log in to join this conversation.