Does Redhat Offer a light weight distribution of openshift that can run of edge?

[charith1994]

I am currrently experimenting  with  linux foundation Edge computing  Solution Edgex Foundry and want to deploy it using kubenetes.I have done it using k3s and other light weight kubernetes distributions.I  wonder whether I can intergrate it with openshift at the edge.Does Redhat offer any guide or strip down version of  openshift that can run on edge devices? 

[flozano · Accepted solution]

It is on the roadmap of OpenShift having the ability to deploy lightier singe-node clusters for edge servers. It is already supported since 4.6 the ability to deploy a remote compute node. But nothing "stripped down" such as k3s AFAIK. That is not real Kubernetes. It doesn't use etcd, for example. It does not behave like a real Kubernetes cluster in many subtle and also significant ways.

As an alternative to really small edge servers, Red Hat is adding capabilities to RHEL8 as a container host such as ease of update, over the air, immutable OS images, based on technology from Atomic and CoreOS.

[mnk_dk]

"It does not behave like a real Kubernetes cluster in many subtle and also significant ways."

Can you please elaborate? I am also looking into light weight K8s solutions, and would like to know what to look out for.

[flozano · Accepted solution]

K3s is not real Kubernetes (K8s). It switches control plane components, for example etcd is replaced by kine or sqllite. Networking and storage components are also replaced without using the CNI and CSI standard APIs.

Unfortunately most people talking about K3s are selling K3s services and distributions so they mislead you into believing that it is the same as Kubernetes and avoid discussing the potential gotchas of switching those components, as if didn't create any incompatibility and other concerns such as maintainability ones. No change to any software is all good with no bad and the lack of information about "gotchas" makes me very suspicious.

It is not that k3s is bad software. It could be the right choice for many scenarios. But declaring it is "Kubernetes" without caveats is like declaring that you could replace a Linux distribution such as RHEL, SuSE, or Ubuntu with a BSD-derived one such as NetBSD and "everything would be the same, after all it's all UNIX and open source".

It is true that K3s is "Certified Kubernetes". But as you can see in the docs of the certification test tool, this only means that it passes a *minimal* set of requirements. You must check what you need beyond that minimal and how K3s will provide (or not). As they say, the devil is on the details.

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-...

Here's the list of tests from the latest conformance testing tool. While it includes "low level" things we don't teach in Red Hat Training courses, such as the downward API, it does not test other important things such as CronJobs, PVCs and RBAC.

https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md

[rootinit]

Red Hat has released an early product called "MicroShift", which is OpenShift for minimal/edge systems. Check out their official website: https://microshift.io

[flozano · Accepted solution]

Clarification remark: Microshift is NOT a product. You cannot buy a subscription to Microshift, open customer support tickets to Red Hat, not rely on a SLA for critical compatibility, stability, performance, and security bug fixes. At least not yet.

Microshift is a community project that some Red Hat employees are working on, as many other Red Hat employees work on many other open source projects, not always because they are the upstream of a Red Hat product. Yes Red Hat hears its customer's requests for lightier OpenShift in far Edge deployments and these employees are working on paid time, it is not an "off hour" project for them. And there are other contributors beyond Red Hat employees.

Comparing Microshift with K3s, Microshift does NOT replace any standard component of Kubernetes (nor of OpenShift). So you're still using etc and standard Kubernetes networking and storage. When Microshift uses OpenShift extensions and components, such as the OpenShift ingress and route controller, it uses the same container images as OpenShift, or more acuratelly, of OKD.

Microshift becomes smaller and lightier than OpenShift by:

1. Careful and opinionated configuration, including some settings that are available in upstream Kubernetes but not exposed by OpenShift cluster operators.

2. Delegating to the host OS things that OpenShift handles with extra layers, such as SDN.

3. Removal of components that are not required by typical edge scenarios, such as the OLM and MCO.

4. Custom building of trimmed down Kubernetes binaries. Kubernetes itself is highly modular and offer many features designed to be turned on or off at compilation and start up time. When a feature is expected to be off for typical Edge scenarios, it is just not included in the build of Microshift.

In the end, Microshift may feel to a sysadmin very different from standard OpenShift but it should feel to developers very close to it. Especially to ISV and hardware partners, which could keep using standard Kubernetes APIs to integrate with Microshift.

More important, by not replacing control plane components, Microshift benefits from any bugfixes and improvements from Kubernetes and OpenShift while reducing the risk of introducing issues and behaviors that are unique to it.