Helm chart , Templates or Kustomization file ?

[Chetan_Tiwary_]

Hi All , I have a question ( complex  :smileyhappy: ) about templates in Openshift.

I was wondering when is it  appropriate to use a template, a helm chart  or a kustomization file ?

Please share your inputs on this important topic. 

Thanks !!

[joelbirchler]

Templates are no longer recommended. We teach the topic, because a student might need to use or update a template that was already provided.

Helm charts were created as a way to package up an application for distribution. If I was working on a platform  system (such as a database or tool) that I wanted others to consume, then a helm chart would be approapriate. Helm is great when you want to create something for another team to configure within a set of paramaters.

Kustomize is a feature built into kubectl and oc, that solves a similar problem as helm charts. In my opinion, the best use-case for this is an app team that needs to configure that application with different parameters for different environments (qa, stage, prod).

So, in my opinion, I would choose kustomize for parameterization of an app that my team fully owns. I would choose helm for when the application is being distributed to other teams.

As a quick aside, if I am packaging my application for use by other teams and wanted more advanced coordination or simply wanted it to appear in the operator hub, I would create an operator instead. The operator might use helm underneath the hood.

[flozano]

All of them (helm, helm, and kustomize) originates outside of Kubernetes proper and only kustomize become integreated into the kubectl tool, so part of "standard" or "vanilla" Kubernetes. And when discussing them you must also talk about operators, as all of them are kubernetes-native options to automate and simplify application deployment.

All are open source so nothing prevents any Kubernetes vendor from supporting any of them, and OpenShift supports all four. Helm came late to OpenShift because the first releases had security issues and would grant any developer cluster admin access to any cluster. Helm 3 solved these issues and is not only supported by OpenShift.

Recent releases include certified helm charts in the developer catalog as well as operators and templates. Kustomize is not something you'd use for an "application marketplace" but it is useful to complement all approaches and also valuable by itself.

Templates

Templates are the simplest to use and quicker to learn. They are also arguably the less powerfull. I don't agree with "not recommended" and I didn't hear that fom OpenShift product management -- they declare templates as "feature complete" and so will not get new features nor enhancements, but they are fully supported with no EoL date set. Unfortunately no Kubernetes distro other than OpenShift adopted templates so they are an OpenShift-only alternative.

Templates provide simple string substitution, validation, and generation of parameters so a template allows basic customizability between deployment environments such as dev and prod or geos. Templates can be stored as kubernetes API resources and AFAIK is the only alternative than can work entirely from inside an OpenShift cluster. All others require additional infrastructure outside of a cluster (or deployed into a cluster).

Helm

Helm packages kubernets manifests into a "chart" and charts are stored in repositories which are basically http servers. When you deploy an application from a chart, helm keeps track of the chart, its release, and parameters in a kubernetes secret or a developer's workstation file (I'd recomend the first). Helm keeps a history of that tracking data for every change you make to a deployed application. You could even uninstall an application and later reinstall it from that recorded data.

Charts can also depend on other charts and follow these dependencies during application installation and update. Helm allows changing parameters of deployed applications and updating these to new versions of a chart.

Helm also provide simple hooks to run during installation and update of applications, so you could do things such as fire a Kubernetes job to load a database with test data or update a database schema.

Helm chart parameters (called values) are complex YAML structure and could embeded complete text and binary files. That makes helm charts very customizable but can also make using them very challenging as chart developers try to allow for "all possible" customizations and introspection/documentation is not very good.

Kustomize

While helm was designed to "package" applications into units that "easy" to consume, kustomize was designed to manage patching and generating Kubernetes resource manifests that you then feed into kubectl apply (or oc apply). It works on a file tree with many "layers" (folders) and each layer could represent a different customization, for example prod and dev environments.

As file trees, kustomize layers are naturally managed on Git repos and integrate seamless into GitOps workflows -- and most GitOps tools such as ArgoCD and Flux assume you're using kustomize layers.

The layering approach allows arbirarily complex customizations, and generators make it easy to perform tasks such as embeding a TLS certificate into a secret. From a parametrization perspective you could say that kustomize is as powerful as helm, and you could emulate hooks by simply including kubernetes job manifests on your layers, but I don't know how close or how far that would be from helm hooks. And kustomize provides no dependency between file trees.

Managing helm applications and their versions/values could become very complex and kustomize helps a lot there. If you wish/need you could also use kustomize to manage templates and operators, or just the CRs from operators, as all of them rely on YAML files.

Operators

Operators package potentialy complex logic to install, update, scale, manage, monitor, backup, recover from failures and potentially all other aspects of "operationalizing" an application. An application deployed from an operator could be as easy to manage as a service from a cloud provider.

Operators define custom resources and these custom resources provide arbitrarily complex customization of an application deployment. Nothing prevents you from embeding complete configuration files inside a custom resources, as with helm values, but It is recommended that operators are "oppinionated" because allowing arbiraty customization makes the operator logic more complex, so you trade flexibility with easy of management and automation.

A custom resource can provide an easier syntax and smarter defaults compared to raw configuration files from a containerized application or helm values because an operator acts on every change to a custom resource while a helm chart hook acts only during installation and updates. There's no helm hook for "change the configuration of that application" you'd be required to unsinstall and reinstall or update the application to a new release of its chart. So operators are able to apply configuration changes without downtime.

Operators can define logic that is far more complex than helm chart hooks. The Operator SDK defines an operator maturity model that classifies the feature set of an operator into levels of power, complexity, and "self-management" and allow using  helm charts as input to generate an operator of level 1. Other levels require custom operator logic, beyond the capabilities of helm.

Operators can depend on other operators and custom resources are versioned. The Operator Life Cycle Manager install, update, and introspect operators. It's a shame that many community projects and some companies don't package their operators for the OLM and instead provide helm charts to deploy their operators(!).

If you package your operator for the OLM, then your operator is published as a catalog container image that includes dependencies, update graphs, and other metadata.

My Recommendations for an OpenShift administrator

A number of OpenShift add-ons still rely on templates as the supported way to deploy applications, so as an OpenShift administrator you must learn them.

I also recomend learning kustomize first and with more depth than helm. Current community and supported chart catalogs are not that large and, well, operators might suplant helm as a way of deploying "packaged" applications from ISVs. Not to mention that most GitOps tools rely on kustomize as a prerequisite.

Of course, if your organization is developing helm charts for internal applications, or you're buying apps from an ISV that provides helm chart, them learning helm becomes a priority but don't stop there.

An OpenShift admin must understand how to use the OLM to manage operators and I also recomend that they learn how to use Kustomize to manage custom resources from operators.

I'll let someone more atune with current development practices voice recommendations for OpenShift developers. I'll just say that, if you're an ISV or HW vendor, learn not only operators but how to package for the OLM. It'll serve you long-term better than helm alone.