Openshift 4.6.8 - Baremetal - Bootstrap node Fails to Start

[sam31]

Can SSH to bootstrap and Master nodes, but the Bootstrap node fails to initiate the Kubernetes

Any idea what seems to be going wrong ?

Getting the following error :-

# journalctl -b -f -u bootkube.service
...
...
 13:15:51 bootstrap.openshift4.example.lo bootkube.sh[769]: Starting cluster-bootstrap...
 13:15:52 bootstrap.openshift4.example.lo bootkube.sh[769]: Trying to pull registry.svc.ci.openshift.org/origin/4.6-2021-01-18-060154@sha256:e4fca6e7d9d2df952886ad9d8296a04713996fa42355addb675f52dbd522b421...
 13:15:58 bootstrap.openshift4.example.lo bootkube.sh[769]: Getting image source signatures
 13:16:02 bootstrap.openshift4.example.lo bootkube.sh[769]: Copying blob sha256:07697aebc83bd9d07a1c0a0e699ec8b8a78ec83da53fde8fd268afe0768fd8a2
 13:16:02 bootstrap.openshift4.example.lo bootkube.sh[769]: Copying blob sha256:48ed3bfd822646e50676cd7606af43e984db141bb1755904362f1eb64684c68a
 13:16:02 bootstrap.openshift4.example.lo bootkube.sh[769]: Copying blob sha256:d327c1598329494579ba3d62999df41f11bff9a2bfad57fb49b30324404ac42a
 13:16:02 bootstrap.openshift4.example.lo bootkube.sh[769]: Copying blob sha256:013253fab02289008c1c3b1cf3062ff08e2898b1e9862a0b8766ac8d42e5ab99
 13:16:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Copying blob sha256:b6bf10db219d963c78a833b260a96bd3bc1972248b06e4c4fcb8820d7a8943cb
 13:16:14 bootstrap.openshift4.example.lo bootkube.sh[769]: Copying config sha256:78d0d76b9b4bc37748db9b79f37632528b4cb3286c28ae9239755487353bdc06
 13:16:14 bootstrap.openshift4.example.lo bootkube.sh[769]: Writing manifest to image destination
 13:16:14 bootstrap.openshift4.example.lo bootkube.sh[769]: Storing signatures
 13:16:17 bootstrap.openshift4.example.lo podman[6649]: 2021-01-18 13:16:17.381973916 +0000 UTC m=+25.402634298 image pull
 13:16:17 bootstrap.openshift4.example.lo bootkube.sh[769]: 78d0d76b9b4bc37748db9b79f37632528b4cb3286c28ae9239755487353bdc06
 13:16:17 bootstrap.openshift4.example.lo podman[6779]: 2021-01-18 13:16:17.489854734 +0000 UTC m=+0.089026898 container create 75bbebbfbb3d17fff13c604b3bde5e2e57b3f9688a71f121e26e7c17198775e5 (image=registry.svc.ci.openshift.org/origin/4.6-2021-01-18-060154@sha256:e4fca6e7d9d2df952886ad9d8296a04713996fa42355addb675f52dbd522b421, name=random)
 13:16:17 bootstrap.openshift4.example.lo podman[6779]: 2021-01-18 13:16:17.781045114 +0000 UTC m=+0.380217296 container init 75bbebbfbb3d17fff13c604b3bde5e2e57b3f9688a71f121e26e7c17198775e5 (image=registry.svc.ci.openshift.org/origin/4.6-2021-01-18-060154@sha256:e4fca6e7d9d2df952886ad9d8296a04713996fa42355addb675f52dbd522b421, name=random)
 13:16:17 bootstrap.openshift4.example.lo podman[6779]: 2021-01-18 13:16:17.787128686 +0000 UTC m=+0.386300858 container start 75bbebbfbb3d17fff13c604b3bde5e2e57b3f9688a71f121e26e7c17198775e5 (image=registry.svc.ci.openshift.org/origin/4.6-2021-01-18-060154@sha256:e4fca6e7d9d2df952886ad9d8296a04713996fa42355addb675f52dbd522b421, name=random)
 13:16:17 bootstrap.openshift4.example.lo podman[6779]: 2021-01-18 13:16:17.787192343 +0000 UTC m=+0.386364745 container attach 75bbebbfbb3d17fff13c604b3bde5e2e57b3f9688a71f121e26e7c17198775e5 (image=registry.svc.ci.openshift.org/origin/4.6-2021-01-18-060154@sha256:e4fca6e7d9d2df952886ad9d8296a04713996fa42355addb675f52dbd522b421, name=random)
 13:16:17 bootstrap.openshift4.example.lo bootkube.sh[769]: Starting temporary bootstrap control plane...
 13:16:17 bootstrap.openshift4.example.lo bootkube.sh[769]: E0204 13:16:17.803393       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://localhost:6443/api/v1/pods: dial tcp [::1]:6443: connect: connection refused
 13:16:58 bootstrap.openshift4.example.lo bootkube.sh[769]: [#206] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
 13:16:59 bootstrap.openshift4.example.lo bootkube.sh[769]: [#207] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
 13:16:59 bootstrap.openshift4.example.lo bootkube.sh[769]: [#208] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
 13:16:59 bootstrap.openshift4.example.lo bootkube.sh[769]: [#209] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
 13:16:59 bootstrap.openshift4.example.lo bootkube.sh[769]: [#210] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "0000_00_cluster-version-operator_00_namespace.yaml" namespaces.v1./openshift-cluster-version -n
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "0001_00_cluster-version-operator_03_service.yaml" services.v1./cluster-version-operator -n openshift-cluster-version
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "00_openshift-etcd-ns.yaml" namespaces.v1./openshift-etcd -n
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "00_openshift-kube-apiserver-ns.yaml" namespaces.v1./openshift-kube-apiserver -n
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "00_openshift-kube-apiserver-operator-ns.yaml" namespaces.v1./openshift-kube-apiserver-operator -n
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "00_openshift-kube-controller-manager-ns.yaml" namespaces.v1./openshift-kube-controller-manager -n
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "00_openshift-kube-controller-manager-operator-ns.yaml" namespaces.v1./openshift-kube-controller-manager-operator -n
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "00_openshift-kube-scheduler-ns.yaml" namespaces.v1./openshift-kube-scheduler -n
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "04-openshift-machine-config-operator.yaml" namespaces.v1./openshift-machine-config-operator -n
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "99_cloud-creds-secret.yaml" secrets.v1./vsphere-creds -n kube-system
 13:17:03 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "99_kubeadmin-password-secret.yaml" secrets.v1./kubeadmin -n kube-system
 13:17:04 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "99_openshift-cluster-api_master-user-data-secret.yaml" secrets.v1./master-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
 13:17:04 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "99_openshift-cluster-api_worker-user-data-secret.yaml" secrets.v1./worker-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
 13:17:04 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "cco-namespace.yaml" namespaces.v1./openshift-cloud-credential-operator -n
 13:17:04 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "cloud-provider-config.yaml" configmaps.v1./cloud-provider-config -n openshift-config: namespaces "openshift-config" not found
 13:17:04 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "cluster-config.yaml" configmaps.v1./cluster-config-v1 -n kube-system
 13:17:05 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "cluster-ingress-00-namespace.yaml" namespaces.v1./openshift-ingress-operator -n
 13:17:05 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "configmap-admin-kubeconfig-client-ca.yaml" configmaps.v1./admin-kubeconfig-client-ca -n openshift-config: namespaces "openshift-config" not found
 13:17:05 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "configmap-csr-controller-ca.yaml" configmaps.v1./csr-controller-ca -n openshift-config-managed: namespaces "openshift-config-managed" not found
 13:17:05 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "configmap-initial-etcd-serving-ca.yaml" configmaps.v1./initial-etcd-ca -n openshift-config: namespaces "openshift-config" not found
 13:17:05 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "configmap-sa-token-signing-certs.yaml" configmaps.v1./sa-token-signing-certs -n openshift-config-managed: namespaces "openshift-config-managed" not found
 13:17:06 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "etcd-ca-bundle-configmap.yaml" configmaps.v1./etcd-ca-bundle -n openshift-config: namespaces "openshift-config" not found
 13:17:06 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "etcd-client-secret.yaml" secrets.v1./etcd-client -n openshift-config: namespaces "openshift-config" not found
 13:17:06 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "etcd-host-service-endpoints.yaml" endpoints.v1./host-etcd -n openshift-etcd
 13:17:06 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "etcd-host-service.yaml" services.v1./host-etcd -n openshift-etcd
 13:17:06 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "etcd-metric-client-secret.yaml" secrets.v1./etcd-metric-client -n openshift-config: namespaces "openshift-config" not found
 13:17:07 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "etcd-metric-serving-ca-configmap.yaml" configmaps.v1./etcd-metric-serving-ca -n openshift-config: namespaces "openshift-config" not found
 13:17:07 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "etcd-metric-signer-secret.yaml" secrets.v1./etcd-metric-signer -n openshift-config: namespaces "openshift-config" not found
 13:17:07 bootstrap.openshift4.example.lo bootkube.sh[769]: Skipped "etcd-namespace.yaml" namespaces.v1./openshift-etcd -n  as it already exists
 13:17:07 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "etcd-service.yaml" services.v1./etcd -n openshift-etcd
 13:17:08 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "etcd-serving-ca-configmap.yaml" configmaps.v1./etcd-serving-ca -n openshift-config: namespaces "openshift-config" not found
 13:17:08 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "etcd-signer-secret.yaml" secrets.v1./etcd-signer -n openshift-config: namespaces "openshift-config" not found
 13:17:08 bootstrap.openshift4.example.lo bootkube.sh[769]: Failed to create "kube-apiserver-serving-ca-configmap.yaml" configmaps.v1./initial-kube-apiserver-server-ca -n openshift-config: namespaces "openshift-config" not found
 13:17:08 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "kube-cloud-config.yaml" secrets.v1./kube-cloud-cfg -n kube-system
 13:17:08 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "kube-system-configmap-root-ca.yaml" configmaps.v1./root-ca -n kube-system
 13:17:09 bootstrap.openshift4.example.lo bootkube.sh[769]: Created "machine-config-server-tls-secret.yaml" secrets.v1./machine-config-server-tls -n openshift-machine-config-operator

[AlejandroC]

Hi Sam!

I've found those error logs in some scenarios:

1) I used the same ignition file for bootstrap and master nodes

2) There was a mistake in the install-config.yaml file (wrong pull-secret or insuficient credentials)

In other cases, reading those errors temporarily can be a normal scenario, but I assume the bootstrap node is never installed based on your initial post.

I would recommend you to clean all the cluster machines and start over (never reuse installed nodes when generating new ignition files). Review that you are pointing the bootstrap node to bootstrap.ign and the master nodes to master.ign.

It would be also useful to see your install-config.yaml file and the output of:

journalctl -b -f -u release-image.service -u bootkube.service

To discard any issues when pulling the installation images.

[sam31 · Accepted solution]

Thanks @AlejandroC for responding, correct the error was in "install-config.yaml", needed to add the following :-

networking:
clusterNetwork:
- cidr: 172.30.0.0/15
hostPrefix: 18
networkType: OpenShiftSDN
serviceNetwork:
- 172.28.0.0/15
machineNetwork:
- cidr: 172.16.0.0/24 ------->>>>> Needed to add this bit (The subnet used Mgmt (Primary) interface of nodes by Nodes) / Else it defaults to 10.0.0.0/16, hence the mismatch

[AlejandroC]

Thanks for sharing Sam!