In case other students run into the same issue, during Lab: OpenShift Logging, I hit a bug described in:
- Loki ingesters 0/1 in RHOCP 4
where the ingester pod for loki was not able to become ready:
[student@workstation logging-review]$ oc get po
NAME READY STATUS RESTARTS AGE
cluster-logging-operator-554849f7dd-g4d7j 1/1 Running 1 17h
collector-2n64n 1/1 Running 0 5m46s
collector-4xj56 1/1 Running 0 5m47s
collector-9qdw9 1/1 Running 0 5m46s
collector-dt926 1/1 Running 0 5m47s
collector-hk559 1/1 Running 0 5m47s
collector-tkw6b 1/1 Running 0 5m47s
logging-loki-compactor-0 1/1 Running 0 6m10s
logging-loki-distributor-6c64dc7796-7n6rt 1/1 Running 0 6m10s
logging-loki-gateway-7cd549bc89-w26r7 2/2 Running 0 6m9s
logging-loki-gateway-7cd549bc89-zhws2 2/2 Running 1 (6m8s ago) 6m9s
logging-loki-index-gateway-0 1/1 Running 0 6m9s
logging-loki-ingester-0 0/1 Running 0 6m10s
logging-loki-querier-788c8965dc-ckwwg 1/1 Running 0 6m10s
logging-loki-query-frontend-67885d5589-dnfz7 1/1 Running 0 6m10s
logging-view-plugin-5b9b5b7bdc-lwlzx 1/1 Running 0 5m59s
I was able to solve it with:
oc --as system:admin -n openshift-logging exec pod/logging-loki-ingester-0 -- bash -c 'rm -rf /tmp/wal/0*'
oc --as system:admin -n openshift-logging exec pod/logging-loki-ingester-0 -- bash -c 'rm -rf /tmp/wal/checkpoint.*'
oc --as system:admin -n openshift-logging delete pod/logging-loki-ingester-0
cf. Loki ingester pod logging-loki-ingester-0 does not become Ready
- deleting loki instance alone did not solve the issue.
I think I hit the issue because I created and deleted several instances of loki, CLF and CL during the lab.
