Lab "backup-restore": hooks skipped during application-consistent backup?

[donaldsebleung]

This is the 2nd time I've attempted the guided exercise "backup-restore".

The 1st time I was able to complete the application-consistent backup successfully with all rows preserved at the timestamp the backup was created. However, this time round, my DB was corrupt twice(!) with the application-consistent backup even though I have followed the instructions to the best of my ability and took note of the following details:

1. Ensure the label selectors include both the MariaDB app "app=mariadb" and the Kubernetes namespace itself "kubernetes.io/metadata.name=database"

2. Ensure "namespace" is in the includedResources to preserve the OpenShift project UID range

3. Ensure "pvc" / "pv" are in the includedResources to backup the database volumes

4. Ensure "pod" / "pods" are in the includedResources for the backup hooks - tried both singular "pod" and plural "pods" form

5. Double-checked the commands in the backup hooks match the exercise instructions

Unfortunately, with the screenshots below, the restored DB in namespace "database-backup" still has corrupt application logs and I am missing the last few rows from the backup timestamp. This suggests the backup hooks were skipped somehow.

The backup logs related to the hookPhase also show msg="stdout: " and msg="stderr: " with no output - does this mean the backup hooks were skipped, and why?

I'm sure there's something simple I missed so would appreciate anyone pointing out my issue, thanks!

[TobyTheHutt]

Hi @donaldsebleung 

Did the backup and the restore both finish with a success status? Both backups and restores can finish even though they encounter issues on the way. So it's important to verify that pre- and post-tasks ran without any issues for both.

[donaldsebleung]

Both the backup and restore operations finished successfully with "Completed" status. Unfortunately, I am not able to troubleshoot further beyond the provided screenshots as I have since run the "lab finish" command to clean up the resources.

The course notes mention that the backup hooks are skipped without warning or error if no pods are selected for executing the backup hooks, e.g. because the "pods" resource is missing from the includedResources.

I've included the "pods" resource under includedResources as mentioned in the course and lab notes, but I'm wondering if they might be skipped since my ".spec.template.hooks.resources" do not explicitly include label selectors to indicate which pod(s) to run the backup hooks on?

[TobyTheHutt · Accepted solution]

@donaldsebleung I see that you left out `sync;` at the end of your pre-hook which is supposed to flush all remaining dirty pages to storage. It's likely not the cause of your issues, but good to know anyway.

If the issue occurs again, provide some more logs from the backup/restore process, specifically from when the hooks were run.

I made some other assumptions in earlier versions of this post, regarding potential flaws in the overall backup strategy, which have been wrong. I did not consider that we use CSI snapshots, which suits the backup scenario.

[donaldsebleung]

Thank you! I knew I was missing something simple (-:

Having mentioned it, I do remember including the "sync" command in my 1st attempt now which escaped my attention the 2nd time round somehow. Will be more careful reading the instructions going forward and probably re-attempt the guided exercise a 3rd time before my scheduled exam on 2025-12-17.