Red Hat Linux Interview Series 32

[Chetan_Tiwary_]

Q.)  How do I configure kdump on RHEL systems?

Bonus Q.) Explain this :

INFO: task <process>:<pid> blocked for more than 120 seconds.  

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 

How to enable collection of a vmcore, via a panic, when a block task event is encountered?

Q.)  How will you troubleshoot this issue :  kdump service restart failed with below error :

Your running kernel is using more than 70% of the amount of space you reserved for kdump, you should consider increasing your crashkernel reservation

Q.) How will you check / analyze virtual memory info from the core dump ?

Level L3 and above

I'll be posting a series of Linux-related questions covering various skill levels. Feel free to share your insights and expertise. Your contributions will benefit learners at all stages, from those in current roles to those preparing for Linux interviews.

[Trevor · Accepted solution]

Let's have a look at that last question first:

Q.) How will you check / analyze virtual memory info from the core dump ?

To address this question, I'm going to recommend crash - a dedicated
debugging tool on Linux systems, that allows the examination of
various aspects of the kernel memory, including virtual addresses,
page tables, and memory regions, within the core dump file.  The
crash tool can provide detailed information about the virtual memory
state of the Linux system at the time of a crash.  Again, this is my
recommendation because the question is making reference to a
core dump (kernel core dump) file.

crash is not only a debugging tool, but it's interactive.  What this means
is that when its launched, you interact with it via its internal prompt.
There are several commands within the crash tool that allow you to
perform an interrogation of a kernel core dump file.  To analyze the 
virtual memory info referenced in this quesiton, the primary crash tool
command will be "vm".  Along with the "vm" command, there are several 
options/switches that allow an expanded interrogation of the virtual
memory information.  The "vm" command shows anything loaded in the
virtual memory at the time of the crash.

If you would like to examine the use of the crash debugging tool, creating
a core dump file would certainly be beneficial. If you'd like to see how to go
about this, let me know, and I'll post the steps that you'll need to follow
in order to put the necessary pieces in place.   

Kernel core dumps are complex! They are not simply copies of system
memory.  In my opinion, a healthy familiarity with kernel memory
management concepts, like "page tables" and "virtual address spaces", 
would be most beneficial in digesting the information provided by the
crash tool.

[Chetan_Tiwary_]

@Trevor wonderfully explained!

[Trevor]

I'd like to now attempt to address that first question:
Q.)  How do I configure kdump on RHEL systems?

Answer:  

To configure kdump on RHEL systems, you've got to complete the

following tasks:

1)  Configuring the Memory Usage

2)  Update the configuration file for the GRUB 2 boot loader

3)  (Optional) - Configuring the Target Type

4)  Enable the Service

Let's have a closer look at each of these.

1)  To configure the amount of memory that is reserved for the kdump kernel, you'll need to edit the /etc/default/grub file.  You'll edit this file by
adding the following parameter to the list of kernel options on the
GRUB_CMDLINE_LINUX  line:

             crashkernel=size
             # where size refers to the amount of memory to reserve

An example of this setting could be:   

             crashkernel=64M     
             # this reserves 64 MB of memory

Note:  If you're using RHEL 9, do not use crashkernel=auto!!!  The
crashkernel=auto  parameter is deprecated (dropped) in RHEL9. 
There's a very good likelihood that there is already a crashkernel= 
parameter set in your /etc/default/grub file.  So, I would just leave that
one intact - unless you just feel like tinkering, or you know that a better
value (size) should be configured.

2) If you did add (or adjust) the crashkernel=size parameter, then you'll
need to rebuild (update) the configuration file for the GRUB 2 boot loader. 
This is achieved using the following command:

               #  grub2-mkconfig  -o  /boot/grub2/grub.cfg

You can verify the new crashkernel=size parameter in the /boot/grub2/grub.cfg file using the following command:

               # grubby  --info  DEFAULT

3) Configuring the Target Type essentially allows you to configure where
you want the vmcore (kernel crash dump) file to be stored.  By default, the
vmcore file will be stored in the /var/crash directory.  Remember, this step
is OPTIONAL!!!  If you desire to change this location (target), have a look
at the "path" option, in the kdump.conf file.  Have fun!

4)  Finally, from the RHCSA-level studies, enable the kdump.service to
start each time the Linux system boots.

There are other things that you'll want to do after configuring kdump, such
as checking the current reserved memory size:

            #  kdumpctl  showmen

or even testing the kdump by manualy crashing the machine.  I won't
bother with those in this post, because that doesn't directly address the
question that I'm responding to, which is: 
"How do I configure kdump on RHEL systems?"

Bonus Coverage:

kdump is a service which provides a crash dumping mechanism. 

The kdump.service enables you to save the contents of the system
memory for analysis.

kdump uses the kexec system call to boot into the second kernel (a
capture kernel) without rebooting; and then captures the contents of the
crashed kernel’s memory (a crash dump or a vmcore) and saves it into a
file. The second kernel resides in a reserved part of the system memory.

kdump is a Linux kernel feature that creates crash dumps to help debug
and identify the cause of a kernel crash. When a kernel crash occurs,
kdump saves a memory image of the system into a file, which can be used for analysis: 

Here's how kdump works:

• Boot into a capture kernel: Kdump uses the kexec system call to boot into a second kernel, called the capture kernel, without restarting the system.
• Capture the crash: The capture kernel captures the contents of the crashed kernel's memory.
• Save the crash dump: The crash dump, also known as a vmcore file, is saved into a file.