cancel
Showing results for 
Search instead for 
Did you mean: 
Trevor
Commander Commander
Commander
  • 288 Views

Kernel-level Issue

What kernel-level issue could possibly be causing
commands to seemingly hang - even ls and pwd?
Users are having logged in. The system load average
is low. Examination of disk issues doesn't produce
any clues. If there is a kernel-level issue that's responsible
for this sluggishness, what can be done to assess it,
without rebooting the system?

Trevor "Red Hat Evangelist" Chandler
Labels (3)
2 Replies
Sunnykumar1
Flight Engineer
Flight Engineer
  • 218 Views

@Trevor 

A likely kernel-level cause is processes stuck in uninterruptible sleep (D state) due to kernel I/O waits, filesystem stalls, or lock contention.

This can be assessed without rebooting by examining process states (ps, top), inspecting kernel stacks (/proc/<pid>/stack), reviewing kernel logs (dmesg, journalctl -k), and using SysRq task dumps (echo t > /proc/sysrq-trigger).

0 Kudos
Chetan_Tiwary_
Community Manager
Community Manager
  • 187 Views

@Trevor Generally when you encounter such issues, as a sysadmin we need to check what is happening in the system, we can try to find some clues in the logs using :

dmesg to check kernel ring buffer messages for any blocked tasks warnings.

You can grep "hung" in /var/log /messages  to see if the processes/threads are getting hung where.

You can also check top  and look for process states as "D" - that is uninterruptible sleep for kernel level I/O blocking.

These  kernel level blocking typically involve uninterruptible I/O waits on blocked resources like filesystems, storage, or network mounts. 

You can use "ps -eo pid,stat | grep "^D " to check the same as well.

Possible culprits for this issue could be related to  NFS, SCSI timeout, disk failures or HBA issues. 

check  df -h  for each mounts to pinpoint the hung mount issue. Then use lsof | grep pid  to catch NFS errros. 

Or use cat /proc/<pid>/stat to track the stuck pid where it is stuck in kernel code. 

These troubleshooting methodology ca definitely help you to find the bottleneck and most importantly the ls and pwd hung as you mentioned hints towwards a mount issue ( NFS ?).

If mount/ NFS issue is there - try remount or unmount to remove the issue otherwise take a kdump before rebooting. 

Join the discussion
You must log in to join this conversation.