What kernel-level issue could possibly be causing
commands to seemingly hang - even ls and pwd?
Users are having logged in. The system load average
is low. Examination of disk issues doesn't produce
any clues. If there is a kernel-level issue that's responsible
for this sluggishness, what can be done to assess it,
without rebooting the system?
A likely kernel-level cause is processes stuck in uninterruptible sleep (D state) due to kernel I/O waits, filesystem stalls, or lock contention.
This can be assessed without rebooting by examining process states (ps, top), inspecting kernel stacks (/proc/<pid>/stack), reviewing kernel logs (dmesg, journalctl -k), and using SysRq task dumps (echo t > /proc/sysrq-trigger).
@Trevor Generally when you encounter such issues, as a sysadmin we need to check what is happening in the system, we can try to find some clues in the logs using :
dmesg to check kernel ring buffer messages for any blocked tasks warnings.
You can grep "hung" in /var/log /messages to see if the processes/threads are getting hung where.
You can also check top and look for process states as "D" - that is uninterruptible sleep for kernel level I/O blocking.
These kernel level blocking typically involve uninterruptible I/O waits on blocked resources like filesystems, storage, or network mounts.
You can use "ps -eo pid,stat | grep "^D " to check the same as well.
Possible culprits for this issue could be related to NFS, SCSI timeout, disk failures or HBA issues.
check df -h for each mounts to pinpoint the hung mount issue. Then use lsof | grep pid to catch NFS errros.
Or use cat /proc/<pid>/stat to track the stuck pid where it is stuck in kernel code.
These troubleshooting methodology ca definitely help you to find the bottleneck and most importantly the ls and pwd hung as you mentioned hints towwards a mount issue ( NFS ?).
If mount/ NFS issue is there - try remount or unmount to remove the issue otherwise take a kdump before rebooting.
Red Hat
Learning Community
A collaborative learning environment, enabling open source skill development.