What tool do you use to diagnose and resolve performance bottlenecks in RHEL ?
I use top , htop and vmstat mostly to deal such situation.
Check System-Wide Performance
top: Overview of CPU, memory, and load.
htop: Interactive, detailed process monitoring.
vmstat: Reports CPU, memory, and I/O statistics.
sar: (sysstat package) – Historical performance metric
Check High CPU Usage
mpstat (sysstat package):Shows CPU usage per core.
pidstat: CPU usage per process.
perf: Detailed performance profiling.
top / htop:
Provides a dynamic, real-time view of system processes and resource usage (CPU, memory, swap).
Helps identify processes consuming excessive resources.1
htop is an interactive improvement over top.
vmstat:
Reports virtual memory statistics, including CPU, memory, I/O, and system activity.2
Useful for identifying memory pressure and I/O bottlenecks.3
iostat:
Provides detailed I/O statistics for disk devices.4
Helps identify disk I/O bottlenecks and assess disk performance.5
mpstat:
Reports CPU utilization statistics for individual processors.6
Helps identify CPU bottlenecks and assess CPU load distribution.7
free:
Displays the amount of free and used memory in the system.
Helps identify memory pressure and swap usage.
sar (System Activity Reporter):
Collects and reports system activity data over time.
Useful for historical analysis and identifying trends.
It records the data that the other tools show in real time.
dstat:
A versatile resource monitoring tool that combines vmstat, iostat, mpstat, and netstat output.
Provides a comprehensive overview of system performance in a single output.
netstat / ss:
Displays network connections, routing tables, and network interface statistics.
Helps identify network bottlenecks and assess network performance.8
ss is a newer and more efficient replacement for netstat.
tcpdump / wireshark:
Captures network traffic for detailed analysis.
Helps identify network latency and packet loss.9
wireshark is a gui version of tcpdump.
iftop:
Displays real-time bandwidth usage on network interfaces.
Helps identify network bandwidth bottlenecks.
strace:
Traces system calls and signals made by a process.
Helps identify system call bottlenecks and debug application issues.
lsof (List Open Files):
Lists open files and the processes that are using them.
Helps identify processes holding onto resources and diagnose file descriptor leaks.
perf:
A powerful performance analysis tool that can profile CPU performance and identify hotspots in code.
Very usefull for developers.
gdb (GNU Debugger):
A debugger that allows you to examine the state of a running program.
Very usefull for developers.
blktrace:
Traces block device I/O operations.
Helps identify disk I/O bottlenecks and assess disk performance.
systemd-journald / journalctl:
Collects and manages system logs.
Helps identify system errors and warnings.
auditd / ausearch:
Audits system events and logs security-related information.
Helps identify security vulnerabilities and track system activity.
Tools for Diagnosing and Resolving Performance Bottlenecks in RHEL
CPU Performance
top / htop – Shows real-time CPU usage per process
mpstat (from sysstat package) – Displays CPU usage across cores
pidstat – Monitors CPU usage per process over time
perf – Provides in-depth CPU performance profiling
Memory Usage
free -m – Displays memory usage and swap utilization
vmstat – Reports memory, CPU, and I/O statistics
sar -r – Collects and reports memory statistics over time
smem – Analyzes memory usage per process
Disk I/O Performance
iostat – Reports disk I/O utilization and latency
iotop – Displays real-time disk usage per process
blktrace / blkstat – Provides detailed block I/O tracing
df -h / du -sh – Checks disk space usage
Network Performance
netstat -s / ss – Displays active network connections and statistics
iftop – Monitors real-time network bandwidth usage
ip -s link – Displays network interface statistics
nload – Provides a graphical real-time network traffic overview
System-Wide Performance Monitoring
dstat – Combines CPU, disk, network, and memory stats in real-time
sar (from sysstat package) – Collects historical performance data
strace – Traces system calls of a process to detect slow operations
lsof – Lists open files to find processes causing resource contention
tuned-adm – Optimizes system performance based on predefined profiles
Hi, in RHEL9 i have really learned to rely on "systemd-cgtop" to show me the realtime usage of resources when i triage bottlenecks. Sometimes limits.conf file needs tweaking also.
free -m helps with servers using swap and need more.
lsof is nice too when things aren't working and you need to find what is the "hold up".....
EDIT: Lastly strace when i am still scratching my head to figure out what is wrong.
Thank you.
Red Hat
Learning Community
A collaborative learning environment, enabling open source skill development.