What tool do you use to diagnose and resolve performance bottlenecks in RHEL ?
Hello,
Best Practices for Performance Tuning in RHELRegular Monitoring: Continuously monitor system performance to track changes and identify new bottlenecks. Use tools like top, htop, and iostat.
In addtion to that, you can use AI-driven tools to track key metrics, monitor application behavior, trace errors and exceptions, understand the factors impacting your application in real time, and ensure your application is always running at peak efficiency.
It helps to proactively identify bottlenecks, resource constraints, and potential failures, ensuring seamless application performance, reduced downtime, and a smooth user experience.
Thanks
Gopinath
I use top, htop, vmstat, iostat, sar, and perf to diagnose CPU, memory, and disk performance bottlenecks in RHEL.
I use top , htop and vmstat mostly to deal such situation.
Check System-Wide Performance
top: Overview of CPU, memory, and load.
htop: Interactive, detailed process monitoring.
vmstat: Reports CPU, memory, and I/O statistics.
sar: (sysstat package) – Historical performance metric
Check High CPU Usage
mpstat (sysstat package):Shows CPU usage per core.
pidstat: CPU usage per process.
perf: Detailed performance profiling.
top / htop:
Provides a dynamic, real-time view of system processes and resource usage (CPU, memory, swap).
Helps identify processes consuming excessive resources.1
htop is an interactive improvement over top.
vmstat:
Reports virtual memory statistics, including CPU, memory, I/O, and system activity.2
Useful for identifying memory pressure and I/O bottlenecks.3
iostat:
Provides detailed I/O statistics for disk devices.4
Helps identify disk I/O bottlenecks and assess disk performance.5
mpstat:
Reports CPU utilization statistics for individual processors.6
Helps identify CPU bottlenecks and assess CPU load distribution.7
free:
Displays the amount of free and used memory in the system.
Helps identify memory pressure and swap usage.
sar (System Activity Reporter):
Collects and reports system activity data over time.
Useful for historical analysis and identifying trends.
It records the data that the other tools show in real time.
dstat:
A versatile resource monitoring tool that combines vmstat, iostat, mpstat, and netstat output.
Provides a comprehensive overview of system performance in a single output.
netstat / ss:
Displays network connections, routing tables, and network interface statistics.
Helps identify network bottlenecks and assess network performance.8
ss is a newer and more efficient replacement for netstat.
tcpdump / wireshark:
Captures network traffic for detailed analysis.
Helps identify network latency and packet loss.9
wireshark is a gui version of tcpdump.
iftop:
Displays real-time bandwidth usage on network interfaces.
Helps identify network bandwidth bottlenecks.
strace:
Traces system calls and signals made by a process.
Helps identify system call bottlenecks and debug application issues.
lsof (List Open Files):
Lists open files and the processes that are using them.
Helps identify processes holding onto resources and diagnose file descriptor leaks.
perf:
A powerful performance analysis tool that can profile CPU performance and identify hotspots in code.
Very usefull for developers.
gdb (GNU Debugger):
A debugger that allows you to examine the state of a running program.
Very usefull for developers.
blktrace:
Traces block device I/O operations.
Helps identify disk I/O bottlenecks and assess disk performance.
systemd-journald / journalctl:
Collects and manages system logs.
Helps identify system errors and warnings.
auditd / ausearch:
Audits system events and logs security-related information.
Helps identify security vulnerabilities and track system activity.
Tools for Diagnosing and Resolving Performance Bottlenecks in RHEL
CPU Performance
top / htop – Shows real-time CPU usage per process
mpstat (from sysstat package) – Displays CPU usage across cores
pidstat – Monitors CPU usage per process over time
perf – Provides in-depth CPU performance profiling
Memory Usage
free -m – Displays memory usage and swap utilization
vmstat – Reports memory, CPU, and I/O statistics
sar -r – Collects and reports memory statistics over time
smem – Analyzes memory usage per process
Disk I/O Performance
iostat – Reports disk I/O utilization and latency
iotop – Displays real-time disk usage per process
blktrace / blkstat – Provides detailed block I/O tracing
df -h / du -sh – Checks disk space usage
Network Performance
netstat -s / ss – Displays active network connections and statistics
iftop – Monitors real-time network bandwidth usage
ip -s link – Displays network interface statistics
nload – Provides a graphical real-time network traffic overview
System-Wide Performance Monitoring
dstat – Combines CPU, disk, network, and memory stats in real-time
sar (from sysstat package) – Collects historical performance data
strace – Traces system calls of a process to detect slow operations
lsof – Lists open files to find processes causing resource contention
tuned-adm – Optimizes system performance based on predefined profiles
Hi, in RHEL9 i have really learned to rely on "systemd-cgtop" to show me the realtime usage of resources when i triage bottlenecks. Sometimes limits.conf file needs tweaking also.
free -m helps with servers using swap and need more.
lsof is nice too when things aren't working and you need to find what is the "hold up".....
EDIT: Lastly strace when i am still scratching my head to figure out what is wrong.
Thank you.
I usually identify performance issues in RHEL use various tools to help pinpoint the cause:
For a broad system overview, dstat provides a combined view of CPU, memory, disk, and network performance.
atop also one of the good tool
top: Provides a real-time view of system resource usage, including CPU, memory, and running processes.
iostat: Displays CPU load and disk I/O statistics.
vmstat: Reports on processes, memory, paging, block I/O, and CPU activity.
free: Checks memory usage.
netstat: Shows network statistics.
iotop: Monitors disk I/O usage by processes.
sar: Collects, reports, and saves system activity information, covering CPU, memory, I/O devices, network, and more
Red Hat
Learning Community
A collaborative learning environment, enabling open source skill development.