Q.) The system shows high load average but the CPU seems mostly idle.
top - 05:20:05 up 1 day, 12:04, 1 user, load average: 4.2, 3.6, 2.2
%Cpu(s): 0.3 us, 0.5 sy, 0.0 ni, 99.1 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
Q.) How to disable the su to root even with provided root password for an otherwise allowed user.
Q.) Server boots old kernel despite installing new one.
Bonus Q.) scp hung in script, but works manually.
Level L2 and above
I'll be posting a series of Linux-related questions covering various skill levels. Feel free to share your insights and expertise. Your contributions will benefit learners at all stages, from those in current roles to those preparing for Linux interviews.
To expound on the use of PAM to disable use of su by an account,
let's take a look at what PAM is, and what it does.
Before I get into the specifics of the PAM configuration that
prevents using su, even for accounts that know the root password,
I'd like to revisit some foundational concepts of PAM.
Let me begin by identifying PAM as a system for handling authentication
and authorization for applications. In the configuration that I performed,
the application is su, and authentication is certainly required when using su.
Now, let's have a look at some key concepts of PAM:
- Modules: PAM relies on dynamically loaded shared libraries called modules
that perform specific authentication tasks (e.g., username/password authentication).
In my configuration example, I'm using the module pam_wheel.so to achieve the
necessary outcome.
- Configuration Files: PAM uses configuration files (typically found in /etc/pam.d/)
to define which modules to use for different services/applications, and how to
manage the authentication process. In my configuration example, I'm using the file
/etc/pam.d/su.
- Function Types: Configuration files specify function types like auth (for authentication),
account (for account management), password (for password changes), and session
(for session management). Looking at my configuration example, you'll see that the
function type that is being specified is auth - because the module that is going to be called is going to perform authentication.
- Control Arguments: These arguments (required, requisite, sufficient, optional), also
referred to as Control Flags, dictate how PAM evaluates the rules within a module stack.
Again, looking at my configuration example, you'll see that I specify the Control Flag
required.
- Stacking: PAM evaluates modules in a specific order - this allows for authentication
scenarios where multiple modules might be involved. In the /etc/pam.d/su configuration
file, there are multiple modules. I don't show this in my configuration example, but
if you were to look at the contents of the /etc/pam.d/su configuration file on your
system, you will see not only multiple modules, but multiple modules that involve authentication.
Remember the function types I mentioned earlier! The function type that I'm
using offers flexibility. PAM allows system admins to easily switch authentication
methods, add new ones, and tailor authentication policies to specific applications
and services. Tailoring an authentication policy to a specific application is exactly
what I did in my configuration example, with the line
auth required pam_wheel.so use_uid
added to the /etc/pam.d/su file.
Now, let's examine the line auth required pam_wheel.so use_uid a little closer:
auth -> function type
required -> control argument
pam_wheel.so -> PAM library module
use_uid -> paramter to used by PAM library module
We've looked briefly at the function type, and control argument. Now,
let's take a closer look at the piece that's providing the magic for this
solution: pam_wheel.so
First, this module, along with a host of other modules, can be found in
the /usr/lib64/security directory.
I'll use the man command to examine the man page for the pam_wheel.so
module: $ man pam_wheel
Looking at the man page for this module, you can see right away this
module's purpose/function. This same man page will explain the purpose/function
of the use_uid option.
Pause!!!! Before reading any further, login to a RHEL system, and
take a look at this man page!!!
Okay, welcome back!
Now, in case you didn't already know this, the su application is PAM-aware.
What does that mean? That means that su relies on PAM to manage the
authentication process through configured modules.
Note: If you ever wish to verify that an application is PAM-aware, you can
use the ldd command to see if the application is linked to the PAM library
(libpam.so).
On my RHEL system, the line auth required pam_wheel.so use_uid already
exist, but it is commented. So, I just needed to remove the hash (#) at the
beginning of the line. Now that I've done that, any user account that wishes
to su to root, MUST be added to the group named wheel - even if the root
password is known.
In summary, all I needed to do to achieve my goal, was to make sure that
the line auth required pam_wheel.so use_uid is (uncommented) in the
/etc/pam.d/su file, and I'm done!!! If I need to have a user account to be
able to su to root, simply add that account name to the wheel group - and
take an extra 30 minutes for lunch!
Additional Commentary:
Earlier, in my writeup, I mentioned the Control Arguments (Control Flags).
I wanted to provide just a little bit more content on those here. I didn't
want to do so earlier because it was very PAM specific, and would have
taken the focus away from the specifics of my configuration example.
All PAM modules generate a success or failure result when called. Control flags tell
PAM what do with the result. Since modules can be stacked in a particular order,
control flags decide how important the success or failure of a particular module is
to the overall goal of authenticating the user to the service.
There are four predefined control flags:
required — The module result must be successful for authentication to continue. If a required module result fails, the user is not notified until results on all modules referencing that interface are completed.
requisite — The module result must be successful for authentication to continue. However, if a requisite module result fails, the user is notified immediately with a message reflecting the first failed required or requisite module.
sufficient — The module result is ignored if it fails. However, if a sufficient flagged module result is successful and no required flagged modules above it have failed, then no other results are required and the user is authenticated to the service.
optional — The module result is ignored. A module flagged as optional only becomes necessary for successful authentication when there are no other modules referencing
the interface.
Chetan, was this: Too much? Not enough? Too high level? Too low level?
Too _____ ? Just right?
Q.) The system shows high load average but the CPU seems mostly idle.
top - 05:20:05 up 1 day, 12:04, 1 user, load average: 4.2, 3.6, 2.2 %Cpu(s): 0.3 us, 0.5 sy, 0.0 ni, 99.1 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
Answer:
This condition is usually an indicator of I/O bottlenecks, memory pressure, or other resource contention.
First, what is load average? Load Average in Linux is a metric that is used by Linux users to keep track of system resources. It also helps you monitor how the system resources are engaged.
In the information above, that is output from the execution of the top command, which shows load average, along with CPU states.
There are five main CPU states:
1) iowait
2) system
3) user
4) nice
5) idle
The output of the top command above is providing us information about each of those CPU states:
- 0.3 us -> user space processes
- 0.5 sy -> system/kernel space processes
- 0.0 ni -> nice (processes with a user-defined priority)
- 99.1 id -> idle CPU time
- 0.1 wa -> wait (time spent waiting for I/O operations)
The other two items on that %Cpu(s) row are:
- 0.0 si -> software interrupts
- 0.0 st -> time stolen from a virtual machine
The other beneficial information in the output of that top command is the load average, which shows the average number of processes that are either running, waiting for CPU, or waiting for I/O. For completeness, let me identify the numbers in the output of that top command as:
- 1-minute average -> 4.2
- 5-minute average -> 3.6
- 15-minute average -> 2.2
Those numbers do NOT show a breakdown of processes running, waiting for CPU, or waiting for I/O!!!
To add a little context to the above load average numbers, if this system has 4 CPUs, that 1-minute average is suggesting that one process is waiting for execution or I/O, indicating potential resource contention! Knowing that the key resources used by a Linux OS are CPU, memory, storage, I/O devices, and network connections, it's just a matter of investigating the non-CPU resouces, to identify what's responsible for this high load average.
Going back to the output provided by the top command, the CPU state that shows an idle time of 99.1, essentially means the high load is likely due to I/O, or other resource issues - not CPU utilization. So, I'll use the iostat utility, to do some analysis of I/O performance. In addition to looking at the I/O performance, I'll investigate the memory usage with the free -m and vmstat tools. I'll leave it to you, to choose your weapon, to examine the storage and network connections.
In summary, a high load average, with a CPU that is mostly idle, often points to I/O bottlenecks, memory pressure, or network delays. Again, an important point to remember is that load average is a function of both CPU and I/O. A high load average doesn't always mean high CPU usage!!!
@Trevor Right, Load average = R (ready to run) + D (waiting on I/O).
A high number doesn't necessarily imply CPU saturation,it can mean many processes are blocked on slow operations.
Q.) How to disable the su to root even with provided root password for an otherwise allowed user.
Answer:
To accomplish the disabling of an account being able to su to root, even when providing the root password, we will leverage PAM (Pluggable Authentication Modules) to control su access.
The steps to implement this setup are:
1. Edit the /etc/pam.d/su file, by adding the following line:
auth required pam_wheel.so use_uid
Note: If this line already exists in the file, simply uncomment the line. In all likelihood, the line does exist in the file.
2. Create a group named "wheel" - if it doesn't already exist; I can't imagine it not existing!
3. Add users to the "wheel" group that should be permitted to su to root. You know how to do this I'm sure, but I'll provide the command anyway:
# usermod -a -G wheel <accountname_to_be_added>
That's it! With this configuration, only users that are members of the "wheel" group will be able to use su to become root - even if they have the root password. I guarantee it!!!
To expound on the use of PAM to disable use of su by an account,
let's take a look at what PAM is, and what it does.
Before I get into the specifics of the PAM configuration that
prevents using su, even for accounts that know the root password,
I'd like to revisit some foundational concepts of PAM.
Let me begin by identifying PAM as a system for handling authentication
and authorization for applications. In the configuration that I performed,
the application is su, and authentication is certainly required when using su.
Now, let's have a look at some key concepts of PAM:
- Modules: PAM relies on dynamically loaded shared libraries called modules
that perform specific authentication tasks (e.g., username/password authentication).
In my configuration example, I'm using the module pam_wheel.so to achieve the
necessary outcome.
- Configuration Files: PAM uses configuration files (typically found in /etc/pam.d/)
to define which modules to use for different services/applications, and how to
manage the authentication process. In my configuration example, I'm using the file
/etc/pam.d/su.
- Function Types: Configuration files specify function types like auth (for authentication),
account (for account management), password (for password changes), and session
(for session management). Looking at my configuration example, you'll see that the
function type that is being specified is auth - because the module that is going to be called is going to perform authentication.
- Control Arguments: These arguments (required, requisite, sufficient, optional), also
referred to as Control Flags, dictate how PAM evaluates the rules within a module stack.
Again, looking at my configuration example, you'll see that I specify the Control Flag
required.
- Stacking: PAM evaluates modules in a specific order - this allows for authentication
scenarios where multiple modules might be involved. In the /etc/pam.d/su configuration
file, there are multiple modules. I don't show this in my configuration example, but
if you were to look at the contents of the /etc/pam.d/su configuration file on your
system, you will see not only multiple modules, but multiple modules that involve authentication.
Remember the function types I mentioned earlier! The function type that I'm
using offers flexibility. PAM allows system admins to easily switch authentication
methods, add new ones, and tailor authentication policies to specific applications
and services. Tailoring an authentication policy to a specific application is exactly
what I did in my configuration example, with the line
auth required pam_wheel.so use_uid
added to the /etc/pam.d/su file.
Now, let's examine the line auth required pam_wheel.so use_uid a little closer:
auth -> function type
required -> control argument
pam_wheel.so -> PAM library module
use_uid -> paramter to used by PAM library module
We've looked briefly at the function type, and control argument. Now,
let's take a closer look at the piece that's providing the magic for this
solution: pam_wheel.so
First, this module, along with a host of other modules, can be found in
the /usr/lib64/security directory.
I'll use the man command to examine the man page for the pam_wheel.so
module: $ man pam_wheel
Looking at the man page for this module, you can see right away this
module's purpose/function. This same man page will explain the purpose/function
of the use_uid option.
Pause!!!! Before reading any further, login to a RHEL system, and
take a look at this man page!!!
Okay, welcome back!
Now, in case you didn't already know this, the su application is PAM-aware.
What does that mean? That means that su relies on PAM to manage the
authentication process through configured modules.
Note: If you ever wish to verify that an application is PAM-aware, you can
use the ldd command to see if the application is linked to the PAM library
(libpam.so).
On my RHEL system, the line auth required pam_wheel.so use_uid already
exist, but it is commented. So, I just needed to remove the hash (#) at the
beginning of the line. Now that I've done that, any user account that wishes
to su to root, MUST be added to the group named wheel - even if the root
password is known.
In summary, all I needed to do to achieve my goal, was to make sure that
the line auth required pam_wheel.so use_uid is (uncommented) in the
/etc/pam.d/su file, and I'm done!!! If I need to have a user account to be
able to su to root, simply add that account name to the wheel group - and
take an extra 30 minutes for lunch!
Additional Commentary:
Earlier, in my writeup, I mentioned the Control Arguments (Control Flags).
I wanted to provide just a little bit more content on those here. I didn't
want to do so earlier because it was very PAM specific, and would have
taken the focus away from the specifics of my configuration example.
All PAM modules generate a success or failure result when called. Control flags tell
PAM what do with the result. Since modules can be stacked in a particular order,
control flags decide how important the success or failure of a particular module is
to the overall goal of authenticating the user to the service.
There are four predefined control flags:
required — The module result must be successful for authentication to continue. If a required module result fails, the user is not notified until results on all modules referencing that interface are completed.
requisite — The module result must be successful for authentication to continue. However, if a requisite module result fails, the user is notified immediately with a message reflecting the first failed required or requisite module.
sufficient — The module result is ignored if it fails. However, if a sufficient flagged module result is successful and no required flagged modules above it have failed, then no other results are required and the user is authenticated to the service.
optional — The module result is ignored. A module flagged as optional only becomes necessary for successful authentication when there are no other modules referencing
the interface.
Chetan, was this: Too much? Not enough? Too high level? Too low level?
Too _____ ? Just right?
@Trevor It is perfect! Just how much was needed ! Thanks a lot!
Q.) Server boots old kernel despite installing new one.
Answer:
Sounds to me like the bootloader ain't configured to boot this new kernel by default.
Execute the following command to verify that the bootloader is in fact defaulting to boot the old kernel:
# grubby --default-kernel
After verifying this, setup your grub config file to boot the new kernel by default. Do this by simply locating the kernel file, in the /boot directory, that you want to have launched by default, and then execute the following command:
# grubby --set-default /boot/<name_of_kernel_file_to_boot>
Reboot your system, to see that the old kernel is no longer be used to boot the system. How does that old saying go, "Out with the old, in with the new"
Bonus Q.) scp hung in script, but works manually.
Answer:
Is the script hanging during the scp login process? I'm going to go after the low-hanging fruit, and say that there's an issue with an environment variable that's being used to reference some pieces of information in the scp command syntax, and that those environment variables aren't being made available to script, when it launches in a new shell.
If I come up with an additional hypothesis, I'll return to pad this initial response!
@Trevor some scenarios like - 1. first time host key exchange prompts 2. Network timeouts .
What would be the alternative tool ?
Red Hat
Learning Community
A collaborative learning environment, enabling open source skill development.