cancel
Showing results for 
Search instead for 
Did you mean: 
Chetan_Tiwary_
Community Manager
Community Manager
  • 473 Views

Troubleshooting - A scientific methodolgy

Troubleshooting issues in Linux - The scientific way !!

The scientific method consists of these steps:

  • Collect relevant information.

  • Create an accurate problem statement.

  • Formulate testable hypotheses.

  • Test each hypothesis.

  • Record and analyze the test results.

  • Fix and verify the problem resolution.

Let us understand this with a simple example of web server not working -

 

0.png

 

Problem given: webserver not working, error connection refused from port 80

Step 1. Collect relevant information:

 

1.png

 

Info collected : a) connection refused from port 80

  1. httpd service was inactive initially
  2. upon restart httpd service failed 
  3. checking the detailed output of service status tells us that :  webserver is using non standard port 82 which could not be bound due to “some issue” ( permission denied )

Step 2. Create an accurate problem statement: 

webserver is not running because Apache is unable to bind to the specified IP address and port 82.

Step 3. Formulate testable hypotheses : 

The problem can occur due to one of the following reasons : 

  • Port conflict - port 82 might already be in use by another application or service on your system.
  • Insufficient Permissions ( files, users etc ).
  • SELinux restrictions ( high probability because port 82 is a non standard port for apache )
  • Firewall restrictions since we got a connection refused error during curl.

Step 4. Test each hypothesis & Step 5. Record and analyze the test results : 

 

 

  • Port conflict : 
 

 2.png

 

 

  • SELinux restrictions ( since it is highly likely the issue)
 

3.png

 

ausearch -m AVC is a command used to search for audit events related to Access  violations in an audited Linux system. AVC violations occur when a process attempts to perform an action that is not allowed by the system's security policy.

We can easily see that port 82 is denied the binding permission.

If you want to see more human friendly information about the violation alert and probable workaround and suggestion on the same - you can use sealert command on the default audit log file location:

4.png

 

 

 The issue is identified now !! port 82 does not have the required SELinux port context for using the httpd service.

 

 Step 6. Fix and verify the problem resolution : 

  1. Find the correct port context for port 82 which can be applied as we use for standard ports 80 & 443 in web server context : 

5.png

 

 

  1. Apply the http_port_t port context to port 82 so that it can be binded to httpd service and then restart the httpd service to apply the changes : 

6.png            This time the httpd service restarts successfully.

 

 

Verify the results

11.png

But it does not work from other server in the same network : 

 

8.png

 

Notice the error : No route to host

That means you need to check whether the firewall allows the port 82 to connect to outside the server:

9.pngNow check from the other server whether you can curl or not the servera webserver :        

Screenshot from 2024-08-29 02-42-04.png

 

 

That's it ! You are now an awesome linux troubleshooter !!

1 Reply
Trevor
Starfighter Starfighter
Starfighter
  • 437 Views

What a beautiful example/lesson!!!  So many nice pieces were addressed - especially my favorite, SELinux!

Being able to troubleshoot an issue on a Linux system is the ultimate measure of a
Sys Admin!!!  The ability to configure things is certainly important, but when a system
isn't functioning, installation and configuration ain't possible.  

As was illustrated so very well in the example, it's so important to know your 
commands, and how to decipher the information that they provide.

Love this exposition!!!!

Trevor "Red Hat Evangelist" Chandler
Join the discussion
You must log in to join this conversation.