Troubleshooting issues in Linux - The scientific way !!
The scientific method consists of these steps:
- Collect relevant information.
- Create an accurate problem statement.
- Formulate testable hypotheses.
- Test each hypothesis.
- Record and analyze the test results.
- Fix and verify the problem resolution.
Let us understand this with a simple example of web server not working -
Problem given: webserver not working, error connection refused from port 80
Step 1. Collect relevant information:
Info collected : a) connection refused from port 80
- httpd service was inactive initially
- upon restart httpd service failed
- checking the detailed output of service status tells us that : webserver is using non standard port 82 which could not be bound due to “some issue” ( permission denied )
Step 2. Create an accurate problem statement:
webserver is not running because Apache is unable to bind to the specified IP address and port 82.
Step 3. Formulate testable hypotheses :
The problem can occur due to one of the following reasons :
- Port conflict - port 82 might already be in use by another application or service on your system.
- Insufficient Permissions ( files, users etc ).
- SELinux restrictions ( high probability because port 82 is a non standard port for apache )
- Firewall restrictions since we got a connection refused error during curl.
Step 4. Test each hypothesis & Step 5. Record and analyze the test results :
- Port conflict :
- SELinux restrictions ( since it is highly likely the issue)
ausearch -m AVC is a command used to search for audit events related to Access violations in an audited Linux system. AVC violations occur when a process attempts to perform an action that is not allowed by the system's security policy.
We can easily see that port 82 is denied the binding permission.
If you want to see more human friendly information about the violation alert and probable workaround and suggestion on the same - you can use sealert command on the default audit log file location:
The issue is identified now !! port 82 does not have the required SELinux port context for using the httpd service.
Step 6. Fix and verify the problem resolution :
- Find the correct port context for port 82 which can be applied as we use for standard ports 80 & 443 in web server context :
- Apply the http_port_t port context to port 82 so that it can be binded to httpd service and then restart the httpd service to apply the changes :
Verify the results :
But it does not work from other server in the same network :
Notice the error : No route to host
That means you need to check whether the firewall allows the port 82 to connect to outside the server:
That's it ! You are now an awesome linux troubleshooter !!
