cancel
Showing results for 
Search instead for 
Did you mean: 
Syed
Community Manager
Community Manager
  • 439 Views

Steps you would take to investigate and mitigate a potential security threat.

200K Contest Question # 5:

A server in your network is experiencing a high volume of traffic from an unknown source. Describe the steps you would take to investigate and mitigate this potential security threat.

Labels (1)
8 Replies
Chetan_Tiwary_
Moderator
Moderator
  • 417 Views

1. Identify the suspicious source IP / server / port using network monitoring tools.

2. Remove that affected server from the LAN.

3. Check logs for information like network protocol , packet surge, time of surge using tools such as sar.

4. Identify the attack type - DoS / DDOS or Brute force.

5. Use firewall , SELinux and SSH configuration to block the incoming traffic.

6. Check for vulnerabilities like privilege escalation, protocol , port , firewall, ssh/login access and update / patch the system with latest security updates.

7. Document the incident and monitor the system post normalisation of services. 

Tags (3)
ipalagin
Flight Engineer Flight Engineer
Flight Engineer
  • 395 Views

Run tcpdump and get that unknown source.

GGPR
Cadet
Cadet
  • 382 Views

First we initiate the investigation from CDN level, analysing the traffic and collecting the IP,method,agent..etc  and payload.

Based on the payload we can implement blocking methods on CDN and server levels.

And also monitoring for some time will help to avoid internal functional errors.

Rahulkrishnan
Mission Specialist
Mission Specialist
  • 382 Views

Below are the step-by-step approach to investigate and mitigate high traffic on a RHEL server:

Identify the Source

  1. Use ss or netstat to identify the source IP addresses of the connections.
  2. Check firewall logs for suspicious activity.

Isolate the Server (Mitigate Immediately)

  1. If the traffic volume is overwhelming, consider temporarily isolating the server from the network. Use firewall rules to block incoming connections.

Analyze Server Logs

  1. Check system logs (syslog, auditd) for any anomalies or error messages related to the high traffic.
  2. Look for entries related to failed login attempts, unusual processes, or resource exhaustion.

Identify Affected Services

  1. Use tools like lsof or netstat to identify which services are experiencing the high traffic.
    This helps narrow down the potential vulnerability.

Secure the Server

  1. Update the system and all installed packages to address any known vulnerabilities.
  2. Focus on recently updated packages or those related to the affected service. We can use yum history to review that.
  3. Review and tighten firewall rules to restrict access to only authorized sources and ports (Make sure it won't affect application working functionalities)

Investigate for Malware

  1. Use system security tools like chkrootkit or rkhunter to scan for potential malware installations.

Analyze Network Traffic

  1. Use network traffic capture tools like tcpdump to analyze the nature of the traffic.
    This can help identify specific attack patterns or protocols used.

Strengthen Authentication

  1. Enforce strong password policies for user accounts on the server.
  2. Consider two-factor authentication for added security.

Consider Additional Security Measures

  1. Implement intrusion detection/prevention systems (IDS/IPS) for real-time threat detection.
  2. For critical servers, consider web application firewalls (WAF) to filter malicious traffic.

By following these steps, we can effectively investigate and mitigate the high traffic threat on the RHEL server.

jgarcia020
Mission Specialist
Mission Specialist
  • 352 Views

The source IP must be identified. This can be done with a sniffer, such as tcpdump, wireshark (tshark), etc.
Then filter the identified IP address.
It is also important to have updated systems.

IvanLabrovic
Flight Engineer
Flight Engineer
  • 330 Views

It is possible the source is a valid source, so we could opt to look for some form of rate-limiting before blocking the source alltogether.

zeet
Flight Engineer
Flight Engineer
  • 244 Views

When dealing with a server experiencing high traffic volumes from an unknown source, it is essential to approach the situation methodically to effectively identify, investigate, and mitigate the potential security threat. Here's how to handle such a scenario, particularly in a Red Hat Linux server environment:

1. Initial Assessment

  • Verify the Symptoms: Confirm the high traffic using monitoring tools such as top, netstat, or iftop to see active connections and traffic volumes.
  • Identify the Nature of Traffic: Determine whether the traffic is legitimate (e.g., due to a recent marketing campaign) or potentially malicious.

2. Gather Information

  • Log Review: Examine logs in /var/log/, such as syslog, auth.log, and apache2/access.log or nginx/access.log for any unusual activity.
  • Analyze Traffic Sources: Use commands like ss or iptraf to identify IP addresses sending excessive requests.

3. Containment

  • Limit Connection Rates: Temporarily restrict the rate of incoming connections using iptables or a similar firewall tool to mitigate the impact on server resources.
  • Block Suspicious IPs: If specific IPs are identified as malicious, block them using firewall rules.

4. Investigate

  • Check for Exploits: Ensure the server is scanned for any signs of compromise, such as unexpected running processes, unusual outbound connections, or unauthorized account access.
  • Dependency and Patch Audit: Make sure all system software and dependencies are up-to-date with the latest security patches.

5. Mitigation

  • Update and Patch Systems: Apply all the latest security patches for the Linux operating system and any installed applications.
  • Enhance Monitoring: Improve monitoring capabilities to detect future incidents early. Tools like fail2ban can be configured to block IPs that exhibit malicious behavior automatically.

6. Recovery and Documentation

  • System Restore: If the server was compromised, consider restoring it from backups after isolating the affected system to prevent further spread.
  • Document the Incident: Keep detailed records of the incident’s timeline, actions taken, and lessons learned to refine future response strategies.

7. Post-Incident Analysis

  • Root Cause Analysis: Determine how the security breach happened and why. Was it a DDoS attack? Was there a vulnerability in the software?
  • Review Security Policies: Adjust security policies and procedures based on findings to prevent similar incidents.
  • Staff Training: Educate the team on the latest threat vectors and encourage regular security training.

8. Ongoing Prevention

  • Regular Audits: Schedule regular security audits to identify and mitigate vulnerabilities.
  • Security Enhancements: Consider implementing more robust security measures such as a Web Application Firewall (WAF), better rate limiting, and advanced intrusion detection systems.

Taking these steps will help ensure that not only is the immediate threat managed, but future vulnerabilities can be prevented, maintaining the integrity and security of your Red Hat Linux server environment.

[root@localhost ~]# Jitendra_Kumar
  • 137 Views

Steps to follow:

1. Use network monitoring tools to analyze the incoming traffic. Look for unidentified IPs, sources to find suspected culprit. 

2. Review server logs to identify the source IP addresses, requested resources, of the suspicious traffic.

3. Use tools like WHOIS to gather more information about the origins of these addresses.

4. Use Firewall to block unwanted to traffic on the server. 

5. Close ports which are not required by any service. 

 

Join the discussion
You must log in to join this conversation.