Does your IDS CPU go spike in the night?
Those single threads killing your IDS? Do you have a single core at 100% and the rest under 10%? Sometimes the issue is not a bad rule or strange traffic. Rule profiling does not always help. The issue could be a new backup or DR sync that the sysadmins started and did not tell you about. Here is a simple way I have found to find the “top talkers” in view of your IDS.
1. Capture the SRC IP/Port and DST IP/Port data. This is a small part of what you would get with “flow” data.
sudo tcpdump -i fp2 -nn > /tmp/top_talkers.txt
- notice we don’t use “-s 0″ or “-vv” as we only need the basics of each packet (src ip/port and dst ip/port)
- notice we are not writing the packets (-w <file>) only the output of the screen to file
- the data we are collecting is smaller than a pcap would be but depending on the amount of data in view of the IDS your disk may fill up quickly. If your CPU is at 100% you only need to capture for about 5 minutes to find your top talkers.
2. Now we need to grab the fields we want (src ip/port and dst ip/port), sort them, and order them based on counts of unique combos.
cat /tmp/top_talkers.txt | sort | uniq -c | sort
- The end of your list will look something like this. The syntax is <count> <src_ip>.<src_port> <dst_ip>.<dst_port>
34903 192.168.1.3.2051 192.168.1.2.42062:
67945 192.168.1.2.42062 192.168.1.3.2051:
3. Work with your sysadmin team on what each system is and what type of traffic is running on those ports. If it happens each night and it is single host to single host we can add a filter so the traffic is spit out before the long list of rules. Remember unless your budget is unlimited your IDS environment is a game of filtering what you don’t think is a risk so you can monitor more traffic that you think might before your max your hardware out.
Please comment if you have another way of doing the same. It is good to share as the vendor does not always have a way to troubleshoot it all.
